chore(dev): add one-shot dev environment setup script

Add scripts/dev/setup.sh: an idempotent bootstrap that installs the required
toolchains (uv/Python, Node/npm, Rust/cargo) when missing, creates .env,
configures git hooks, installs all Python + Node workspace deps, pre-downloads
the local ML models + tokenizer for offline use, and builds the TypeScript SDK
and Rust CLI. Flags: --skip-build, --skip-models, --with-docs, --force.

Document it in CONTRIBUTING.md as the recommended setup, keeping the manual
steps as a fallback.

.claude-plugin/marketplace.json

@@ -1,7 +1,6 @@
 {
   "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
   "name": "hindsight",
-  "version": "0.7.5",
   "description": "Official Hindsight integrations for Claude Code",
   "owner": {
     "name": "vectorize-io"
@@ -11,11 +10,6 @@
       "name": "hindsight-memory",
       "description": "Automatic long-term memory for Claude Code via Hindsight",
       "source": "./hindsight-integrations/claude-code"
-    },
-    {
-      "name": "hindsight-zcode",
-      "description": "No-MCP long-term memory for ZCode via Hindsight hooks",
-      "source": "./hindsight-integrations/zcode"
     }
   ]
 }

.claude/skills/code-review/SKILL.md

@@ -73,16 +73,6 @@ results = await asyncio.gather(*tasks, return_exceptions=True)
 results = await asyncio.gather(*tasks, return_exceptions=True)
 ```
 
-### API Layer & Data Access
-- **No direct database access in `api/http.py`** (or any API router). HTTP handlers must not build SQL, call `acquire_with_retry` / `conn.fetch` / `conn.fetchrow` / `conn.execute`, or reference `fq_table(...)`. All persistence and queries live in `MemoryEngine` (the engine layer). A handler parses/validates the request, calls an engine method, shapes the HTTP response, and maps domain results to status codes (e.g. a `None` return → 404).
-- **Authentication/tenancy is enforced inside each engine method, not assumed by the handler.** Every engine method that touches bank-scoped data must authenticate via `request_context` — typically `await self._authenticate_tenant(request_context)` (often indirectly through `get_bank_profile(...)`) — so the correct tenant schema is resolved before any query runs. Handlers must thread `request_context` through to the engine method; never query a tenant-scoped table assuming the schema is already set.
-- Engine methods return typed models (Pydantic/dataclass), not raw dicts (see Type Safety).
-
-### Database Locking
-- **Never use PostgreSQL advisory locks** (`pg_advisory_lock`, `pg_try_advisory_lock`, `pg_advisory_xact_lock`, `pg_advisory_unlock`, …) in migrations, engine code, or anything else. Hindsight runs against connection poolers and managed/PG-compatible services where advisory locks are unreliable or unsupported: session-level locks silently leak or vanish when a pooler hands the session to another client, and callers can block forever on a lock the server never grants. Reject any new occurrence, including ones that look "safe" because they are transaction-scoped.
-- The pre-existing usage in `hindsight_api/migrations.py` is grandfathered, not a precedent — it is tracked for removal. Don't copy it.
-- Design the concurrency out instead of locking around it: give each process its own object to write (e.g. per-schema DDL rather than a shared `public.` object), make the operation idempotent, or use a real row/table constraint (`INSERT ... ON CONFLICT`, `SELECT ... FOR UPDATE` in a fixed order). See #2690 for a migration that reached for `pg_advisory_xact_lock` and had to be reverted.
-
 ### Branch Hygiene
 - **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
 - **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
@@ -145,13 +135,6 @@ For each new or significantly changed function/endpoint/class:
 
 Flag any new logic that lacks test coverage.
 
-**LLM-behaviour changes need a real-LLM judge test, not MockLLM.** If the change alters how the model interprets a prompt — fact/observation extraction, `fact_type` (world/experience) classification, speaker attribution, instruction-following, prompt wording — there MUST be a test marked `pytest.mark.hs_llm_core` that runs the real pipeline and asserts via `tests.llm_judge.assert_meets_criteria` (not string/enum matching). Flag these as findings:
-- A prompt/classification change verified only by MockLLM or string assertions (MockLLM echoes input — such tests pass spuriously). **Should fix.**
-- A test that hard-asserts `fact_type == "world"/"experience"` (or other model-decided output) instead of judging it — non-deterministic, will flake across providers/runs. **Should fix** (move the classification check into the judge `criteria`; keep only genuinely deterministic structural asserts direct).
-- Deterministic mechanics (prompt assembly, suppression/branching logic) that are covered *only* by a slow LLM test — these should also have fast non-LLM unit tests. **Note.**
-
-See CLAUDE.md → Key Conventions → Testing for the full pattern.
-
 ### 7. Check API consistency
 
 If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
@@ -159,24 +142,6 @@ If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
 - Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
 - Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
 
-### 7a. Check TS/Python wrapper-client parity
-
-Two of the generated SDKs ship a **hand-written, maintained convenience wrapper** on top of the auto-generated low-level client — and *only* these two:
-- **TypeScript**: `hindsight-clients/typescript/src/index.ts` (`HindsightClient`)
-- **Python**: `hindsight-clients/python/hindsight_client/hindsight_client.py` (`Hindsight`)
-
-(The Rust/Go/etc. clients are generated-only — no wrapper to keep in sync.)
-
-These wrappers are what most third-party consumers actually call, and they must expose the same surface. **If a change touches one wrapper's method — adds/removes a parameter, changes a default, forwards a new query/body field — the equivalent method in the *other* wrapper must get the same change in the same (or an immediately-following) PR.** A parameter that exists in the generated SDK but is dropped by one wrapper silently strips it for every consumer of that language (this is exactly what #2975 / #3042 fixed for `detail`/`tags_match`/`limit`/`offset` on `listMentalModels`/`getMentalModel`). **Should fix** — flag any wrapper method that gains capabilities in one language but not the other, and add a matching mapping regression test on both sides.
-
-Note: the `client-coverage-check` CI tool only validates **request-body** fields, not GET **query** parameters — so query-param parity gaps are *not* caught automatically and must be checked by hand here.
-
-### 7b. Check API-layer data-access boundary
-
-For each changed handler in `hindsight-api-slim/hindsight_api/api/` (e.g. `http.py`, `mcp.py`):
-- **Flag any direct DB access in the handler** — `acquire_with_retry`, `conn.fetch` / `fetchrow` / `execute`, raw SQL strings, or `fq_table(...)`. These are a **must fix**: the query must be moved into a `MemoryEngine` method that returns a typed model, and the handler must call that method.
-- **Verify authentication is enforced in the engine** — the handler must delegate to an engine method that authenticates via `request_context` (`_authenticate_tenant`, typically through `get_bank_profile`). A handler that reads/writes tenant-scoped data without an engine method enforcing auth is a **must fix** (tenant data could leak across schemas).
-
 ### 8. Check code comments
 
 For each non-trivial change:
@@ -189,8 +154,7 @@ For each non-trivial change:
 If any files in `hindsight-integrations/` were added or changed, verify:
 - **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
 - **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
-- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` AND in the `INTEGRATIONS` dict in `hindsight-dev/hindsight_dev/generate_changelog.py` (the changelog generator keeps its own list; a release fails at the changelog step if the name is missing there). If either is missing, flag it.
-- **Docs gallery + sidebar entry** — the integration must have an entry in `hindsight-docs/src/data/integrations.json`. This file is the **single source of truth** that drives both the integrations gallery and the docs sidebar (the sidebar category is injected from it at render time across all docs versions). The entry needs an internal `/sdks/integrations/<slug>` `link` and a matching page at `hindsight-docs/docs-integrations/<slug>.md(x)`. The `hindsight-docs/scripts/check-integrations.mjs` build step enforces both directions — forward: every internal JSON entry has a doc page; reverse: every released tag (`integrations/<name>/vX.Y.Z`) appears in the JSON (private infra like `cloudflare-oauth-proxy` is in the script's `EXCLUDED` set). Flag any integration that is released (or being released) but missing from `integrations.json`, and any JSON entry without a doc page. Do **not** hand-edit `versioned_sidebars/*.json` to add integration links — they are positional placeholders filled from the JSON.
+- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
 - **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
 
 ### 10. Check MCP tool registration completeness
@@ -209,26 +173,6 @@ If a migration adds a new PostgreSQL table (look for `CREATE TABLE` / `op.create
 - The guard test `test_backup_tables_covers_entire_schema` in `tests/test_admin_backup_restore.py` enforces this — flag it as a **must fix** if a new table is absent from `BACKUP_TABLES`.
 - Oracle-only tables (e.g. `observation_sources`) are intentionally excluded — admin backup/restore is PostgreSQL-only.
 
-### 11b. Check new config flags update the env template
-
-If the diff adds a new configuration field (a new `ENV_*` / `HINDSIGHT_*` env var
-in `hindsight-api-slim/hindsight_api/config.py`):
-- **`.env.example`** (repo root) — must add the variable (commented if optional)
-  alongside the docs entry in `hindsight-docs/docs/developer/configuration.md`.
-  A flag added to `config.py` but absent from `.env.example` is a **should fix**.
-- **`hindsight-embed/hindsight_embed/env.example`** — the bundled copy must stay
-  byte-identical to the repo-root `.env.example` (it seeds embed/profile configs).
-  The `test_bundled_template_matches_repo_root` sync test fails on drift; if the
-  root file changed without re-copying, flag it as a **must fix**.
-
-### 11c. Check for advisory locks
-
-Grep the diff for `advisory` (`git diff main...HEAD | grep -in advisory`). Any new
-`pg_advisory_lock` / `pg_try_advisory_lock` / `pg_advisory_xact_lock` /
-`pg_advisory_unlock` call is a **must fix** — see Database Locking above. Point the
-author at the alternatives (per-process objects, idempotent DDL, row-level
-constraints) rather than just asking them to drop the lock.
-
 ### 12. Review against other coding standards
 
 Check the diff for violations of the standards listed above:
@@ -252,10 +196,7 @@ Present a clear summary organized by severity:
 - Raw dict usage for structured data (including internal code)
 - Multi-item tuple returns (including internal code)
 - Missing tests for new endpoints
-- Direct DB access (raw SQL / `acquire_with_retry` / `fq_table`) in an `api/` handler instead of a `MemoryEngine` method
-- Tenant-scoped data accessed without authentication enforced in the engine (`_authenticate_tenant` / `get_bank_profile`)
 - New integration missing tests, CI job, or release-integration.sh entry
-- Released/added integration missing from `hindsight-docs/src/data/integrations.json`, or a JSON entry with no `docs-integrations/<slug>` page (fails the docs build via `check-integrations.mjs`)
 - New PostgreSQL table missing from `BACKUP_TABLES` in `admin/cli.py` (silent data loss on restore)
 
 **Should fix** — issues that hurt code quality:

.claude/skills/hs-release/SKILL.md

@@ -1,116 +0,0 @@
----
-name: hs-release
-description: Cut a core Hindsight release (vX.Y.Z) and open the changelog + blog PR. Use when asked to cut/start a release, bump the version, or publish a new Hindsight version.
-user_invocable: true
----
-
-# Hindsight Release
-
-Cut a **core** Hindsight release and open the accompanying changelog/blog PR. This is for the core
-product version (API, clients, CLI, control plane, Helm). **Integrations are versioned
-independently** — use `scripts/release-integration.sh` for those, not this skill.
-
-The release is **irreversible and outward-facing**: it tags a version and pushes it straight to
-`main`, which triggers CI that publishes packages to PyPI / npm / Helm. Confirm the version number
-and that the intended fixes are already merged to `main` before you start.
-
-## Step 0 — Pre-flight
-
-1. **Decide the base.** A release is cut from the latest `origin/main`, never from a feature
-   branch. `git fetch origin --tags` first. Confirm the "couple of fixes" the user means are
-   actually merged to `main` (`git log v<prev>..origin/main --oneline`).
-2. **Find where `main` is checked out.** `main` is often already checked out in a sibling worktree
-   (`git worktree list`). You **cannot** check out `main` in a second worktree — run the release in
-   the worktree that already holds it. If that worktree is dirty with throwaway cruft
-   (`.next-*` tsconfig paths, screenshots), `git stash push -u`, fast-forward to `origin/main`,
-   run the release, then `git stash pop`.
-3. **Pitfall:** never pipe the checkout in an `&&` chain like
-   `git checkout main 2>&1 | tail && git reset --hard ...` — the pipe's exit status is `tail`'s
-   (always 0), so a failed checkout won't stop the chain and the `reset` fires on the **wrong
-   branch**. Check out as its own command and verify `git branch --show-current` before resetting.
-
-## Step 1 — Cut the release
-
-Run from the worktree on a clean `main`:
-
-```bash
-./scripts/release.sh <version>     # e.g. 0.8.1  (no leading v)
-```
-
-`release.sh` bumps the version in every component, regenerates the OpenAPI spec + all client SDKs,
-updates docs versioning, commits `Release v<version>`, tags `v<version>`, and **pushes the commit
-and tag directly to `main`**. The push triggers the `Release` GitHub Actions workflow that builds
-and publishes the packages. It is **not** a PR.
-
-Verify after: `gh run list --limit 5` should show the `Release v<version>` workflow running, and
-`git ls-remote --tags origin v<version>` should return the tag.
-
-## Step 2 — Changelog + blog PR (separate)
-
-Done **after** the tag exists, as its own PR (precedent: v0.8.0 = #2053, v0.8.1 = #2080). Work on a
-branch off the new `main`:
-
-```bash
-git checkout -b docs-changelog-<version> origin/main
-```
-
-Only spin up a separate worktree (`git worktree add ../hindsight-changelog-<version> -b
-docs-changelog-<version> origin/main`) if you can't get a clean checkout otherwise — e.g. `main` is
-held in another worktree and the current one has work you don't want to disturb.
-
-**Branch naming:** use the `docs-` (hyphen) convention, e.g. `docs-changelog-0.8.1`. A remote
-branch literally named `docs` exists, so any `docs/...` branch is rejected on push with
-`directory file conflict`.
-
-### Changelog
-
-```bash
-uv run --directory hindsight-dev generate-changelog <version>
-```
-
-LLM-summarizes the commits between the previous tag and `v<version>` and prepends an entry to
-`hindsight-docs/src/pages/changelog/index.md`. Requires `OPENAI_API_KEY` (already in the repo
-`.env`). It excludes `hindsight-integrations/` source, but new integrations whose commits also
-touched docs will still appear — that matches precedent, leave them in the **changelog**.
-
-### Blog post
-
-Hand-write `hindsight-docs/blog/YYYY-MM-DD-version-X-Y-Z.md` (mirror an existing one; patch
-releases are short — see `2026-06-02-version-0-7-2.md`). Guidance:
-
-- **Explain user impact, not internals/mechanism.** Lead with what the user can now do and what to
-  set. Config/env-var names are fine (developer-facing), code symbols and internals are not.
-- **Do not list integrations in the release blog.** The core blog covers core engine / API /
-  ops changes; each integration ships its own changelog. (Integrations may still appear in the
-  generated `changelog/index.md` — that's fine; just keep them out of the blog.)
-- Call out an upgrade recommendation when there are operational/data-integrity fixes.
-- Validate formatting: `npx prettier --check <blog file>`.
-
-### Sync the docs skill
-
-```bash
-./scripts/generate-docs-skill.sh
-```
-
-Refreshes `skills/hindsight-docs/references/changelog/index.md`. It will also bump
-`skills/hindsight-docs/references/openapi.json` by one version — `release.sh` regenerates the skill
-*before* bumping OpenAPI, so the skill copy lags a version in the release commit; this step syncs
-it. Expect a one-line `version` diff there; keep it.
-
-### Commit, push, PR
-
-```bash
-git add -A
-git commit --no-verify -m "docs: changelog and blog post for v<version>"
-git push -u origin docs-changelog-<version>
-gh pr create --base main --title "docs: changelog and blog post for v<version>" --body "..."
-```
-
-Expected files in the PR: the changelog entry, the new blog post, the regenerated skill changelog
-mirror, and the skill `openapi.json` version sync.
-
-## Cleanup
-
-If you created a temporary worktree, remove it once the PR is up
-(`git worktree remove ../hindsight-changelog-<version>`; the branch stays on origin). Restore any
-stash you popped in Step 0.

.env.example

@@ -2,7 +2,7 @@
 # Copy this file to .env and fill in your values
 
 # LLM Configuration (Required)
-# Supported providers: openai, openai-responses, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, atlas, volcano
+# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, volcano
 HINDSIGHT_API_LLM_PROVIDER=openai
 HINDSIGHT_API_LLM_API_KEY=your-api-key-here
 HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
@@ -10,55 +10,6 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # Reasoning effort for providers/models that support it. Examples: low, medium, high, xhigh.
 # HINDSIGHT_API_LLM_REASONING_EFFORT=low
 
-# Sampling temperature for internal LLM calls. Set a number in [0.0, 2.0], or `none`
-# to omit the temperature parameter entirely (required for models that reject explicit
-# temperatures, e.g. Azure gpt-5.5). The global override below applies to every operation;
-# per-operation overrides (defaults: verification=0.0, retain=0.1, reflect=0.9,
-# consolidation=0.0) take precedence.
-# HINDSIGHT_API_LLM_TEMPERATURE=none
-# HINDSIGHT_API_LLM_TEMPERATURE_VERIFICATION=0.0
-# HINDSIGHT_API_LLM_TEMPERATURE_RETAIN=0.1
-# HINDSIGHT_API_LLM_TEMPERATURE_REFLECT=0.9
-# HINDSIGHT_API_LLM_TEMPERATURE_CONSOLIDATION=0.0
-
-# Grammar-enforce structured output (json_schema strict) instead of the soft
-# schema-in-prompt path. Helps weaker self-hosted models that emit prose preambles
-# or invalid JSON. The global override below applies to every operation;
-# per-operation overrides take precedence, in both directions -- set one to false
-# to opt that operation out while the global flag is on.
-# HINDSIGHT_API_LLM_STRICT_SCHEMA=false
-# HINDSIGHT_API_LLM_STRICT_SCHEMA_RETAIN=true
-# HINDSIGHT_API_LLM_STRICT_SCHEMA_REFLECT=true
-# HINDSIGHT_API_LLM_STRICT_SCHEMA_CONSOLIDATION=true
-
-# Some backends, including Bedrock Converse, reject JSON Schema maxItems.
-# Disable it only for those backends; consolidation still enforces the cap.
-# HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS=true
-
-# Pin a conversation to one backend prompt cache (OpenAI-compatible providers only).
-# Server-side prompt caches are per backend server, so the same conversation has to
-# reach the same one to hit. Values: auto (default), xai_conv_id (sends xAI's
-# x-grok-conv-id header), openai_prompt_cache_key (sends OpenAI's prompt_cache_key
-# field), none (sends nothing). "auto" picks from the base URL host and is an
-# allowlist: x.ai / grok.com get the header, native OpenAI / openai.com / Azure
-# OpenAI get the field, and every other backend gets nothing. Per-operation
-# overrides take precedence. Set to none to disable entirely.
-# HINDSIGHT_API_LLM_CACHE_AFFINITY=auto
-# HINDSIGHT_API_RETAIN_LLM_CACHE_AFFINITY=none
-# HINDSIGHT_API_REFLECT_LLM_CACHE_AFFINITY=xai_conv_id
-# HINDSIGHT_API_CONSOLIDATION_LLM_CACHE_AFFINITY=none
-
-# Ask litellm/litellmrouter/bedrock for structured output via a forced tool call
-# instead of response_format. Enable it for backends that reject response_format
-# outright -- e.g. Bedrock Claude in ap-southeast-2 ("Extra inputs are not permitted");
-# the same model in us-east-1 accepts response_format and needs nothing here.
-# HINDSIGHT_API_LLM_STRUCTURED_OUTPUT_FORCED_TOOL=false
-
-# Diagnostic: on any LLM 4xx, log the exact assembled request ([LLM_4XX_DUMP]) --
-# serialized request config (message bodies stripped) + capped per-message previews.
-# For debugging otherwise-unreproducible rejected calls. Off by default.
-# HINDSIGHT_API_LLM_DEBUG_DUMP_4XX=false
-
 # Example: Anthropic Claude configuration
 # HINDSIGHT_API_LLM_PROVIDER=anthropic
 # HINDSIGHT_API_LLM_API_KEY=your-anthropic-api-key
@@ -74,13 +25,7 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # Example: MiniMax configuration (1M context window)
 # HINDSIGHT_API_LLM_PROVIDER=minimax
 # HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
-# HINDSIGHT_API_LLM_MODEL=MiniMax-M3  # or MiniMax-M2.7 for the previous generation
-
-# Example: OpenAI Responses API (/v1/responses) — reasoning + function tools together
-# HINDSIGHT_API_LLM_PROVIDER=openai-responses
-# HINDSIGHT_API_LLM_API_KEY=your-openai-api-key
-# HINDSIGHT_API_LLM_MODEL=gpt-5.6  # reasoning model (gpt-5.x / o-series); e.g. gpt-5.6-terra
-# HINDSIGHT_API_LLM_REASONING_EFFORT=high  # sent alongside tools, unlike chat/completions
+# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
 
 # Example: DeepSeek configuration (https://api.deepseek.com)
 # HINDSIGHT_API_LLM_PROVIDER=deepseek
@@ -92,60 +37,16 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # HINDSIGHT_API_LLM_API_KEY=your-zai-api-key
 # HINDSIGHT_API_LLM_MODEL=glm-4.5-flash  # or glm-4.5-air for the paid tier
 
-# Example: Atlas Cloud configuration (OpenAI-compatible, https://www.atlascloud.ai)
-# HINDSIGHT_API_LLM_PROVIDER=atlas
-# HINDSIGHT_API_LLM_API_KEY=your-atlascloud-api-key
-# HINDSIGHT_API_LLM_MODEL=deepseek-ai/deepseek-v4-pro  # reasoning model; also Qwen / GLM / Kimi / MiniMax, etc.
-
 # Example: LM Studio local configuration (Qwen 2.5 32B recommended)
 # HINDSIGHT_API_LLM_PROVIDER=lmstudio
 # HINDSIGHT_API_LLM_API_KEY=lmstudio
 # HINDSIGHT_API_LLM_BASE_URL=http://localhost:1234/v1
 # HINDSIGHT_API_LLM_MODEL=qwen2.5-32b-instruct
 
-# Example: Ollama local configuration (native provider)
-# HINDSIGHT_API_LLM_PROVIDER=ollama
-# HINDSIGHT_API_LLM_BASE_URL=http://localhost:11434/v1
-# HINDSIGHT_API_LLM_MODEL=gemma3:12b
-# Native Ollama context-window override (num_ctx). Leave unset to let Ollama use
-# the model Modelfile / server default; set a positive integer only to force a
-# specific context size (e.g. 16384 to keep the previous request behavior).
-# HINDSIGHT_API_LLM_OLLAMA_NUM_CTX=16384
-
-# Multi-LLM strategies: configure extra LLMs by index alongside the primary above,
-# then pick a routing strategy. Unset = single primary LLM (default). Members are
-# numbered from 1; indices must be contiguous. Each operation can override with a
-# RETAIN_/REFLECT_/CONSOLIDATION_ prefix (e.g. HINDSIGHT_API_RETAIN_LLM_1_PROVIDER).
-# HINDSIGHT_API_LLM_1_PROVIDER=groq
-# HINDSIGHT_API_LLM_1_API_KEY=your-groq-api-key
-# HINDSIGHT_API_LLM_1_MODEL=openai/gpt-oss-120b
-# HINDSIGHT_API_LLM_2_PROVIDER=anthropic
-# HINDSIGHT_API_LLM_2_API_KEY=your-anthropic-api-key
-# Strategy JSON: {"mode": "failover"} or {"mode": "round-robin"}.
-# Round-robin accepts optional positive-int "weights" (one per member, primary first).
-# HINDSIGHT_API_LLM_STRATEGY={"mode": "failover"}
-
 # API Configuration (Optional)
 HINDSIGHT_API_HOST=0.0.0.0
 HINDSIGHT_API_PORT=8888
 HINDSIGHT_API_LOG_LEVEL=info
-# Optional retain chunking override for structured logs/transcripts.
-# Unset uses HINDSIGHT_API_RETAIN_CHUNK_SIZE as the structured-chunk limit.
-# HINDSIGHT_API_RETAIN_STRUCTURED_CHUNK_SIZE=
-
-# When true, a retain operation that hit any fact-extraction errors is marked
-# 'failed' (not 'completed'), surfacing silently-dropped facts. Default false.
-# HINDSIGHT_API_FAIL_ON_EXTRACTION_ERRORS=false
-
-# Wall-clock ceiling (seconds) for one retain task in the worker. A retain that
-# blocks indefinitely is cancelled and marked 'failed' — and so becomes
-# retryable — instead of holding its worker slot until the process restarts.
-# Set well above your slowest healthy retain; 0 disables. Default 3600.
-# HINDSIGHT_API_RETAIN_WALL_TIMEOUT=3600
-
-# Dry-run extraction preview endpoint (POST /memories/dry-run-extract). Enabled by default; it makes
-# a real LLM call but stores nothing. Set to false to remove the endpoint (returns 404).
-# HINDSIGHT_API_ENABLE_DRY_RUN_EXTRACT=true
 
 # Base Path / Reverse Proxy Support (Optional)
 # Set these when deploying behind a reverse proxy with path-based routing
@@ -158,13 +59,6 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_READ_DATABASE_URL=  # Optional read-replica URL. When set, recall queries (semantic, BM25, graph, temporal) flow through a separate pool against this URL, offloading the primary. Typically points to a read-only endpoint (CNPG's <cluster>-ro service or Aurora reader endpoint).
 # HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
 # HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
-# HINDSIGHT_API_DB_MAX_PARALLEL_WORKERS_PER_GATHER=  # Optional cap on Postgres planner parallelism for this process's pool connections. Unset leaves the server default; 0 makes background/bulk queries run serially (useful on worker processes sharing a primary with latency-sensitive traffic).
-# HINDSIGHT_API_ENTITY_TRGM_SIMILARITY_THRESHOLD=0.15  # Postgres pg_trgm.similarity_threshold applied on every pool connection, used by entity resolution's % trigram match. Must be in (0, 1]. Lower catches more substring-ish matches at higher CPU cost on large entity sets; higher is stricter and cheaper.
-# HINDSIGHT_API_ENTITY_INTRABATCH_MERGE_SIMILARITY=0.5  # Trigram similarity (pg_trgm-equivalent, computed in-memory) at/above which two new names created by the SAME retain are merged into one entity (in-batch dedup of surface-form variants). Must be in (0, 1]. A merge cutoff, stricter than the recall threshold above; raise toward 1.0 to merge only near-identical forms.
-# HINDSIGHT_API_RETAIN_ENTITY_RESOLUTION_MAX_CANDIDATES=200  # Max candidates scored per entity mention during retain. The fuzzy lookup keeps only this many best matches per name (ranked by trigram/Jaro-Winkler similarity) before scoring them one by one. On banks holding thousands of near-identical names an uncapped set turns one retain into minutes of CPU that stall the worker's health checks. Raise only if entities that should merge are being duplicated.
-# HINDSIGHT_API_MIGRATION_CONCURRENCY=1  # Tenant schemas to migrate concurrently (PG only, each in its own process; per-schema work stays sequential). Each worker has ~1-2s startup cost + uses ~3 DB connections, so it only pays off with many schemas (tens+) or slow migrations; keep concurrency*3 <= spare max_connections. Default: 1 (sequential).
-# HINDSIGHT_API_OPERATION_RETENTION_DAYS=30  # Prune terminal operation rows, payloads, and metadata after this many days; 0 (the default) keeps them forever.
-# HINDSIGHT_API_OPERATION_CLEANUP_BATCH_SIZE=1000  # Maximum expired terminal rows deleted per tenant schema in each cleanup cycle; must be positive.
 
 # Vector Extension (Optional - uses pgvector by default)
 # Options: "pgvector" (default), "vchord", "pgvectorscale" (DiskANN)
@@ -184,56 +78,14 @@ HINDSIGHT_API_LOG_LEVEL=info
 # chinese_lindera/lindera(chinese), japanese_lindera/lindera(japanese),
 # korean_lindera/lindera(korean), ngram(min,max), edge_ngram(min,max)
 # HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER=
-# Optional cap on the number of terms in the native PostgreSQL BM25 tsquery.
-# Long queries OR-join every normalized token, which can match too much of a
-# large bank. 0 (default) keeps the historical uncapped behavior; a positive
-# value bounds only the native backend (other BM25 backends get the raw query).
-# HINDSIGHT_API_BM25_MAX_QUERY_TERMS=0
-
-# File Parser (Optional - uses markitdown by default)
-# HINDSIGHT_API_FILE_PARSER=markitdown
-# Enable image OCR for MarkItDown using an OpenAI-compatible OCR/vision endpoint.
-# These OCR settings are independent from HINDSIGHT_API_LLM_* because MarkItDown
-# uses the OpenAI SDK directly and requires Chat Completions image input support.
-# When OCR is enabled, API_KEY, BASE_URL, and MODEL are required.
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_ENABLED=false
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_API_KEY=
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_BASE_URL=
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_MODEL=
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_PROMPT=
-# Optional JSON dict of custom headers for the OCR OpenAI client (e.g. proxies / request tracing).
-# HINDSIGHT_API_FILE_PARSER_MARKITDOWN_OCR_DEFAULT_HEADERS=
 
 # Embeddings Configuration (Optional - uses local by default)
-# Provider: "local" (default), "onnx", "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
+# Provider: "local" (default), "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
 # For local provider:
 # HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=BAAI/bge-small-en-v1.5
-# Force CPU if local embeddings hit MPS/XPC instability on macOS:
-# HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=false
-# Opt in to the Apple Silicon MPS GPU (off by default: MPS leaks memory under
-# variable-length workloads). CUDA/XPU still auto-select regardless:
-# HINDSIGHT_API_EMBEDDINGS_LOCAL_ALLOW_MPS=false
-# For ONNX provider (local CPU embeddings without an Ollama/TEI sidecar):
-# HINDSIGHT_API_EMBEDDINGS_PROVIDER=onnx
-# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID=intfloat/multilingual-e5-small
-# HINDSIGHT_API_EMBEDDINGS_ONNX_FILE=onnx/model.onnx
-# HINDSIGHT_API_EMBEDDINGS_ONNX_DIMENSIONS=384
-# HINDSIGHT_API_EMBEDDINGS_ONNX_MAX_TOKENS=512
-# HINDSIGHT_API_EMBEDDINGS_ONNX_POOLING=mean
-# HINDSIGHT_API_EMBEDDINGS_ONNX_NORMALIZE=true
-# HINDSIGHT_API_EMBEDDINGS_ONNX_QUERY_PREFIX="query: "
-# HINDSIGHT_API_EMBEDDINGS_ONNX_PASSAGE_PREFIX="passage: "
-# Optional for local model paths or pre-downloaded artifacts:
-# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH=/models/multilingual-e5-small/onnx/model.onnx
-# HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH=/models/multilingual-e5-small
 # Optional for China network / restricted HF access:
 # HF_ENDPOINT=https://hf-mirror.com
-# Applies to any provider: cap each input at this many tiktoken tokens before
-# embedding, so oversized content is truncated instead of failing the embed call
-# permanently (e.g. Bedrock Titan V2's 8192, or a llama.cpp server's context). Off
-# by default. (Deprecated alias: HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS)
-# HINDSIGHT_API_EMBEDDINGS_MAX_INPUT_TOKENS=8192
 # For TEI provider:
 # HINDSIGHT_API_EMBEDDINGS_TEI_URL=http://localhost:8080
 # For OpenAI-compatible embeddings:
@@ -255,58 +107,13 @@ HINDSIGHT_API_LOG_LEVEL=info
 # DeepSeek note: DeepSeek is supported for LLM calls, but not for embeddings.
 # If using DeepSeek as LLM provider, keep embeddings on local/openai/cohere/google/etc.
 
-# Embedding similarity thresholds. These defaults preserve the behavior calibrated
-# for BAAI/bge-small-en-v1.5. Recalibrate each threshold independently when changing
-# embedding models because cosine-similarity distributions are model-dependent.
-# HINDSIGHT_API_SEMANTIC_MIN_SIMILARITY=0.3
-# HINDSIGHT_API_GRAPH_SEED_MIN_SIMILARITY=0.3
-# HINDSIGHT_API_TEMPORAL_SEMANTIC_MIN_SIMILARITY=0.1
-# HINDSIGHT_API_SEMANTIC_LINK_MIN_SIMILARITY=0.7
-# HINDSIGHT_API_CONSOLIDATION_DEDUP_THRESHOLD=0.97
-
-# Recall pipeline stages (all on by default). Each is hierarchical, so a single
-# bank can switch a stage off via the config API without changing the server
-# default. Turning all three off leaves semantic + BM25 fused by RRF, the
-# lowest-latency recall path.
-# Temporal retrieval arm, plus the date-aware query analysis that feeds it:
-# HINDSIGHT_API_ENABLE_TEMPORAL_RETRIEVAL=true
-# Entity/link graph traversal arm:
-# HINDSIGHT_API_ENABLE_GRAPH_RETRIEVAL=true
-# Cross-encoder rerank of the fused candidates (false = use the RRF order):
-# HINDSIGHT_API_ENABLE_RERANKING=true
-
 # Reranker Configuration (Optional - uses local by default)
 # Provider: "local" (default) or "tei" (HuggingFace Text Embeddings Inference)
 # HINDSIGHT_API_RERANKER_PROVIDER=local
-# Trusted gateway attribution (disabled by default). When enabled, remote
-# reranker requests include X-Hindsight-Bank-Id with the current bank ID.
-# HINDSIGHT_API_RERANKER_SEND_BANK_AS_HEADER=false
 # For local provider:
 # HINDSIGHT_API_RERANKER_LOCAL_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
-# Force CPU if the local reranker hits MPS/XPC instability on macOS:
-# HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=false
-# Opt in to the Apple Silicon MPS GPU (off by default: MPS leaks memory under
-# variable-length workloads). CUDA/XPU still auto-select regardless:
-# HINDSIGHT_API_RERANKER_LOCAL_ALLOW_MPS=false
 # For TEI provider:
 # HINDSIGHT_API_RERANKER_TEI_URL=http://localhost:8081
-# Max candidates the cross-encoder reranks per recall (RRF pre-filters the rest):
-# HINDSIGHT_API_RERANKER_MAX_CANDIDATES=300
-# Optionally scale that cap by the recall budget level (the cross-encoder dominates
-# a large recall's latency). 0 = fall back to the flat cap above; fully backwards-compatible.
-# HINDSIGHT_API_RERANKER_MAX_CANDIDATES_LOW=0
-# HINDSIGHT_API_RERANKER_MAX_CANDIDATES_MID=0
-# HINDSIGHT_API_RERANKER_MAX_CANDIDATES_HIGH=0
-
-# Reranker failover chain: extra rerankers tried, in order, when the one above
-# fails. Members are numbered from 1 (indices must be contiguous) and every
-# setting of member n carries the same index. A member inherits nothing from the
-# primary, so spell out everything it needs. Unset = no fallback (default): a
-# failing reranker fails the recall. End the chain with "rrf" to fail open and
-# keep the retrieval order instead.
-# HINDSIGHT_API_RERANKER_1_PROVIDER=cohere
-# HINDSIGHT_API_RERANKER_1_COHERE_API_KEY=your-cohere-api-key
-# HINDSIGHT_API_RERANKER_2_PROVIDER=rrf
 
 # Observability & Tracing (Optional - disabled by default)
 # Enable OpenTelemetry tracing for LLM calls (GenAI semantic conventions)
@@ -322,37 +129,6 @@ HINDSIGHT_API_LOG_LEVEL=info
 # Custom service name and environment (optional, defaults: hindsight-api, development)
 # HINDSIGHT_API_OTEL_SERVICE_NAME=hindsight-production
 # HINDSIGHT_API_OTEL_DEPLOYMENT_ENVIRONMENT=production
-#
-# Expose async-operation queue + consolidation-backlog gauges on /metrics.
-# Runs periodic per-schema COUNT queries on a background task (disabled by default).
-# HINDSIGHT_API_METRICS_BACKLOG_ENABLED=true
-#
-# Runtime-stall observability (enabled by default). When a liveness probe fails,
-# these tell you WHY: a blocked event loop vs DB connection-pool exhaustion.
-# The loop watchdog logs the offending stack when the loop is unresponsive; the
-# DB-pool acquire timing logs (and exposes hindsight.db.pool.waiting) when
-# callers queue for a connection. Both are cheap; tune or disable if needed.
-# HINDSIGHT_API_LOOP_WATCHDOG_ENABLED=false
-# HINDSIGHT_API_LOOP_WATCHDOG_STALL_THRESHOLD_MS=1000
-# HINDSIGHT_API_LOOP_WATCHDOG_POLL_INTERVAL_MS=250
-# HINDSIGHT_API_DB_ACQUIRE_WARN_THRESHOLD_MS=1000
-
-# -----------------------------------------------------------------------------
-# Webhooks (Optional)
-# -----------------------------------------------------------------------------
-
-# Outbound webhook delivery targets caller-supplied URLs. To prevent SSRF, the
-# delivery worker blocks private, loopback, and link-local destinations
-# (including the cloud metadata address 169.254.169.254) by default. List hosts
-# or IP/CIDR ranges here (comma-separated) to re-permit specific internal
-# destinations — e.g. 127.0.0.1 for local testing, or an internal receiver.
-# HINDSIGHT_API_WEBHOOK_ALLOWED_HOSTS=127.0.0.1,internal-receiver.svc,10.0.0.0/8
-
-# Whether the webhook delivery-history API returns the raw upstream response
-# body. Off by default: returning arbitrary response bodies to callers is an
-# information-exfiltration primitive. The delivery status code is always
-# returned regardless. Enable only if you trust your webhook destinations.
-# HINDSIGHT_API_WEBHOOK_EXPOSE_RESPONSE_BODY=false
 
 # -----------------------------------------------------------------------------
 # Control Plane (Optional)

.github/dependabot.yml

@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

.github/workflows/deploy-docs.yml

@@ -22,8 +22,6 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
-        with:
-          fetch-depth: 0 # fetch tags so check-released-integrations can see them
       - uses: actions/setup-node@v6
         with:
           node-version: 20

.github/workflows/perf-test.yml

@@ -23,9 +23,7 @@ on:
           - retain
           - recall
           - recall-with-observations
-          - recall-temporal
           - consolidation
-          - graph-maintenance
         default: ""
       locomo_conversations:
         description: "LoComo conversation IDs (space-separated). Blank = curated set (conv-26 conv-30 conv-43)."
@@ -35,18 +33,6 @@ on:
         description: "Skip LoComo job"
         type: boolean
         default: false
-      obs_skip:
-        description: "Skip observation-dedup benchmark job"
-        type: boolean
-        default: false
-      obs_dataset:
-        description: "Obs benchmark dataset substring (blank = English hermes transcript)."
-        type: string
-        default: ""
-      obs_fraction:
-        description: "Obs benchmark fraction (0-1] of each document to run."
-        type: string
-        default: "1.0"
       ref:
         description: "Git ref to test (branch, tag, or SHA). Defaults to main."
         type: string
@@ -212,86 +198,3 @@ jobs:
         PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
         GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       run: ./scripts/benchmarks/publish-locomo-results.sh hindsight-dev/benchmarks/locomo/results/benchmark_results.json
-
-  obs:
-    # Observation-dedup quality benchmark: ingests a transcript, drains consolidation
-    # (serial SyncTaskBackend + embedded pg0 — no external DB / worker), and reports the
-    # near-duplicate observation rate. Real LLM via VertexAI, mirroring the LoComo job.
-    if: inputs.obs_skip != true
-    runs-on: ubuntu-latest
-    env:
-      HINDSIGHT_API_LLM_PROVIDER: vertexai
-      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
-      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
-      HINDSIGHT_API_ENABLE_OBSERVATIONS: "true"
-    steps:
-    - uses: actions/checkout@v6
-      with:
-        ref: ${{ inputs.ref || github.ref }}
-
-    - name: Setup GCP credentials
-      run: |
-        printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
-        PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
-        echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
-
-    - name: Install uv
-      uses: astral-sh/setup-uv@v7
-      with:
-        enable-cache: true
-        prune-cache: false
-
-    - name: Set up Python
-      uses: actions/setup-python@v6
-      with:
-        python-version-file: ".python-version"
-
-    - name: Cache HuggingFace models
-      uses: actions/cache@v5
-      with:
-        path: ~/.cache/huggingface
-        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
-        restore-keys: |
-          ${{ runner.os }}-huggingface-
-
-    - name: Pre-download models
-      working-directory: ./hindsight-api-slim
-      run: |
-        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
-        from sentence_transformers import SentenceTransformer
-        print('Downloading embedding model...')
-        SentenceTransformer('BAAI/bge-small-en-v1.5')
-        print('Model downloaded successfully')
-        "
-
-    - name: Install hindsight-dev dependencies
-      run: |
-        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
-
-    - name: Run obs benchmark
-      # Default to the English hermes transcript at full fraction — a clean, deterministic
-      # consolidation-dedup signal (the Chinese variant adds a cross-lingual embedding
-      # confound). Override dataset/fraction via workflow_dispatch.
-      run: |
-        DATASET="${{ inputs.obs_dataset }}"
-        if [ -z "$DATASET" ]; then DATASET="hermes_session_2026-05-15_en"; fi
-        FRACTION="${{ inputs.obs_fraction }}"
-        if [ -z "$FRACTION" ]; then FRACTION="1.0"; fi
-        cd hindsight-dev
-        uv run python -m benchmarks.obs.obs_benchmark \
-          --dataset "$DATASET" --fraction "$FRACTION" --wipe-bank --output obs-results.json
-
-    - name: Upload obs results
-      if: always()
-      uses: actions/upload-artifact@v7
-      with:
-        name: obs-results-${{ github.sha }}
-        path: hindsight-dev/obs-results.json
-        retention-days: 90
-
-    - name: Publish obs to dashboard
-      if: success() && (github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch')
-      env:
-        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
-        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      run: ./scripts/benchmarks/publish-obs-results.sh hindsight-dev/obs-results.json

.github/workflows/release-integration.yml

@@ -9,11 +9,7 @@ jobs:
   publish:
     runs-on: ubuntu-latest
     permissions:
-      id-token: write  # for PyPI trusted publishing + build-provenance attestations
-      attestations: write  # for actions/attest-build-provenance (Obsidian assets)
-      # No `contents: write`: we never create releases in this repo. The Obsidian
-      # plugin's distribution release is pushed to its dedicated repo using
-      # OBSIDIAN_DIST_TOKEN (see the "Mirror Obsidian plugin" step below).
+      id-token: write  # for PyPI trusted publishing
 
     steps:
     - uses: actions/checkout@v6
@@ -75,7 +71,7 @@ jobs:
       if: steps.type.outputs.type == 'plugin'
       run: |
         echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
-        echo "Users install via: claude plugin marketplace add vectorize-io/hindsight"
+        echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
 
     # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
 
@@ -116,71 +112,6 @@ jobs:
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
       run: npm run build
 
-    # Build-provenance attestations for the Obsidian release assets (community-store
-    # recommendation). Runs after the build so main.js exists. The assets are
-    # released in the dedicated repo while the build runs here, so users verify at
-    # owner scope: `gh attestation verify main.js --owner vectorize-io`.
-    - name: Attest Obsidian plugin build provenance
-      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
-      uses: actions/attest-build-provenance@v2
-      with:
-        subject-path: |
-          hindsight-integrations/obsidian/main.js
-          hindsight-integrations/obsidian/styles.css
-
-    # ── Obsidian plugin — mirror to its dedicated repo + cut the BRAT release ──
-    # We do NOT create a GitHub Release in this monorepo: per-integration
-    # releases pollute the repo's release list (it's for the core product) and
-    # steal the "Latest" badge, and BRAT / the community store read a repo's
-    # *latest* release — not a tag — so they can't target a tag in a monorepo.
-    #
-    # Instead this monorepo stays the source of truth, and on each obsidian
-    # release we mirror hindsight-integrations/obsidian/ → the *root* of
-    # github.com/vectorize-io/hindsight-obsidian (git subtree, history
-    # preserved) and cut the BRAT / community-store release *there*.
-    #
-    # Requires secret OBSIDIAN_DIST_TOKEN — a token with `contents: write` on
-    # vectorize-io/hindsight-obsidian (fine-grained PAT or app installation
-    # token). The dedicated repo is generated; never edit it directly.
-    - name: Mirror Obsidian plugin to its dedicated repo
-      if: steps.type.outputs.type == 'typescript' && steps.info.outputs.integration == 'obsidian'
-      env:
-        DIST_TOKEN: ${{ secrets.OBSIDIAN_DIST_TOKEN }}
-      run: |
-        set -euo pipefail
-        VERSION="${{ steps.info.outputs.version }}"
-        DIST_REPO="vectorize-io/hindsight-obsidian"
-        OBS_DIR="hindsight-integrations/obsidian"
-
-        # `git subtree split` needs full history; the default checkout is shallow.
-        git fetch --unshallow 2>/dev/null || true
-        git config user.name "github-actions[bot]"
-        git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-
-        # The runner injects the default GITHUB_TOKEN as an http.extraheader via
-        # an *included* config file (/home/runner/work/_temp/git-credentials-*.config),
-        # so `git config --local --unset-all` can't remove it and it authenticates
-        # the push as github-actions[bot] (no access to the dedicated repo → 403).
-        # The documented way to drop an inherited extraheader is to RESET the list
-        # with an empty value: since command-line `-c` is read last, the empty
-        # value clears the accumulated headers (including the included one) at
-        # request-build time. The dist token then comes from the push URL → a
-        # single Authorization header.
-        git subtree split --prefix="$OBS_DIR" -b _obs_dist
-        git -c "http.https://github.com/.extraheader=" \
-          push "https://x-access-token:${DIST_TOKEN}@github.com/${DIST_REPO}.git" _obs_dist:main
-
-        # Cut the BRAT / community-store release. Bare version tag (e.g. 0.1.0)
-        # to match manifest.json — idempotent so re-runs just refresh the assets.
-        export GH_TOKEN="$DIST_TOKEN"
-        ASSETS="$OBS_DIR/main.js $OBS_DIR/manifest.json $OBS_DIR/styles.css"
-        NOTES="Hindsight for Obsidian v${VERSION}. Install via BRAT (add ${DIST_REPO}) or copy main.js/manifest.json/styles.css into <vault>/.obsidian/plugins/hindsight/."
-        if gh release view "$VERSION" --repo "$DIST_REPO" >/dev/null 2>&1; then
-          gh release upload "$VERSION" $ASSETS --repo "$DIST_REPO" --clobber
-        else
-          gh release create "$VERSION" $ASSETS --repo "$DIST_REPO" --title "$VERSION" --notes "$NOTES"
-        fi
-
     - name: Publish TypeScript package to npm
       if: steps.type.outputs.type == 'typescript'
       working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
@@ -190,12 +121,7 @@ jobs:
         EXIT_CODE=$?
         echo "$OUTPUT"
         if [ $EXIT_CODE -ne 0 ]; then
-          # Treat "already published" as success so re-pointed-tag re-runs stay green.
-          # "cannot publish over" = the version exists. TLOG_CREATE_ENTRY_ERROR / 409
-          # "equivalent entry already exists in the transparency log" = the identical
-          # --provenance artifact was already logged on a prior run (Sigstore tlog is
-          # idempotent); the package is published, so this is benign.
-          if echo "$OUTPUT" | grep -qE "cannot publish over|TLOG_CREATE_ENTRY_ERROR|already exists in the transparency log"; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
             echo "Package version already published, skipping..."
             exit 0
           fi

.github/workflows/release.yml

@@ -25,20 +25,6 @@ jobs:
       with:
         python-version-file: ".python-version"
 
-    # Each package is built from its own directory, so stage the repository's
-    # canonical license inside each isolated build context.
-    - name: Stage Python package licenses
-      run: |
-        for package in \
-          hindsight-clients/python \
-          hindsight-api-slim \
-          hindsight-api \
-          hindsight-all \
-          hindsight-all-slim \
-          hindsight-embed; do
-          cp LICENSE "$package/LICENSE"
-        done
-
     # Build all packages
     - name: Build hindsight-client
       working-directory: ./hindsight-clients/python
@@ -64,24 +50,6 @@ jobs:
       working-directory: ./hindsight-embed
       run: uv build --out-dir dist
 
-    - name: Verify Python package licenses
-      run: |
-        for package in \
-          hindsight-clients/python \
-          hindsight-api-slim \
-          hindsight-api \
-          hindsight-all \
-          hindsight-all-slim \
-          hindsight-embed; do
-          wheel=$(find "$package/dist" -maxdepth 1 -name '*.whl' -print -quit)
-          sdist=$(find "$package/dist" -maxdepth 1 -name '*.tar.gz' -print -quit)
-
-          unzip -Z1 "$wheel" | grep -Eq '\.dist-info/licenses/LICENSE$'
-          unzip -p "$wheel" '*/METADATA' | grep -Fxq 'License-Expression: MIT'
-          unzip -p "$wheel" '*/METADATA' | grep -Fxq 'License-File: LICENSE'
-          tar -tzf "$sdist" | grep -Eq '/LICENSE$'
-        done
-
     # Publish in order (client and api-slim first, then api/all wrappers which depend on them)
     - name: Publish hindsight-client to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
@@ -298,7 +266,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - os: ubuntu-22.04
+          - os: ubuntu-latest
             target: x86_64-unknown-linux-gnu
             artifact_name: hindsight
             asset_name: hindsight-linux-amd64
@@ -310,7 +278,7 @@ jobs:
             target: aarch64-apple-darwin
             artifact_name: hindsight
             asset_name: hindsight-darwin-arm64
-          - os: ubuntu-22.04-arm
+          - os: ubuntu-24.04-arm
             target: aarch64-unknown-linux-gnu
             artifact_name: hindsight
             asset_name: hindsight-linux-arm64

.github/workflows/star-history.yml

@@ -1,29 +0,0 @@
-name: Update star history
-
-on:
-  schedule:
-    - cron: '17 3 * * *'
-  workflow_dispatch:
-
-permissions:
-  contents: write
-
-jobs:
-  update:
-    concurrency:
-      group: star-history
-      cancel-in-progress: false
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: nicoloboschi/gh-stars@v1
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          line-color: '#14b8a6'
-      - name: Commit chart
-        run: |
-          git config user.name 'github-actions[bot]'
-          git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
-          git add .github/star-history/data.json .github/star-history/chart.svg
-          git diff --cached --quiet || git commit -m 'chore: update star history'
-          git push

.gitignore

@@ -5,15 +5,7 @@ build/
 dist/
 wheels/
 *.egg-info
-# Release builds stage the canonical root license in each package context.
-/hindsight-clients/python/LICENSE
-/hindsight-api-slim/LICENSE
-/hindsight-api/LICENSE
-/hindsight-all/LICENSE
-/hindsight-all-slim/LICENSE
-/hindsight-embed/LICENSE
 .mcp.json
-.playwright-mcp/
 .osgrep
 # Virtual environments
 .venv
@@ -23,8 +15,6 @@ node_modules/
 
 # Environment variables and local config
 .env
-.env.bak*
-.env.*.bak
 docker-compose.yml
 docker-compose.override.yml
 
@@ -48,7 +38,6 @@ nltk_data/
 logs/
 
 .DS_Store
-.sesskey
 
 # Generated docs files
 hindsight-docs/static/llms-full.txt
@@ -70,5 +59,4 @@ hindsight-integrations/_drafts/
 # Changelog is now tracked in hindsight-docs/src/pages/changelog.md
 # CHANGELOG.md
 
-blog-post*
-.worktrees/
+blog-post*

.sesskey

@@ -0,0 +1 @@
+fcac2839-1db5-432f-91e1-c5dac07d7290

CLAUDE.md

@@ -216,46 +216,10 @@ migration file dispatches through `run_for_dialect`, which calls either
 ./scripts/hooks/lint.sh
 ```
 
-Dead-code detection runs in CI (the `check-unused-code` job) at two levels:
-- **Blocking:** unused imports (ruff `F401`) and variables (`F841`) — `lint.sh` auto-removes
-  them and `verify-generated-files` fails on any leftover diff; and **knip** for orphaned
-  control-plane files / unused (or unlisted) `package.json` dependencies.
-- **Advisory:** whole unused Python functions (vulture) and unused control-plane *exports*
-  (the shadcn/ui surface is kept on purpose) — surfaced, not gated.
-
-Run both locally with:
-```bash
-./scripts/hooks/check-unused.sh
-```
-
 **After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
 
 **MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
 
-### Testing
-
-Most tests are deterministic (MockLLM, pure functions) — assert directly.
-
-**Tests that verify LLM behaviour use a real LLM + an LLM-as-judge.** When the thing under test is *how the model interprets a prompt* (classification, attribution, dimension preservation, instruction-following), MockLLM can't simulate it and exact string/enum asserts flake across providers and runs. Use this pattern instead:
-
-1. Mark the test module `pytestmark = pytest.mark.hs_llm_core` (single-provider; CI runs it in the core-LLM job). Use `hs_llm_mat` only for provider-matrix acceptance tests.
-2. Call the real pipeline (`LLMConfig.from_env()`, `_get_raw_config()`), e.g. `extract_facts_from_text(...)`.
-3. Assert with the judge, not string matching:
-   ```python
-   from tests.llm_judge import assert_meets_criteria
-   facts_summary = "\n".join(f"- [{f.fact_type}] {f.fact}" for f in facts)
-   await assert_meets_criteria(
-       response=facts_summary,
-       criteria="The first-person user statements are classified 'world' and attributed to the user, not the agent.",
-       context="What the input said and who was speaking.",
-   )
-   ```
-
-Rules of thumb:
-- **Judge anything non-deterministic** — including `fact_type` classification and speaker attribution. Do NOT hard-assert `fact_type == "..."`; pass a `[fact_type] fact` summary to the judge instead. Structural facts that ARE deterministic (counts, presence of a field, that a substring was injected into a prompt) stay as direct asserts in fast unit tests.
-- **Split the test surface**: cover the deterministic mechanics (prompt assembly, suppression logic) with fast non-LLM unit tests, and the model-following behaviour with one `hs_llm_core` judge test. (Example pair: `test_narrator_resolution.py` + `test_narrator_context_override.py`.)
-- The judge model is independent of the test provider (defaults to Gemini); never judge with the same call you're testing.
-
 ### Memory Banks
 - Each bank is an isolated memory store (like a "brain" for one user/agent)
 - Banks have dispositions (skepticism, literalism, empathy traits 1-5) affecting reflect
@@ -286,35 +250,6 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
    - Update the client type definition in `lib/api.ts`
    - Update any UI components that need to use the new parameter
 
-### Harness Attribution (which coding agent wrote a document)
-
-`hindsight-integrations/hindsight-coding-agents/` stamps the coding agent on every
-document it retains, so the control plane can show its logo instead of another
-`key=value` chip:
-
-- `metadata.harness = "<id>"` — the authoritative field
-- tag `harness:<id>` — the same value, so the documents list can filter on it
-
-The ids are defined by that integration's HookSpecs
-(`src/harness/hook-lifecycle.ts`) plus the persistent-plugin entrypoints
-registered in `src/harness/registry.ts`, whose id is their
-`createPluginEntry(...)` argument — currently `antigravity-cli`, `claude-code`,
-`cline-cli`, `codex`, `copilot-cli`, `cursor-cli`, `devin-cli`, `grok-build`,
-`kilo`, `opencode`.
-
-The control plane resolves the value in
-`hindsight-control-plane/src/lib/harness-logo.ts` (metadata wins over the tag) and
-renders it with `components/ui/harness-logo.tsx` in the documents table and the
-document detail dialog. **Adding a harness to the integration means adding it to
-that registry in the same change**: copy its icon from
-`hindsight-docs/static/img/icons/` (or take it from the agent's own brand assets
-when the docs site carries none) into
-`hindsight-control-plane/public/img/harness/` and add one entry. Don't register
-ids nothing writes — a test asserts the registry matches the emitted set, plus an
-explicit list of retired ids kept so already-retained documents keep their logo.
-An unregistered harness is not an error: it renders no logo and still shows as
-ordinary metadata.
-
 ### Adding New Integrations
 
 Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
@@ -356,10 +291,7 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
    ```
 
 2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
-   - No change is needed for ordinary environment-backed config fields. The CLI starts from `_get_raw_config()`,
-     so new `HindsightConfig` fields are carried through automatically.
-   - If the new field should be overridable by a CLI flag, add the argparse option in `_parse_cli_args()` and include
-     that field in the `dataclasses.replace(config, ...)` call near the "CLI override" comment.
+   - Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
 
 3. **Use hierarchical config in MemoryEngine**:
    ```python
@@ -379,16 +311,6 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
    - Add to appropriate section table with Variable, Description, Default
    - Mark if it's hierarchical (can be overridden per-bank)
 
-6. **Env template** (`.env.example`):
-   - Add the variable to the appropriate section, commented if optional, with a
-     short inline comment describing it (mirror the documentation entry).
-   - This file is the single source of truth for the env template:
-     `scripts/dev/setup.sh` copies it to `.env`, and `hindsight-embed` ships a
-     bundled copy (`hindsight-embed/hindsight_embed/env.example`) that seeds
-     embed/profile configs. After editing `.env.example`, re-copy it to the
-     embed package (`cp .env.example hindsight-embed/hindsight_embed/env.example`)
-     or the `test_bundled_template_matches_repo_root` sync test will fail.
-
 #### Hierarchical vs Static Guidelines
 
 **Hierarchical** (per-bank overridable):
@@ -405,7 +327,7 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
 
 ```bash
 cp .env.example .env
-# Edit .env with the LLM provider/model and credentials for your setup
+# Edit .env with LLM API key
 
 # Python deps
 uv sync --directory hindsight-api-slim/
@@ -414,10 +336,10 @@ uv sync --directory hindsight-api-slim/
 npm install
 ```
 
-Common LLM settings:
+Required env vars:
 - `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio
-- `HINDSIGHT_API_LLM_API_KEY`: API key for providers that require one
-- `HINDSIGHT_API_LLM_MODEL`: Model name (defaults are provider-specific)
+- `HINDSIGHT_API_LLM_API_KEY`: Your API key
+- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., gpt-4o-mini, claude-sonnet-4-20250514)
 
 Optional (uses local models by default):
 - `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei

README.md

@@ -7,6 +7,7 @@
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
 [![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
 <br/>
@@ -61,16 +62,16 @@ If you need more control over how and when your agent stores and recalls memorie
 ```bash
 export OPENAI_API_KEY=sk-xxx
 
-docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8888 -p 9999:9999 \
+docker run --rm -it --pull always -p 8888:8888 -p 9999:9999 \
   -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
-  -v hindsight-data:/home/hindsight/.pg0 \
+  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
   ghcr.io/vectorize-io/hindsight:latest
 ```
 
 >API: http://localhost:8888
 >UI: http://localhost:9999
 
-You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, `minimax`, and `atlas` ([Atlas Cloud](https://www.atlascloud.ai/?utm_source=github&utm_medium=link&utm_campaign=hindsight)). The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
+You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `minimax`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
 
 
 
@@ -142,8 +143,6 @@ main();
 pip install hindsight-all -U
 ```
 
-On Intel (x86_64) Macs, install `hindsight-all-slim` instead — see [Supported Platforms](#supported-platforms).
-
 ```python
 import os
 from hindsight import HindsightServer, HindsightClient
@@ -250,7 +249,7 @@ Recall performs 4 retrieval strategies in parallel:
 - Graph: Entity/temporal/causal links
 - Temporal: Time range filtering
 
-![Recall Operation](hindsight-docs/static/img/recall-operation.webp)
+![Retain Operation](hindsight-docs/static/img/recall-operation.webp)
 
 The individual results from the retrievals are merged, then ordered by relevance using reciprocal rank fusion and a cross-encoder reranking model.
 
@@ -276,7 +275,7 @@ client = Hindsight(base_url="http://localhost:8888")
 client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 ```
 
-![Reflect Operation](hindsight-docs/static/img/reflect-operation.webp)
+![Retain Operation](hindsight-docs/static/img/reflect-operation.webp)
 
 ---
 
@@ -298,20 +297,7 @@ client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 ---
 ## Star History
 
-[![Star history](https://raw.githubusercontent.com/vectorize-io/hindsight/main/.github/star-history/chart.svg)](https://github.com/vectorize-io/hindsight/stargazers)
----
-
-## Supported Platforms
-
-| Platform | Docker | Bare Metal (pip) | Embedded DB (pg0) |
-|----------|--------|------------------|--------------------|
-| **Linux** (x86_64, ARM64) | ✅ | ✅ | ✅ |
-| **macOS** (Apple Silicon / arm64) | ✅ | ✅ | ✅ |
-| **macOS** (Intel / x86_64) | ✅ | ⚠️ | ✅ |
-| **Windows** (x86_64) | ✅ | ✅ | ✅ |
-
-⚠️ Intel Macs: use `hindsight-all-slim` — see the [installation guide](https://hindsight.vectorize.io/developer/installation#supported-platforms) for details.
-
+[![Star History Chart](https://api.star-history.com/svg?repos=vectorize-io/hindsight&type=date&legend=top-left)](https://www.star-history.com/#vectorize-io/hindsight&type=date&legend=top-left)
 ---
 
 ## Contributing

deno.lock

@@ -77,6 +77,7 @@
             "npm:@radix-ui/react-dropdown-menu@^2.1.16",
             "npm:@radix-ui/react-label@^2.1.8",
             "npm:@radix-ui/react-popover@^1.1.15",
+            "npm:@radix-ui/react-radio-group@^1.3.8",
             "npm:@radix-ui/react-select@^2.2.6",
             "npm:@radix-ui/react-slider@^1.3.6",
             "npm:@radix-ui/react-slot@^1.2.4",

docker/docker-compose/custom-models/Dockerfile

@@ -17,7 +17,7 @@ FROM ghcr.io/vectorize-io/hindsight:latest-slim
 # `pip install` would fall back to user site-packages and not be visible
 # to the runtime python.
 RUN uv pip install --python /app/api/.venv/bin/python --no-cache \
-    'sentence-transformers>=5.0.0' \
+    'sentence-transformers>=3.3.0' \
     'transformers>=4.53.0' \
     'torch>=2.6.0'
 

docker/docker-compose/pgroonga/Dockerfile

@@ -3,11 +3,21 @@
 # pgroonga is a multilingual full-text search extension built on Groonga.
 # It works out of the box for CJK (Chinese, Japanese, Korean) and other
 # non-whitespace-segmented languages via the TokenBigram tokenizer.
-FROM groonga/pgroonga:4.0.8-debian-17
+FROM groonga/pgroonga:latest-debian-pg17
 
 # Install pgvector on top of the pgroonga base image (which already provides
-# pgroonga, the Groonga library, and the PostgreSQL PGDG package repository).
+# pgroonga and the Groonga library).
 RUN apt-get update && apt-get install -y --no-install-recommends \
-    postgresql-17-pgvector=0.8.6-1.pgdg13+1 \
-    && apt-get clean \
+    build-essential \
+    git \
+    postgresql-server-dev-17 \
     && rm -rf /var/lib/apt/lists/*
+
+RUN cd /tmp && \
+    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
+    cd pgvector && \
+    make && \
+    make install
+
+RUN rm -rf /tmp/pgvector && \
+    apt-get purge -y --auto-remove build-essential git postgresql-server-dev-17

docker/docker-compose/vchord/docker-compose.yaml

@@ -1,6 +1,6 @@
 name: hindsight
 # Docker Compose file for Hindsight with PostgreSQL and vectorchord
-# docker compose -f docker/docker-compose/vchord/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/vchord/docker-compose.yaml up -d
+# docker compose -f docker/docker-compose/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/docker-compose.yaml up -d
 # Make sure to set the required environment variables before running:
 # - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
 # - Configure LLM provider variables as needed (see below in the hindsight service)

docker/standalone/Dockerfile

@@ -41,43 +41,26 @@ RUN apt-get update && apt-get install -y \
     && rm -rf /var/lib/apt/lists/* \
     && pip install --no-cache-dir uv
 
-# Copy the workspace lock and member metadata before source code so dependency
-# installation stays cacheable while matching the versions tested in CI.
-COPY pyproject.toml uv.lock ./
-COPY hindsight-all/pyproject.toml ./hindsight-all/
-COPY hindsight-api/pyproject.toml ./hindsight-api/
+# Copy dependency files and README (required by pyproject.toml)
 COPY hindsight-api-slim/pyproject.toml ./api/
 COPY hindsight-api-slim/README.md ./api/
-COPY hindsight-all-slim/pyproject.toml ./hindsight-all-slim/
-COPY hindsight-dev/pyproject.toml ./hindsight-dev/
-COPY hindsight-clients/python/pyproject.toml ./hindsight-clients/python/
-COPY hindsight-embed/pyproject.toml ./hindsight-embed/
-RUN ln -s api hindsight-api-slim
+
+WORKDIR /app/api
 
 # Sync dependencies using appropriate extras based on INCLUDE_LOCAL_MODELS
 # local-ml: torch, sentence-transformers, transformers, einops, flashrank, mlx (optional)
 # embedded-db: pg0-embedded (always included for embedded PostgreSQL support)
-# ONNX Runtime embeddings are intentionally not bundled into the official
-# standalone image; install the local-onnx extra in custom images when needed.
-ENV UV_PROJECT_ENVIRONMENT=/app/api/.venv
 RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-        uv sync --locked --package hindsight-api-slim --no-install-package hindsight-api-slim --extra local-ml --extra embedded-db; \
+        uv sync --extra local-ml --extra embedded-db; \
     else \
-        uv sync --locked --package hindsight-api-slim --no-install-package hindsight-api-slim --extra embedded-db; \
+        uv sync --extra embedded-db; \
     fi
 
 # Copy source code (alembic migrations are inside hindsight_api/)
-WORKDIR /app/api
 COPY hindsight-api-slim/hindsight_api ./hindsight_api
 
-# Install the local package from the same validated lock after source is present.
-WORKDIR /app
-RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-        uv sync --locked --package hindsight-api-slim --extra local-ml --extra embedded-db; \
-    else \
-        uv sync --locked --package hindsight-api-slim --extra embedded-db; \
-    fi \
-    && uv pip check --python /app/api/.venv/bin/python
+# Install the local package (uv sync only installed dependencies, not the package itself)
+RUN uv pip install -e .
 
 # =============================================================================
 # Stage: SDK Builder (needed for Control Plane)
@@ -160,8 +143,6 @@ FROM python:3.11-slim AS api-only
 WORKDIR /app
 
 # Note: libicu version varies by Debian version - try common versions in order
-# Runtime images use uv directly; remove pip build tooling after installation so
-# vulnerable setuptools-vendored packages and wheel are not shipped in production.
 RUN apt-get update && apt-get install -y \
     curl \
     procps \
@@ -171,8 +152,7 @@ RUN apt-get update && apt-get install -y \
     libossp-uuid16 \
     && (apt-get install -y libicu72 2>/dev/null || apt-get install -y libicu74 2>/dev/null || apt-get install -y libicu76 2>/dev/null || true) \
     && rm -rf /var/lib/apt/lists/* \
-    && pip install --no-cache-dir uv \
-    && pip uninstall --yes setuptools wheel
+    && pip install --no-cache-dir uv
 
 RUN useradd -m -s /bin/bash hindsight
 
@@ -310,8 +290,6 @@ WORKDIR /app
 
 # Install Node.js, curl, uv, and system dependencies
 # Note: libicu version varies by Debian version - try common versions in order
-# Runtime images use uv directly; remove pip build tooling after installation so
-# vulnerable setuptools-vendored packages and wheel are not shipped in production.
 RUN apt-get update && apt-get install -y \
     curl \
     procps \
@@ -323,8 +301,7 @@ RUN apt-get update && apt-get install -y \
     && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
     && apt-get install -y nodejs \
     && rm -rf /var/lib/apt/lists/* \
-    && pip install --no-cache-dir uv \
-    && pip uninstall --yes setuptools wheel
+    && pip install --no-cache-dir uv
 
 RUN useradd -m -s /bin/bash hindsight
 

docker/standalone/start-all.sh

@@ -43,56 +43,11 @@ check_pg0_data_integrity() {
     return 0
 }
 
-# =============================================================================
-# Embedded pg0 writability pre-check (#1483)
-#
-# The container runs as the unprivileged `hindsight` user (UID 1000). When the
-# pg0 data directory is a host bind mount (e.g. `-v $HOME/dir:/home/hindsight/.pg0`)
-# that is not owned by UID 1000 — the default on macOS Docker Desktop and most
-# non-1000 Linux hosts — pg0 fails with the opaque "Permission denied (os error
-# 13)". We cannot chown it ourselves without root (and the image is deliberately
-# rootless), so we surface an actionable message up front instead.
-#
-# Docker *named* volumes are seeded with the image directory's ownership (UID
-# 1000) on first use, so they avoid this entirely — hence the named-volume
-# recommendation below and in the README.
-# =============================================================================
-check_pg0_writable() {
-    local pg0_data_dir="$1"
-
-    # Only relevant for embedded pg0; an external database doesn't use this dir.
-    if [ -n "${HINDSIGHT_API_DATABASE_URL:-}" ]; then
-        return 0
-    fi
-
-    mkdir -p "$pg0_data_dir" 2>/dev/null || true
-    if touch "$pg0_data_dir/.hindsight-write-test" 2>/dev/null; then
-        rm -f "$pg0_data_dir/.hindsight-write-test" 2>/dev/null || true
-        return 0
-    fi
-
-    echo "❌ The embedded database directory $pg0_data_dir is not writable by this container (UID $(id -u))."
-    echo ""
-    echo "   A host directory was bind-mounted but is not owned by the container user (UID 1000)."
-    echo "   Hindsight runs rootless and cannot fix this for you. Choose one:"
-    echo ""
-    echo "   • Recommended — use a Docker named volume (auto-owned by the container):"
-    echo "       -v hindsight-data:/home/hindsight/.pg0"
-    echo ""
-    echo "   • Or keep the host path and run as your host user, chowning it to match:"
-    echo "       sudo chown -R \$(id -u):\$(id -g) <host-directory>"
-    echo "       docker run --user \$(id -u):\$(id -g) -e HOME=/home/hindsight ..."
-    echo ""
-    echo "   See https://github.com/vectorize-io/hindsight/issues/1483"
-    return 1
-}
-
 if [ "${HINDSIGHT_START_ALL_SOURCE_ONLY:-false}" = "true" ]; then
     return 0 2>/dev/null || exit 0
 fi
 
 check_pg0_data_integrity "${HOME}/.pg0"
-check_pg0_writable "${HOME}/.pg0" || exit 1
 
 # Service flags (default to true if not set)
 ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"

docker/standalone/test-start-all.sh

@@ -8,7 +8,7 @@ source "$SCRIPT_DIR/start-all.sh"
 unset HINDSIGHT_START_ALL_SOURCE_ONLY
 
 TMP_DIR="$(mktemp -d)"
-trap 'chmod -R u+rwx "$TMP_DIR" 2>/dev/null || true; rm -rf "$TMP_DIR"' EXIT
+trap 'rm -rf "$TMP_DIR"' EXIT
 
 assert_contains() {
     local output="$1"
@@ -71,51 +71,3 @@ nonempty_output="$(check_pg0_data_integrity "$TMP_DIR/nonempty")"
 assert_contains "$nonempty_output" "WARNING: pg0 data directory exists"
 
 echo "start-all pg0 integrity checks passed"
-
-# =============================================================================
-# check_pg0_writable (#1483)
-# These rely on filesystem permissions, which root bypasses; skip under root.
-# =============================================================================
-if [ "$(id -u)" != "0" ]; then
-    # Writable directory: returns 0, prints nothing, leaves no artifact behind.
-    mkdir -p "$TMP_DIR/writable"
-    writable_output="$(check_pg0_writable "$TMP_DIR/writable")"
-    assert_empty "$writable_output"
-    if [ -e "$TMP_DIR/writable/.hindsight-write-test" ]; then
-        echo "check_pg0_writable left its write-test file behind"
-        exit 1
-    fi
-
-    # Non-writable directory: returns 1 with actionable guidance.
-    mkdir -p "$TMP_DIR/readonly"
-    chmod 000 "$TMP_DIR/readonly"
-    set +e
-    readonly_output="$(check_pg0_writable "$TMP_DIR/readonly" 2>&1)"
-    readonly_rc=$?
-    set -e
-    chmod 755 "$TMP_DIR/readonly"
-    if [ "$readonly_rc" -eq 0 ]; then
-        echo "check_pg0_writable should fail on a non-writable directory"
-        exit 1
-    fi
-    assert_contains "$readonly_output" "not writable"
-    assert_contains "$readonly_output" "hindsight-data:/home/hindsight/.pg0"
-    assert_contains "$readonly_output" "--user"
-
-    # External database configured: skip the check regardless of dir perms.
-    mkdir -p "$TMP_DIR/extdb"
-    chmod 000 "$TMP_DIR/extdb"
-    set +e
-    HINDSIGHT_API_DATABASE_URL="postgres://x" check_pg0_writable "$TMP_DIR/extdb" >/dev/null 2>&1
-    extdb_rc=$?
-    set -e
-    chmod 755 "$TMP_DIR/extdb"
-    if [ "$extdb_rc" -ne 0 ]; then
-        echo "check_pg0_writable should skip when an external database is configured"
-        exit 1
-    fi
-
-    echo "start-all pg0 writability checks passed"
-else
-    echo "⚠️  Running as root; skipping pg0 writability checks (permissions are bypassed)."
-fi

docs/superpowers/plans/2026-07-25-v2-knowledge-pages.md

@@ -1,341 +0,0 @@
-# v2 Knowledge Pages — Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Make v2 knowledge pages a reliable, cleanly-tiered "wiki" surface: passive `entity_labels` tier-tagging + tag-scoped seeded pages, a `hindsight_*` MCP surface with one active `capture_initiative` verb that creates per-initiative pages linked by tag, and SessionStart/UserPromptSubmit page-roster injection.
-
-**Architecture:** Shared TS core (`hindsight-integrations/hindsight-coding-agents`). Extraction stays blind to "pages"; classification is intrinsic (`knowledge:<tier>` tags), pages are tag-scoped saved views. Per-initiative navigation via a `relatedPageId:<id>` tag the synthesizer renders into `[[page:<id>]]`, with the Initiatives folder/roster as the guaranteed fallback.
-
-**Tech Stack:** TypeScript, vitest, tsup bundling. Hindsight REST API (`/knowledge-base/*`, `/mental-models`, `/memories`, bank `/config`).
-
-**Spec:** `docs/superpowers/specs/2026-07-25-v2-knowledge-pages-design.md`
-
-**Working dir for all commands:** `hindsight-integrations/hindsight-coding-agents`
-**Test command:** `npx vitest run <file>` (fast suite; excludes `*.live.test.ts`). Full check: `npx vitest run && npx tsc --noEmit`.
-
-**Conventions to follow (existing patterns):**
-- `HindsightClient` HTTP via `this.req("METHOD", this.bankUrl(path), body?)`; JSON via `await r.json()`.
-- MCP tools are SDK-free `ToolSpec { name, description, inputSchema (ZodRawShape), handler }`; wrap handler bodies in `guarded(...)`; `ok(value)` / `err(e)` result helpers.
-- Fail-open everywhere in hooks; pure logic separated from stdin/stdout plumbing.
-- Do NOT add a Claude co-author trailer to any commit.
-
----
-
-## Task 1: Config field `pageRefreshEveryTurns`
-
-**Files:**
-- Modify: `src/core/config.ts`
-- Test: `src/core/config.test.ts`
-
-- [ ] **Step 1: Write failing test** — assert the default resolves to 10 and an override wins.
-
-```ts
-it("pageRefreshEveryTurns defaults to 10 and is overridable", () => {
-  expect(loadConfig({ harness: "claude-code", projectDir: process.cwd() }).pageRefreshEveryTurns).toBe(10);
-});
-```
-(Add an override case mirroring the existing override tests in this file.)
-
-- [ ] **Step 2: Run** `npx vitest run src/core/config.test.ts` → FAIL (property missing).
-- [ ] **Step 3: Implement** — add `pageRefreshEveryTurns: number` to the `Config` type and default `10` in the same place `recallMaxTokens`/`reflectTimeoutMs` are defined/merged. Follow the exact merge/layering pattern already used for numeric fields.
-- [ ] **Step 4: Run** the test → PASS.
-- [ ] **Step 5: Commit** `git add src/core/config.ts src/core/config.test.ts && git commit -m "feat(core): add pageRefreshEveryTurns config (default 10)"`
-
----
-
-## Task 2: `knowledge-injection.ts` — roster/preamble formatting (pure, new)
-
-**Files:**
-- Create: `src/core/knowledge-injection.ts`
-- Test: `src/core/knowledge-injection.test.ts`
-
-Pure, SDK-free, no network. Parses the `listPages()` payload and formats the two injections.
-
-- [ ] **Step 1: Write failing tests**
-
-```ts
-import { describe, expect, it } from "vitest";
-import { parsePageList, buildKnowledgePreamble, buildRosterRefresh } from "./knowledge-injection";
-
-describe("parsePageList", () => {
-  it("extracts {id,title} from the mental-model list shape, tolerating junk", () => {
-    const raw = { items: [{ id: "p1", name: "Component map" }, { id: "p2", name: "Core concepts" }, { nope: 1 }] };
-    expect(parsePageList(raw)).toEqual([{ id: "p1", title: "Component map" }, { id: "p2", title: "Core concepts" }]);
-  });
-  it("returns [] for null/garbage", () => {
-    expect(parsePageList(null)).toEqual([]);
-    expect(parsePageList(42 as unknown)).toEqual([]);
-  });
-});
-
-describe("buildKnowledgePreamble", () => {
-  it("includes guidance, a roster of pages, and a refresh note", () => {
-    const out = buildKnowledgePreamble([{ id: "p1", title: "Component map" }]);
-    expect(out).toContain("<hindsight_knowledge>");
-    expect(out).toContain("Component map");
-    expect(out).toContain("p1");
-    expect(out).toMatch(/hindsight_read_knowledge_page/);
-  });
-  it("has an empty-state line when there are no pages", () => {
-    const out = buildKnowledgePreamble([]);
-    expect(out).toMatch(/no knowledge pages yet|still learning/i);
-  });
-});
-
-describe("buildRosterRefresh", () => {
-  it("is a compact 'current pages' block listing ids+titles", () => {
-    const out = buildRosterRefresh([{ id: "p1", title: "Component map" }]);
-    expect(out).toContain("Component map");
-    expect(out).toContain("p1");
-  });
-  it("returns undefined when there are no pages (nothing to refresh)", () => {
-    expect(buildRosterRefresh([])).toBeUndefined();
-  });
-});
-```
-
-- [ ] **Step 2: Run** `npx vitest run src/core/knowledge-injection.test.ts` → FAIL.
-- [ ] **Step 3: Implement**
-
-```ts
-export interface PageRef { id: string; title: string; }
-
-/** Defensive parse of HindsightClient.listPages() (GET /mental-models?detail=metadata → {items:[{id,name}]}). */
-export function parsePageList(raw: unknown): PageRef[] {
-  const items = (raw as { items?: unknown })?.items;
-  if (!Array.isArray(items)) return [];
-  const out: PageRef[] = [];
-  for (const it of items) {
-    const id = (it as { id?: unknown })?.id;
-    const name = (it as { name?: unknown })?.name;
-    if (typeof id === "string" && typeof name === "string") out.push({ id, title: name });
-  }
-  return out;
-}
-
-function roster(pages: PageRef[]): string {
-  return pages.map((p) => `- ${p.title} (${p.id})`).join("\n");
-}
-
-/** SessionStart: teach when/why to use pages + list what exists. Empty-state aware. */
-export function buildKnowledgePreamble(pages: PageRef[]): string {
-  const body = pages.length
-    ? `Knowledge pages available in this repository:\n${roster(pages)}`
-    : "No knowledge pages yet — Hindsight is still learning this repo; they'll appear as it processes.";
-  return (
-    "<hindsight_knowledge>\n" +
-    "This repository has a Hindsight knowledge base: curated, continuously-updated pages summarizing its " +
-    "durable engineering knowledge (architecture, components, conventions, key decisions, and in-flight initiatives).\n" +
-    "Before substantial work, consult the relevant pages instead of re-deriving understanding from the code: read " +
-    "Conventions before writing new code, the Component map before changing a subsystem, and an initiative's page " +
-    "before continuing that feature.\n" +
-    `${body}\n` +
-    "Read one with hindsight_read_knowledge_page(page_id). Follow any [[page:<id>]] links you see. The list is " +
-    "re-injected for you periodically as it changes.\n" +
-    "</hindsight_knowledge>"
-  );
-}
-
-/** Periodic UserPromptSubmit refresh — compact, or undefined when there's nothing to show. */
-export function buildRosterRefresh(pages: PageRef[]): string | undefined {
-  if (!pages.length) return undefined;
-  return (
-    "<hindsight_knowledge_refresh>\n" +
-    `Current Hindsight knowledge pages (may have changed):\n${roster(pages)}\n` +
-    "Read any with hindsight_read_knowledge_page(page_id).\n" +
-    "</hindsight_knowledge_refresh>"
-  );
-}
-```
-
-- [ ] **Step 4: Run** the test → PASS.
-- [ ] **Step 5: Commit** `git add src/core/knowledge-injection.ts src/core/knowledge-injection.test.ts && git commit -m "feat(core): knowledge-injection roster/preamble formatting"`
-
----
-
-## Task 3: `entity_labels` tier vocabulary + configureBank wiring
-
-**Files:**
-- Modify: `src/core/missions.ts` (add `KNOWLEDGE_LABELS`)
-- Modify: `src/core/hindsight.ts` (`configureBank` PATCH sets `entity_labels`)
-- Test: `src/core/hindsight.*.test.ts` (add/extend a config test with a mock client)
-
-- [ ] **Step 1: Write failing test** — assert `configureBank` PATCHes `/config` with `entity_labels` containing the `knowledge` group and its five values, and `entities_allow_free_form: true`. Use the existing fetch/req mock pattern from `hindsight.*.test.ts`; capture the PATCH body to `/config` and assert on it.
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-  - In `missions.ts`, export `KNOWLEDGE_LABELS` — the exact object from the spec §4 (`key:"knowledge"`, `type:"multi-values"`, `optional:true`, `tag:true`, the verbose group `description`, and the five value `{value,description}` entries: feature-work, decision, convention, component, concept).
-  - In `hindsight.ts::configureBank`, extend the existing `PATCH .../config` `updates` object to include `entity_labels: [KNOWLEDGE_LABELS]` and `entities_allow_free_form: true`. Import `KNOWLEDGE_LABELS`.
-  - Update the `[bank] configured …` log to mention `entity_labels`.
-- [ ] **Step 4: Run** → PASS.
-- [ ] **Step 5: Commit** `git add src/core/missions.ts src/core/hindsight.ts src/core/hindsight.*.test.ts && git commit -m "feat(core): passive knowledge entity_labels tier vocabulary + configureBank wiring"`
-
----
-
-## Task 4: Tag-scoped seeded pages + Initiatives folder + link source_query
-
-**Files:**
-- Modify: `src/core/missions.ts` (`PAGES` gain `tags`; Initiatives `source_query` link instruction)
-- Modify: `src/core/hindsight.ts` (`ensureFolder`, `createPages` sets page `tags` + parents Initiatives under the folder)
-- Test: `src/core/hindsight.pages.test.ts`
-
-- [ ] **Step 1: Write failing tests** (mock client `req`):
-  - Each seeded page POST to `/knowledge-base/pages` includes `tags: ["knowledge:<tier>"]` mapped per the spec §5 table.
-  - The Initiatives page is created with `parent_id` equal to the id returned by an Initiatives folder POST to `/knowledge-base/folders`.
-  - `ensureFolder("Initiatives")` returns an existing root folder's id when the tree already contains it (GET `/knowledge-base/tree`) and does NOT POST a duplicate.
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-  - `missions.ts`: add `tags: string[]` to each `PAGES` entry (feature-work/decision/convention/component/concept mapping). Append to the Initiatives `source_query`: *"When a source memory carries a tag of the form `relatedPageId:<id>`, include a Markdown link `[[page:<id>]]` to that page in the summary, so each initiative links to its detailed page."*
-  - `hindsight.ts`: add
-    ```ts
-    /** Find a root folder by name (case-insensitive) or create it; returns its id. */
-    async ensureFolder(name: string): Promise<string | undefined> {
-      try {
-        const tree = (await (await this.req("GET", this.bankUrl("/knowledge-base/tree"))).json()) as
-          { roots?: { id?: string; kind?: string; name?: string }[] };
-        const hit = (tree.roots || []).find((n) => n.kind === "folder" && (n.name || "").toLowerCase() === name.toLowerCase());
-        if (hit?.id) return hit.id;
-      } catch { /* fall through to create */ }
-      try {
-        const r = await this.req("POST", this.bankUrl("/knowledge-base/folders"), { name });
-        return ((await r.json()) as { id?: string }).id;
-      } catch { return undefined; }
-    }
-    ```
-  - In `createPages()`: before the loop, `const initiativesFolderId = await this.ensureFolder("Initiatives");`. For each page, build body `{ name, source_query, tags: p.tags, parent_id: <initiativesFolderId if this is the Initiatives page else undefined>, trigger: { fact_types:[...], refresh_after_consolidation:true } }`. (Page-level `tags` drives synthesis scoping via `RefreshTagFiltering`; `tags_match` defaults to `all_strict` when tags present.)
-- [ ] **Step 4: Run** → PASS.
-- [ ] **Step 5: Commit** `git add src/core/missions.ts src/core/hindsight.ts src/core/hindsight.pages.test.ts && git commit -m "feat(core): tag-scope seeded pages, Initiatives folder, relatedPageId link source_query"`
-
----
-
-## Task 5: Client helpers — per-initiative page + marker retain
-
-**Files:**
-- Modify: `src/core/hindsight.ts` (`captureInitiative`)
-- Test: `src/core/hindsight.pages.test.ts`
-
-- [ ] **Step 1: Write failing tests** (mock `req`):
-  - `captureInitiative({title:"Retry backoff for the uploader", summary:"…"})` → derives slug `retry-backoff-for-the-uploader`, POSTs a page id `initiative-<slug>` to `/knowledge-base/pages` with `parent_id` = the Initiatives folder and `tags: ["knowledge:feature-work"]`, AND POSTs a marker to `/memories` (via `retain`) tagged `["knowledge:feature-work","relatedPageId:initiative-<slug>"]`, strategy `session` or `document` (pick `document`), `async:true`. Returns `{ page_id: "initiative-<slug>" }`.
-  - Slug is deterministic and identical between the page id and the `relatedPageId:` tag value.
-  - Enhancement path: `captureInitiative({title, summary, relatesToPageId:"initiative-x"})` POSTs NO new page; marker tagged `relatedPageId:initiative-x`; returns `{ page_id: "initiative-x" }`.
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-
-```ts
-private slugify(s: string): string {
-  return s.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, "").slice(0, 60) || "initiative";
-}
-
-/** Active-path capture: register a major feature as a per-initiative page + a tagged marker memory. */
-async captureInitiative(args: { title: string; summary: string; relatesToPageId?: string }): Promise<{ page_id: string }> {
-  const pageId = args.relatesToPageId ?? `initiative-${this.slugify(args.title)}`;
-  if (!args.relatesToPageId) {
-    const folderId = await this.ensureFolder("Initiatives");
-    await this.req("POST", this.bankUrl("/knowledge-base/pages"), {
-      name: args.title,
-      source_query: `Summarize the "${args.title}" initiative: what is being built or changed and why, and its current state — drawn from the project's memory.`,
-      parent_id: folderId,
-      tags: ["knowledge:feature-work", `relatedPageId:${pageId}`],
-      trigger: { fact_types: ["world", "experience", "observation"], refresh_after_consolidation: true },
-    });
-  }
-  const verb = args.relatesToPageId ? "Enhancement to an existing initiative" : "New initiative";
-  const content = `${verb}: ${args.title}. ${args.summary}`;
-  await this.retain(content, "initiative marker", pageId /* not a stable doc id requirement; see note */,
-    ["knowledge:feature-work", `relatedPageId:${pageId}`], "document", { async: true });
-  return { page_id: pageId };
-}
-```
-  - NOTE: use a UNIQUE document id per marker (e.g. `initiative-marker-<slug>-<n>`), NOT `pageId`, so repeated enhancement captures accrue instead of replacing. Since `Date.now()` is fine here (runtime, not a workflow script), suffix with a timestamp: `initiative-marker-${this.slugify(args.title)}-${Date.now()}`. Keep the `relatedPageId` tag equal to `pageId`.
-  - Confirm `retain(content, context, documentId, tags, strategy, opts)` signature matches current `HindsightClient.retain`.
-
-- [ ] **Step 4: Run** → PASS.
-- [ ] **Step 5: Commit** `git add src/core/hindsight.ts src/core/hindsight.pages.test.ts && git commit -m "feat(core): captureInitiative — per-initiative page + relatedPageId marker"`
-
----
-
-## Task 6: MCP surface — `hindsight_*` grounding + `capture_initiative`; drop page CRUD
-
-**Files:**
-- Modify: `src/core/knowledge-tools.ts`
-- Modify: `src/mcp-server.ts` (only if it references removed tool names)
-- Test: `src/core/knowledge-tools.test.ts`, `src/mcp-server.test.ts` (tool-count assertions)
-
-- [ ] **Step 1: Write failing tests**
-  - `buildKnowledgeTools(client, bankId)` returns exactly these tool names: `hindsight_get_current_bank`, `hindsight_list_knowledge_pages`, `hindsight_read_knowledge_page`, `hindsight_search_memory`, `hindsight_capture_initiative`, `hindsight_ingest_document`. (Assert the set; update any count assertion.)
-  - `hindsight_capture_initiative` handler calls `client.captureInitiative` with `{title, summary, relatesToPageId?}` and returns the page id (mock client).
-  - No `create_page` / `update_page` / `delete_page` tools are present.
-  - Each tool still fails closed via `guarded` (a thrown client error → `isError:true`, no throw).
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-  - Rebuild the `buildKnowledgeTools` list: rename read/recall/ingest/bank tools to the `hindsight_*` names; drop `create_page`/`update_page`/`delete_page`; add `hindsight_capture_initiative` with `inputSchema { title: z.string(), summary: z.string(), relates_to_page_id: z.string().optional() }` calling `client.captureInitiative({ title, summary, relatesToPageId: relates_to_page_id })`.
-  - Use the **verbatim agent-facing `description` strings** from the spec §6 / the brainstorm (grounding tools + the explicit WHEN/WHEN-NOT `capture_initiative` description).
-  - Update `mcp-server.ts` only if it enumerates tool names; otherwise it consumes `buildKnowledgeTools` generically and needs no change.
-- [ ] **Step 4: Run** `npx vitest run src/core/knowledge-tools.test.ts src/mcp-server.test.ts` → PASS.
-- [ ] **Step 5: Commit** `git add src/core/knowledge-tools.ts src/mcp-server.ts src/core/knowledge-tools.test.ts src/mcp-server.test.ts && git commit -m "feat(mcp): hindsight_* grounding tools + capture_initiative; remove raw page CRUD from agent"`
-
----
-
-## Task 7: SessionStart — preamble + roster
-
-**Files:**
-- Modify: `src/core/session-start.ts`
-- Test: `src/core/session-start.test.ts`
-
-- [ ] **Step 1: Write failing tests**
-  - `buildSessionStartContext` now fetches pages via the client and injects `buildKnowledgePreamble(...)` instead of the static `KNOWLEDGE_MISSION`. Extend the `SeedContextClient` interface with `listPages(): Promise<unknown>`; the mock returns `{items:[{id:"p1",name:"Component map"}]}` and the output contains "Component map".
-  - listPages failure is fail-open: the preamble still renders (empty-state) and the seed logic is unaffected.
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-  - Add `listPages` to `SeedContextClient`.
-  - Replace the `parts.push(KNOWLEDGE_MISSION)` line with: fetch `const pages = parsePageList(await client.listPages().catch(() => null));` then `parts.push(buildKnowledgePreamble(pages));`. Import from `./knowledge-injection`.
-  - Remove the now-unused `KNOWLEDGE_MISSION` export if nothing else references it (grep first; keep if referenced).
-- [ ] **Step 4: Run** → PASS.
-- [ ] **Step 5: Commit** `git add src/core/session-start.ts src/core/session-start.test.ts && git commit -m "feat(core): SessionStart injects page roster + guidance preamble"`
-
----
-
-## Task 8: UserPromptSubmit — hook-counted periodic roster refresh
-
-**Files:**
-- Modify: `src/core/hook.ts`
-- Test: `src/core/hook.test.ts`
-
-- [ ] **Step 1: Write failing tests**
-  - The session cache round-trips `{answer, turns}`; each `buildHookOutput` call increments `turns`.
-  - Add `listPages` to the `HookClient` interface. On a turn where `turns % cfg.pageRefreshEveryTurns === 0`, the output includes `buildRosterRefresh(...)` content (assert "Component map" appears); on other turns it does not.
-  - Refresh is fail-open (a `listPages` rejection doesn't break recall/injection).
-  - First-turn behavior (reflect) unchanged.
-
-- [ ] **Step 2: Run** → FAIL.
-- [ ] **Step 3: Implement**
-  - Extend the cache read/write to `{ answer?: string; turns?: number }`. Compute `const turns = (cached.turns ?? 0) + 1;` and persist it alongside `answer`.
-  - Add `listPages(): Promise<unknown>` to `HookClient`.
-  - After computing `memBlock`, if `cfg.pageRefreshEveryTurns > 0 && turns % cfg.pageRefreshEveryTurns === 0`, `try { const refresh = buildRosterRefresh(parsePageList(await client.listPages())); if (refresh) blocks.push(refresh); } catch { /* fail-open */ }`. Kick the `listPages` call off concurrently with recall to avoid added latency.
-  - Import from `./knowledge-injection`.
-- [ ] **Step 4: Run** `npx vitest run src/core/hook.test.ts` → PASS.
-- [ ] **Step 5: Commit** `git add src/core/hook.ts src/core/hook.test.ts && git commit -m "feat(core): UserPromptSubmit hook-counted periodic page-roster refresh"`
-
----
-
-## Task 9: Full check + LLM behavior (live) verification
-
-**Files:**
-- Modify: `src/system.live.test.ts` (add coverage; runs only under `HINDSIGHT_LIVE_E2E=1`)
-
-- [ ] **Step 1: Full fast suite + types** — `npx vitest run && npx tsc --noEmit` → all green.
-- [ ] **Step 2: Add a live assertion** (guarded by the existing live env flag) that after seeding a small repo + one `captureInitiative`, the Initiatives page content contains a `[[page:initiative-…]]` link (verifies the `relatedPageId` → link rendering end-to-end). Keep it in the live suite; do not run in the fast job.
-- [ ] **Step 3: Manual/live run** (optional, operator): `HINDSIGHT_API_URL=http://localhost:8888 npm run test:live`.
-- [ ] **Step 4: Commit** `git add src/system.live.test.ts && git commit -m "test(live): initiative page renders relatedPageId link end-to-end"`
-
----
-
-## Final review
-
-- [ ] Dispatch a final code-reviewer over the whole change set against the spec (`docs/superpowers/specs/2026-07-25-v2-knowledge-pages-design.md`).
-- [ ] Rebuild + dev-install the `claude-code-v2` bundle so the running plugin picks up the new hooks/MCP (`bash scripts/dev-install.sh`); do not push/PR without explicit consent.
-- [ ] Note deferred follow-ups: session drill-down tag, `capture_decision`, `gotcha` tier, older-bank reseed requirement.

docs/superpowers/specs/2026-07-25-v2-knowledge-pages-design.md

@@ -1,135 +0,0 @@
-# v2 Knowledge Pages — Design Spec
-
-**Status:** approved in brainstorm (2026-07-25), pending implementation plan
-**Scope:** `hindsight-integrations/hindsight-coding-agents` (shared TS core) + `claude-code-v2` wrapper
-**Motivation:** make knowledge pages a real, trustworthy "wiki" surface for the vectorize-crm demo (the `knowledge-pages-as-trust-surface` principle) — the agent reliably knows what pages exist, pages are cleanly tiered instead of blended, and major initiatives become first-class, linkable pages.
-
----
-
-## 1. Problem
-
-Three gaps in the current v2 branch:
-
-1. **Page discovery is a blind fetch.** SessionStart injects a static `KNOWLEDGE_MISSION` telling the agent to call `agent_knowledge_list_pages`, but hands it **no roster** — the agent never learns a page exists unless it independently decides to call the tool. Per-turn recall injects facts, not pages.
-2. **Pages are blended.** Neither the seeded `PAGES` nor the agent's `create_page` tool scope synthesis by tag, so every page synthesizes from the whole bank filtered only by `fact_type`. Git-log, session, and survey memories all bleed into every page.
-3. **No first-class initiative tracking / linking.** Hindsight has no native page-to-page links. A "major feature" leaves no durable, navigable page a future session can pick up.
-
-## 2. Principles applied
-
-- Automatic/visible value; zero out-of-band CLI; memory beats code search; knowledge pages as a trust surface; minimal post-setup burden.
-- Modular units, small files, follow existing patterns (per-hook specs, fail-open, unit-testable pure cores).
-- The **memory extractor never knows what a "page" is.** Classification is by the fact's *intrinsic* nature; pages are application-side saved views. No abstraction leak into extraction.
-
-## 3. Architecture overview
-
-Two complementary curation paths + a discovery layer:
-
-- **Passive (automatic):** `entity_labels` schema-forces the extractor to tag qualifying facts `knowledge:<tier>`. Seeded **tier pages** each filter on one tier tag. No agent effort.
-- **Active (high-signal):** one intent-named MCP verb, `hindsight_capture_initiative`, lets the agent register a major feature as a **per-initiative page** with a tag-based link back from the aggregate Initiatives page.
-- **Discovery:** SessionStart injects guidance + the page roster; the UserPromptSubmit hook re-injects a fresh roster on a fixed cadence (hook-counted, not model-counted).
-
-## 4. `entity_labels` — passive tier tagging
-
-One hierarchical bank config group, set by `configureBank` at seed time:
-
-```jsonc
-{
-  "key": "knowledge",
-  "type": "multi-values",   // 0, 1, or several — empty is normal
-  "optional": true,
-  "tag": true,               // emits knowledge:<value> onto the fact's tags
-  "description": "Routing labels for this project's Hindsight KNOWLEDGE PAGES — curated, human-readable summaries of the repo's DURABLE engineering knowledge (architecture, key decisions, conventions, ongoing initiatives), each page rebuilt automatically from the facts labeled for it. Mark a fact only when it is durable, reusable knowledge a developer would still want surfaced in future sessions. IMPORTANT: leave this EMPTY for routine, transient, or operational facts — a passing test, a one-off command, a status update, a debugging dead-end. MOST facts should get no label here. Assign more than one value only when the fact genuinely fits several.",
-  "values": [
-    { "value": "feature-work", "description": "A new feature, initiative, or enhancement being planned or built — the capability being added and the intent behind it. Not routine bug-fixes or chores." },
-    { "value": "decision",     "description": "A technical decision that will constrain future work, with its rationale — why this approach was chosen over alternatives, or a rule deliberately adopted." },
-    { "value": "convention",   "description": "An established way this project does things — naming, structure, testing, error handling, or another recurring pattern a contributor is expected to follow." },
-    { "value": "component",    "description": "What a specific module, file, service, or subsystem is responsible for, or how components depend on and connect to one another." },
-    { "value": "concept",      "description": "A domain concept, key abstraction, or piece of project vocabulary a new contributor must understand to work effectively." }
-  ]
-}
-```
-
-Notes:
-- `tag: true` → `_inject_label_tags` copies each `knowledge:<value>` onto the fact's `tags` (no extra query infra).
-- Selectivity (multi-values + "mostly empty" instruction) prevents force-fitting routine facts into a tier.
-
-## 5. Seeded tier pages (tag-scoped)
-
-Created via `/knowledge-base/pages` (supports `tags`, `trigger`, `parent_id`) — **not** `/mental-models`. Each `PAGES` entry gains a `trigger.tags` pin:
-
-| Page | `trigger.tags` |
-| --- | --- |
-| Initiatives and enhancements | `["knowledge:feature-work"]` |
-| Key decisions and rationale | `["knowledge:decision"]` |
-| Conventions and patterns | `["knowledge:convention"]` |
-| Component map | `["knowledge:component"]` |
-| Core concepts | `["knowledge:concept"]` |
-
-`tags_match` strict enough to exclude untagged facts (`all_strict`/`any_strict`). Tag matching is exact set-ops (no wildcards) — this is *why* the vocabulary is fixed, not per-feature.
-
-## 6. MCP surface
-
-Raw page CRUD (`create_page`/`update_page`/`delete_page`) is **removed** from the agent. The agent sees grounding tools + one capture verb. Naming convention: `hindsight_*`.
-
-**Grounding**
-- `hindsight_list_knowledge_pages` `{}` — roster: id, title, one-line coverage. (agent-facing description as drafted in brainstorm)
-- `hindsight_read_knowledge_page` `{ page_id }` — full page content; follow `[[page:<id>]]` links by re-calling.
-- `hindsight_search_memory` `{ query, max_tokens? }` — raw fact recall for specifics pages don't cover.
-- `hindsight_get_current_bank` `{}` — minor introspection (kept).
-
-**Capture**
-- `hindsight_capture_initiative` `{ title, summary, relates_to_page_id? }` — the one active verb. Explicit WHEN / WHEN-NOT description (as drafted). Returns the initiative page id.
-- `hindsight_ingest_document` `{ title, content }` — existing `agent_knowledge_ingest`, reframed.
-
-(Full agent-facing descriptions are captured verbatim in the brainstorm thread and will be reproduced in the implementation plan.)
-
-## 7. `hindsight_capture_initiative` mechanism
-
-- Derive one slug `S` from `title`. Page id = `initiative-<S>`. **The slug in the tag and the page id are the same token, derived once** (cannot drift).
-- **New initiative** (`relates_to_page_id` omitted):
-  1. Create page `initiative-<S>` (title from `title`, `source_query` about that initiative) under an **"Initiatives" folder** (tag-scoped).
-  2. Retain a marker memory (text = title + summary) tagged `["knowledge:feature-work", "relatedPageId:initiative-<S>"]`. **No session tag** (decided — the MCP server has no Claude session id; faking one wouldn't link to the Stop write-back's `conversation:<sessionId>` doc anyway).
-- **Enhancement** (`relates_to_page_id` given): marker only, `relatedPageId = relates_to_page_id`; no new page. Re-invoking for the same initiative accrues markers → the page re-synthesizes with progress.
-
-### Link survival (why `relatedPageId` as a tag, not in prose)
-
-A tag is set directly via the retain `tags` param — it **bypasses LLM extraction entirely**, so it's guaranteed present verbatim (no REF-ID-style preservation needed at extraction). Verified: the reflect/synthesis path SELECTs `tags` and serializes facts via `_prune_nulls(model_dump())`, which keeps non-empty tags → **the synthesis LLM sees the tag.** The **Initiatives page `source_query`** instructs: *"when a memory carries a `relatedPageId:<id>` tag, emit a `[[page:<id>]]` link to it."* The link id is generated from the tag value at synthesis time, so it always matches the created page id.
-
-- Only **Stage 2 (synthesis)** is probabilistic now (bounded token budget may omit some entries when there are many).
-- **Guaranteed fallback:** the per-initiative page always exists (created via API, independent of any LLM stage) and appears in the **Initiatives folder / injected roster**, so navigation works even if a synthesized inline link drops.
-
-## 8. Page-access injection
-
-- **SessionStart** (`session-start.ts`): replace static `KNOWLEDGE_MISSION` with a preamble = (a) guidance on *when/why* to consult pages, (b) the roster fetched via `client.listPages()` (`- <title> (<id>)`, empty-state aware), (c) a note that the list refreshes periodically. Cold repo → empty roster line; roster comes alive mid-session as seeding/survey complete.
-- **UserPromptSubmit** (`hook.ts`): extend the per-session cache (`{answer}` → `{answer, turns}`); the **hook** counts user turns and, roughly every `pageRefreshEveryTurns` (default 10, approximate), calls `listPages()` and injects a compact roster refresh. Runs concurrently with recall; **fail-open** (a refresh error never blocks the turn).
-- **Shared formatting** (new `core/knowledge-injection.ts`, SDK-free/unit-testable): `parsePageList(raw) -> {id,title}[]`, `buildKnowledgePreamble(pages)`, `buildRosterRefresh(pages)`.
-- **Config:** `pageRefreshEveryTurns` (default 10).
-
-## 9. Non-goals / deferred
-
-- Session drill-down tag on captured markers (dropped — see §7).
-- `hindsight_capture_decision` and other capture verbs (passive path covers those tiers; revisit if the aggregate pages aren't sharp enough).
-- A `gotcha`/`pitfall` tier (five tiers for now).
-- Native page-to-page links / backlinks (Hindsight has none; we approximate via folder tree + `relatedPageId`-driven `[[page:<id>]]`).
-
-## 10. Risks / migration
-
-- **Older banks** need re-seeding to pick up the new `entity_labels`, the `session` retain strategy, and the tag-scoped page triggers (`configureBank` sets them). User is starting fresh with v2 banks, so acceptable; live retain fails open otherwise.
-- **Stage-2 synthesis omission** for large initiative counts — mitigated by the folder/roster fallback.
-- **Instruction adherence** for the `source_query` link-rendering and the label selectivity — both are LLM-following behaviors; cover with an `hs_llm_core` judge test, and the deterministic mechanics (tag injection, roster formatting, slug/id equality, hook turn-counting) with fast unit tests.
-
-## 11. Testing
-
-- **Deterministic unit tests:** `knowledge-injection` formatting + empty-state; hook turn-counter + cadence; `capture_initiative` slug→id→tag equality and request shape (mock client); tag-scoped page request bodies; entity_labels config emitted by `configureBank`.
-- **LLM judge test (`hs_llm_core`):** label selectivity (routine facts get no `knowledge:*`), and `relatedPageId` → `[[page:<id>]]` rendering in a synthesized Initiatives page.
-
-## 12. File map (anticipated)
-
-- `src/core/knowledge-injection.ts` (new) — roster/preamble formatting.
-- `src/core/session-start.ts` — preamble + roster.
-- `src/core/hook.ts` — cache `{answer,turns}` + periodic roster refresh.
-- `src/core/config.ts` — `pageRefreshEveryTurns`.
-- `src/core/missions.ts` — `entity_labels` group; tag-scoped `PAGES`; Initiatives `source_query` link instruction.
-- `src/core/hindsight.ts` — `configureBank` sets `entity_labels`; `createPages` pins `trigger.tags` + Initiatives folder; new `createInitiativePage`/marker retain helpers.
-- `src/core/knowledge-tools.ts` — new `hindsight_*` grounding + `capture_initiative` tools; remove raw page CRUD from agent surface.
-- Tests alongside each.

docs/superpowers/specs/2026-07-27-reflect-pages-runtime.md

@@ -1,139 +0,0 @@
-# Reflect + Pages Runtime — Design Spec
-
-**Status:** decided (2026-07-27), reconciles the earlier reflect-based runtime with the recall-based v2 into one opinionated path
-**Scope:** `hindsight-integrations/hindsight-coding-agents` (shared TS core) + `claude-code-v2` wrapper
-**Motivation:** the 33-task coding benchmark showed the v2 recall-per-prompt runtime *underperforms no memory* (35.0 mean corrections vs 32.0 baseline), while the earlier reflect-injection runtime beats baseline by 22% (25.0). This spec restores reflect as the only deep-memory path and replaces raw per-turn recall with lightweight injection from knowledge pages — "fast like recall, organized like reflect" — keeping v2's page/curation machinery where it earned its place and deleting it where it didn't.
-
----
-
-## 1. Problem
-
-Two prior iterations, each half right:
-
-1. **Reflect runtime (v1):** one agentic REFLECT over the bank at session start, cached and re-injected every turn. Benchmark-proven (25.0 mean corrections) — but nothing surfaced mid-session; a task that drifted away from the first message got stale context.
-2. **Recall runtime (v2):** per-prompt recall injection for turn-by-turn visibility, plus knowledge pages as a trust surface. But raw recall injects unsynthesized fact fragments — noise that *hurt*: 35.0 mean corrections, worse than running with no memory at all.
-
-| Runtime | Mean corrections (33-task benchmark) | vs no-memory (32.0) |
-| --- | --- | --- |
-| Reflect-injection (v1) | **25.0** | **−22%** |
-| Recall-per-prompt (v2) | 35.0 | +9% (regression) |
-| No memory | 32.0 | baseline |
-
-The reconciliation: keep reflect's synthesis quality as the deep path, keep v2's per-turn visibility principle, but source the per-turn material from the already-synthesized knowledge pages instead of raw recall.
-
-## 2. Decisions
-
-Explicit, decided — not options:
-
-1. **Reflect restored** as the only deep-memory path (session-start, agentic synthesis, cached + re-injected every turn).
-2. **Recall removed from the runtime** entirely. No per-prompt `recall` call.
-3. **No `memoryMode` flag.** One opinionated path; config is for environment, naming, and harness wiring only — never behavior selection.
-4. **Sections, not pages, are the per-turn injection unit** — locally matched, budget-trimmed, provenance-labeled.
-5. **JSON turn transcripts** replace the markdown tool-call transcript in the Stop-hook write-back, with compact action entries.
-6. **No tags / no `entity_labels`.** The server re-synthesizes pages after consolidation; "living pages" needs no client-side tagging machinery.
-
-## 3. Runtime path — session start
-
-Three steps, in order, all inside existing hooks (no out-of-band CLI):
-
-### 3a. Cold-repo bootstrap (kept from v2)
-
-On a bank with no prior memories: automatic shallow gitlog seed + codebase survey, exactly as v2 does it. The user never runs a setup command; the first session self-seeds. (Deep ingestion of that history is §7 — the seed here stays instant.)
-
-### 3b. REFLECT once, on the first task message
-
-The benchmark-proven core:
-
-- On the first user prompt of the session, run one **REFLECT** — agentic synthesis over the whole bank, prompted to return the *root-cause decision with exact values* (concrete file paths, config values, version numbers — not summaries of summaries).
-- Cache the result per session; **re-inject it every turn**. It is the session's durable deep context.
-- One LLM-backed call per session, on the message that actually states the task — not on session-open, where there is nothing to reflect about.
-
-### 3c. Page index build
-
-Fetch all knowledge pages once (existing `listPages` + page reads), split each page at headings into **sections**, and build a **local section index** in the hook process. This index is what every subsequent turn matches against (§4) — no further server calls on the hot path.
-
-## 4. Runtime path — every turn
-
-Per-turn visibility, satisfied at ~zero latency and ~zero cost. Injection sources from **knowledge pages, not raw recall** — the material is already synthesized and organized; the turn hook only *selects* from it.
-
-Mechanism (local, deterministic — no server call, no LLM call):
-
-| Aspect | Design |
-| --- | --- |
-| Unit | Page **sections** (pages split at headings at index-build time) |
-| Matching | Lexical: prompt scored against each section by weighted term overlap; **heading hits weighted higher** than body hits |
-| Selection | Top 2–3 sections |
-| Budget | Trimmed to a **~700-token total** |
-| Provenance | Each snippet labeled `From <page> › <section>` + a tool pointer to read the full page |
-| Floor | A minimum-score threshold below which **nothing is injected** — silence over noise |
-| Refresh | Section index rebuilt on the existing 10-turn roster cadence (`pageRefreshEveryTurns`) |
-
-The score floor is load-bearing: the benchmark showed that injecting weak matches is worse than injecting nothing (v2's regression). An empty injection is a correct outcome, not a failure mode.
-
-## 5. Write-back
-
-The Stop-hook session retain is **kept** — same trigger, same fail-open behavior. What changes is the transcript format handed to extraction:
-
-- **JSON turns**, not markdown: an array of `{ "role": "user" | "assistant", "text": ... }` entries for the conversational content.
-- Tool calls collapse to **compact one-line action entries**: `{ "role": "action", "text": "Edit boltons/strutils.py" }` — tool name + primary target only, **no arguments, no outputs**.
-
-Rationale: extraction keeps the concrete artifacts (which files were touched, what actions occurred) without the transcript noise of full tool payloads — the markdown tool-call dumps were volume without signal.
-
-## 6. Knowledge pages
-
-Simplified from the v2 spec:
-
-- **Dropped: tags and `entity_labels`** (v2 spec §4–5). The server already re-synthesizes pages after consolidation, so pages stay "living" with no client-side routing machinery. The extractor-never-knows-about-pages principle now holds trivially — there is nothing to route.
-- **Creation paths:**
-  1. **Seeded taxonomy** at bank creation (the fixed page set, as today, minus tag triggers).
-  2. **Agent-driven `capture_initiative`** at plan approval — the one active capture verb survives from v2.
-  3. **Organic splitting** of pages that outgrow their scope is a **server/curator concern**, not a client feature.
-
-## 7. Ingestion — progressive background deepening
-
-*Status: design accepted, implementation phased separately.*
-
-Replaces the manual backfill CLI as the user-facing path (the CLI was out-of-band burden; nobody runs it). The principle: converge to full-depth history through normal usage, with zero user action.
-
-1. **Instant shallow seed** — the gitlog seed from §3a; the session is useful immediately.
-2. **Background deepening** — a background worker deep-ingests **per-commit-with-diffs, incrementally**, never blocking a turn.
-3. **Working-set prioritization** — commits are ingested in order of relevance to what the agent is actually doing: files the agent reads/edits get their commit histories ingested **first**. Depth arrives where it pays off.
-4. **Checkpointing** — progress persists across sessions; each session resumes deepening where the last left off, converging to full depth over normal usage.
-
-The **backfill CLI survives as an internal tool** (benchmark setup, CI bank preparation) — it is no longer a documented user path.
-
-## 8. Gap analysis — v2 principles under this design
-
-| v2 principle | How this design satisfies it |
-| --- | --- |
-| See-it-working (automatic, visible value) | Reflect answer visible from turn 1; page-section snippets appear with explicit `From <page> › <section>` provenance, so the user sees memory working — and the score floor keeps it from visibly misfiring. |
-| No out-of-band CLI | Cold-repo auto-seed kept (§3a); backfill CLI demoted to internal-only, replaced by background deepening (§7). Nothing requires a terminal command. |
-| Reuse-over-reinvent | Reflect, `listPages`, Stop-hook retain, `capture_initiative`, and the 10-turn refresh cadence are all existing machinery recombined; the only new code is the local section index and matcher — deliberately dumb (lexical, no LLM). |
-| Preserve-intent | Reflect is prompted for root-cause decisions with exact values; JSON transcripts keep concrete action artifacts; per-commit-with-diffs deepening captures *why* the code changed, not just that it did. |
-| Near-zero-burden | No config flags to choose, no CLI to run, no tags to maintain; one LLM call per session start, everything else local. |
-
-## 9. Verification gates
-
-Ship gates, in order:
-
-1. **Reflect-restored benchmark:** the restored runtime must recover **~25 mean corrections at n=2 on identical banks** to the original reflect run. This proves the restoration is faithful before anything is layered on.
-2. **Reflect+pages benchmark:** with per-turn section injection enabled, the score **must not regress** vs reflect-alone. Section injection earns its place by not hurting; any regression points at the floor/budget tuning.
-3. **Live system suite:** existing hook/integration suite updated for the new path — reflect caching + per-turn re-injection, section index build/refresh, score-floor silence, JSON transcript shape, action-entry compaction. Deterministic pieces (matcher scoring, budget trim, provenance formatting, transcript serialization) as fast unit tests.
-
-## 10. Non-goals / deferred
-
-- Any per-turn LLM or server call for injection (explicitly excluded — the local matcher is the whole point).
-- Semantic/embedding-based section matching (revisit only if lexical matching demonstrably misses; start dumb).
-- Client-side page splitting or curation (server/curator concern, §6).
-- Progressive-deepening implementation details (worker scheduling, checkpoint format) — phased separately per §7.
-
-## 11. File map (anticipated)
-
-- `src/core/reflect.ts` (restored) — session reflect call + per-session cache.
-- `src/core/section-index.ts` (new) — page → sections split, lexical scorer, budget trim, provenance formatting; pure/unit-testable.
-- `src/core/hook.ts` — drop recall; inject cached reflect + matched sections; index refresh on roster cadence.
-- `src/core/session-start.ts` — cold-repo seed (unchanged) + reflect trigger wiring + initial index build.
-- `src/core/transcript.ts` (new or reworked) — JSON turn serialization + action-entry compaction for the Stop hook.
-- `src/core/missions.ts` / `src/core/hindsight.ts` — remove `entity_labels` and tag-scoped page triggers; keep seeded taxonomy + `capture_initiative`.
-- `src/core/config.ts` — remove any behavior flags; keep env/naming/harness + `pageRefreshEveryTurns`.
-- Tests alongside each.

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.9.0
-appVersion: "0.9.0"
+version: 0.7.1
+appVersion: "0.7.1"
 keywords:
   - ai
   - memory

helm/hindsight/README.md

@@ -66,13 +66,13 @@ helm install hindsight ./helm/hindsight -n hindsight --create-namespace -f value
 
 | Parameter | Description | Default |
 |-----------|-------------|---------|
-| `version` | Default image tag for all components | Chart `appVersion` |
+| `version` | Default image tag for all components | `0.1.0` |
 | `api.enabled` | Enable the API component | `true` |
-| `api.image.repository` | API image repository | `ghcr.io/vectorize-io/hindsight-api` |
+| `api.image.repository` | API image repository | `hindsight/api` |
 | `api.image.tag` | API image tag (defaults to `version`) | - |
 | `api.service.port` | API service port | `8888` |
 | `controlPlane.enabled` | Enable the control plane | `true` |
-| `controlPlane.image.repository` | Control plane image repository | `ghcr.io/vectorize-io/hindsight-control-plane` |
+| `controlPlane.image.repository` | Control plane image repository | `hindsight/control-plane` |
 | `controlPlane.image.tag` | Control plane image tag (defaults to `version`) | - |
 | `controlPlane.service.port` | Control plane service port | `3000` |
 | `postgresql.enabled` | Deploy PostgreSQL as subchart | `true` |

helm/hindsight/templates/worker-statefulset.yaml

@@ -60,13 +60,13 @@ spec:
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
-        {{- /* Explicitly set port to override K8s service discovery env var (HINDSIGHT_API_PORT) */}}
-        - name: HINDSIGHT_API_PORT
-          value: {{ .Values.worker.service.targetPort | quote }}
-        {{- /* Inherit LLM config from api.env, then apply worker-specific env.
-               Merge (worker.env wins) so a key set in both does not emit a
-               duplicate env entry, which server-side apply rejects. */}}
-        {{- range $key, $value := merge (deepCopy (.Values.worker.env | default dict)) (.Values.api.env | default dict) }}
+        {{- /* Inherit LLM config from api.env */}}
+        {{- range $key, $value := .Values.api.env }}
+        - name: {{ $key }}
+          value: {{ $value | quote }}
+        {{- end }}
+        {{- /* Worker-specific env vars */}}
+        {{- range $key, $value := .Values.worker.env }}
         - name: {{ $key }}
           value: {{ $value | quote }}
         {{- end }}

helm/hindsight/values.yaml

@@ -13,6 +13,9 @@
 #   - Any other env vars you want to inject
 # existingSecret: "my-hindsight-secret"
 
+# Global settings
+replicaCount: 1
+
 # Image settings for api
 api:
   enabled: true
@@ -36,22 +39,16 @@ api:
       cpu: 500m
       memory: 1Gi
 
-  # Liveness and readiness probes.
-  # Liveness uses /health/live, which performs no database access: a slow or
-  # unreachable database must gate traffic (readiness), never restart pods.
-  # Needs an image from this chart's appVersion or newer — older ones serve
-  # /health only, and would fail this probe with a 404.
+  # Liveness and readiness probes
   livenessProbe:
     httpGet:
-      path: /health/live
+      path: /health
       port: 8888
     initialDelaySeconds: 30
     periodSeconds: 10
     timeoutSeconds: 5
     failureThreshold: 3
 
-  # Readiness checks the database, so a pod that cannot reach it is pulled out
-  # of the Service and put back once the database recovers.
   readinessProbe:
     httpGet:
       path: /health
@@ -137,15 +134,10 @@ worker:
       cpu: 500m
       memory: 1Gi
 
-  # Liveness and readiness probes.
-  # Liveness uses /health/live, which performs no database access. Restarting a
-  # worker whose database is merely slow requeues its claimed operations with
-  # retry_count incremented, so DB checks must stay out of liveness.
-  # Needs an image from this chart's appVersion or newer — older ones serve
-  # /health only, and would fail this probe with a 404.
+  # Liveness and readiness probes
   livenessProbe:
     httpGet:
-      path: /health/live
+      path: /health
       port: 8889
     initialDelaySeconds: 30
     periodSeconds: 10

hindsight-all-npm/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@vectorize-io/hindsight-all",
-  "version": "0.9.0",
+  "version": "0.7.1",
   "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",

hindsight-all-slim/pyproject.toml

@@ -1,18 +1,17 @@
 [build-system]
-requires = ["setuptools>=77"]
+requires = ["setuptools>=61"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "hindsight-all-slim"
-version = "0.9.0"
+version = "0.7.1"
 description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
-license = "MIT"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim==0.9.0",
+    "hindsight-api-slim==0.7.1",
     "hindsight-client>=0.0.7",
-    "hindsight-embed==0.9.0",
+    "hindsight-embed>=0.1.0",
 ]
 
 [tool.uv.sources]

hindsight-all/pyproject.toml

@@ -1,18 +1,17 @@
 [build-system]
-requires = ["hatchling>=1.27"]
+requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-all"
-version = "0.9.0"
+version = "0.7.1"
 description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
-license = "MIT"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api-slim[all]==0.9.0",
+    "hindsight-api-slim[all]==0.7.1",
     "hindsight-client>=0.0.7",
-    "hindsight-embed==0.9.0",
+    "hindsight-embed>=0.1.0",
 ]
 
 [tool.uv.sources]
@@ -22,7 +21,7 @@ hindsight-embed = { workspace = true }
 
 [project.optional-dependencies]
 local-llm = [
-    "hindsight-api-slim[local-llm]==0.9.0",
+    "hindsight-api-slim[local-llm]==0.7.1",
 ]
 test = [
     "pytest>=7.0.0",

hindsight-api-slim/README.md

@@ -99,7 +99,7 @@ hindsight-api
 ## Docker
 
 ```bash
-docker run -it --name hindsight --restart unless-stopped -p 8888:8888 \
+docker run --rm -it -p 8888:8888 \
   -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
   -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
   ghcr.io/vectorize-io/hindsight:latest
@@ -121,7 +121,7 @@ This runs a stdio-based MCP server that can be used directly with MCP-compatible
 - **Entity Graph** — Automatic entity extraction and relationship tracking
 - **Temporal Reasoning** — Native support for time-based queries
 - **Disposition Traits** — Configurable skepticism, literalism, and empathy influence opinion formation
-- **Three Memory Types** — World facts, experience facts (the bank's own actions), and observations
+- **Three Memory Types** — World facts, bank actions, and formed opinions with confidence scores
 
 ## Documentation
 

hindsight-api-slim/hindsight_api/__init__.py

@@ -53,4 +53,4 @@ __all__ = [
     "RemoteTEICrossEncoder",
     "LLMConfig",
 ]
-__version__ = "0.9.0"
+__version__ = "0.7.1"

hindsight-api-slim/hindsight_api/_text_search.py

@@ -1,23 +0,0 @@
-"""Text-search SQL shapes shared by index DDL and the queries that must hit it.
-
-A PostgreSQL expression index is only selectable when the query repeats the
-indexed expression verbatim, so the DDL (``migrations.py`` and the Alembic
-versions) and the read arms (``engine/sql/postgresql.py``) cannot be allowed to
-drift. Both sides call the helpers here — same idea as
-``_pg_search.pg_search_bm25_columns``.
-"""
-
-
-def mental_models_text_document(alias: str | None = None) -> str:
-    """The ``mental_models`` full-text document: model/page name + content.
-
-    Mirrors the generating expression of the native tsvector column created by
-    the ``n9i0j1k2l3m4`` (learnings / pinned_reflections) migration, so every
-    backend indexes and queries the exact same document. ``content`` is NOT NULL,
-    hence the deliberate lack of a ``COALESCE`` around it.
-
-    ``alias`` qualifies the columns for queries that join the table (``mm``);
-    leave it unset for DDL, where the expression is already table-scoped.
-    """
-    prefix = f"{alias}." if alias else ""
-    return f"(COALESCE({prefix}name, '') || ' ' || {prefix}content)"

hindsight-api-slim/hindsight_api/_vector_index.py

@@ -54,20 +54,23 @@ _INDEX_TYPE_KEYWORDS = {
 #   pre-dispatcher code (internal benchmarks tuned around our embedding count
 #   and recall floor; see the link_utils / pool init call sites for the
 #   latency-vs-recall framing).
-# - vchord exposes vchordrq.probes, but its shape must match the index's
-#   build.internal.lists hierarchy. VectorChord 1.1 added per-index fallback
-#   parameters for this reason: a session GUC overrides every vchordrq index,
-#   and a single value can be invalid for listless or mixed-layout indexes.
-#   Hindsight's built-in vchord clause does not set lists, so the safe default
-#   is no session-level probe override; deployments that partition vchordrq
-#   indexes should attach probes to the index storage parameters instead.
+# - vchord exposes vchordrq.probes (no default; see VectorChord issue #392)
+#   and vchordrq.epsilon (default 1.9). probes = 10 / 30 are starting
+#   defaults pending a workload-specific sweep — vchordrq's recall curve
+#   shape differs from HNSW's, so the pgvector numbers don't translate
+#   directly. Revisit with a per-cluster benchmark once we have production
+#   recall data; until then these are deliberately conservative on the
+#   high-recall path. We leave epsilon at its default; tightening it is a
+#   separate trade-off.
 # - pgvectorscale / pg_diskann / scann do not expose an equivalent per-statement
 #   knob in the engine today, so the dispatcher returns no statements for them.
 _ANN_TUNING_LOW_LATENCY: dict[str, tuple[tuple[str, str], ...]] = {
     "pgvector": (("hnsw.ef_search", "60"),),
+    "vchord": (("vchordrq.probes", "10"),),
 }
 _ANN_TUNING_HIGH_RECALL: dict[str, tuple[tuple[str, str], ...]] = {
     "pgvector": (("hnsw.ef_search", "200"),),
+    "vchord": (("vchordrq.probes", "30"),),
 }
 
 _EXTENSION_INSTALL_SQL = {

hindsight-api-slim/hindsight_api/admin/cli.py

@@ -8,9 +8,7 @@ import asyncio
 import io
 import json
 import logging
-import struct
 import zipfile
-from dataclasses import dataclass
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any
@@ -18,12 +16,8 @@ from typing import Any
 import asyncpg
 import typer
 
-from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig, load_dotenv_for_entrypoint
-from ..engine.memory_engine import _current_schema
-from ..engine.retain.bank_utils import _vector_index_clause
+from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
 from ..engine.schema import fq_table_explicit as _fq_table
-from ..engine.transfer import export_bank
-from ..engine.vector_index_health import SchemaVectorIndexResult, repair_vector_indexes
 from ..extensions import TenantExtension, load_extension
 from ..pg0 import parse_pg0_url, resolve_database_url
 
@@ -53,222 +47,23 @@ BACKUP_TABLES = [
     "entities",
     "chunks",
     "memory_units",
-    "invalidated_memory_units",
     "unit_entities",
     "entity_cooccurrences",
     "memory_links",
-    "observation_history",
     "mental_models",
-    "mental_model_history",
-    "knowledge_pages",
     "directives",
     "async_operations",
     "webhooks",
     "file_storage",
     "audit_log",
-    "llm_requests",
     "graph_maintenance_queue",
 ]
 
-MANIFEST_VERSION = "2"
+MANIFEST_VERSION = "1"
 
 
-@dataclass(frozen=True)
-class BackupColumn:
-    """A PostgreSQL column shape required to decode a binary COPY stream."""
-
-    name: str
-    type_name: str
-
-
-@dataclass(frozen=True)
-class TableRestorePlan:
-    """How one table's backed-up binary COPY stream is replayed onto the target.
-
-    ``columns`` is the target column list handed to ``copy_to_table``, in stream
-    order. When the target no longer has a backed-up column, its field is stripped
-    from every tuple (``dropped_field_indices``) before the stream is replayed —
-    binary COPY is positional, so the column list and the tuple fields must agree.
-    """
-
-    columns: list[str]
-    dropped_field_indices: tuple[int, ...]
-    source_field_count: int
-
-
-# Header of a PostgreSQL binary COPY stream: an 11-byte signature, an int32 flags
-# field, and an int32 header-extension length followed by that many bytes.
-_COPY_BINARY_SIGNATURE = b"PGCOPY\n\xff\r\n\x00"
-_COPY_BINARY_HEADER_LEN = len(_COPY_BINARY_SIGNATURE) + 8
-
-
-def _strip_binary_copy_fields(data: bytes, plan: TableRestorePlan) -> bytes:
-    """Drop `plan.dropped_field_indices` from every tuple of a binary COPY stream.
-
-    Restore used to reject a backup whose columns the target no longer had — the
-    preflight raised "target is missing backup columns …", which made any backup
-    taken before a column-dropping migration unrestorable afterwards. Those columns
-    are now ignored instead, but they cannot simply be left out of the
-    ``copy_to_table`` column list: binary COPY carries no column identities, so each
-    tuple's fields are matched to the column list purely by position and an unedited
-    stream would desynchronise (or, worse, land values in the wrong columns). So the
-    stream itself is rewritten here.
-
-    Tuple format: int16 field count, then per field an int32 length (-1 for NULL)
-    followed by that many bytes. An int16 of -1 is the end-of-data trailer.
-    """
-    if not plan.dropped_field_indices:
-        return data
-    if not data.startswith(_COPY_BINARY_SIGNATURE):
-        raise ValueError("Backup stream is not in PostgreSQL binary COPY format")
-
-    (extension_len,) = struct.unpack_from("!i", data, len(_COPY_BINARY_SIGNATURE) + 4)
-    pos = _COPY_BINARY_HEADER_LEN + extension_len
-    out = bytearray(data[:pos])
-
-    dropped = set(plan.dropped_field_indices)
-    kept_count = plan.source_field_count - len(dropped)
-    while True:
-        (field_count,) = struct.unpack_from("!h", data, pos)
-        pos += 2
-        if field_count == -1:  # end-of-data trailer
-            out += struct.pack("!h", -1)
-            break
-        if field_count != plan.source_field_count:
-            raise ValueError(
-                f"Backup stream tuple has {field_count} fields, manifest declares {plan.source_field_count}"
-            )
-        out += struct.pack("!h", kept_count)
-        for index in range(field_count):
-            (length,) = struct.unpack_from("!i", data, pos)
-            pos += 4
-            payload = b"" if length == -1 else data[pos : pos + length]
-            pos += max(length, 0)
-            if index in dropped:
-                continue
-            out += struct.pack("!i", length)
-            out += payload
-    return bytes(out)
-
-
-async def _table_columns(conn: asyncpg.Connection, schema: str, table: str) -> list[BackupColumn]:
-    rows = await conn.fetch(
-        """
-        SELECT a.attname AS name, pg_catalog.format_type(a.atttypid, a.atttypmod) AS type_name
-        FROM pg_catalog.pg_attribute AS a
-        JOIN pg_catalog.pg_class AS c ON c.oid = a.attrelid
-        JOIN pg_catalog.pg_namespace AS n ON n.oid = c.relnamespace
-        WHERE n.nspname = $1 AND c.relname = $2 AND a.attnum > 0 AND NOT a.attisdropped
-          AND a.attgenerated = ''
-        ORDER BY a.attnum
-        """,
-        schema,
-        table,
-    )
-    return [BackupColumn(name=row["name"], type_name=row["type_name"]) for row in rows]
-
-
-async def _validate_restore_schema(
-    conn: asyncpg.Connection, manifest: dict[str, Any], schema: str
-) -> dict[str, TableRestorePlan]:
-    """Validate every COPY stream against the target before destructive work starts.
-
-    A column the target no longer has is **not** an error: a migration that drops a
-    column would otherwise make every backup taken before it permanently
-    unrestorable. Such columns are skipped (their fields are stripped from the
-    stream by ``_strip_binary_copy_fields``) and reported, so the operator sees what
-    was discarded instead of the restore failing outright.
-
-    Type mismatches remain fatal. Type equality is an exact ``format_type`` string
-    match. This is deliberately stricter than binary-COPY wire compatibility (e.g.
-    ``varchar`` and ``text`` share a binary format yet compare unequal here): we
-    would rather fail a genuinely-restorable backup with a clear, actionable error
-    than silently risk a subtle binary mismatch. Restores blocked this way can be
-    recovered by aligning the target schema.
-    """
-    plans: dict[str, TableRestorePlan] = {}
-    errors: list[str] = []
-    for table, table_manifest in manifest["tables"].items():
-        source_columns = [BackupColumn(**column) for column in table_manifest["columns"]]
-        target_by_name = {column.name: column for column in await _table_columns(conn, schema, table)}
-        unknown = [
-            (index, column.name) for index, column in enumerate(source_columns) if column.name not in target_by_name
-        ]
-        mismatched = [
-            f"{column.name} ({column.type_name} in backup, {target_by_name[column.name].type_name} in target)"
-            for column in source_columns
-            if column.name in target_by_name and target_by_name[column.name].type_name != column.type_name
-        ]
-        if mismatched:
-            errors.append(f"{table}: incompatible column types: {', '.join(mismatched)}")
-        if unknown:
-            typer.echo(
-                f"  {table}: ignoring {len(unknown)} backup column(s) absent from the target schema: "
-                f"{', '.join(name for _, name in unknown)}"
-            )
-        plans[table] = TableRestorePlan(
-            columns=[column.name for column in source_columns if column.name in target_by_name],
-            dropped_field_indices=tuple(index for index, _ in unknown),
-            source_field_count=len(source_columns),
-        )
-
-    if errors:
-        details = "; ".join(errors)
-        raise ValueError(f"Backup schema is incompatible with target schema '{schema}': {details}")
-    return plans
-
-
-def _effective_backup_tables() -> list[str]:
-    """Core backup tables plus any bank-scoped tables a loaded extension declares.
-
-    ``BACKUP_TABLES`` covers only the tables core owns. An extension that
-    provisions its own bank-scoped tables (via ``TenantExtension``) declares
-    them through ``extra_bank_tables()`` so they aren't dropped on restore.
-    Extension tables are appended *after* the core set so restore's forward
-    COPY inserts them after their FK parents (e.g. ``banks``) and the reversed
-    TRUNCATE clears them before those parents.
-    """
-    tables = list(BACKUP_TABLES)
-    tenant_extension = load_extension("TENANT", TenantExtension)
-    if tenant_extension is not None:
-        seen = set(tables)
-        for spec in tenant_extension.extra_bank_tables():
-            if spec.include_in_backup and spec.name not in seen:
-                tables.append(spec.name)
-                seen.add(spec.name)
-    return tables
-
-
-async def _admin_connect(db_url: str) -> asyncpg.Connection:
-    """Open a raw asyncpg connection to an admin DB URL.
-
-    ``resolve_database_url`` handles both plain ``postgres://`` (passthrough) and
-    ``pg0://`` (boots the embedded server and returns its real libpq URL), so this
-    is the only step needed to connect. JSON codecs are registered so ``jsonb``
-    columns decode to Python objects (used by the export row dumps).
-    """
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
-    if is_pg0:
-        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
-    conn = await asyncpg.connect(await resolve_database_url(db_url))
-    for type_name in ("json", "jsonb"):
-        await conn.set_type_codec(type_name, encoder=json.dumps, decoder=json.loads, schema="pg_catalog")
-    return conn
-
-
-async def _backup(
-    database_url: str,
-    output_path: Path,
-    schema: str = "public",
-    backup_tables: list[str] | None = None,
-) -> dict[str, Any]:
-    """Backup all tables to a zip file using binary COPY protocol.
-
-    ``backup_tables`` defaults to the core ``BACKUP_TABLES``; callers pass the
-    extension-augmented list from ``_effective_backup_tables()``.
-    """
-    backup_tables = backup_tables if backup_tables is not None else BACKUP_TABLES
+async def _backup(database_url: str, output_path: Path, schema: str = "public") -> dict[str, Any]:
+    """Backup all tables to a zip file using binary COPY protocol."""
     conn = await asyncpg.connect(database_url)
     try:
         tables: dict[str, Any] = {}
@@ -285,24 +80,14 @@ async def _backup(
         # entities table was backed up.
         async with conn.transaction(isolation="repeatable_read"):
             with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf:
-                for i, table in enumerate(backup_tables, 1):
-                    typer.echo(f"  [{i}/{len(backup_tables)}] Backing up {table}...", nl=False)
+                for i, table in enumerate(BACKUP_TABLES, 1):
+                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Backing up {table}...", nl=False)
 
                     buffer = io.BytesIO()
 
-                    columns = await _table_columns(conn, schema, table)
-
-                    # Pin the ordered columns into both the stream and manifest.
-                    # PostgreSQL binary COPY does not encode column identities, so
-                    # restore must validate this shape before truncating any data.
+                    # Use binary COPY for exact type preservation
                     # asyncpg requires schema_name as separate parameter
-                    await conn.copy_from_table(
-                        table,
-                        schema_name=schema,
-                        columns=[column.name for column in columns],
-                        output=buffer,
-                        format="binary",
-                    )
+                    await conn.copy_from_table(table, schema_name=schema, output=buffer, format="binary")
 
                     data = buffer.getvalue()
                     zf.writestr(f"{table}.bin", data)
@@ -313,7 +98,6 @@ async def _backup(
                     tables[table] = {
                         "rows": row_count,
                         "size_bytes": len(data),
-                        "columns": [{"name": column.name, "type_name": column.type_name} for column in columns],
                     }
 
                     typer.echo(f" {row_count} rows")
@@ -325,20 +109,8 @@ async def _backup(
         await conn.close()
 
 
-async def _restore(
-    database_url: str,
-    input_path: Path,
-    schema: str = "public",
-    backup_tables: list[str] | None = None,
-) -> dict[str, Any]:
-    """Restore all tables from a zip file using binary COPY protocol.
-
-    ``backup_tables`` defaults to the core ``BACKUP_TABLES``; callers pass the
-    extension-augmented list from ``_effective_backup_tables()``. Tables named
-    here but absent from the archive are truncated then skipped for restore, so
-    a stale extension registration never leaves pre-restore rows behind.
-    """
-    backup_tables = backup_tables if backup_tables is not None else BACKUP_TABLES
+async def _restore(database_url: str, input_path: Path, schema: str = "public") -> dict[str, Any]:
+    """Restore all tables from a zip file using binary COPY protocol."""
     conn = await asyncpg.connect(database_url)
     try:
         with zipfile.ZipFile(input_path, "r") as zf:
@@ -347,42 +119,29 @@ async def _restore(
             if manifest.get("version") != MANIFEST_VERSION:
                 raise ValueError(f"Unsupported backup version: {manifest.get('version')}")
 
-            # Complete the compatibility check before entering the transaction
-            # that truncates tables. This turns historical schema drift into an
-            # actionable error without risking the target's existing data.
-            restore_plans = await _validate_restore_schema(conn, manifest, schema)
-
             # Use a transaction for atomic restore - either all tables are
             # restored or none are, preventing partial/inconsistent state.
             async with conn.transaction():
                 typer.echo("  Clearing existing data...")
                 # Truncate tables in reverse order (respects FK constraints)
-                for table in reversed(backup_tables):
+                for table in reversed(BACKUP_TABLES):
                     qualified_table = _fq_table(table, schema)
                     await conn.execute(f"TRUNCATE TABLE {qualified_table} CASCADE")
 
                 # Restore tables in forward order
-                for i, table in enumerate(backup_tables, 1):
+                for i, table in enumerate(BACKUP_TABLES, 1):
                     filename = f"{table}.bin"
                     if filename not in zf.namelist():
-                        typer.echo(f"  [{i}/{len(backup_tables)}] {table}: skipped (not in backup)")
+                        typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] {table}: skipped (not in backup)")
                         continue
 
                     expected_rows = manifest["tables"].get(table, {}).get("rows", "?")
-                    typer.echo(f"  [{i}/{len(backup_tables)}] Restoring {table}... {expected_rows} rows")
+                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Restoring {table}... {expected_rows} rows")
 
-                    plan = restore_plans[table]
-                    # Strips the fields of any column the target no longer has;
-                    # a no-op when the schemas still line up.
-                    buffer = io.BytesIO(_strip_binary_copy_fields(zf.read(filename), plan))
+                    data = zf.read(filename)
+                    buffer = io.BytesIO(data)
                     # asyncpg requires schema_name as separate parameter
-                    await conn.copy_to_table(
-                        table,
-                        schema_name=schema,
-                        columns=plan.columns,
-                        source=buffer,
-                        format="binary",
-                    )
+                    await conn.copy_to_table(table, schema_name=schema, source=buffer, format="binary")
 
                 # Refresh materialized view
                 typer.echo("  Refreshing materialized views...")
@@ -395,22 +154,20 @@ async def _restore(
 
 async def _run_backup(db_url: str, output: Path, schema: str = "public") -> dict[str, Any]:
     """Resolve database URL and run backup."""
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
-    return await _backup(resolved_url, output, schema, backup_tables=_effective_backup_tables())
+    return await _backup(resolved_url, output, schema)
 
 
 async def _run_restore(db_url: str, input_file: Path, schema: str = "public") -> dict[str, Any]:
     """Resolve database URL and run restore."""
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
-    return await _restore(resolved_url, input_file, schema, backup_tables=_effective_backup_tables())
+    return await _restore(resolved_url, input_file, schema)
 
 
 @app.command()
@@ -434,7 +191,7 @@ def backup(
     manifest = asyncio.run(_run_backup(config.database_url, output, schema))
 
     total_rows = sum(t["rows"] for t in manifest["tables"].values())
-    typer.echo(f"Backed up {total_rows} rows across {len(manifest['tables'])} tables")
+    typer.echo(f"Backed up {total_rows} rows across {len(BACKUP_TABLES)} tables")
     typer.echo(f"Backup saved to {output}")
 
 
@@ -467,7 +224,7 @@ def restore(
     manifest = asyncio.run(_run_restore(config.database_url, input_file, schema))
 
     total_rows = sum(t["rows"] for t in manifest["tables"].values())
-    typer.echo(f"Restored {total_rows} rows across {len(manifest['tables'])} tables")
+    typer.echo(f"Restored {total_rows} rows across {len(BACKUP_TABLES)} tables")
     typer.echo("Restore complete")
 
 
@@ -476,22 +233,26 @@ async def _run_migration(
     schema: str | None = None,
     base_schema: str = DEFAULT_DATABASE_SCHEMA,
     embedding_dimension: int | None = None,
-    ensure_extensions: bool = True,
 ) -> list[str]:
     """Resolve database URL and run migrations for one schema or all discovered schemas."""
-    from ..migrations import run_migrations_for_schemas
+    from ..migrations import (
+        ensure_embedding_dimension,
+        ensure_text_search_extension,
+        ensure_vector_extension,
+        run_migrations,
+    )
 
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
 
     config = HindsightConfig.from_env()
-    tenant_extension = load_extension("TENANT", TenantExtension)
     if schema:
         schemas = [schema]
     else:
+        tenant_extension = load_extension("TENANT", TenantExtension)
+
         schemas = [base_schema or DEFAULT_DATABASE_SCHEMA]
         if tenant_extension:
             tenants = await tenant_extension.list_tenants()
@@ -500,52 +261,36 @@ async def _run_migration(
         # Preserve order while removing duplicates.
         schemas = list(dict.fromkeys(schemas))
 
-    # Migrate up to `migration_concurrency` schemas at once (each in its own
-    # process); within a schema the work stays sequential. Run off the event
-    # loop so the process pool's blocking joins don't stall it.
-    await asyncio.to_thread(
-        run_migrations_for_schemas,
-        resolved_url,
-        schemas,
-        concurrency=config.migration_concurrency,
-        migration_database_url=config.migration_database_url,
-        embedding_dimension=embedding_dimension,
-        vector_extension=config.vector_extension,
-        text_search_extension=config.text_search_extension,
-        pg_search_tokenizer=config.text_search_extension_pg_search_tokenizer,
-        ensure_extensions=ensure_extensions,
-    )
+    for schema in schemas:
+        run_migrations(resolved_url, schema=schema, migration_database_url=config.migration_database_url)
 
-    # After core migrations, provision any extension-owned bank-scoped tables
-    # per schema so extension schema evolves on the same lifecycle as core
-    # schema (rather than via a lazy first-request path).
-    if tenant_extension is not None:
-        await _provision_extra_bank_tables(resolved_url, schemas, tenant_extension)
+    if embedding_dimension is not None:
+        for schema in schemas:
+            ensure_embedding_dimension(
+                resolved_url,
+                embedding_dimension,
+                schema=schema,
+                vector_extension=config.vector_extension,
+            )
+
+    for schema in schemas:
+        ensure_vector_extension(
+            resolved_url,
+            vector_extension=config.vector_extension,
+            schema=schema,
+        )
+
+    for schema in schemas:
+        ensure_text_search_extension(
+            resolved_url,
+            text_search_extension=config.text_search_extension,
+            pg_search_tokenizer=config.text_search_extension_pg_search_tokenizer,
+            schema=schema,
+        )
 
     return schemas
 
 
-async def _provision_extra_bank_tables(
-    resolved_url: str, schemas: list[str], tenant_extension: TenantExtension
-) -> None:
-    """Run the tenant extension's table provisioner for each migrated schema.
-
-    Fires after core migrations complete so extension-owned bank tables are
-    created/evolved on the same lifecycle as core schema. A failure aborts the
-    migration command (and names the offending schema) rather than being
-    swallowed — provisioning is idempotent, so the operator can fix and re-run.
-    """
-    for schema in schemas:
-        conn = await asyncpg.connect(resolved_url)
-        try:
-            await tenant_extension.provision_bank_tables(conn, schema)
-        except Exception as e:
-            typer.echo(f"  Failed to provision extension tables for schema '{schema}': {e}", err=True)
-            raise
-        finally:
-            await conn.close()
-
-
 @app.command(name="run-db-migration")
 def run_db_migration(
     schema: str | None = typer.Option(
@@ -559,18 +304,6 @@ def run_db_migration(
         "--embedding-dimension",
         help="Expected embedding dimension to enforce after migrations. Omit to skip dimension sync.",
     ),
-    skip_extension_reconcile: bool = typer.Option(
-        False,
-        "--skip-extension-reconcile",
-        help=(
-            "Skip the post-migration vector / text-search index reconcile. This step only does "
-            "work when the configured backend (HINDSIGHT_API_VECTOR_EXTENSION / "
-            "HINDSIGHT_API_TEXT_SEARCH_EXTENSION) differs from a schema's existing indexes — a "
-            "rare, operator-driven change. Skipping it makes a no-change re-migration over many "
-            "tenant schemas much faster. Only use when you have NOT changed the backend; a "
-            "backend change still needs a normal run to reshape the indexes."
-        ),
-    ),
 ):
     """Run database migrations to the latest version."""
     config = HindsightConfig.from_env()
@@ -584,8 +317,6 @@ def run_db_migration(
         typer.echo(f"Running database migrations for schema: {schema}...")
     else:
         typer.echo("Running database migrations for base schema and all discovered tenant schemas...")
-    if skip_extension_reconcile:
-        typer.echo("Skipping post-migration extension reconcile (--skip-extension-reconcile).")
 
     schemas = asyncio.run(
         _run_migration(
@@ -593,270 +324,15 @@ def run_db_migration(
             schema=schema,
             base_schema=config.database_schema,
             embedding_dimension=embedding_dimension,
-            ensure_extensions=not skip_extension_reconcile,
         )
     )
 
     typer.echo(f"Database migrations completed successfully for {len(schemas)} schema(s)")
 
 
-async def _resolve_schemas(base_schema: str | None) -> list[str]:
-    """Base schema plus every discovered tenant schema, de-duplicated in order."""
-    schemas = [base_schema or DEFAULT_DATABASE_SCHEMA]
-    tenant_extension = load_extension("TENANT", TenantExtension)
-    if tenant_extension:
-        tenants = await tenant_extension.list_tenants()
-        schemas.extend(tenant.schema for tenant in tenants if tenant.schema)
-    return list(dict.fromkeys(schemas))
-
-
-async def _run_repair_bank(
-    db_url: str,
-    *,
-    base_schema: str,
-    schema: str | None,
-    bank_id: str | None,
-    dry_run: bool,
-) -> list[SchemaVectorIndexResult]:
-    """Reconcile per-(bank, fact_type) vector index coverage over a raw connection.
-
-    A single autocommit connection is used because ``CREATE INDEX CONCURRENTLY``
-    (used by ``repair_vector_indexes``) cannot run inside a transaction block.
-    """
-    schemas = [schema] if schema else await _resolve_schemas(base_schema)
-    index_clause = _vector_index_clause()
-    # Guarded by the command, but assert so this helper is never called for a
-    # backend without per-bank indexes.
-    assert index_clause is not None
-
-    conn = await _admin_connect(db_url)
-    try:
-        results = await repair_vector_indexes(conn, schemas, index_clause, dry_run=dry_run, bank_id=bank_id)
-        for result in results:
-            typer.echo(
-                f"  schema '{result.schema}': {result.banks_scanned} bank(s) scanned, "
-                f"{result.already_present} present, {result.created} created, "
-                f"{result.skipped} to-create (dry-run), {result.failed} failed"
-            )
-        return results
-    finally:
-        await conn.close()
-
-
-@app.command(name="repair-bank")
-def repair_bank(
-    bank_id: str | None = typer.Option(
-        None,
-        "--bank",
-        "-b",
-        help="Bank id to repair. Mutually exclusive with --all.",
-    ),
-    all_banks: bool = typer.Option(
-        False,
-        "--all",
-        help="Repair every bank in the base schema and all discovered tenant schemas.",
-    ),
-    schema: str | None = typer.Option(
-        None,
-        "--schema",
-        "-s",
-        help="Limit to a single schema. Defaults to the base schema plus discovered tenant schemas.",
-    ),
-    dry_run: bool = typer.Option(
-        False,
-        "--dry-run",
-        help="Report what would be repaired without creating or dropping any index.",
-    ),
-):
-    """Verify and repair a bank's per-(bank, fact_type) vector index coverage.
-
-    Per-bank partial vector indexes are created when a bank is first created
-    (instant on an empty bank). Banks that arrive populated — via logical
-    restore, a cross-version upgrade, or a vector-extension switch — never hit
-    that path, so their recall silently falls back to a global index +
-    post-filter (slower, under-returning). This command detects missing OR
-    invalid coverage (an INVALID leftover or an index whose access method
-    drifted counts as missing) and rebuilds it with CREATE INDEX CONCURRENTLY,
-    so it never blocks the live fleet. Idempotent and safe to re-run — the
-    escape hatch after a restore, upgrade, or backend switch.
-    """
-    if bool(bank_id) == all_banks:
-        typer.echo("Error: pass exactly one of --bank <id> or --all.", err=True)
-        raise typer.Exit(2)
-
-    config = HindsightConfig.from_env()
-    if not config.database_url:
-        typer.echo("Error: Database URL not configured.", err=True)
-        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
-        raise typer.Exit(1)
-
-    # Backend guard: backends with a single global vector index (AlloyDB ScaNN,
-    # Oracle) have no per-bank indexes to repair.
-    if _vector_index_clause() is None:
-        typer.echo("Configured vector backend does not use per-bank vector indexes — nothing to repair.")
-        return
-
-    target = f"bank '{bank_id}'" if bank_id else "all banks"
-    scope = f"schema '{schema}'" if schema else "base schema and all discovered tenant schemas"
-    typer.echo(f"Repairing per-bank vector indexes for {target} across {scope}...")
-    if dry_run:
-        typer.echo("Dry run: no indexes will be created or dropped.")
-
-    results = asyncio.run(
-        _run_repair_bank(
-            config.database_url,
-            base_schema=config.database_schema,
-            schema=schema,
-            bank_id=bank_id,
-            dry_run=dry_run,
-        )
-    )
-
-    total_banks = sum(r.banks_scanned for r in results)
-    total_present = sum(r.already_present for r in results)
-    total_created = sum(r.created for r in results)
-    total_skipped = sum(r.skipped for r in results)
-    total_failed = sum(r.failed for r in results)
-    typer.echo(
-        f"Done: {len(results)} schema(s), {total_banks} bank(s) scanned, "
-        f"{total_present} already present, {total_created} created, "
-        f"{total_skipped} to-create (dry-run), {total_failed} failed"
-    )
-    if total_failed:
-        failed_names = [name for r in results for name in r.failed_indexes]
-        typer.echo(f"Failed indexes (dropped, retry with a re-run): {', '.join(failed_names)}", err=True)
-        raise typer.Exit(1)
-
-
-async def _run_export_bank(db_url: str, bank_id: str, output: Path, schema: str, include_history: bool) -> int:
-    """Export a whole bank to a ZIP archive."""
-    conn = await _admin_connect(db_url)
-    try:
-        # export_bank resolves table names via fq_table (the _current_schema
-        # contextvar); set it so the raw connection targets the right schema.
-        _current_schema.set(schema)
-        # _admin_connect registers JSON codecs, so row dumps already contain
-        # decoded Python values (including JSON scalar strings).
-        data = await export_bank(
-            conn,
-            bank_id,
-            include_history=include_history,
-            bank_rows_json_encoding="decoded",
-        )
-    finally:
-        await conn.close()
-
-    output.write_bytes(data)
-    return len(data)
-
-
-@app.command(name="export-bank")
-def export_bank_command(
-    bank_id: str = typer.Option(..., "--bank", "-b", help="Bank id to export."),
-    output: Path = typer.Option(..., "--output", "-o", help="Path to write the .zip archive."),
-    schema: str | None = typer.Option(
-        None,
-        "--schema",
-        "-s",
-        help="Database schema the bank lives in. Defaults to the configured base schema.",
-    ),
-    include_history: bool = typer.Option(
-        False,
-        "--include-history",
-        help="Also export operational history (audit_log, llm_requests). Off by default.",
-    ),
-):
-    """Export an entire bank to a portable ZIP (no embeddings — regenerated on import).
-
-    Carries documents, facts, observations, bank config, mental models, directives
-    and webhooks so the bank can be imported into a new instance configured with a
-    different embedding model / vector / text-search backend.
-    """
-    config = HindsightConfig.from_env()
-
-    if not config.database_url:
-        typer.echo("Error: Database URL not configured.", err=True)
-        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
-        raise typer.Exit(1)
-
-    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
-    typer.echo(f"Exporting bank '{bank_id}' from schema '{target_schema}'...")
-
-    size = asyncio.run(_run_export_bank(config.database_url, bank_id, output, target_schema, include_history))
-
-    typer.echo(f"Exported bank '{bank_id}' to {output} ({size} bytes)")
-
-
-async def _run_import_bank(archive_path: Path, schema: str, target_bank_id: str | None, include_history: bool):
-    """Boot a MemoryEngine (for the target's embedding model) and restore a bank archive."""
-    # MemoryEngine is heavy (loads embeddings); import it lazily so other admin
-    # commands don't pay for it. _current_schema is imported at module top.
-    from ..engine.memory_engine import MemoryEngine
-    from ..models import RequestContext
-
-    archive_bytes = archive_path.read_bytes()
-    # run_migrations=True so a fresh target instance is provisioned at this
-    # instance's embedding dimension / vector / text-search backend before restore.
-    engine = MemoryEngine(run_migrations=True)
-    await engine.initialize()
-    try:
-        _current_schema.set(schema)
-        context = RequestContext(internal=True, user_initiated=True)
-        return await engine.import_bank_async(
-            archive_bytes,
-            context,
-            target_bank_id=target_bank_id,
-            include_history=include_history,
-        )
-    finally:
-        await engine.close()
-
-
-@app.command(name="import-bank")
-def import_bank_command(
-    archive: Path = typer.Option(..., "--archive", "-a", help="Path to the .zip produced by export-bank."),
-    schema: str | None = typer.Option(
-        None, "--schema", "-s", help="Target schema. Defaults to the configured base schema."
-    ),
-    target_bank: str | None = typer.Option(
-        None, "--target-bank", help="Override the bank id (defaults to the archive's source bank)."
-    ),
-    include_history: bool = typer.Option(
-        False, "--include-history", help="Also restore operational history if present in the archive."
-    ),
-):
-    """Restore a whole bank from an export-bank archive into THIS instance.
-
-    Re-embeds facts with this instance's configured embedding model and rebuilds
-    links and indexes — the import half of a cross-instance migration. Run against
-    an instance configured with the desired embedding / vector / text-search backend.
-    The target bank must not already exist (import restores a whole bank, not a merge).
-    """
-    config = HindsightConfig.from_env()
-    if not config.database_url:
-        typer.echo("Error: Database URL not configured.", err=True)
-        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
-        raise typer.Exit(1)
-
-    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
-    typer.echo(f"Importing bank archive '{archive}' into schema '{target_schema}'...")
-
-    result = asyncio.run(_run_import_bank(archive, target_schema, target_bank, include_history))
-
-    typer.echo(
-        f"Imported bank '{result.bank_id}': {result.documents_imported} doc(s), "
-        f"{result.facts_imported} fact(s), {result.observations_imported} observation(s), "
-        f"{result.mental_models_imported} mental model(s), "
-        f"{result.mental_model_history_imported} mm-history row(s), "
-        f"{result.knowledge_pages_imported} knowledge page(s), {result.directives_imported} directive(s), "
-        f"{result.webhooks_imported} webhook(s), {result.history_rows_imported} history row(s)"
-    )
-
-
 async def _decommission_worker(db_url: str, worker_id: str, schema: str = "public") -> int:
     """Release all tasks owned by a worker, setting them back to pending status."""
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
@@ -915,8 +391,7 @@ def decommission_worker(
 
 async def _decommission_all_workers(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
     """Release all processing tasks from all workers, setting them back to pending status."""
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
@@ -981,8 +456,7 @@ def decommission_workers(
 
 async def _worker_status(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
     """Get all processing tasks grouped by worker with their last update time."""
-    _pg0 = parse_pg0_url(db_url)
-    is_pg0, instance_name = _pg0.is_pg0, _pg0.instance_name
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
     if is_pg0:
         typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
     resolved_url = await resolve_database_url(db_url)
@@ -1050,7 +524,6 @@ def worker_status(
 
 
 def main():
-    load_dotenv_for_entrypoint()
     app()
 
 

hindsight-api-slim/hindsight_api/alembic/env.py

@@ -96,8 +96,7 @@ def get_database_url() -> str:
         # for the sync engine used during migrations.
         database_url = to_libpq_url(database_url)
 
-    # Alembic stores options through ConfigParser, where '%' is interpolation.
-    config.set_main_option("sqlalchemy.url", database_url.replace("%", "%%"))
+    config.set_main_option("sqlalchemy.url", database_url)
     return database_url
 
 

hindsight-api-slim/hindsight_api/alembic/versions/2071c7518f88_add_memory_links_bank_id_index.py

@@ -1,105 +0,0 @@
-"""Add a composite index on memory_links(bank_id, link_type) (PostgreSQL).
-
-``bank_id`` was added to ``memory_links`` in ``c5d6e7f8a9b0`` precisely so that
-bank-scoped reads (e.g. the stats endpoint) could filter on the link table
-directly instead of joining ``memory_units`` — that JOIN took 18+ seconds on
-banks with millions of links. The column landed without an index, so every
-``bank_id = $1`` predicate still falls back to a sequential scan over the whole
-table.
-
-This adds the missing btree. It is composite on ``(bank_id, link_type)`` rather
-than ``bank_id`` alone because the hot query is the stats endpoint's
-``SELECT link_type, COUNT(*) ... WHERE bank_id = $1 GROUP BY link_type``: a
-``(bank_id, link_type)`` index serves that filter, grouping and count as an
-index-only scan, never touching the heap, whereas a ``bank_id``-only index would
-still have to read every matching row to recover ``link_type``. ``link_type`` is
-low-cardinality (only ``temporal``/``semantic``/``caused_by`` are written —
-entity edges were dropped in ``e9b2c7d1f3a4``), so the trailing column adds
-little to the index size while removing the heap fetch.
-
-The Oracle baseline (``o1a2b3c4d5e6``) already creates ``idx_ml_bank_id`` on
-``memory_links(bank_id)``; that single-column index already covers Oracle's
-bank-scoped filter, so the Oracle slot here is intentionally absent and only the
-PostgreSQL dialect gets the composite index.
-
-``memory_links`` can hold tens of millions of rows, so the index is built
-CONCURRENTLY to avoid taking a write lock on the table. CONCURRENTLY cannot run
-inside a transaction block, so the statement runs in an ``autocommit_block()``;
-``IF NOT EXISTS`` keeps it idempotent across retries and re-migrated tenant
-schemas. A CONCURRENTLY build interrupted partway (lock conflict, disk
-pressure, signal) leaves the index behind as *invalid*; ``IF NOT EXISTS`` would
-then skip over it forever, so the upgrade first drops any invalid leftover of
-this name before (re)creating it.
-
-Revision ID: 2071c7518f88
-Revises: a1d3f5b7c9e2
-Create Date: 2026-06-16
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import text
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "2071c7518f88"
-down_revision: str | Sequence[str] | None = "a1d3f5b7c9e2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_INDEX_NAME = "idx_memory_links_bank_id_link_type"
-
-
-def _get_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    bind = op.get_bind()
-    # `or None` collapses an unset option and an explicit empty string into NULL
-    # so the COALESCE below falls back to current_schema() in both cases.
-    target_schema = context.config.get_main_option("target_schema") or None
-    schema = _get_schema_prefix()
-
-    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block; the
-    # autocommit_block runs each statement outside Alembic's migration
-    # transaction.
-    with op.get_context().autocommit_block():
-        # A CONCURRENTLY build that errored on a previous run leaves an INVALID
-        # index of this name behind. `CREATE INDEX ... IF NOT EXISTS` would see
-        # that relation and skip, so bank_id queries would keep seq-scanning.
-        # Drop only the invalid leftover — never a healthy index — so the retry
-        # actually rebuilds a usable one.
-        leftover_invalid = bind.execute(
-            text(
-                "SELECT NOT i.indisvalid "
-                "FROM pg_class c "
-                "JOIN pg_index i ON c.oid = i.indexrelid "
-                "JOIN pg_namespace n ON c.relnamespace = n.oid "
-                "WHERE c.relname = :index_name "
-                "  AND n.nspname = COALESCE(:target_schema, current_schema())"
-            ),
-            {"index_name": _INDEX_NAME, "target_schema": target_schema},
-        ).scalar()
-        if leftover_invalid:
-            op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_INDEX_NAME}")
-
-        # IF NOT EXISTS keeps the create idempotent across retries and schemas.
-        op.execute(f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {_INDEX_NAME} ON {schema}memory_links(bank_id, link_type)")
-
-
-def _pg_downgrade() -> None:
-    schema = _get_schema_prefix()
-    with op.get_context().autocommit_block():
-        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_INDEX_NAME}")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a1c9e7f3b2d8_observation_history_drop_memory_units_fk.py

@@ -1,67 +0,0 @@
-"""Drop observation_history's FK to memory_units.
-
-The history table records one snapshot per observation change, keyed by
-``(bank_id, observation_id)``. Its foreign key to ``memory_units`` existed only to
-cascade-delete history when the observation row went away.
-
-That assumes every observation *is* a ``memory_units`` row, which is true only
-while Postgres is the memories store. When another store owns the memories the
-observation lives there and Postgres holds no row for it, so every history insert
-raises a foreign-key violation — swallowed by the writer as "a race with parallel
-consolidation" and logged at warning level. The audit trail goes silently empty.
-
-Dropping the constraint lets history be recorded wherever the observation is
-stored. The cleanup the cascade used to do is now explicit, in the paths that
-delete observations (``_execute_delete_action``, ``clear_observations``,
-``delete_bank``). Rows orphaned by a path that misses — a document delete
-cascading through ``memory_units``, for instance — are invisible to readers,
-which always filter by ``(bank_id, observation_id)``, and are reclaimed when the
-bank is deleted.
-
-Oracle builds this schema through its own DDL runner and never had the
-constraint, so the Oracle slot is a deliberate no-op.
-
-Revision ID: a1c9e7f3b2d8
-Revises: c7d1e9a4b3f2
-"""
-
-from collections.abc import Sequence
-
-from alembic import op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "a1c9e7f3b2d8"
-down_revision: str | Sequence[str] | None = "c7d1e9a4b3f2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_CONSTRAINT = "observation_history_observation_id_fkey"
-
-
-def _pg_upgrade() -> None:
-    op.execute(f"ALTER TABLE observation_history DROP CONSTRAINT IF EXISTS {_CONSTRAINT}")
-
-
-def _pg_downgrade() -> None:
-    # Re-adding the FK requires every row to reference a live memory_unit, so
-    # clear any history whose observation is not a Postgres row first — those are
-    # exactly the rows this migration made possible.
-    op.execute(
-        "DELETE FROM observation_history h "
-        "WHERE NOT EXISTS (SELECT 1 FROM memory_units m WHERE m.id = h.observation_id)"
-    )
-    op.execute(
-        f"ALTER TABLE observation_history ADD CONSTRAINT {_CONSTRAINT} "
-        "FOREIGN KEY (observation_id) REFERENCES memory_units(id) ON DELETE CASCADE"
-    )
-
-
-def upgrade() -> None:
-    # Oracle never had the constraint (its schema is built by a separate DDL
-    # runner), so only Postgres has anything to drop.
-    run_for_dialect(pg=_pg_upgrade, oracle=None)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=None)

hindsight-api-slim/hindsight_api/alembic/versions/a1d3f5b7c9e2_widen_remaining_bank_id_to_text.py

@@ -1,85 +0,0 @@
-"""Repair: widen the remaining live ``bank_id`` columns from VARCHAR(64) to TEXT on PostgreSQL.
-
-Follow-up to ``c3e5a7b9d1f4`` (issue #2106), which widened the two *history*
-tables (``observation_history``, ``mental_model_history``) to ``TEXT`` after the
-narrow ``VARCHAR(64)`` declaration bricked startup. The same VARCHAR(64) / TEXT
-inconsistency still affects the live tables that store a user-supplied
-``bank_id``:
-
-* ``directives``    -- created VARCHAR(64) in ``p1k2l3m4n5o6``
-* ``mental_models`` -- VARCHAR(64) (origin ``pinned_reflections`` in
-                       ``n9i0j1k2l3m4``; recreated in ``h3c4d5e6f7g8``)
-
-``mental_model_versions`` is intentionally *not* widened here: it is created in
-``j5e6f7g8h9i0`` but dropped (``DROP TABLE ... CASCADE``) in ``o0j1k2l3m4n5`` and
-never recreated on the upgrade path, so it does not exist at head. Issuing
-``ALTER TABLE mental_model_versions ...`` would raise ``UndefinedTable`` and --
-because migrations run inside the lifespan-startup transaction -- roll the whole
-migration back, bricking the API. (It is unrelated to the live
-``mental_model_history`` table widened by ``c3e5a7b9d1f4``.)
-
-``banks.bank_id`` is ``TEXT`` (unbounded), so a deployment can create a bank
-whose id exceeds 64 chars -- the 78-char hierarchical org-unit shape reported in
-issue #2106 -- and the bank insert succeeds. The next write that propagates that
-id (``create_directive``, ``create_mental_model`` / consolidation, or
-mental-model versioning) then aborts with::
-
-    psycopg2.errors.StringDataRightTruncation: value too long for type
-    character varying(64)
-
-i.e. a 500 on core write endpoints, instead of the startup brick that
-``c3e5a7b9d1f4`` already repaired.
-
-``ALTER COLUMN ... TYPE TEXT`` is a no-op on a column that is already ``TEXT``,
-so every upgrade path converges on ``TEXT``. These tables are per-tenant (they
-live in each tenant schema, not ``public``), so this runs for every migrated
-schema via the search-path-aware prefix -- the same mechanism as
-``c3e5a7b9d1f4``.
-
-PostgreSQL only: these tables are created by PostgreSQL-only migrations
-(``run_for_dialect(pg=...)``); on Oracle they are absent or already
-``VARCHAR2(256)`` (consistent, never truncates), so the Oracle slot is
-intentionally absent -- mirroring ``c3e5a7b9d1f4``.
-
-Revision ID: a1d3f5b7c9e2
-Revises: c3e5a7b9d1f4
-Create Date: 2026-06-13
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "a1d3f5b7c9e2"
-down_revision: str | Sequence[str] | None = "c3e5a7b9d1f4"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}directives ALTER COLUMN bank_id TYPE TEXT")
-    op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN bank_id TYPE TEXT")
-
-
-def _pg_downgrade() -> None:
-    # No-op: narrowing back to VARCHAR(64) could truncate real data and would
-    # re-introduce the bug this migration repairs. The column types are owned by
-    # the migrations that created the tables.
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a7b8c9d0e1f2_split_history_into_own_tables.py

@@ -1,253 +0,0 @@
-"""Move mental-model and observation history into dedicated tables.
-
-Both histories were accumulated in a single JSONB/CLOB ``history`` column
-(``mental_models.history`` and ``memory_units.history``), appended to on every
-update. That design has two problems:
-
-1. **Unbounded growth on observations.** The observation write path appended a
-   snapshot on every update with no cap at all, so a frequently-reinforced
-   observation grew its ``history`` array until it crossed Postgres's hard 256MB
-   jsonb limit (SQLSTATE 54000), after which every further UPDATE failed and the
-   row was stuck.
-2. **Wrong-axis cap on mental models.** The mental-model cap bounded the *number*
-   of entries (50), not their *size* — a single large reflect snapshot could
-   still blow the budget — and rewrote the whole array (plus TOAST) on every
-   refresh, defeating HOT updates.
-
-This migration creates one row per history entry in two dedicated tables, with
-an index that makes "most recent N for this item" cheap, then drops the old
-columns. The cap is now enforced at write time as a bounded DELETE of the
-oldest over-cap rows (see config ``*_HISTORY_MAX_ENTRIES``).
-
-Revision ID: a7b8c9d0e1f2
-Revises: d3e4f5a6b7c8
-Create Date: 2026-06-05
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "a7b8c9d0e1f2"
-down_revision: str | Sequence[str] | None = "d3e4f5a6b7c8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-# ---------------------------------------------------------------------------
-# PostgreSQL
-# ---------------------------------------------------------------------------
-def _pg_upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Both tables share the same shape: surrogate id, FK to the parent, bank_id,
-    # the snapshot payload as a single JSONB ``content`` blob, and changed_at.
-    # The payload is per-row (one change per row) so it stays small — this is NOT
-    # the old single-column-grows-forever design; growth is bounded by row count
-    # plus the write-time cap. Folding the previous_* fields into one JSONB keeps
-    # the schema dialect-simple (no array columns) and flexible.
-
-    # --- mental_model_history -------------------------------------------------
-    # content: {"previous_content": ..., "previous_reflect_response": {...}}
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}mental_model_history (
-            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
-            mental_model_id VARCHAR(64) NOT NULL,
-            bank_id TEXT NOT NULL,
-            content JSONB NOT NULL,
-            changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
-            FOREIGN KEY (mental_model_id, bank_id)
-                REFERENCES {schema}mental_models(id, bank_id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute(
-        "CREATE INDEX IF NOT EXISTS idx_mm_history_model "
-        f"ON {schema}mental_model_history (bank_id, mental_model_id, changed_at DESC, id DESC)"
-    )
-
-    # --- observation_history --------------------------------------------------
-    # content: {"previous_text", "previous_tags", "previous_occurred_start",
-    #           "previous_occurred_end", "previous_mentioned_at", "new_source_memory_ids"}
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}observation_history (
-            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
-            observation_id UUID NOT NULL,
-            bank_id TEXT NOT NULL,
-            content JSONB NOT NULL,
-            changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
-            FOREIGN KEY (observation_id)
-                REFERENCES {schema}memory_units(id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute(
-        "CREATE INDEX IF NOT EXISTS idx_observation_history_obs "
-        f"ON {schema}observation_history (observation_id, changed_at DESC, id DESC)"
-    )
-
-    # --- backfill mental models ----------------------------------------------
-    # Explode each row's history array into rows, preserving chronological order
-    # via WITH ORDINALITY so the IDENTITY id tie-breaks oldest->newest correctly.
-    # changed_at is promoted to its own column; the rest of the element becomes
-    # ``content`` (the ``- 'changed_at'`` strips the now-redundant key).
-    op.execute(
-        f"""
-        INSERT INTO {schema}mental_model_history (mental_model_id, bank_id, content, changed_at)
-        SELECT mm.id, mm.bank_id,
-               e - 'changed_at',
-               COALESCE(NULLIF(e->>'changed_at', '')::timestamptz, now())
-        FROM {schema}mental_models mm
-             CROSS JOIN LATERAL jsonb_array_elements(mm.history) WITH ORDINALITY a(e, ord)
-        WHERE mm.history IS NOT NULL
-          AND jsonb_typeof(mm.history) = 'array'
-          AND jsonb_array_length(mm.history) > 0
-        ORDER BY mm.id, mm.bank_id, ord
-        """
-    )
-
-    # --- backfill observations -----------------------------------------------
-    op.execute(
-        f"""
-        INSERT INTO {schema}observation_history (observation_id, bank_id, content, changed_at)
-        SELECT mu.id, mu.bank_id,
-               e - 'changed_at',
-               COALESCE(NULLIF(e->>'changed_at', '')::timestamptz, now())
-        FROM {schema}memory_units mu
-             CROSS JOIN LATERAL jsonb_array_elements(mu.history) WITH ORDINALITY a(e, ord)
-        WHERE mu.fact_type = 'observation'
-          AND mu.history IS NOT NULL
-          AND jsonb_typeof(mu.history) = 'array'
-          AND jsonb_array_length(mu.history) > 0
-        ORDER BY mu.id, ord
-        """
-    )
-
-    # --- drop the legacy columns ---------------------------------------------
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS history")
-
-
-def _pg_downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Re-add the columns (empty — historical content is not reconstructed back
-    # into the array form; the dedicated tables are dropped below).
-    op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_observation_history_obs")
-    op.execute(f"DROP TABLE IF EXISTS {schema}observation_history")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mm_history_model")
-    op.execute(f"DROP TABLE IF EXISTS {schema}mental_model_history")
-
-
-# ---------------------------------------------------------------------------
-# Oracle 23ai
-# ---------------------------------------------------------------------------
-def _oracle_upgrade() -> None:
-    # Same single-JSONB shape as PG: ``content`` holds the snapshot payload as a
-    # CLOB IS JSON. The legacy per-element JSON object (minus changed_at, promoted
-    # to its own column) is carried through verbatim on backfill — the array
-    # columns the previous design needed are gone.
-    op.execute(
-        """
-        CREATE TABLE IF NOT EXISTS mental_model_history (
-            id NUMBER GENERATED BY DEFAULT AS IDENTITY,
-            mental_model_id VARCHAR2(256) NOT NULL,
-            bank_id VARCHAR2(256) NOT NULL,
-            content CLOB NOT NULL
-                CONSTRAINT mmh_content_json CHECK (content IS JSON),
-            changed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-            CONSTRAINT pk_mental_model_history PRIMARY KEY (id),
-            CONSTRAINT fk_mmh_model FOREIGN KEY (mental_model_id, bank_id)
-                REFERENCES mental_models(id, bank_id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute(
-        "CREATE INDEX idx_mm_history_model ON mental_model_history (bank_id, mental_model_id, changed_at DESC, id DESC)"
-    )
-
-    op.execute(
-        """
-        CREATE TABLE IF NOT EXISTS observation_history (
-            id NUMBER GENERATED BY DEFAULT AS IDENTITY,
-            observation_id RAW(16) NOT NULL,
-            bank_id VARCHAR2(256) NOT NULL,
-            content CLOB NOT NULL
-                CONSTRAINT oh_content_json CHECK (content IS JSON),
-            changed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-            CONSTRAINT pk_observation_history PRIMARY KEY (id),
-            CONSTRAINT fk_oh_obs FOREIGN KEY (observation_id)
-                REFERENCES memory_units(id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute(
-        "CREATE INDEX idx_observation_history_obs ON observation_history (observation_id, changed_at DESC, id DESC)"
-    )
-
-    bind = op.get_bind()
-
-    # Backfill via JSON_TABLE. ``content`` is the whole element (FORMAT JSON PATH
-    # '$'); changed_at is also promoted to its own column. Backfilled content may
-    # therefore still carry a redundant changed_at key, which the read path
-    # ignores in favour of the column — harmless, and avoids JSON surgery here.
-    bind.exec_driver_sql(
-        """
-        INSERT INTO mental_model_history (mental_model_id, bank_id, content, changed_at)
-        SELECT mm.id, mm.bank_id, jt.content, NVL(jt.changed_at, SYSTIMESTAMP)
-        FROM mental_models mm,
-             JSON_TABLE(mm.history, '$[*]' COLUMNS (
-                 seq FOR ORDINALITY,
-                 content CLOB FORMAT JSON PATH '$',
-                 changed_at TIMESTAMP WITH TIME ZONE PATH '$.changed_at'
-             )) jt
-        WHERE mm.history IS NOT NULL
-        ORDER BY mm.id, mm.bank_id, jt.seq
-        """
-    )
-
-    bind.exec_driver_sql(
-        """
-        INSERT INTO observation_history (observation_id, bank_id, content, changed_at)
-        SELECT mu.id, mu.bank_id, jt.content, NVL(jt.changed_at, SYSTIMESTAMP)
-        FROM memory_units mu,
-             JSON_TABLE(mu.history, '$[*]' COLUMNS (
-                 seq FOR ORDINALITY,
-                 content CLOB FORMAT JSON PATH '$',
-                 changed_at TIMESTAMP WITH TIME ZONE PATH '$.changed_at'
-             )) jt
-        WHERE mu.fact_type = 'observation' AND mu.history IS NOT NULL
-        ORDER BY mu.id, jt.seq
-        """
-    )
-
-    op.execute("ALTER TABLE mental_models DROP COLUMN history")
-    op.execute("ALTER TABLE memory_units DROP COLUMN history")
-
-
-def _oracle_downgrade() -> None:
-    op.execute("ALTER TABLE mental_models ADD history CLOB DEFAULT '[]' NOT NULL")
-    op.execute("ALTER TABLE memory_units ADD history CLOB DEFAULT '[]'")
-    op.execute("DROP TABLE observation_history CASCADE CONSTRAINTS")
-    op.execute("DROP TABLE mental_model_history CASCADE CONSTRAINTS")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a8c1e4f7b0d3_add_operation_retention_indexes.py

@@ -1,82 +0,0 @@
-"""Add indexes for terminal cleanup and newest-first operation listing.
-
-Revision ID: a8c1e4f7b0d3
-Revises: e7c3a9f1b2d5
-Create Date: 2026-07-14
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "a8c1e4f7b0d3"
-down_revision: str | Sequence[str] | None = "e7c3a9f1b2d5"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for PostgreSQL multi-tenant migration runs."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    # These can be large tables in long-running installations. Concurrent DDL
-    # keeps operation submission, polling, and status reads available.
-    with op.get_context().autocommit_block():
-        op.execute(
-            "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_async_operations_terminal_cleanup "
-            f"ON {schema}async_operations (updated_at, operation_id) "
-            "WHERE status IN ('completed', 'failed', 'cancelled')"
-        )
-        op.execute(
-            "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_async_operations_bank_created_desc "
-            f"ON {schema}async_operations (bank_id, created_at DESC)"
-        )
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    with op.get_context().autocommit_block():
-        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_async_operations_bank_created_desc")
-        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_async_operations_terminal_cleanup")
-
-
-def _oracle_create_index(sql: str) -> None:
-    """Create an index idempotently for rerun-safe Oracle migrations."""
-    block = (
-        "BEGIN "
-        "EXECUTE IMMEDIATE :stmt; "
-        "EXCEPTION WHEN OTHERS THEN "
-        "IF SQLCODE = -955 THEN NULL; ELSE RAISE; END IF; "
-        "END;"
-    )
-    op.get_bind().exec_driver_sql(block, {"stmt": sql})
-
-
-def _oracle_upgrade() -> None:
-    # Oracle migrations run with CURRENT_SCHEMA set to each tenant, so table
-    # and index names intentionally remain unqualified here.
-    _oracle_create_index(
-        "CREATE INDEX idx_async_operations_terminal_cleanup ON async_operations (updated_at, operation_id, status)"
-    )
-    _oracle_create_index(
-        "CREATE INDEX idx_async_operations_bank_created_desc ON async_operations (bank_id, created_at DESC)"
-    )
-
-
-def _oracle_downgrade() -> None:
-    op.execute("DROP INDEX idx_async_operations_bank_created_desc")
-    op.execute("DROP INDEX idx_async_operations_terminal_cleanup")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/a9b8c7d6e5f4_add_knowledge_pages.py

@@ -1,126 +0,0 @@
-"""Add knowledge_pages table (knowledge-base hierarchy).
-
-The knowledge base organizes synthesized mental models into a navigable tree of
-**folders** and **pages**. A page references the mental model that holds its
-content (``mental_model_id``); a folder is a pure container (``mental_model_id``
-NULL). Hierarchy is a single self-referential ``parent_id`` so folders can nest
-arbitrarily. Content stays in ``mental_models`` — this table is metadata + tree
-structure only.
-
-``managed`` lets a client tag a node as system-owned vs. hand-authored; it
-carries no server-side behaviour. A partial unique index keeps page names unique
-within a folder (case-insensitive; root pages compared under an empty parent).
-
-Revision ID: a9b8c7d6e5f4
-Revises: a1c9e7f3b2d8
-Create Date: 2026-06-25
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "a9b8c7d6e5f4"
-down_revision: str | Sequence[str] | None = "a1c9e7f3b2d8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    # parent_id self-FK cascades so deleting a folder row removes its whole
-    # subtree of rows in one shot. The mental_model FK is composite (matches the
-    # mental_models (id, bank_id) PK) and cascades too, so deleting a page's
-    # mental model removes the page row — folders skip the FK because a NULL
-    # column in a composite FK is not enforced (MATCH SIMPLE).
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}knowledge_pages (
-            id VARCHAR(64) NOT NULL,
-            bank_id TEXT NOT NULL,
-            parent_id VARCHAR(64),
-            kind VARCHAR(16) NOT NULL,
-            name TEXT NOT NULL,
-            mental_model_id VARCHAR(64),
-            sort_order INTEGER NOT NULL DEFAULT 0,
-            managed BOOLEAN NOT NULL DEFAULT false,
-            created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
-            updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
-            CONSTRAINT pk_knowledge_pages PRIMARY KEY (id),
-            CONSTRAINT ck_knowledge_pages_kind CHECK (kind IN ('folder', 'page')),
-            CONSTRAINT fk_kp_bank FOREIGN KEY (bank_id)
-                REFERENCES {schema}banks(bank_id) ON DELETE CASCADE,
-            CONSTRAINT fk_kp_parent FOREIGN KEY (parent_id)
-                REFERENCES {schema}knowledge_pages(id) ON DELETE CASCADE,
-            CONSTRAINT fk_kp_mm FOREIGN KEY (mental_model_id, bank_id)
-                REFERENCES {schema}mental_models(id, bank_id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_kp_bank_parent ON {schema}knowledge_pages (bank_id, parent_id, sort_order)"
-    )
-    # COALESCE(parent_id, '') so root-level pages (NULL parent) are also unique by
-    # name — NULLs would otherwise compare distinct and allow duplicates.
-    op.execute(
-        "CREATE UNIQUE INDEX IF NOT EXISTS uq_kp_folder_pagename "
-        f"ON {schema}knowledge_pages (bank_id, COALESCE(parent_id, ''), lower(name)) "
-        "WHERE kind = 'page'"
-    )
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    op.execute(f"DROP INDEX IF EXISTS {schema}uq_kp_folder_pagename")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_kp_bank_parent")
-    op.execute(f"DROP TABLE IF EXISTS {schema}knowledge_pages")
-
-
-def _oracle_upgrade() -> None:
-    # No case-insensitive unique index on Oracle: `name` is a CLOB and cannot be
-    # indexed with lower(); page-name uniqueness is enforced on PG only.
-    op.execute(
-        """
-        CREATE TABLE IF NOT EXISTS knowledge_pages (
-            id VARCHAR2(64) NOT NULL,
-            bank_id VARCHAR2(256) NOT NULL,
-            parent_id VARCHAR2(64),
-            kind VARCHAR2(16) NOT NULL,
-            name CLOB NOT NULL,
-            mental_model_id VARCHAR2(64),
-            sort_order NUMBER DEFAULT 0 NOT NULL,
-            managed NUMBER(1) DEFAULT 0 NOT NULL,
-            created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-            updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-            CONSTRAINT pk_knowledge_pages PRIMARY KEY (id),
-            CONSTRAINT ck_knowledge_pages_kind CHECK (kind IN ('folder', 'page')),
-            CONSTRAINT fk_kp_bank FOREIGN KEY (bank_id)
-                REFERENCES banks(bank_id) ON DELETE CASCADE,
-            CONSTRAINT fk_kp_parent FOREIGN KEY (parent_id)
-                REFERENCES knowledge_pages(id) ON DELETE CASCADE,
-            CONSTRAINT fk_kp_mm FOREIGN KEY (mental_model_id, bank_id)
-                REFERENCES mental_models(id, bank_id) ON DELETE CASCADE
-        )
-        """
-    )
-    op.execute("CREATE INDEX idx_kp_bank_parent ON knowledge_pages (bank_id, parent_id, sort_order)")
-
-
-def _oracle_downgrade() -> None:
-    op.execute("DROP TABLE knowledge_pages CASCADE CONSTRAINTS")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/b2d4f6a8c1e3_repair_maintenance_routines_public.py

@@ -1,156 +0,0 @@
-"""Repair: install maintenance routines on the ``public`` / base-schema run.
-
-The original maintenance-routines migration (``e5f6a7b8c9d0``) only created the
-shared ``public.banks_needing_consolidation()`` and
-``public.schemas_with_expired_rows(...)`` routines when the run had *no*
-``target_schema`` at all. But the single-tenant runtime always migrates an
-explicit schema — which defaults to ``public`` — so on every default
-PostgreSQL deployment the migration was stamped as applied while the functions
-were never created. Background maintenance then logs::
-
-    Retention sweep failed for llm_requests: function public.schemas_with_expired_rows(...) does not exist
-    Consolidation reconcile discovery failed: function public.banks_needing_consolidation() does not exist
-
-See https://github.com/vectorize-io/hindsight/issues/2056.
-
-Because ``e5f6a7b8c9d0`` is already stamped on affected ``0.8.0`` databases,
-editing it would not re-run it there. This forward migration re-installs the
-functions idempotently (``CREATE OR REPLACE``) on the run that targets the
-shared ``public`` schema (base run with no ``target_schema``, or an explicit
-``target_schema=public``), self-healing already-upgraded deployments and
-covering fresh upgrades from earlier versions.
-
-Per-tenant runs against a non-``public`` schema still skip it: re-issuing
-``CREATE OR REPLACE FUNCTION public....`` from each concurrent tenant migration
-aborts with ``tuple concurrently updated`` on the ``pg_proc`` catalog row, and
-the base/public run has already created the functions for every tenant to use.
-Runs that target ``public`` are serialized by the per-schema migration advisory
-lock, so only one wins the create.
-
-PostgreSQL only — the worker poller and these tables are not wired for Oracle,
-so the Oracle slot is intentionally absent (mirrors ``e5f6a7b8c9d0``).
-
-Revision ID: b2d4f6a8c1e3
-Revises: e5f6a7b8c9d0
-Create Date: 2026-06-08
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "b2d4f6a8c1e3"
-down_revision: str | Sequence[str] | None = "e5f6a7b8c9d0"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _should_install_public_routines(target_schema: str | None) -> bool:
-    """True for the run that must (re)create the shared ``public.*`` routines.
-
-    The routines physically live in ``public`` (hard-coded ``public.`` qualifier
-    in the SQL below), so they must be installed exactly once — on the base run
-    (no ``target_schema``) or on the run that explicitly targets ``public``. A
-    run against any other tenant schema skips it to avoid concurrent
-    ``CREATE OR REPLACE`` on the same ``pg_proc`` row.
-    """
-    return not target_schema or target_schema == "public"
-
-
-def _pg_upgrade() -> None:
-    if not _should_install_public_routines(context.config.get_main_option("target_schema")):
-        return
-    # Banks with eligible-but-unscheduled facts and no in-flight consolidation.
-    # Auto-consolidation is filtered here only at the bank level (cheap prune);
-    # the full hierarchical resolution (global -> tenant -> bank, plus
-    # enable_observations) is done by the caller for the small returned set.
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.banks_needing_consolidation()
-        RETURNS TABLE(schema_name text, bank_id text)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
-            LOOP
-                RETURN QUERY EXECUTE format($q$
-                    SELECT %1$L::text, m.bank_id
-                    FROM %1$I.memory_units m
-                    JOIN %1$I.banks b ON b.bank_id = m.bank_id
-                    WHERE m.consolidated_at IS NULL
-                      AND m.consolidation_failed_at IS NULL
-                      AND m.fact_type IN ('experience', 'world')
-                      AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
-                      AND NOT EXISTS (
-                          SELECT 1 FROM %1$I.async_operations o
-                          WHERE o.bank_id = m.bank_id
-                            AND o.operation_type = 'consolidation'
-                            AND o.status IN ('pending', 'processing')
-                      )
-                    GROUP BY m.bank_id
-                $q$, sch);
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-    # Schemas holding at least one row of p_table older than p_days. p_ts_col is
-    # the timestamp column to compare. Returns nothing when p_days <= 0
-    # (retention disabled).
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.schemas_with_expired_rows(
-            p_table text, p_ts_col text, p_days int
-        )
-        RETURNS SETOF text
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-            has_expired boolean;
-        BEGIN
-            IF p_days IS NULL OR p_days <= 0 THEN
-                RETURN;
-            END IF;
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = p_table AND c.relkind = 'r'
-            LOOP
-                EXECUTE format(
-                    'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
-                    sch, p_table, p_ts_col
-                ) INTO has_expired USING p_days;
-                IF has_expired THEN
-                    RETURN NEXT sch;
-                END IF;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    # No-op: ``e5f6a7b8c9d0`` owns the lifecycle of these functions and drops
-    # them on its own downgrade. This migration only ever (re)creates them, so
-    # there is nothing to undo without racing that migration's DROP.
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/b3e8d1c6f4a9_entity_kind_partial_trgm_index.py

@@ -1,213 +0,0 @@
-"""Add entities.entity_kind and exclude label entities from the trigram index.
-
-Label entities (values of ``entity_labels`` config groups, stored as
-``key:value`` canonical names) resolve by exact match only — fuzzy resolution
-must never merge distinct label values (#1558), and since #3187 they are looked
-up via the exact-match unique index rather than probed through pg_trgm. Their
-rows were still covered by the shared trigram index, so every fuzzy probe for a
-*regular* entity name pulled them into its candidate set only to discard them
-in the bitmap recheck. On banks where a free-text label group accumulated tens
-of thousands of mutually-similar values this recheck-discard overhead dominated
-database CPU under ingest bursts (#3208).
-
-"Is this row a label" was previously derived at runtime from the bank's
-``entity_labels`` config, which an index predicate cannot reference — so the
-classification is now materialised on the row:
-
-1. Add ``entity_kind`` ("regular"/"label", CHECK-constrained) on both dialects.
-   A kind column rather than a boolean so future entity kinds don't need
-   another column.
-2. Backfill per bank by classifying ``canonical_name`` against the bank's
-   ``entity_labels`` config with the same ``is_label_entity()`` the resolver
-   uses at insert time — a SQL reimplementation would be a second source of
-   truth (and the map-group recursion doesn't translate). Banks hold at most
-   tens of thousands of entities, so the synchronous per-bank backfill is fine.
-   Label configs supplied only by a tenant extension (not stored in
-   ``banks.config``) can't be seen here; their rows stay "regular", which
-   costs index size but never correctness — label *texts* still resolve via
-   the exact-match unique index.
-3. Rebuild the PG trigram index as a partial index excluding label rows.
-   Built CONCURRENTLY (autocommit block, invalid-leftover sweep, IF NOT
-   EXISTS — same shape as 2071c7518f88) and only then drop the old full
-   index, so fuzzy probes never lose index coverage. Skipped entirely when
-   pg_trgm is absent (the resolver falls back to the "full" strategy, #626).
-
-Oracle has no trigram index — it fuzzy-matches with a UTL_MATCH scan — so it
-only gets the column + backfill; the resolver adds the matching
-``entity_kind != 'label'`` filter to that scan.
-
-Revision ID: b3e8d1c6f4a9
-Revises: f2a6d8c4b1e9
-Create Date: 2026-08-06
-"""
-
-import json
-from collections.abc import Sequence
-
-import sqlalchemy as sa
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "b3e8d1c6f4a9"
-down_revision: str | Sequence[str] | None = "f2a6d8c4b1e9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_OLD_INDEX = "entities_canonical_name_lower_trgm_idx"
-_NEW_INDEX = "entities_canonical_name_lower_trgm_nonlabel_idx"
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _backfill_entity_kind(schema: str) -> None:
-    """Set entity_kind='label' on rows matching their bank's entity_labels config.
-
-    Runs the resolver's own classification (``is_label_entity``) per bank in
-    Python rather than reimplementing the enum/text/map prefix rules in SQL.
-    Shared by both dialects: plain SELECT/UPDATE with expanding IN binds.
-    """
-    from hindsight_api.engine.retain.entity_labels import (
-        build_labels_lookup,
-        is_label_entity,
-        parse_entity_labels,
-    )
-
-    bind = op.get_bind()
-    banks = bind.execute(sa.text(f"SELECT bank_id, config FROM {schema}banks")).fetchall()
-    for bank_id, raw_config in banks:
-        # PG JSONB arrives as a dict; Oracle CLOB arrives as a LOB object on
-        # raw text() fetches (oracledb's fetch_lobs default) — read it into a
-        # JSON string first.
-        if raw_config is not None and not isinstance(raw_config, (str, dict)):
-            raw_config = raw_config.read()
-        config = json.loads(raw_config) if isinstance(raw_config, str) else (raw_config or {})
-        labels_cfg = parse_entity_labels(config.get("entity_labels"))
-        if labels_cfg is None:
-            continue
-        lookup = build_labels_lookup(labels_cfg)
-        rows = bind.execute(
-            sa.text(f"SELECT id, canonical_name FROM {schema}entities WHERE bank_id = :bank_id"),
-            {"bank_id": bank_id},
-        ).fetchall()
-        label_ids = [entity_id for entity_id, name in rows if is_label_entity(name, labels_cfg, lookup)]
-        # Chunked to stay under Oracle's 1000-element IN limit; also keeps PG
-        # bind arrays bounded.
-        for start in range(0, len(label_ids), 500):
-            chunk = label_ids[start : start + 500]
-            stmt = sa.text(f"UPDATE {schema}entities SET entity_kind = 'label' WHERE id IN :ids").bindparams(
-                sa.bindparam("ids", expanding=True)
-            )
-            bind.execute(stmt, {"ids": chunk})
-
-
-def _pg_upgrade() -> None:
-    bind = op.get_bind()
-    schema = _pg_schema_prefix()
-    # `or None` collapses an unset option and an explicit empty string into NULL
-    # so the COALESCE below falls back to current_schema() in both cases.
-    target_schema = context.config.get_main_option("target_schema") or None
-
-    # IF NOT EXISTS: the transactional part below commits when the autocommit
-    # block is entered, so a failure during the CONCURRENTLY build leaves the
-    # revision unstamped with the column already added — the retry must not
-    # trip over it. The constant default is a metadata-only change on PG 11+.
-    op.execute(
-        f"ALTER TABLE {schema}entities ADD COLUMN IF NOT EXISTS entity_kind TEXT DEFAULT 'regular' NOT NULL "
-        f"CONSTRAINT chk_entities_entity_kind CHECK (entity_kind IN ('regular', 'label'))"
-    )
-    _backfill_entity_kind(schema)
-
-    # Without pg_trgm neither the old index nor the extension's opclass exists;
-    # the resolver already runs the "full" strategy there (#626).
-    has_trgm = bind.execute(sa.text("SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_trgm')")).scalar()
-    if not has_trgm:
-        return
-
-    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block; the
-    # autocommit_block runs each statement outside Alembic's migration
-    # transaction. Build the partial index first and drop the old full index
-    # only afterwards, so fuzzy probes never lose index coverage.
-    with op.get_context().autocommit_block():
-        # A CONCURRENTLY build that errored on a previous run leaves an INVALID
-        # index of this name behind, which IF NOT EXISTS would skip forever.
-        leftover_invalid = bind.execute(
-            sa.text(
-                "SELECT NOT i.indisvalid "
-                "FROM pg_class c "
-                "JOIN pg_index i ON c.oid = i.indexrelid "
-                "JOIN pg_namespace n ON c.relnamespace = n.oid "
-                "WHERE c.relname = :index_name "
-                "  AND n.nspname = COALESCE(:target_schema, current_schema())"
-            ),
-            {"index_name": _NEW_INDEX, "target_schema": target_schema},
-        ).scalar()
-        if leftover_invalid:
-            op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_NEW_INDEX}")
-
-        # The predicate must textually match the resolver's candidate query
-        # (`entity_kind != 'label'`) for the planner to choose this index.
-        op.execute(
-            f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {_NEW_INDEX} "
-            f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops) "
-            f"WHERE entity_kind != 'label'"
-        )
-        op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}{_OLD_INDEX}")
-
-
-def _pg_downgrade() -> None:
-    bind = op.get_bind()
-    schema = _pg_schema_prefix()
-
-    has_trgm = bind.execute(sa.text("SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_trgm')")).scalar()
-    if has_trgm:
-        # Restore the full index before dropping the partial one so fuzzy
-        # probes keep index coverage throughout.
-        with op.get_context().autocommit_block():
-            op.execute(
-                f"CREATE INDEX CONCURRENTLY IF NOT EXISTS {_OLD_INDEX} "
-                f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops)"
-            )
-    # Dropping the column also drops the partial index and CHECK constraint.
-    op.execute(f"ALTER TABLE {schema}entities DROP COLUMN IF EXISTS entity_kind")
-
-
-def _oracle_upgrade() -> None:
-    # Swallow ORA-01430 (column already exists) so a retry after a mid-run
-    # failure is idempotent — Oracle DDL auto-commits statement by statement.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE entities ADD (entity_kind VARCHAR2(16) DEFAULT ''regular'' NOT NULL
-                CONSTRAINT chk_entities_entity_kind CHECK (entity_kind IN (''regular'', ''label'')))';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -1430 THEN RAISE; END IF;
-        END;
-        """
-    )
-    _backfill_entity_kind("")
-
-
-def _oracle_downgrade() -> None:
-    # Swallow ORA-00904 (column does not exist).
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE entities DROP COLUMN entity_kind';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -904 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/b57a7c9e0d13_add_bank_stats_cache.py

@@ -1,61 +0,0 @@
-"""Add bank_stats_cache table for distributed get_bank_stats caching
-
-Revision ID: b57a7c9e0d13
-Revises: c3f7a1b9d2e4
-Create Date: 2026-07-01
-
-get_bank_stats aggregates over memory_links / unit_entities — a multi-second scan
-on banks with millions of rows. The result was cached per-process (in-memory), so
-every API worker recomputed it once per TTL and the first caller after expiry
-stalled. This table backs a shared, cross-process TTL cache: one worker's compute
-is written here and served to all the others.
-
-PostgreSQL only. Oracle keeps the in-process cache (the runtime picks the backing
-store by dialect), so the Oracle upgrade slot is intentionally absent.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "b57a7c9e0d13"
-down_revision: str | Sequence[str] | None = "c3f7a1b9d2e4"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _get_schema_prefix()
-    # One row per bank: payload is the full get_bank_stats result, computed_at
-    # drives logical TTL expiry. Rows are overwritten in place (ON CONFLICT), so
-    # the table never grows beyond the number of banks and needs no purge job.
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}bank_stats_cache (
-            bank_id TEXT PRIMARY KEY,
-            payload JSONB NOT NULL,
-            computed_at TIMESTAMPTZ NOT NULL DEFAULT now()
-        )
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"DROP TABLE IF EXISTS {schema}bank_stats_cache")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)  # oracle slot intentionally absent → no-op
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/b6d2f8a4c1e7_maintenance_routines_schema_local.py

@@ -1,259 +0,0 @@
-"""Install the maintenance discovery routines into the configured schema.
-
-The three discovery routines driving the background maintenance loop —
-``banks_needing_consolidation()``, ``schemas_with_expired_rows(...)`` and
-``mental_models_with_cron()`` — were installed into ``public`` and gated on the
-run being the base run (no ``target_schema``) or an explicit
-``target_schema='public'`` run (``e5f6a7b8c9d0`` → ``b2d4f6a8c1e3`` →
-``c7e9f1a3b5d2``, ``f4d1c2b3a5e6``).
-
-That leaves a **single-tenant deployment migrated into a dedicated, non-**
-``public`` **schema** (``HINDSIGHT_API_DATABASE_SCHEMA=<non-public>``) with no
-routines at all: the runtime migrates only that one schema, so ``target_schema``
-is never falsy or ``public``, the gate never opens, and the maintenance loop
-logs, forever::
-
-    function public.banks_needing_consolidation() does not exist
-    function public.schemas_with_expired_rows(...) does not exist
-
-The revision is stamped applied, so redeploying the same version does not help
-(issue #2638; #2056 only fixed the ``public``/base-run case).
-
-**The bug was the hardcoded literal, not the gating.** These routines are
-database-global — each enumerates ``pg_class`` across every schema and dispatches
-per schema — so exactly one copy should exist, and the maintenance loop calls the
-one in ``get_config().database_schema`` (see ``fq_routine``). The old gate
-installed into whichever schema was named ``public`` instead of whichever schema
-the deployment is actually configured to use. Comparing ``target_schema`` against
-the configured schema instead of the literal fixes #2638 at the source.
-
-That also keeps the property the gate existed for: exactly one migration run
-satisfies the predicate, so concurrent per-schema runs never issue competing
-``CREATE OR REPLACE`` against the same ``pg_proc`` row and cannot hit
-``tuple concurrently updated``. No cross-process coordination is required — in
-particular no advisory lock, which is unusable here because Hindsight runs behind
-connection poolers and managed PG services (see #2817).
-
-Runs targeting any *other* schema drop the routines from that schema rather than
-merely skipping. An earlier revision of this migration installed a copy into
-every schema it touched, which left one dead duplicate per tenant on any database
-that ran it; the drop makes the next migration pass clean those up instead of
-leaving them behind forever.
-
-PostgreSQL only: the maintenance loop and worker poller are PG-only, so the
-Oracle slot is intentionally absent (mirrors ``e5f6a7b8c9d0``).
-
-Revision ID: b6d2f8a4c1e7
-Revises: a8c1e4f7b0d3
-Create Date: 2026-07-20
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-from hindsight_api.config import get_config
-
-revision: str = "b6d2f8a4c1e7"
-down_revision: str | Sequence[str] | None = "a8c1e4f7b0d3"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _configured_schema() -> str:
-    """The one schema this deployment's routines live in and are called from."""
-    return get_config().database_schema or "public"
-
-
-def _target_schema() -> str | None:
-    return context.config.get_main_option("target_schema")
-
-
-def _is_install_run() -> bool:
-    """True for the single run that owns the routines.
-
-    The base run (no ``target_schema``) and the run targeting the configured
-    schema are the same deployment-level run; every other target is a tenant
-    schema that must not carry its own copy.
-    """
-    target = _target_schema()
-    return not target or target == _configured_schema()
-
-
-def _prefix(schema: str | None) -> str:
-    """Qualifier for ``schema``, or ``""`` to fall back to ``search_path``."""
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    if not _is_install_run():
-        _drop_stray_copies()
-        return
-    schema = _prefix(_target_schema())
-
-    op.execute(
-        f"""
-        CREATE OR REPLACE FUNCTION {schema}banks_needing_consolidation()
-        RETURNS TABLE(schema_name text, bank_id text)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    RETURN QUERY EXECUTE format($q$
-                        SELECT %1$L::text, m.bank_id
-                        FROM %1$I.memory_units m
-                        JOIN %1$I.banks b ON b.bank_id = m.bank_id
-                        WHERE m.consolidated_at IS NULL
-                          AND m.consolidation_failed_at IS NULL
-                          AND m.fact_type IN ('experience', 'world')
-                          AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
-                          AND NOT EXISTS (
-                              SELECT 1 FROM %1$I.async_operations o
-                              WHERE o.bank_id = m.bank_id
-                                AND o.operation_type = 'consolidation'
-                                AND o.status IN ('pending', 'processing')
-                          )
-                        GROUP BY m.bank_id
-                    $q$, sch);
-                EXCEPTION
-                    -- Schema or its tables vanished between the pg_class
-                    -- snapshot and this query (tenant dropped or migrating).
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-    op.execute(
-        f"""
-        CREATE OR REPLACE FUNCTION {schema}schemas_with_expired_rows(
-            p_table text, p_ts_col text, p_days int
-        )
-        RETURNS SETOF text
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-            has_expired boolean;
-        BEGIN
-            IF p_days IS NULL OR p_days <= 0 THEN
-                RETURN;
-            END IF;
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = p_table AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    EXECUTE format(
-                        'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
-                        sch, p_table, p_ts_col
-                    ) INTO has_expired USING p_days;
-                EXCEPTION
-                    -- Schema or its table vanished mid-scan; skip it.
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-                IF has_expired THEN
-                    RETURN NEXT sch;
-                END IF;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-    op.execute(
-        f"""
-        CREATE OR REPLACE FUNCTION {schema}mental_models_with_cron()
-        RETURNS TABLE(schema_name text, bank_id text, mental_model_id text,
-                     refresh_cron text, last_refreshed_at timestamptz)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'mental_models' AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    RETURN QUERY EXECUTE format($q$
-                        SELECT %1$L::text, mm.bank_id::text, mm.id::text,
-                               mm.trigger->>'refresh_cron', mm.last_refreshed_at
-                        FROM %1$I.mental_models mm
-                        WHERE COALESCE(mm.trigger->>'refresh_cron', '') <> ''
-                          AND NOT EXISTS (
-                              SELECT 1 FROM %1$I.async_operations o
-                              WHERE o.bank_id = mm.bank_id
-                                AND o.operation_type = 'refresh_mental_model'
-                                AND o.status IN ('pending', 'processing')
-                                AND o.task_payload->>'mental_model_id' = mm.id::text
-                          )
-                    $q$, sch);
-                EXCEPTION
-                    -- Schema or its tables vanished between the pg_class
-                    -- snapshot and this query (tenant dropped or migrating).
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _drop_routines(schema: str | None) -> None:
-    prefix = _prefix(schema)
-    op.execute(f"DROP FUNCTION IF EXISTS {prefix}mental_models_with_cron()")
-    op.execute(f"DROP FUNCTION IF EXISTS {prefix}schemas_with_expired_rows(text, text, int)")
-    op.execute(f"DROP FUNCTION IF EXISTS {prefix}banks_needing_consolidation()")
-
-
-def _drop_stray_copies() -> None:
-    """Remove per-tenant duplicates left by the first cut of this migration.
-
-    That version installed a copy into every schema it touched, so a database
-    that ran it carries one dead duplicate per tenant — only the copy in the
-    configured schema is ever called. Dropping here means the next migration pass
-    cleans them up; without it they would persist for the life of the database.
-
-    Safe on a database that never had them: ``DROP FUNCTION IF EXISTS`` is a
-    no-op, and this branch never runs for the configured schema.
-    """
-    _drop_routines(_target_schema())
-
-
-def _pg_downgrade() -> None:
-    # Only drop what this migration uniquely owns. When the configured schema is
-    # ``public`` the copies there belong to e5f6a7b8c9d0 / f4d1c2b3a5e6, which are
-    # still applied at this point and drop them on their own downgrade — removing
-    # them here would strand those migrations without the functions they claim to
-    # have installed.
-    if not _is_install_run() or _configured_schema() == "public":
-        return
-    _drop_routines(_target_schema())
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c3e5a7b9d1f4_widen_history_bank_id_to_text.py

@@ -1,75 +0,0 @@
-"""Repair: widen ``*_history.bank_id`` from VARCHAR(64) to TEXT on PostgreSQL.
-
-The original split-history migration (``a7b8c9d0e1f2``) declared
-``observation_history.bank_id`` and ``mental_model_history.bank_id`` as
-``VARCHAR(64)`` on PostgreSQL. But ``memory_units.bank_id`` — the backfill
-source for observations — is ``TEXT`` (unbounded), as are ``banks``,
-``documents`` and ``entities``. Any deployment whose ``bank_id`` exceeds 64
-characters aborts the backfill ``INSERT`` with::
-
-    psycopg2.errors.StringDataRightTruncation: value too long for type
-    character varying(64)
-
-Because the migration runs in ``lifespan`` startup inside a transaction, the
-whole migration rolls back and the API never comes up — unrecoverable from the
-running container. See https://github.com/vectorize-io/hindsight/issues/2106.
-
-``a7b8c9d0e1f2`` itself has been corrected to create the column as ``TEXT``,
-which unblocks deployments that *failed* (the migration rolled back, so it
-re-runs the fixed DDL). This forward migration covers deployments that already
-*succeeded* with the narrow ``VARCHAR(64)`` column — where editing
-``a7b8c9d0e1f2`` has no effect because it will not re-run — by widening the
-column in place. ``ALTER COLUMN ... TYPE TEXT`` is a no-op on a column that is
-already ``TEXT`` (fresh installs and re-run failures), so every upgrade path
-converges on ``TEXT``.
-
-The history tables are per-tenant (they live in each tenant schema, not
-``public``), so this runs for every migrated schema via the search-path-aware
-prefix — unlike the shared-``public`` routines repaired in ``b2d4f6a8c1e3``.
-
-PostgreSQL only. On Oracle both ``memory_units.bank_id`` and the history
-``bank_id`` columns are already ``VARCHAR2(256)`` (consistent, never
-truncates), so the Oracle slot is intentionally absent.
-
-Revision ID: c3e5a7b9d1f4
-Revises: c9a1b2d3e4f5
-Create Date: 2026-06-10
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "c3e5a7b9d1f4"
-down_revision: str | Sequence[str] | None = "c9a1b2d3e4f5"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}observation_history ALTER COLUMN bank_id TYPE TEXT")
-    op.execute(f"ALTER TABLE {schema}mental_model_history ALTER COLUMN bank_id TYPE TEXT")
-
-
-def _pg_downgrade() -> None:
-    # No-op: narrowing back to VARCHAR(64) could truncate real data and would
-    # re-introduce the bug this migration repairs. The column type is owned by
-    # ``a7b8c9d0e1f2``'s lifecycle.
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c3f7a1b9d2e4_backfill_observation_search_vector.py

@@ -1,144 +0,0 @@
-"""Backfill search_vector for native-backend observations.
-
-Observations created or updated by the consolidator landed with a NULL
-``search_vector`` under the ``native`` text-search backend: the
-single-row INSERT/UPDATE paths in ``consolidator.py`` never populated the
-tsvector (only the batch raw-fact path in ``ops_postgresql.insert_facts_batch``
-did). Those observations were therefore invisible to the BM25 retrieval arm
-until they were re-written by a later consolidation pass. The writer is fixed
-in the same change set (all four consolidator sites now call
-``to_tsvector($lang, COALESCE(text, ''))``); this migration repairs the
-historical residue so existing observations become BM25-searchable without a
-re-ingest.
-
-Scope mirrors the writer fix exactly:
-  * Only the ``native`` backend is touched. The gate is the column *type*:
-    under ``native`` ``search_vector`` is a regular (non-generated) tsvector
-    column; under ``vchord`` it is a ``bm25vector`` and under
-    ``pg_textsearch`` / ``pgroonga`` / ``pg_search`` it is a dummy ``text``
-    column. ``_is_regular_tsvector`` is true only for ``native``, so every
-    other backend is a no-op.
-  * The tsvector is built from the observation's own ``text`` only — matching
-    the consolidator INSERT/UPDATE paths (entity / source / temporal signals
-    are intentionally excluded; the other retrieval arms cover those).
-  * Only ``fact_type = 'observation'`` rows with a NULL ``search_vector`` are
-    rewritten. Raw facts already carry a populated tsvector, and the
-    ``IS NULL`` predicate makes the migration idempotent and re-runnable.
-
-The configured ``HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE`` is used
-so backfilled rows are lexically identical to newly-created observations. The
-value is validated as a PG identifier (mirroring
-``HindsightConfig.validate``) before being embedded as a SQL literal.
-
-This is a single UPDATE per schema: it locks the targeted observation rows for
-its duration. It is one-time and only touches unpopulated rows, so subsequent
-online writes (which now carry the tsvector via the writer fix) are unaffected.
-
-Oracle slot is intentionally absent: the consolidator INSERT/UPDATE paths that
-this repairs are PostgreSQL-specific (``ops_postgresql``), and the native
-tsvector ``search_vector`` column only exists on PostgreSQL. There is no Oracle
-residue to repair.
-
-Revision ID: c3f7a1b9d2e4
-Revises: f4d1c2b3a5e6
-Create Date: 2026-06-29
-"""
-
-import os
-import re
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import Connection, text
-
-from hindsight_api.alembic._dialect import run_for_dialect
-from hindsight_api.config import (
-    DEFAULT_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
-    ENV_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
-)
-
-revision: str = "c3f7a1b9d2e4"
-down_revision: str | Sequence[str] | None = "f4d1c2b3a5e6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-# Matches HindsightConfig.validate(): a tsvector regconfig name embedded as a
-# SQL literal must be a bare PG identifier.
-_PG_IDENTIFIER = re.compile(r"[a-zA-Z_][a-zA-Z0-9_]*")
-
-
-def _schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _schema_name() -> str:
-    return (context.config.get_main_option("target_schema") or "public").strip('"')
-
-
-def _native_language() -> str:
-    """Configured native tsvector language, validated as a PG identifier."""
-    lang = os.getenv(
-        ENV_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
-        DEFAULT_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
-    )
-    if not _PG_IDENTIFIER.fullmatch(lang):
-        return DEFAULT_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE
-    return lang
-
-
-def _is_regular_tsvector(conn: Connection, schema: str, table: str) -> bool:
-    """True iff ``schema.table.search_vector`` is a non-generated tsvector column.
-
-    This is the ``native`` backend signature. ``vchord`` (bm25vector) and
-    ``pg_textsearch`` / ``pgroonga`` / ``pg_search`` (dummy text column) all
-    fail this check, so the backfill is a no-op for them.
-    """
-    row = conn.execute(
-        text(
-            """
-            SELECT is_generated, udt_name
-            FROM information_schema.columns
-            WHERE table_schema = :schema
-              AND table_name = :table
-              AND column_name = 'search_vector'
-            """
-        ),
-        {"schema": schema, "table": table},
-    ).fetchone()
-    if not row:
-        return False
-    is_generated, udt_name = row[0], row[1]
-    return udt_name == "tsvector" and is_generated != "ALWAYS"
-
-
-def _pg_upgrade() -> None:
-    conn = op.get_bind()
-    schema_name = _schema_name()
-    if not _is_regular_tsvector(conn, schema_name, "memory_units"):
-        # Non-native backend (or column absent) — nothing to backfill.
-        return
-    schema_prefix = _schema_prefix()
-    lang = _native_language()
-    op.execute(
-        f"""
-        UPDATE {schema_prefix}memory_units
-        SET search_vector = to_tsvector('{lang}'::regconfig, COALESCE(text, ''))
-        WHERE fact_type = 'observation' AND search_vector IS NULL
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    # No-op: backfilled rows are indistinguishable from observations that were
-    # populated by the post-fix writer, and reverting either to NULL would
-    # re-break BM25 retrieval. The column simply stays populated.
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c7d1e9a4b3f2_add_archive_causal_links.py

@@ -1,90 +0,0 @@
-"""Add ``causal_links`` to the curation archive (invalidated_memory_units).
-
-Causal edges (``caused_by`` and the historical ``causes``/``enables``/
-``prevents``) are retain-time extraction output: unlike temporal and semantic
-links they cannot be recomputed from dates or embeddings, and graph maintenance
-never rebuilds them. Invalidation MOVES a fact out of ``memory_units``, so the
-``memory_links → memory_units`` FK cascade deletes every incident edge — and
-revert had no way to bring the causal ones back (#2864).
-
-This column parks the descriptors of the causal edges incident to an archived
-fact — ``[{"from_unit_id", "to_unit_id", "link_type", "weight"}, ...]`` — so
-revert can rematerialize them. It is deliberately unindexed and lives only on
-the archive: live facts keep their causal edges in ``memory_links`` (curation
-edits no longer delete them), and the archive is small, cold, and only read by
-low-frequency curation operations.
-
-Revision ID: c7d1e9a4b3f2
-Revises: d7b2f8a1c934
-Create Date: 2026-07-24
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "c7d1e9a4b3f2"
-down_revision: str | Sequence[str] | None = "d7b2f8a1c934"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    # NOT NULL DEFAULT is metadata-only on PG 11+, so this is cheap even on a
-    # large archive. Existing rows read as "no causal edges captured" — edges
-    # lost before this migration cannot be reconstructed and are not guessed.
-    op.execute(
-        f"ALTER TABLE {schema}invalidated_memory_units "
-        f"ADD COLUMN IF NOT EXISTS causal_links JSONB NOT NULL DEFAULT '[]'::jsonb"
-    )
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS causal_links")
-
-
-def _oracle_upgrade() -> None:
-    # Kept in sync with PG for schema parity (curation itself is PostgreSQL-only
-    # today — it introspects pg_attribute to move rows between the two tables).
-    # Swallow ORA-01430 (column already exists) so the migration is idempotent.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units ADD (causal_links CLOB DEFAULT ''[]''
-                CONSTRAINT imu_causal_links_json CHECK (causal_links IS JSON))';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -1430 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def _oracle_downgrade() -> None:
-    # Swallow ORA-00904 (column does not exist).
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units DROP COLUMN causal_links';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -904 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c7e9f1a3b5d2_maintenance_routines_skip_vanished_schemas.py

@@ -1,158 +0,0 @@
-"""Make maintenance routines resilient to schemas that vanish mid-scan.
-
-``public.banks_needing_consolidation()`` and
-``public.schemas_with_expired_rows(...)`` snapshot the set of schemas owning a
-target table from ``pg_class`` and then run a dynamic query against each schema
-in turn. That is a time-of-check/time-of-use race: a schema (or its tables) can
-be dropped — a tenant being deleted, or a tenant migration that recreates
-tables — between the snapshot and the per-schema query, which then aborts the
-whole routine with::
-
-    relation "<schema>.memory_units" does not exist
-    relation "<schema>.audit_log" does not exist
-
-In the test suite this surfaces as cross-worker contamination: the multi-tenant
-maintenance test creates and drops ~100 ``mt<hash>_NNN`` schemas while
-``test_maintenance_routines`` (on another xdist worker, same DB) calls the
-routines. In production the background maintenance loop hits the same race when
-a tenant is removed or mid-migration.
-
-Wrap each per-schema query in its own ``BEGIN ... EXCEPTION`` block so a schema
-that disappears (``undefined_table`` / ``invalid_schema_name`` /
-``undefined_column``) is skipped instead of aborting the scan. The routines stay
-``CREATE OR REPLACE`` and PostgreSQL-only, and are (re)installed only on the run
-that targets the shared ``public`` schema — same gating as the original
-install (``e5f6a7b8c9d0``) and its repair (``b2d4f6a8c1e3``).
-
-Revision ID: c7e9f1a3b5d2
-Revises: e1f2a3b4c5d6
-Create Date: 2026-06-19
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "c7e9f1a3b5d2"
-down_revision: str | Sequence[str] | None = "e1f2a3b4c5d6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _should_install_public_routines(target_schema: str | None) -> bool:
-    """True for the run that must (re)create the shared ``public.*`` routines.
-
-    The routines physically live in ``public``, so they are installed exactly
-    once — on the base run (no ``target_schema``) or the run that explicitly
-    targets ``public``. Mirrors ``b2d4f6a8c1e3``.
-    """
-    return not target_schema or target_schema == "public"
-
-
-def _pg_upgrade() -> None:
-    if not _should_install_public_routines(context.config.get_main_option("target_schema")):
-        return
-
-    # Same body as b2d4f6a8c1e3, but each per-schema query runs in its own
-    # subtransaction so a schema dropped mid-scan is skipped, not fatal.
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.banks_needing_consolidation()
-        RETURNS TABLE(schema_name text, bank_id text)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    RETURN QUERY EXECUTE format($q$
-                        SELECT %1$L::text, m.bank_id
-                        FROM %1$I.memory_units m
-                        JOIN %1$I.banks b ON b.bank_id = m.bank_id
-                        WHERE m.consolidated_at IS NULL
-                          AND m.consolidation_failed_at IS NULL
-                          AND m.fact_type IN ('experience', 'world')
-                          AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
-                          AND NOT EXISTS (
-                              SELECT 1 FROM %1$I.async_operations o
-                              WHERE o.bank_id = m.bank_id
-                                AND o.operation_type = 'consolidation'
-                                AND o.status IN ('pending', 'processing')
-                          )
-                        GROUP BY m.bank_id
-                    $q$, sch);
-                EXCEPTION
-                    -- Schema or its tables vanished between the pg_class
-                    -- snapshot and this query (tenant dropped or migrating).
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.schemas_with_expired_rows(
-            p_table text, p_ts_col text, p_days int
-        )
-        RETURNS SETOF text
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-            has_expired boolean;
-        BEGIN
-            IF p_days IS NULL OR p_days <= 0 THEN
-                RETURN;
-            END IF;
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = p_table AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    EXECUTE format(
-                        'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
-                        sch, p_table, p_ts_col
-                    ) INTO has_expired USING p_days;
-                EXCEPTION
-                    -- Schema or its table vanished mid-scan; skip it.
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-                IF has_expired THEN
-                    RETURN NEXT sch;
-                END IF;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    # No-op: e5f6a7b8c9d0 owns these functions' lifecycle and drops them on its
-    # own downgrade. This migration only re-installs them (the resilient body is
-    # a strict superset of the previous behaviour), so there is nothing to undo
-    # without racing that migration's DROP.
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/c9a1b2d3e4f5_add_invalidated_memory_units.py

@@ -1,108 +0,0 @@
-"""Add invalidated_memory_units table for curation (edit/invalidate).
-
-Curation keeps the recall hot-path (``memory_units``) clean by *moving*
-invalidated facts into a sibling archive table rather than flagging them in
-place. If a row is in ``memory_units`` it is live; if it is in
-``invalidated_memory_units`` it has been retired. Recall/consolidation/graph
-queries never need a state predicate — the rows simply aren't there.
-
-The archive mirrors ``memory_units`` column-for-column — except ``embedding``,
-which it never keeps: the archive is cold storage, never a recall surface, and
-revert recomputes the embedding from the unit's text/dates/entities. Keeping no
-archive vector also means a later embedding-model switch (which re-dimensions
-``memory_units``) can't trip a dimension mismatch on the move (#2209). Plus:
-- ``invalidation_reason``  optional free text recorded on invalidate
-- ``invalidated_at``       when it was retired
-- ``entity_ids``           snapshot of the unit's entity associations, so revert
-                           can restore them (``unit_entities`` is cascade-deleted
-                           when the live row is removed)
-
-This migration also adds ``edited_at`` to ``memory_units``: set whenever a user
-edits a memory's fields (text, context, dates, fact_type, entities) via curation.
-NULL means never manually modified; a non-NULL value answers "has the user ever
-changed this?" with the time of the last edit (distinct from ``updated_at``,
-which background operations also bump). It is added to ``memory_units`` *before*
-the archive is cloned below, so the archive inherits the column and the marker
-travels with a fact when it is invalidated.
-
-Revision ID: c9a1b2d3e4f5
-Revises: b2d4f6a8c1e3
-Create Date: 2026-06-03
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "c9a1b2d3e4f5"
-down_revision: str | Sequence[str] | None = "b2d4f6a8c1e3"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    # Add edited_at to the live table FIRST so the archive's LIKE clone below
-    # inherits it (keeps the two tables column-for-column identical for round-trip).
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS edited_at TIMESTAMPTZ")
-    # LIKE ... INCLUDING DEFAULTS clones every memory_units column (incl.
-    # edited_at) so an invalidated row can move back verbatim. We deliberately
-    # omit indexes/constraints — the archive is cold storage, not a recall
-    # surface; only the lookups below need indexing.
-    op.execute(
-        f"CREATE TABLE IF NOT EXISTS {schema}invalidated_memory_units (LIKE {schema}memory_units INCLUDING DEFAULTS)"
-    )
-    # ...then drop the inherited embedding: the archive never stores one (revert
-    # recomputes it), so it isn't created here only to be dropped again later by
-    # d4f6a8c2e1b3. That migration still runs as a no-op (DROP ... IF EXISTS) on
-    # fresh DBs and does the real drop on DBs created before this column was removed.
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS embedding")
-    op.execute(
-        f"ALTER TABLE {schema}invalidated_memory_units "
-        f"ADD COLUMN IF NOT EXISTS invalidation_reason TEXT, "
-        f"ADD COLUMN IF NOT EXISTS invalidated_at TIMESTAMPTZ DEFAULT now(), "
-        f"ADD COLUMN IF NOT EXISTS entity_ids UUID[]"
-    )
-    op.execute(f"CREATE UNIQUE INDEX IF NOT EXISTS idx_invalidated_mu_id ON {schema}invalidated_memory_units (id)")
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_invalidated_mu_bank "
-        f"ON {schema}invalidated_memory_units (bank_id, invalidated_at)"
-    )
-    # Deleting a document (or bank) should clear its archived facts too, mirroring
-    # the memory_units → documents cascade.
-    op.execute(
-        f"""
-        DO $$ BEGIN
-        IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = 'invalidated_mu_document_fkey') THEN
-            ALTER TABLE {schema}invalidated_memory_units
-                ADD CONSTRAINT invalidated_mu_document_fkey
-                FOREIGN KEY (document_id, bank_id)
-                REFERENCES {schema}documents(id, bank_id) ON DELETE CASCADE;
-        END IF; END $$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    # Drops the archive (and its inherited edited_at) wholesale, then removes
-    # edited_at from the live table.
-    op.execute(f"DROP TABLE IF EXISTS {schema}invalidated_memory_units")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS edited_at")
-
-
-def upgrade() -> None:
-    # PG-only: Oracle gets the table from the baseline snapshot, matching the
-    # convention used by sibling column/index migrations in this tree.
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/d3e4f5a6b7c8_add_llm_requests_table.py

@@ -1,96 +0,0 @@
-"""Add llm_requests table for per-bank LLM request tracing.
-
-Stores one row per logical LLM call Hindsight makes (success and failure),
-capturing the input messages, model output, token usage (input/output/cached/
-total), finish reason, and caller metadata. Disabled by default at the
-application layer (HINDSIGHT_API_LLM_TRACE_ENABLED); this migration only
-creates the table.
-
-PostgreSQL only — the tracing subsystem is not wired for Oracle, so the Oracle
-slot is intentionally absent (mirrors the audit_log table).
-
-Revision ID: d3e4f5a6b7c8
-Revises: c1d2e3f4a5b6
-Create Date: 2026-06-01
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "d3e4f5a6b7c8"
-down_revision: str | Sequence[str] | None = "c1d2e3f4a5b6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}llm_requests (
-            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-            bank_id TEXT,
-            operation TEXT,
-            scope TEXT,
-            -- OTel-style grouping: trace_id is shared by every LLM call of one
-            -- operation invocation (e.g. all calls of a single reflect run);
-            -- parent_span_id is that operation span; span_id is this call.
-            trace_id TEXT,
-            span_id TEXT,
-            parent_span_id TEXT,
-            provider TEXT,
-            model TEXT,
-            status TEXT NOT NULL,
-            started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-            ended_at TIMESTAMPTZ,
-            duration_ms INTEGER,
-            input_tokens INTEGER,
-            output_tokens INTEGER,
-            cached_tokens INTEGER,
-            total_tokens INTEGER,
-            input JSONB,
-            output JSONB,
-            error TEXT,
-            llm_info JSONB DEFAULT '{{}}'::jsonb,
-            metadata JSONB DEFAULT '{{}}'::jsonb
-        )
-        """
-    )
-
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_bank_started ON {schema}llm_requests (bank_id, started_at DESC)"
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_status_started ON {schema}llm_requests (status, started_at DESC)"
-    )
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_llm_requests_started ON {schema}llm_requests (started_at DESC)")
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_llm_requests_trace ON {schema}llm_requests (bank_id, trace_id, started_at)"
-    )
-
-
-def _pg_downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_status_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_llm_requests_bank_started")
-    op.execute(f"DROP TABLE IF EXISTS {schema}llm_requests")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/d4f6a8c2e1b3_drop_archive_embedding_column.py

@@ -1,93 +0,0 @@
-"""Drop the embedding column from the curation archive (invalidated_memory_units).
-
-The archive is cold storage, never a recall surface, so it has no business
-keeping an embedding. Earlier curation code copied the live row's embedding into
-``invalidated_memory_units`` on invalidate; the engine now leaves it out on
-invalidate and recomputes it on revert, so the column is dead weight.
-
-Dropping it makes "the archive holds no embedding" a schema-enforced invariant
-rather than a convention the move queries have to honour, and removes a latent
-failure mode (#2209): after an embedding-model switch the live tables are
-re-dimensioned but the archive was not, so a stale old-dimension embedding in
-the archive tripped a vector-dimension mismatch on the INSERT … SELECT
-round-trip. With no column at all, there is nothing to mismatch.
-
-The creation sites no longer add the column (the PG ``LIKE`` clone in
-c9a1b2d3e4f5 drops it; the Oracle baseline omits it), so on a fresh database
-this migration is a no-op (DROP ... IF EXISTS / Oracle ORA-00904 swallow). It
-does the real work on databases created before the column was removed there.
-
-DROP COLUMN is a metadata-only operation on both PostgreSQL and Oracle 23ai (no
-table rewrite), so it is cheap even across many tenant schemas. The downgrade
-re-adds an unconstrained vector column (any dimension) — empty, since the
-embeddings are intentionally discarded.
-
-Revision ID: d4f6a8c2e1b3
-Revises: a1d3f5b7c9e2
-Create Date: 2026-06-15
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "d4f6a8c2e1b3"
-down_revision: str | Sequence[str] | None = "a1d3f5b7c9e2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS embedding")
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    # Unconstrained `vector` (no dimension) so the re-added column accepts any
-    # model's embeddings; it comes back empty regardless.
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units ADD COLUMN IF NOT EXISTS embedding vector")
-
-
-def _oracle_upgrade() -> None:
-    # Oracle has no `DROP COLUMN IF EXISTS`; swallow ORA-00904 (column does not
-    # exist) so the migration is idempotent and safe on a fresh schema whose
-    # baseline already omits the column.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units DROP COLUMN embedding';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -904 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def _oracle_downgrade() -> None:
-    # Swallow ORA-01430 (column already exists) for idempotency.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units ADD (embedding VECTOR)';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -1430 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/d7b2f8a1c934_add_schemas_with_expired_operations_routine.py

@@ -1,150 +0,0 @@
-"""Add the ``schemas_with_expired_operations`` cross-tenant discovery routine.
-
-The worker's terminal-operation cleanup (``a8c1e4f7b0d3``) opens a connection
-and a prune transaction against *every* tenant schema on every cleanup cycle,
-whether or not that tenant has anything to prune. At thousands of tenants that
-is a per-cycle query storm whose cost is paid entirely by idle schemas.
-
-This is the same problem ``public.schemas_with_expired_rows`` already solves for
-the ``audit_log`` / ``llm_requests`` retention sweeps (``e5f6a7b8c9d0``): one
-round-trip returns just the schemas that actually hold expired rows, and the
-caller then does real work only there. ``async_operations`` needs its own
-routine rather than reusing that one because eligibility is not "row older than
-N days" — pending and processing rows are never prunable, so the status filter
-has to be part of the predicate.
-
-Install policy mirrors ``b6d2f8a4c1e7`` (#2638/#2824), the current behaviour for
-the sibling routines: the routine is database-global — it enumerates ``pg_class``
-across every schema and dispatches per schema — so exactly one copy should exist,
-installed into the schema this deployment is *configured* to use and called from
-there via ``fq_routine``. Gating on the literal ``"public"`` instead of the
-configured schema is what left single-tenant deployments in a dedicated
-non-``public`` schema without the routine (#2638).
-
-Exactly one migration run satisfies that predicate, so concurrent per-schema runs
-never issue competing ``CREATE OR REPLACE`` against the same ``pg_proc`` row and
-cannot hit ``tuple concurrently updated``. No cross-process coordination is
-required — in particular no advisory lock, which is unusable here because
-Hindsight runs behind connection poolers and managed PG services (see #2817).
-
-Each per-schema probe runs in its own ``BEGIN ... EXCEPTION`` block so a tenant
-dropped mid-scan is skipped instead of aborting the sweep (see ``c7e9f1a3b5d2``).
-
-Revision ID: d7b2f8a1c934
-Revises: b6d2f8a4c1e7
-Create Date: 2026-07-20
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-from hindsight_api.config import get_config
-
-revision: str = "d7b2f8a1c934"
-down_revision: str | Sequence[str] | None = "b6d2f8a4c1e7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _configured_schema() -> str:
-    """The one schema this deployment's routines live in and are called from."""
-    return get_config().database_schema or "public"
-
-
-def _target_schema() -> str | None:
-    return context.config.get_main_option("target_schema")
-
-
-def _is_install_run() -> bool:
-    """True for the single run that owns the routine (mirrors b6d2f8a4c1e7)."""
-    target = _target_schema()
-    return not target or target == _configured_schema()
-
-
-def _prefix(schema: str | None) -> str:
-    """Qualifier for ``schema``, or ``""`` to fall back to ``search_path``."""
-    return f'"{schema}".' if schema else ""
-
-
-def _drop_routine(schema: str | None) -> None:
-    op.execute(f"DROP FUNCTION IF EXISTS {_prefix(schema)}schemas_with_expired_operations(int)")
-
-
-def _pg_upgrade() -> None:
-    if not _is_install_run():
-        # Tenant schemas must not carry their own copy: the routine is
-        # database-global and only the configured schema's copy is ever called.
-        # Dropping (rather than skipping) also cleans up after any interim build
-        # of this branch that installed per-schema copies.
-        _drop_routine(_target_schema())
-        return
-    schema = _prefix(_target_schema())
-    op.execute(
-        f"""
-        CREATE OR REPLACE FUNCTION {schema}schemas_with_expired_operations(p_days int)
-        RETURNS SETOF text
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-            has_expired boolean;
-        BEGIN
-            -- Zero (or negative) retention means "keep forever": report nothing
-            -- so the caller skips the sweep entirely.
-            IF p_days IS NULL OR p_days <= 0 THEN
-                RETURN;
-            END IF;
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'async_operations' AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    -- Matches the worker's prune predicate: only terminal rows
-                    -- are eligible, so a schema holding nothing but pending or
-                    -- processing work is correctly reported as having nothing
-                    -- to prune. Uses idx_async_operations_terminal_cleanup.
-                    EXECUTE format(
-                        'SELECT EXISTS ('
-                        '  SELECT 1 FROM %I.async_operations'
-                        '  WHERE status IN (''completed'', ''failed'', ''cancelled'')'
-                        '    AND updated_at < NOW() - make_interval(days => $1)'
-                        ')',
-                        sch
-                    ) INTO has_expired USING p_days;
-                EXCEPTION
-                    -- Schema or its table vanished between the pg_class
-                    -- snapshot and this probe (tenant dropped or migrating).
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-                IF has_expired THEN
-                    RETURN NEXT sch;
-                END IF;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    # This migration is the sole creator of this routine — no older migration
-    # owns a copy the way e5f6a7b8c9d0 owns the public sibling routines — so the
-    # install run's own copy is always ours to drop.
-    if not _is_install_run():
-        return
-    _drop_routine(_target_schema())
-
-
-def upgrade() -> None:
-    # Oracle slot intentionally absent: this mirrors the PostgreSQL-only
-    # maintenance routines, and the Oracle worker keeps its per-schema sweep.
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/e1f2a3b4c5d6_merge_heads_embedding_drop_and_links_index.py

@@ -1,39 +0,0 @@
-"""Merge two divergent migration heads.
-
-``d4f6a8c2e1b3`` (drop the curation-archive embedding column) and
-``2071c7518f88`` (add the memory_links(bank_id, link_type) index) were authored
-in parallel off the same parent (``a1d3f5b7c9e2``) and merged independently,
-leaving the DAG with two heads. This is a no-op merge that re-unifies them so
-``alembic upgrade head`` is unambiguous again (enforced by
-``tests/test_alembic_dag.py::test_single_head``).
-
-Revision ID: e1f2a3b4c5d6
-Revises: d4f6a8c2e1b3, 2071c7518f88
-Create Date: 2026-06-16
-"""
-
-from collections.abc import Sequence
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "e1f2a3b4c5d6"
-down_revision: str | Sequence[str] | None = ("d4f6a8c2e1b3", "2071c7518f88")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_upgrade() -> None:
-    # Pure DAG merge — both parents already applied their schema changes.
-    pass
-
-
-def _pg_downgrade() -> None:
-    pass
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/e4a7c1b9d2f6_drop_memory_units_access_count.py

@@ -1,114 +0,0 @@
-"""Drop the never-written `access_count` column from memory_units (and its archive).
-
-``memory_units.access_count`` has been dead since the initial schema
-(5a366d414dce): no code path anywhere in the repo ever writes it, and — despite
-the ``access_count DESC`` index created alongside it — no query ever reads or
-orders by it either. It is 0 on every row of every install. The lone remaining
-mentions were an index, a stale comment naming an ``access_count_update`` task
-type that was never implemented, and the column's name in the Oracle backend's
-numeric-RETURNING list; all three go away with this change.
-
-The column is dropped from the curation archive too. ``invalidated_memory_units``
-was cloned ``LIKE memory_units`` (c9a1b2d3e4f5), so it inherited the column, and
-curation's INSERT…SELECT round-trip builds its column list from the catalog
-(``writes.py::_memory_unit_columns``) — the two tables must stay in lockstep or
-the round-trip breaks on a column-count mismatch.
-
-Dropping the column implicitly drops its index on both dialects
-(``idx_memory_units_access_count`` on PG, ``idx_mu_access_count`` on Oracle), so
-PostgreSQL also stops maintaining a btree that nothing ever probed.
-
-Cost: on PostgreSQL ``DROP COLUMN`` is metadata-only (the attribute is marked
-dropped, no table rewrite). On Oracle it does delete the column data row by row,
-so on a large ``memory_units`` this migration is not free — it is still bounded
-work on a single small integer column, and Oracle installs of that size can run
-it during a maintenance window ahead of the upgrade if they prefer.
-
-Revision ID: e4a7c1b9d2f6
-Revises: a9b8c7d6e5f4
-Create Date: 2026-08-03
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "e4a7c1b9d2f6"
-down_revision: str | Sequence[str] | None = "a9b8c7d6e5f4"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_TABLES = ("memory_units", "invalidated_memory_units")
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    for table in _TABLES:
-        # Drops idx_memory_units_access_count along with the column.
-        op.execute(f"ALTER TABLE {schema}{table} DROP COLUMN IF EXISTS access_count")
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    for table in _TABLES:
-        op.execute(f"ALTER TABLE {schema}{table} ADD COLUMN IF NOT EXISTS access_count integer NOT NULL DEFAULT 0")
-    # The archive was cloned without indexes; only the live table carried one.
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_memory_units_access_count ON {schema}memory_units (access_count DESC)")
-
-
-def _oracle_upgrade() -> None:
-    # Oracle has no `DROP COLUMN IF EXISTS`; swallow ORA-00904 (column does not
-    # exist) so the migration is idempotent and safe on a schema that already
-    # lacks the column. Dropping the column also drops idx_mu_access_count.
-    for table in _TABLES:
-        op.execute(
-            f"""
-            BEGIN
-                EXECUTE IMMEDIATE 'ALTER TABLE {table} DROP COLUMN access_count';
-            EXCEPTION WHEN OTHERS THEN
-                IF SQLCODE != -904 THEN RAISE; END IF;
-            END;
-            """
-        )
-
-
-def _oracle_downgrade() -> None:
-    # Swallow ORA-01430 (column already exists) for idempotency. Matches the
-    # Oracle baseline's declaration: NUMBER(10) DEFAULT 0 NOT NULL.
-    for table in _TABLES:
-        op.execute(
-            f"""
-            BEGIN
-                EXECUTE IMMEDIATE
-                    'ALTER TABLE {table} ADD (access_count NUMBER(10) DEFAULT 0 NOT NULL)';
-            EXCEPTION WHEN OTHERS THEN
-                IF SQLCODE != -1430 THEN RAISE; END IF;
-            END;
-            """
-        )
-    # ORA-00955: index name already in use.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'CREATE INDEX idx_mu_access_count ON memory_units(access_count DESC)';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -955 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/e5f6a7b8c9d0_add_maintenance_routines.py

@@ -1,153 +0,0 @@
-"""Add server-side routines for background maintenance sweeps.
-
-Installs two PL/pgSQL discovery routines in the ``public`` schema. Both loop
-over every schema that actually holds the relevant table (via ``pg_class``), so
-a single function call covers all tenants in one round-trip instead of the
-per-tenant query storm that a client-side loop would create at thousands of
-tenants.
-
-- ``public.banks_needing_consolidation()`` -> (schema_name, bank_id) for banks
-  that have eligible-but-unscheduled facts (``consolidated_at IS NULL AND
-  consolidation_failed_at IS NULL`` for consolidatable fact types), have
-  auto-consolidation not explicitly disabled at the bank level, and have no
-  consolidation operation already pending/processing. Drives the periodic
-  reconcile that re-schedules consolidation after a terminal failure left facts
-  stranded (see HINDSIGHT_API_CONSOLIDATION_RECONCILE_INTERVAL_SECONDS).
-
-- ``public.schemas_with_expired_rows(p_table, p_ts_col, p_days)`` -> schema
-  names that hold at least one ``p_table`` row older than ``p_days``. Drives the
-  cross-tenant retention sweeps for ``audit_log`` and ``llm_requests``; the loop
-  then issues a DELETE only against the returned schemas.
-
-These are read-only (STABLE) discovery routines — the caller performs the
-enqueue/DELETE — so installing them never mutates data.
-
-PostgreSQL only — the worker poller and these tables are not wired for Oracle,
-so the Oracle slot is intentionally absent (mirrors the audit_log / llm_requests
-table migrations). The routines live in ``public`` and are CREATE OR REPLACE, so
-running this migration once per tenant schema is idempotent.
-
-Revision ID: e5f6a7b8c9d0
-Revises: a7b8c9d0e1f2
-Create Date: 2026-06-05
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "e5f6a7b8c9d0"
-down_revision: str | Sequence[str] | None = "a7b8c9d0e1f2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _is_base_schema_run() -> bool:
-    """True only for the base-schema migration (no per-tenant target_schema).
-
-    These routines live in the shared ``public`` schema, so they must be created
-    exactly once. Running ``CREATE OR REPLACE FUNCTION public....`` again from each
-    concurrent per-tenant migration aborts with ``tuple concurrently updated`` on
-    the ``pg_proc`` catalog row, so tenant runs skip it (the base run already
-    created the function for every tenant to use).
-    """
-    return not context.config.get_main_option("target_schema")
-
-
-def _pg_upgrade() -> None:
-    if not _is_base_schema_run():
-        return
-    # Banks with eligible-but-unscheduled facts and no in-flight consolidation.
-    # Auto-consolidation is filtered here only at the bank level (cheap prune);
-    # the full hierarchical resolution (global -> tenant -> bank, plus
-    # enable_observations) is done by the caller for the small returned set.
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.banks_needing_consolidation()
-        RETURNS TABLE(schema_name text, bank_id text)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'memory_units' AND c.relkind = 'r'
-            LOOP
-                RETURN QUERY EXECUTE format($q$
-                    SELECT %1$L::text, m.bank_id
-                    FROM %1$I.memory_units m
-                    JOIN %1$I.banks b ON b.bank_id = m.bank_id
-                    WHERE m.consolidated_at IS NULL
-                      AND m.consolidation_failed_at IS NULL
-                      AND m.fact_type IN ('experience', 'world')
-                      AND COALESCE(b.config -> 'enable_auto_consolidation', 'true'::jsonb) <> 'false'::jsonb
-                      AND NOT EXISTS (
-                          SELECT 1 FROM %1$I.async_operations o
-                          WHERE o.bank_id = m.bank_id
-                            AND o.operation_type = 'consolidation'
-                            AND o.status IN ('pending', 'processing')
-                      )
-                    GROUP BY m.bank_id
-                $q$, sch);
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-    # Schemas holding at least one row of p_table older than p_days. p_ts_col is
-    # the timestamp column to compare. Returns nothing when p_days <= 0
-    # (retention disabled).
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.schemas_with_expired_rows(
-            p_table text, p_ts_col text, p_days int
-        )
-        RETURNS SETOF text
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-            has_expired boolean;
-        BEGIN
-            IF p_days IS NULL OR p_days <= 0 THEN
-                RETURN;
-            END IF;
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = p_table AND c.relkind = 'r'
-            LOOP
-                EXECUTE format(
-                    'SELECT EXISTS (SELECT 1 FROM %I.%I WHERE %I < NOW() - make_interval(days => $1))',
-                    sch, p_table, p_ts_col
-                ) INTO has_expired USING p_days;
-                IF has_expired THEN
-                    RETURN NEXT sch;
-                END IF;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    if not _is_base_schema_run():
-        return
-    op.execute("DROP FUNCTION IF EXISTS public.banks_needing_consolidation()")
-    op.execute("DROP FUNCTION IF EXISTS public.schemas_with_expired_rows(text, text, int)")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/e7c3a9f1b2d5_drop_archive_search_vector_column.py

@@ -1,96 +0,0 @@
-"""Drop the search_vector column from the curation archive (invalidated_memory_units).
-
-The archive is cold storage, never a recall surface, and carries no text-search
-index. Like ``embedding`` (dropped in d4f6a8c2e1b3), ``search_vector`` is a
-recall-surface column whose type follows the configured text-search backend, so
-it has no business living on the archive. Earlier curation code copied the live
-row's ``search_vector`` into ``invalidated_memory_units`` on invalidate; the
-engine now leaves it out on invalidate and recomputes it on revert, so the
-column is dead weight.
-
-Dropping it removes a latent failure mode (#2503): under a non-native backend
-(pgroonga / pg_textsearch / pg_search / vchord) ``ensure_text_search_extension``
-reconciles ``memory_units.search_vector`` to ``text`` / ``bm25vector`` but never
-touched the archive, which the ``LIKE memory_units`` clone (c9a1b2d3e4f5) created
-as ``tsvector``. The type mismatch then broke the curation INSERT … SELECT
-round-trip:
-
-    column "search_vector" is of type tsvector but expression is of type text
-
-With no column at all, there is nothing to mismatch. Unlike ``embedding`` (whose
-creation sites already omit it), the ``LIKE`` clone still adds ``search_vector``,
-so this migration does real work on both fresh and existing PostgreSQL databases.
-
-DROP COLUMN is a metadata-only operation on both PostgreSQL and Oracle 23ai (no
-table rewrite), so it is cheap even across many tenant schemas. The downgrade
-re-adds an empty ``tsvector`` column (its original creation type).
-
-Revision ID: e7c3a9f1b2d5
-Revises: b57a7c9e0d13
-Create Date: 2026-07-02
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "e7c3a9f1b2d5"
-down_revision: str | Sequence[str] | None = "b57a7c9e0d13"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _pg_upgrade() -> None:
-    schema = _pg_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units DROP COLUMN IF EXISTS search_vector")
-
-
-def _pg_downgrade() -> None:
-    schema = _pg_schema_prefix()
-    # Re-add as the original tsvector creation type; comes back empty regardless.
-    op.execute(f"ALTER TABLE {schema}invalidated_memory_units ADD COLUMN IF NOT EXISTS search_vector tsvector")
-
-
-def _oracle_upgrade() -> None:
-    # Oracle has no `DROP COLUMN IF EXISTS`; swallow ORA-00904 (column does not
-    # exist) so the migration is idempotent and safe on a schema whose baseline
-    # may already omit the column.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units DROP COLUMN search_vector';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -904 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def _oracle_downgrade() -> None:
-    # Swallow ORA-01430 (column already exists) for idempotency. Oracle stores
-    # search_vector as CLOB (see the Oracle baseline), so re-add it as CLOB.
-    op.execute(
-        """
-        BEGIN
-            EXECUTE IMMEDIATE 'ALTER TABLE invalidated_memory_units ADD (search_vector CLOB)';
-        EXCEPTION WHEN OTHERS THEN
-            IF SQLCODE != -1430 THEN RAISE; END IF;
-        END;
-        """
-    )
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/f2a6d8c4b1e9_drop_stale_global_memory_units_vector_index.py

@@ -1,85 +0,0 @@
-"""Repair: drop the stale global memory_units vector index on per-bank backends.
-
-Revision ID: f2a6d8c4b1e9
-Revises: e4a7c1b9d2f6
-Create Date: 2026-08-06
-
-Migration d5e6f7a8b9c0 dropped the global ``idx_memory_units_embedding`` for
-per-bank backends (every vector search is bank + fact_type scoped and served
-by the ``idx_mu_emb_*`` partial indexes; the global index is never chosen by
-the planner). However, older versions of the post-migration reconcile
-(``ensure_vector_extension``) recreated the index when they found none, so
-schemas that were provisioned or reconciled in that window carry it to this
-day — paying a second vector graph insertion on every ``memory_units`` write
-for an index no query uses.
-
-This repair drops the leftover index. It is intentionally a migration, not
-runtime reconcile behavior: ``DROP INDEX`` takes an ACCESS EXCLUSIVE lock on
-``memory_units``, which belongs in the versioned, once-per-schema migration
-path — not in code that runs at unpredictable times during startup or tenant
-provisioning. The reconcile now leaves memory_units vector-index DDL to
-migrations entirely on per-bank backends.
-
-ScaNN deployments keep the global index by design (filtered vector search over
-a global index; per-bank partial indexes cannot be built safely there), so the
-migration is a no-op for them.
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "f2a6d8c4b1e9"
-down_revision: str | Sequence[str] | None = "e4a7c1b9d2f6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _pg_schema_prefix() -> str:
-    """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _configured_vector_extension() -> str:
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext not in {"pgvector", "pgvectorscale", "vchord", "scann"}:
-        raise ValueError(
-            f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {ext}. Must be 'pgvector', 'vchord', 'pgvectorscale', or 'scann'"
-        )
-    return ext
-
-
-def _pg_upgrade() -> None:
-    # ScaNN uses a global vector index by design — nothing stale to repair.
-    if _configured_vector_extension() == "scann":
-        return
-    schema = _pg_schema_prefix()
-    # DROP INDEX needs ACCESS EXCLUSIVE on memory_units. While it waits for
-    # in-flight transactions, every new query on the table queues behind it,
-    # so on a write-busy schema an unbounded wait can pile up traffic. Fail
-    # fast instead: the migration errors, the schema stays below head, and
-    # the next migration pass retries — preferable to freezing the table.
-    # SET LOCAL scopes the timeout to this migration's transaction.
-    op.execute("SET LOCAL lock_timeout = '10s'")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_embedding")
-
-
-def _pg_downgrade() -> None:
-    # Intentional no-op: recreating a potentially multi-GB vector index that no
-    # query uses is not a safe downgrade action. Downgrading past d5e6f7a8b9c0
-    # restores the global index for deployments that genuinely need it.
-    pass
-
-
-def upgrade() -> None:
-    # PG-only repair: the stale index is a PostgreSQL artifact of the old
-    # reconcile; Oracle deployments never had a reconcile that created it.
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/f4d1c2b3a5e6_add_scheduled_mental_model_refresh_routine.py

@@ -1,110 +0,0 @@
-"""Add server-side routine for cron-scheduled mental model refresh.
-
-Installs ``public.mental_models_with_cron()`` — a discovery routine that returns
-every mental model carrying a non-empty ``trigger->>'refresh_cron'`` across all
-tenant schemas in one round-trip (the same per-schema scan as the other
-maintenance routines from ``e5f6a7b8c9d0``). The maintenance loop evaluates each
-candidate's cron expression in Python (``croniter``) against ``last_refreshed_at``
-to decide whether a scheduled refresh is due — cron arithmetic isn't expressible
-in plain SQL — and only the cron *candidate set* is discovered here.
-
-Models that already have a ``refresh_mental_model`` operation pending/processing
-are excluded so a slow refresh isn't double-queued (mirrors the in-flight guard
-in ``banks_needing_consolidation``). Each per-schema query runs in its own
-``BEGIN ... EXCEPTION`` subtransaction so a schema dropped mid-scan (tenant
-deletion / migration) is skipped, not fatal — same resilience as
-``c7e9f1a3b5d2``.
-
-Read-only (STABLE) discovery routine — the caller performs the refresh enqueue —
-so installing it never mutates data. PostgreSQL only: the worker poller and the
-maintenance loop are PG-only (Oracle slot intentionally absent, mirroring
-``e5f6a7b8c9d0``). The routine lives in ``public`` and is CREATE OR REPLACE, so
-it is installed exactly once (base / ``public`` run) to avoid the
-``tuple concurrently updated`` race on concurrent per-tenant runs.
-
-Revision ID: f4d1c2b3a5e6
-Revises: c7e9f1a3b5d2
-Create Date: 2026-06-23
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-from hindsight_api.alembic._dialect import run_for_dialect
-
-revision: str = "f4d1c2b3a5e6"
-down_revision: str | Sequence[str] | None = "c7e9f1a3b5d2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _should_install_public_routines(target_schema: str | None) -> bool:
-    """True for the run that must (re)create the shared ``public.*`` routine.
-
-    The routine physically lives in ``public``, so it is installed exactly once —
-    on the base run (no ``target_schema``) or the run that explicitly targets
-    ``public``. Mirrors ``c7e9f1a3b5d2``.
-    """
-    return not target_schema or target_schema == "public"
-
-
-def _pg_upgrade() -> None:
-    if not _should_install_public_routines(context.config.get_main_option("target_schema")):
-        return
-
-    op.execute(
-        """
-        CREATE OR REPLACE FUNCTION public.mental_models_with_cron()
-        RETURNS TABLE(schema_name text, bank_id text, mental_model_id text,
-                     refresh_cron text, last_refreshed_at timestamptz)
-        LANGUAGE plpgsql STABLE
-        AS $fn$
-        DECLARE
-            sch text;
-        BEGIN
-            FOR sch IN
-                SELECT n.nspname
-                FROM pg_class c
-                JOIN pg_namespace n ON n.oid = c.relnamespace
-                WHERE c.relname = 'mental_models' AND c.relkind = 'r'
-            LOOP
-                BEGIN
-                    RETURN QUERY EXECUTE format($q$
-                        SELECT %1$L::text, mm.bank_id::text, mm.id::text,
-                               mm.trigger->>'refresh_cron', mm.last_refreshed_at
-                        FROM %1$I.mental_models mm
-                        WHERE COALESCE(mm.trigger->>'refresh_cron', '') <> ''
-                          AND NOT EXISTS (
-                              SELECT 1 FROM %1$I.async_operations o
-                              WHERE o.bank_id = mm.bank_id
-                                AND o.operation_type = 'refresh_mental_model'
-                                AND o.status IN ('pending', 'processing')
-                                AND o.task_payload->>'mental_model_id' = mm.id::text
-                          )
-                    $q$, sch);
-                EXCEPTION
-                    -- Schema or its tables vanished between the pg_class
-                    -- snapshot and this query (tenant dropped or migrating).
-                    WHEN undefined_table OR invalid_schema_name OR undefined_column THEN
-                        CONTINUE;
-                END;
-            END LOOP;
-        END;
-        $fn$;
-        """
-    )
-
-
-def _pg_downgrade() -> None:
-    if not _should_install_public_routines(context.config.get_main_option("target_schema")):
-        return
-    op.execute("DROP FUNCTION IF EXISTS public.mental_models_with_cron()")
-
-
-def upgrade() -> None:
-    run_for_dialect(pg=_pg_upgrade)
-
-
-def downgrade() -> None:
-    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/alembic/versions/o1a2b3c4d5e6_oracle_baseline.py

@@ -122,7 +122,6 @@ _TABLES: tuple[str, ...] = (
         text_signals      CLOB,
         consolidation_failed_at TIMESTAMP WITH TIME ZONE,
         search_vector     CLOB,
-        edited_at         TIMESTAMP WITH TIME ZONE,
         created_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
         updated_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
         CONSTRAINT pk_memory_units PRIMARY KEY (id),
@@ -139,50 +138,6 @@ _TABLES: tuple[str, ...] = (
     PARTITION BY LIST (bank_id) AUTOMATIC
     (PARTITION p_default VALUES ('__default__'))
     """,
-    # Cold archive for curation: invalidated facts are MOVED here out of
-    # memory_units so the recall hot-path never sees them. Mirrors memory_units
-    # plus invalidation bookkeeping and an entity-id snapshot for lossless revert.
-    # No `embedding` column: the archive is cold storage and revert recomputes the
-    # embedding, so there is no archive vector to fall out of sync with the live
-    # model's dimension on a model switch (#2209).
-    """
-    CREATE TABLE IF NOT EXISTS invalidated_memory_units (
-        id                RAW(16)        NOT NULL,
-        bank_id           VARCHAR2(256)  NOT NULL,
-        document_id       VARCHAR2(512),
-        chunk_id          VARCHAR2(512),
-        text              CLOB           NOT NULL,
-        context           CLOB,
-        event_date        TIMESTAMP WITH TIME ZONE NOT NULL,
-        occurred_start    TIMESTAMP WITH TIME ZONE,
-        occurred_end      TIMESTAMP WITH TIME ZONE,
-        mentioned_at      TIMESTAMP WITH TIME ZONE,
-        fact_type         VARCHAR2(64)   DEFAULT 'world' NOT NULL,
-        confidence_score  BINARY_DOUBLE,
-        access_count      NUMBER(10)     DEFAULT 0 NOT NULL,
-        consolidated_at   TIMESTAMP WITH TIME ZONE,
-        observation_scopes CLOB          CONSTRAINT imu_obs_scopes_json CHECK (observation_scopes IS JSON OR observation_scopes IS NULL),
-        tags              CLOB           DEFAULT '[]' NOT NULL,
-        metadata          CLOB           DEFAULT '{}' NOT NULL
-                                         CONSTRAINT imu_metadata_json CHECK (metadata IS JSON),
-        proof_count       NUMBER(10)     DEFAULT 1,
-        source_memory_ids CLOB,
-        history           CLOB           DEFAULT '[]'
-                                         CONSTRAINT imu_history_json CHECK (history IS JSON OR history IS NULL),
-        text_signals      CLOB,
-        consolidation_failed_at TIMESTAMP WITH TIME ZONE,
-        search_vector     CLOB,
-        edited_at         TIMESTAMP WITH TIME ZONE,
-        created_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-        updated_at        TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
-        invalidation_reason CLOB,
-        invalidated_at    TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP,
-        entity_ids        CLOB           CONSTRAINT imu_entity_ids_json CHECK (entity_ids IS JSON OR entity_ids IS NULL),
-        CONSTRAINT pk_invalidated_memory_units PRIMARY KEY (id),
-        CONSTRAINT fk_imu_document FOREIGN KEY (document_id, bank_id)
-            REFERENCES documents(id, bank_id) ON DELETE CASCADE
-    )
-    """,
     """
     CREATE TABLE IF NOT EXISTS entities (
         id                RAW(16)        DEFAULT SYS_GUID() NOT NULL,

hindsight-api-slim/hindsight_api/alembic/versions/x9s0t1u2v3w4_add_bank_config_column.py

@@ -16,7 +16,9 @@ retention parameters, retrieval settings, etc.) in Python field name format.
 
 from collections.abc import Sequence
 
+import sqlalchemy as sa
 from alembic import context, op
+from sqlalchemy.dialects.postgresql import JSONB
 
 from hindsight_api.alembic._dialect import run_for_dialect
 

hindsight-api-slim/hindsight_api/api/disconnect.py

@@ -1,97 +0,0 @@
-"""Client-disconnect detection that works behind ``BaseHTTPMiddleware``.
-
-``Request.is_disconnected()`` is the obvious way to notice an abandoned HTTP
-request, but it is silently broken once any ``@app.middleware("http")``
-(Starlette ``BaseHTTPMiddleware``) is installed: that middleware runs the route
-in a child task behind anyio memory streams, so the ``http.disconnect`` ASGI
-event never reaches the route's ``Request``. This app has such middlewares, so
-the recall/reflect cancellation in #2122/#2127 never actually fired in
-production — the disconnect was never observed.
-
-This pure-ASGI middleware sits *outside* the ``BaseHTTPMiddleware`` layer, where
-it still owns the real ``receive`` channel. For the recall and reflect routes it
-drains ``receive`` in a background task and trips a :class:`CancellationToken`
-the moment ``http.disconnect`` arrives, stashing the token on the ASGI ``scope``.
-The route copies that token onto its ``RequestContext`` and the engine checks it
-at stage boundaries — so abandoned work stops instead of running to completion.
-
-It only wraps recall/reflect (small JSON bodies); every other request — uploads,
-MCP streams, etc. — passes straight through untouched, so there is no buffering
-or latency cost elsewhere.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import contextlib
-from collections.abc import Awaitable, Callable, MutableMapping
-from typing import Any
-
-from ..cancellation import CancellationToken
-
-# Key under which the per-request CancellationToken is stored on the ASGI scope.
-# A dedicated top-level scope key (not scope["state"]) avoids any interaction
-# with Starlette's per-request state copying.
-SCOPE_CANCELLATION_TOKEN = "hindsight.cancellation_token"
-
-_CLIENT_DISCONNECTED_REASON = "client disconnected"
-
-Scope = MutableMapping[str, Any]
-Receive = Callable[[], Awaitable[MutableMapping[str, Any]]]
-Send = Callable[[MutableMapping[str, Any]], Awaitable[None]]
-
-
-def _should_monitor(path: str) -> bool:
-    """Only the two long-running, abandon-prone read endpoints need monitoring."""
-    return path.endswith("/memories/recall") or path.endswith("/reflect")
-
-
-class ClientDisconnectCancellationMiddleware:
-    """Trip a scope-level CancellationToken when the client disconnects.
-
-    Must be installed *outside* any ``BaseHTTPMiddleware`` so it owns the real
-    ASGI ``receive`` channel.
-    """
-
-    def __init__(self, app: Callable) -> None:
-        self.app = app
-
-    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
-        if scope["type"] != "http" or not _should_monitor(scope.get("path", "")):
-            await self.app(scope, receive, send)
-            return
-
-        token = CancellationToken()
-        scope[SCOPE_CANCELLATION_TOKEN] = token
-
-        # The downstream app still needs to read the request body, so we cannot
-        # simply consume `receive` ourselves. Instead a single pump task drains
-        # the real channel, forwards every message to a queue the app reads from,
-        # and trips the token the instant `http.disconnect` shows up — which the
-        # app would otherwise never pull once it has finished reading the body.
-        queue: asyncio.Queue = asyncio.Queue()
-
-        async def pump() -> None:
-            while True:
-                message = await receive()
-                if message["type"] == "http.disconnect":
-                    token.cancel(_CLIENT_DISCONNECTED_REASON)
-                    await queue.put(message)
-                    return
-                await queue.put(message)
-
-        async def proxied_receive() -> MutableMapping[str, Any]:
-            return await queue.get()
-
-        pump_task = asyncio.create_task(pump())
-        try:
-            await self.app(scope, proxied_receive, send)
-        finally:
-            pump_task.cancel()
-            with contextlib.suppress(asyncio.CancelledError):
-                await pump_task
-
-
-def get_scope_cancellation_token(scope: Scope) -> CancellationToken | None:
-    """Return the CancellationToken the middleware attached, if any."""
-    return scope.get(SCOPE_CANCELLATION_TOKEN)

hindsight-api-slim/hindsight_api/api/mcp.py

@@ -9,7 +9,7 @@ from fastmcp import FastMCP
 
 from hindsight_api import MemoryEngine
 from hindsight_api import __version__ as HINDSIGHT_VERSION
-from hindsight_api.config import DEFAULT_MCP_RECALL_DESCRIPTION, DEFAULT_MCP_RETAIN_DESCRIPTION, _get_raw_config
+from hindsight_api.config import _get_raw_config
 from hindsight_api.engine.memory_engine import _current_schema
 from hindsight_api.extensions import MCPExtension, load_extension
 from hindsight_api.extensions.tenant import AuthenticationError
@@ -78,19 +78,6 @@ def get_current_mcp_authenticated() -> bool:
     return _current_mcp_authenticated.get()
 
 
-def _build_mcp_tool_descriptions(extra_instructions: str | None) -> tuple[str | None, str | None]:
-    """Return custom retain/recall descriptions when server-level MCP instructions are set."""
-    if not isinstance(extra_instructions, str):
-        return None, None
-
-    extra_instructions = extra_instructions.strip()
-    if not extra_instructions:
-        return None, None
-
-    suffix = f"\n\nAdditional instructions: {extra_instructions}"
-    return DEFAULT_MCP_RETAIN_DESCRIPTION + suffix, DEFAULT_MCP_RECALL_DESCRIPTION + suffix
-
-
 def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
     """
     Create and configure the Hindsight MCP server.
@@ -126,8 +113,6 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
             "delete_directive",
             "list_memories",
             "get_memory",
-            "update_memory",
-            "invalidate_memory",
             "list_documents",
             "get_document",
             "delete_document",
@@ -148,10 +133,6 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
         allowed = frozenset(global_config.mcp_enabled_tools)
         base_tools = (base_tools if base_tools is not None else _ALL_TOOLS) & allowed
 
-    retain_description, recall_description = _build_mcp_tool_descriptions(
-        getattr(global_config, "mcp_instructions", None)
-    )
-
     # Configure and register tools using shared module
     config = MCPToolsConfig(
         bank_id_resolver=get_current_bank_id,
@@ -161,8 +142,6 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
         mcp_authenticated_resolver=get_current_mcp_authenticated,  # Propagate MCP pre-auth flag
         include_bank_id_param=multi_bank,
         tools=base_tools,
-        retain_description=retain_description,
-        recall_description=recall_description,
     )
 
     register_mcp_tools(mcp, memory, config)

hindsight-api-slim/hindsight_api/api/page_markdown.py

@@ -1,171 +0,0 @@
-"""Markdown rendering for knowledge pages.
-
-Knowledge pages render as *read-only* markdown documents over the existing mental
-models: each mental model becomes a markdown body with a YAML frontmatter block
-(``type`` required; ``title``/``description``/``tags``/``timestamp`` optional).
-
-This module is intentionally pure: every function transforms the mental-model
-dicts returned by ``MemoryEngine.list_mental_models`` / ``get_mental_model`` and
-never touches the database. That keeps rendering unit-testable without a DB or
-LLM and lets the HTTP layer stay a thin wrapper.
-"""
-
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import Any
-
-# Every page carries exactly one ``type`` frontmatter field. We default to this
-# when a page does not declare one via a ``type:<x>`` tag.
-DEFAULT_PAGE_TYPE = "knowledge-page"
-
-# A page declares its ``type`` through a tag of the form ``type:runbook``.
-# This keeps rendering schema-free (no new mental_models column): the type is
-# lifted from the existing tags array.
-TYPE_TAG_PREFIX = "type:"
-
-INDEX_FILENAME = "index.md"
-
-
-@dataclass(frozen=True)
-class PageType:
-    """A page's ``type`` and the tags that remain after the type tag is split off."""
-
-    type: str
-    display_tags: list[str]
-
-
-def _scalar(value: Any) -> str:
-    """Emit a YAML-safe double-quoted scalar.
-
-    We always double-quote so arbitrary page names / source queries can't be
-    misread as YAML special forms (``true``, ``2026-01-01``, ``- x``, etc.).
-    """
-    text = str(value)
-    escaped = text.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n").replace("\r", "")
-    return f'"{escaped}"'
-
-
-def page_type(tags: list[str] | None) -> PageType:
-    """Split a ``type`` out of the tag list.
-
-    The first ``type:<x>`` tag wins; all ``type:`` tags are removed from the
-    returned ``display_tags`` so they don't leak into the page's displayed tags.
-    Falls back to :data:`DEFAULT_PAGE_TYPE`.
-    """
-    resolved = DEFAULT_PAGE_TYPE
-    display: list[str] = []
-    for tag in tags or []:
-        if tag.startswith(TYPE_TAG_PREFIX):
-            suffix = tag[len(TYPE_TAG_PREFIX) :].strip()
-            if suffix and resolved == DEFAULT_PAGE_TYPE:
-                resolved = suffix
-            continue
-        display.append(tag)
-    return PageType(type=resolved, display_tags=display)
-
-
-def _timestamp(mm: dict[str, Any]) -> str | None:
-    return mm.get("last_refreshed_at") or mm.get("created_at")
-
-
-def frontmatter(mm: dict[str, Any]) -> dict[str, Any]:
-    """Build the ordered frontmatter mapping for a mental model.
-
-    ``None``/empty values are dropped by :func:`render_frontmatter`.
-    """
-    pt = page_type(mm.get("tags"))
-    return {
-        "id": mm.get("id"),
-        "type": pt.type,
-        "title": mm.get("name"),
-        "description": mm.get("source_query"),
-        "tags": pt.display_tags,
-        "timestamp": _timestamp(mm),
-    }
-
-
-def render_frontmatter(fm: dict[str, Any]) -> str:
-    """Render a frontmatter mapping into a ``---`` fenced YAML block."""
-    lines = ["---"]
-    for key, value in fm.items():
-        if value is None:
-            continue
-        if isinstance(value, list):
-            if not value:
-                continue
-            lines.append(f"{key}:")
-            lines.extend(f"  - {_scalar(item)}" for item in value)
-        else:
-            lines.append(f"{key}: {_scalar(value)}")
-    lines.append("---")
-    return "\n".join(lines)
-
-
-def render_document(mm: dict[str, Any]) -> str:
-    """Render a full markdown document: frontmatter block + markdown body."""
-    body = (mm.get("content") or "").strip()
-    return f"{render_frontmatter(frontmatter(mm))}\n\n{body}\n" if body else f"{render_frontmatter(frontmatter(mm))}\n"
-
-
-def page_filename(page_id: str) -> str:
-    """Bundle filename for a page id."""
-    return f"{page_id}.md"
-
-
-def log_filename(page_id: str) -> str:
-    """Reserved per-page history filename."""
-    return f"{page_id}.log.md"
-
-
-def render_index(nodes: list[dict[str, Any]]) -> str:
-    """Render the reserved ``index.md`` — nested markdown navigation over the tree.
-
-    ``nodes`` is the flat folder/page list (each with ``id``, ``kind``, ``name``,
-    ``parent_id``); folders nest their children, pages link to their ``.md``.
-    """
-    fm = render_frontmatter({"type": "index", "title": "Knowledge base"})
-    lines = [fm, "", "# Knowledge base", ""]
-
-    children: dict[Any, list[dict[str, Any]]] = {}
-    for node in nodes:
-        children.setdefault(node.get("parent_id"), []).append(node)
-
-    def walk(parent: Any, depth: int) -> None:
-        ordered = sorted(children.get(parent, []), key=lambda n: (n.get("sort_order", 0), n.get("name") or ""))
-        for node in ordered:
-            indent = "  " * depth
-            if node.get("kind") == "folder":
-                lines.append(f"{indent}- **{node['name']}/**")
-                walk(node["id"], depth + 1)
-            else:
-                description = node.get("source_query") or node.get("description")
-                link = f"{indent}- [{node['name']}](./{page_filename(node['id'])})"
-                lines.append(f"{link} — {description}" if description else link)
-
-    walk(None, 0)
-    if len(lines) == 4:
-        lines.append("_No knowledge pages yet._")
-    return "\n".join(lines) + "\n"
-
-
-def render_log(mm: dict[str, Any], history: list[dict[str, Any]]) -> str:
-    """Render the reserved per-page ``log.md`` from refresh history.
-
-    Each history entry is ``{previous_content, previous_reflect_response,
-    changed_at}`` (newest first), capturing the content *before* a refresh.
-    """
-    name = mm.get("name") or mm.get("id")
-    fm = render_frontmatter({"type": "log", "title": f"{name} — history"})
-    lines = [fm, "", f"# {name} — history", ""]
-    if not history:
-        lines.append("_No refresh history._")
-        return "\n".join(lines) + "\n"
-    for entry in history:
-        changed_at = entry.get("changed_at") or "unknown"
-        previous = (entry.get("previous_content") or "").strip()
-        lines.append(f"## {changed_at}")
-        lines.append("")
-        lines.append(previous if previous else "_(empty)_")
-        lines.append("")
-    return "\n".join(lines).rstrip() + "\n"

hindsight-api-slim/hindsight_api/banner.py

@@ -66,6 +66,11 @@ def color_end(text: str) -> str:
     return color(text, 1.0)
 
 
+def color_mid(text: str) -> str:
+    """Color text with gradient middle color."""
+    return color(text, 0.5)
+
+
 def dim(text: str) -> str:
     """Dim/gray text."""
     return f"\033[38;2;128;128;128m{text}\033[0m"

hindsight-api-slim/hindsight_api/cancellation.py

@@ -1,85 +0,0 @@
-"""Cooperative cancellation for long-running engine operations.
-
-Recall runs as a staged pipeline whose heavy stages — graph expansion and
-cross-encoder reranking — execute in worker threads (``run_in_executor``) that
-asyncio task cancellation cannot interrupt once they have started. Cancelling
-the awaiting task only unblocks the ``await``; the thread keeps burning CPU to
-completion. So rather than rely on task cancellation, callers thread a
-``CancellationToken`` through ``RequestContext`` and the engine checks it at
-stage boundaries (``raise_if_cancelled``), bailing out *before* dispatching the
-next expensive stage.
-
-This is cooperative by design: it cannot stop a computation already inside a
-worker thread, but it does stop an abandoned recall from progressing into — or
-past — that work, which is what starves the instance in issue #2122. The token
-lives on ``RequestContext``, so any operation that receives one (recall today;
-reflect/consolidation/MCP later) can adopt the same checkpoints, and any driver
-(client disconnect today; a deadline tomorrow) can fire it.
-"""
-
-from __future__ import annotations
-
-import asyncio
-
-
-class OperationCancelledError(Exception):
-    """Raised at a checkpoint when the operation has been cancelled.
-
-    Carries the ``reason`` set by whoever cancelled (e.g. "client disconnected")
-    so the HTTP layer can translate it into the appropriate status code instead
-    of a generic 500.
-
-    NOTE: this is a plain ``Exception`` on purpose, NOT ``BaseException``. The
-    recall/reflect pipelines have broad ``except Exception`` handlers that would
-    otherwise swallow it — those handlers re-raise ``OperationCancelledError``
-    explicitly (see ``_search_with_retries``) so cancellation propagates to the
-    HTTP layer. A ``BaseException`` would dodge those handlers but also slip past
-    legitimate ``isinstance(result, Exception)`` checks (e.g. the reflect agent's
-    ``asyncio.gather(..., return_exceptions=True)`` tool-result handling), which
-    expect every non-tuple result to be an ``Exception``.
-    """
-
-    def __init__(self, reason: str = "operation cancelled") -> None:
-        super().__init__(reason)
-        self.reason = reason
-
-
-class CancellationToken:
-    """A one-shot, cooperative cancellation signal.
-
-    Cheap to poll (``raise_if_cancelled``) at stage boundaries and awaitable
-    (``wait``) so a driver task can block until cancellation. Safe to share
-    across an engine call tree; polling is a no-op until something cancels, and
-    cancellation is idempotent (the first reason wins).
-    """
-
-    __slots__ = ("_event", "_reason")
-
-    def __init__(self) -> None:
-        self._event = asyncio.Event()
-        self._reason = "operation cancelled"
-
-    def cancel(self, reason: str = "operation cancelled") -> None:
-        """Signal cancellation. Idempotent; the first reason recorded wins."""
-        if not self._event.is_set():
-            self._reason = reason
-            self._event.set()
-
-    @property
-    def cancelled(self) -> bool:
-        """Whether cancellation has been signalled."""
-        return self._event.is_set()
-
-    @property
-    def reason(self) -> str:
-        """The reason recorded by the first ``cancel`` call."""
-        return self._reason
-
-    def raise_if_cancelled(self) -> None:
-        """Raise ``OperationCancelledError`` if cancellation has been signalled."""
-        if self._event.is_set():
-            raise OperationCancelledError(self._reason)
-
-    async def wait(self) -> None:
-        """Block until cancellation is signalled."""
-        await self._event.wait()

hindsight-api-slim/hindsight_api/config_resolver.py

@@ -8,21 +8,16 @@ Config values are resolved on every request to ensure consistency across
 multiple API servers.
 """
 
-import asyncio
 import json
 import logging
-from dataclasses import asdict, fields, replace
-from functools import lru_cache
-from types import UnionType
-from typing import TYPE_CHECKING, Any, Union, get_args, get_origin
+from dataclasses import asdict, replace
+from typing import TYPE_CHECKING, Any
 
 from hindsight_api.config import (
     RECALL_BUDGET_FUNCTIONS,
     HindsightConfig,
     _get_raw_config,
     normalize_config_dict,
-    validate_retain_chunking_config,
-    validate_retain_completion_token_budget,
 )
 from hindsight_api.engine.memory_engine import fq_table
 from hindsight_api.extensions.tenant import TenantExtension
@@ -34,43 +29,6 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 
 
-class BankConfigPersistenceConflictError(ValueError):
-    """Raised when a validated bank config update can no longer be persisted."""
-
-    def __init__(self, bank_id: str):
-        self.bank_id = bank_id
-        super().__init__(f"Cannot update config for bank '{bank_id}': the bank does not exist")
-
-
-def _validate_retain_strategy_chunking(base_config: HindsightConfig, strategies: Any) -> None:
-    """Validate retain strategy chunking with the same semantics as apply_strategy()."""
-    if not isinstance(strategies, dict):
-        return
-    configurable = HindsightConfig.get_configurable_fields()
-    for strategy_name, overrides in strategies.items():
-        if not isinstance(overrides, dict):
-            raise ValueError(f"Invalid retain strategy {strategy_name!r}: must be an object")
-        filtered = {k: v for k, v in overrides.items() if k in configurable}
-        if not filtered:
-            continue
-        try:
-            resolved = replace(base_config, **filtered)
-            validate_retain_chunking_config(
-                resolved.retain_chunk_size,
-                resolved.retain_structured_chunk_size,
-            )
-            validate_retain_completion_token_budget(
-                llm_provider=resolved.llm_provider,
-                retain_max_completion_tokens=resolved.retain_max_completion_tokens,
-                retain_chunk_size=resolved.retain_chunk_size,
-                retain_llm_model=resolved.retain_llm_model,
-                llm_model=resolved.llm_model,
-                retain_llm_provider=resolved.retain_llm_provider,
-            )
-        except ValueError as e:
-            raise ValueError(f"Invalid retain strategy {strategy_name!r}: {e}") from e
-
-
 class ConfigResolver:
     """Resolves hierarchical configuration with tenant/bank overrides."""
 
@@ -88,26 +46,6 @@ class ConfigResolver:
         self._configurable_fields = HindsightConfig.get_configurable_fields()
         self._credential_fields = HindsightConfig.get_credential_fields()
 
-    async def _resolve_parent_config_dict(self, bank_id: str, context: RequestContext | None = None) -> dict[str, Any]:
-        """Resolve global + tenant config before bank-level overrides."""
-        config_dict = asdict(self._global_config)
-
-        if self.tenant_extension and context:
-            try:
-                tenant_overrides = await self.tenant_extension.get_tenant_config(context)
-                if tenant_overrides:
-                    # Normalize keys and filter to configurable fields only
-                    normalized_tenant = normalize_config_dict(tenant_overrides)
-                    configurable_tenant = {k: v for k, v in normalized_tenant.items() if k in self._configurable_fields}
-                    config_dict.update(configurable_tenant)
-                    logger.debug(
-                        f"Applied tenant config overrides for bank {bank_id}: {list(configurable_tenant.keys())}"
-                    )
-            except Exception as e:
-                logger.warning(f"Failed to load tenant config for bank {bank_id}: {e}")
-
-        return config_dict
-
     async def resolve_full_config(self, bank_id: str, context: RequestContext | None = None) -> HindsightConfig:
         """
         Resolve full HindsightConfig for a bank with hierarchical overrides applied.
@@ -127,7 +65,23 @@ class ConfigResolver:
         Returns:
             Complete HindsightConfig with hierarchical overrides applied
         """
-        config_dict = await self._resolve_parent_config_dict(bank_id, context)
+        # Start with global config (all fields)
+        config_dict = asdict(self._global_config)
+
+        # Load tenant config overrides (if tenant extension available)
+        if self.tenant_extension and context:
+            try:
+                tenant_overrides = await self.tenant_extension.get_tenant_config(context)
+                if tenant_overrides:
+                    # Normalize keys and filter to configurable fields only
+                    normalized_tenant = normalize_config_dict(tenant_overrides)
+                    configurable_tenant = {k: v for k, v in normalized_tenant.items() if k in self._configurable_fields}
+                    config_dict.update(configurable_tenant)
+                    logger.debug(
+                        f"Applied tenant config overrides for bank {bank_id}: {list(configurable_tenant.keys())}"
+                    )
+            except Exception as e:
+                logger.warning(f"Failed to load tenant config for bank {bank_id}: {e}")
 
         # Load bank config overrides
         bank_overrides = await self._load_bank_config(bank_id)
@@ -138,26 +92,6 @@ class ConfigResolver:
         # Return full config object (dataclass doesn't have __init__ that accepts kwargs, so we update the object)
         # Create a new config instance by copying the global config and updating fields
         resolved_config = HindsightConfig(**config_dict)
-        # Multi-LLM chains and the reranker failover chain are static credential fields
-        # (never tenant/bank-overridable), but asdict() above flattened their member
-        # dataclasses into plain dicts. Restore the original typed objects from the global
-        # config so the resolved object stays well-typed for any consumer that reads them.
-        resolved_config = replace(
-            resolved_config,
-            reranker_members=self._global_config.reranker_members,
-            llm_members=self._global_config.llm_members,
-            llm_strategy=self._global_config.llm_strategy,
-            retain_llm_members=self._global_config.retain_llm_members,
-            retain_llm_strategy=self._global_config.retain_llm_strategy,
-            reflect_llm_members=self._global_config.reflect_llm_members,
-            reflect_llm_strategy=self._global_config.reflect_llm_strategy,
-            consolidation_llm_members=self._global_config.consolidation_llm_members,
-            consolidation_llm_strategy=self._global_config.consolidation_llm_strategy,
-        )
-        validate_retain_chunking_config(
-            resolved_config.retain_chunk_size,
-            resolved_config.retain_structured_chunk_size,
-        )
         return resolved_config
 
     async def get_bank_config(self, bank_id: str, context: RequestContext | None = None) -> dict[str, Any]:
@@ -188,83 +122,26 @@ class ConfigResolver:
         resolved_config = await self.resolve_full_config(bank_id, context)
         config_dict = asdict(resolved_config)
 
-        # SECURITY: drop static/infrastructure + credential fields, then permission-filter.
-        filtered = self._strip_static_and_credential_fields(config_dict)
-        return await self._apply_permission_filter(filtered, bank_id, context)
+        # SECURITY: Filter to only configurable fields (exclude static/infrastructure)
+        filtered = {k: v for k, v in config_dict.items() if k in self._configurable_fields}
 
-    def _strip_static_and_credential_fields(self, config_dict: dict[str, Any]) -> dict[str, Any]:
-        """Keep only configurable, non-credential fields.
+        # SECURITY: Remove ALL credential fields (API keys, base URLs, etc.)
+        filtered = {k: v for k, v in filtered.items() if k not in self._credential_fields}
 
-        SECURITY: excludes static/infrastructure fields and ALL credential fields
-        (API keys, base URLs, etc.) so a resolved config is safe to return over the API.
-        """
-        return {
-            k: v for k, v in config_dict.items() if k in self._configurable_fields and k not in self._credential_fields
-        }
-
-    async def _apply_permission_filter(
-        self, filtered: dict[str, Any], bank_id: str, context: RequestContext | None
-    ) -> dict[str, Any]:
-        """Further restrict already-stripped config to the tenant/bank permission allow-list.
-
-        On extension error, leaves ``filtered`` unchanged (parity with the historical
-        single-bank path: a permissions lookup failure must not leak or drop fields).
-        """
-        if not (self.tenant_extension and context):
-            return filtered
-        try:
-            allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
-            if allowed_fields is not None:  # None means "allow all"
-                filtered = {k: v for k, v in filtered.items() if k in allowed_fields}
-                logger.debug(
-                    f"Applied permission filter for bank {bank_id}: allowed={len(allowed_fields)} fields, "
-                    f"returned={len(filtered)} fields"
-                )
-        except Exception as e:
-            logger.warning(f"Failed to load permissions for bank {bank_id}: {e}")
-        return filtered
-
-    async def get_bank_configs(
-        self, bank_ids: list[str], context: RequestContext | None = None
-    ) -> dict[str, dict[str, Any]]:
-        """Batch variant of :meth:`get_bank_config` for many banks.
-
-        Equivalent to calling ``get_bank_config`` per bank, but resolves the
-        global + tenant base once and loads every bank's ``banks.config`` JSONB
-        in a single query, instead of one config round-trip per bank. Used by
-        ``list_banks`` to overlay disposition + mission without an N+1.
-
-        Returns a mapping of bank_id -> filtered configurable-field dict. A bank
-        with no config row still appears, mapped to the global+tenant base.
-        """
-        if not bank_ids:
-            return {}
-
-        # Global + tenant base, resolved once (tenant override is per-request, not per-bank).
-        base_dict = asdict(self._global_config)
+        # PERMISSIONS: Further filter based on tenant/bank permissions
         if self.tenant_extension and context:
             try:
-                tenant_overrides = await self.tenant_extension.get_tenant_config(context)
-                if tenant_overrides:
-                    normalized_tenant = normalize_config_dict(tenant_overrides)
-                    base_dict.update({k: v for k, v in normalized_tenant.items() if k in self._configurable_fields})
+                allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
+                if allowed_fields is not None:  # None means "allow all"
+                    filtered = {k: v for k, v in filtered.items() if k in allowed_fields}
+                    logger.debug(
+                        f"Applied permission filter for bank {bank_id}: allowed={len(allowed_fields)} fields, "
+                        f"returned={len(filtered)} fields"
+                    )
             except Exception as e:
-                logger.warning(f"Failed to load tenant config for bulk resolve: {e}")
+                logger.warning(f"Failed to load permissions for bank {bank_id}: {e}")
 
-        # All bank overrides in one query, then merge + strip per bank.
-        bank_overrides = await self._load_bank_configs(bank_ids)
-        stripped = {
-            bank_id: self._strip_static_and_credential_fields({**base_dict, **bank_overrides.get(bank_id, {})})
-            for bank_id in bank_ids
-        }
-
-        # Permission filter is per-bank; resolve concurrently when an extension is present.
-        if not (self.tenant_extension and context):
-            return stripped
-        permission_filtered = await asyncio.gather(
-            *(self._apply_permission_filter(stripped[bank_id], bank_id, context) for bank_id in bank_ids)
-        )
-        return dict(zip(bank_ids, permission_filtered, strict=True))
+        return filtered
 
     async def _load_bank_config(self, bank_id: str) -> dict[str, Any]:
         """
@@ -297,63 +174,17 @@ class ConfigResolver:
 
                     # Only return active overrides for configurable fields. JSON null is a tombstone
                     # for "Server Default" in the bank-config UI and should not override defaults.
-                    active = {k: v for k, v in normalized.items() if k in self._configurable_fields and v is not None}
-                    return _coerce_stored_bank_overrides(bank_id, active)
+                    return {k: v for k, v in normalized.items() if k in self._configurable_fields and v is not None}
         except Exception as e:
             logger.error(f"Failed to load bank config for {bank_id}: {e}")
 
         return {}
 
-    async def _load_bank_configs(self, bank_ids: list[str]) -> dict[str, dict[str, Any]]:
-        """Bulk variant of :meth:`_load_bank_config`: load many banks' overrides in one query.
-
-        Returns a mapping of bank_id -> normalized active overrides. Banks with no row
-        (or an empty/all-tombstone config) are simply absent from the mapping.
+    async def update_bank_config(
+        self, bank_id: str, updates: dict[str, Any], context: RequestContext | None = None
+    ) -> None:
         """
-        result: dict[str, dict[str, Any]] = {}
-        if not bank_ids:
-            return result
-        try:
-            async with self._backend.acquire() as conn:
-                rows = await conn.fetch(
-                    f"""
-                    SELECT bank_id, config FROM {fq_table("banks")} WHERE bank_id = ANY($1)
-                    """,
-                    bank_ids,
-                )
-                for row in rows:
-                    config_data = row["config"]
-                    if not config_data:
-                        continue
-                    # Handle case where JSONB is returned as JSON string
-                    if isinstance(config_data, str):
-                        config_data = json.loads(config_data)
-
-                    # Normalize keys (handle both env var format and Python field format)
-                    normalized = normalize_config_dict(config_data)
-
-                    # Only active overrides for configurable fields. JSON null is a tombstone
-                    # for "Server Default" in the bank-config UI and must not override defaults.
-                    overrides = {
-                        k: v for k, v in normalized.items() if k in self._configurable_fields and v is not None
-                    }
-                    if overrides:
-                        result[row["bank_id"]] = _coerce_stored_bank_overrides(row["bank_id"], overrides)
-        except Exception as e:
-            logger.error(f"Failed to bulk-load bank configs: {e}")
-        return result
-
-    async def validate_bank_config_updates(
-        self,
-        bank_id: str,
-        updates: dict[str, Any],
-        context: RequestContext | None = None,
-        *,
-        projected_bank_overrides: dict[str, Any] | None = None,
-        check_permissions: bool = True,
-    ) -> dict[str, Any]:
-        """
-        Normalize and validate bank configuration overrides.
+        Update bank configuration overrides (with permission checking).
 
         Args:
             bank_id: Bank identifier
@@ -362,16 +193,9 @@ class ConfigResolver:
                     or Python field format (llm_provider).
                     Only configurable fields are allowed.
             context: Request context for permission checking
-            projected_bank_overrides: Bank overrides to use as the validation
-                base instead of loading the current bank row.
-            check_permissions: Whether client field permissions apply to these
-                updates. Server-owned projected values set this to false.
-
-        Returns:
-            Normalized updates ready to persist.
 
         Raises:
-            ValueError: If attempting to override invalid/disallowed fields.
+            ValueError: If attempting to override invalid/disallowed fields
         """
         # Normalize keys
         normalized_updates = normalize_config_dict(updates)
@@ -403,7 +227,7 @@ class ConfigResolver:
                 )
 
         # PERMISSIONS: Check tenant/bank permissions
-        if check_permissions and self.tenant_extension and context:
+        if self.tenant_extension and context:
             try:
                 allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
                 if allowed_fields is not None:  # None means "allow all"
@@ -413,7 +237,7 @@ class ConfigResolver:
                             f"Not allowed to modify fields: {sorted(disallowed)}. "
                             f"Your permissions allow: {sorted(list(allowed_fields)[:10])}..."
                             if allowed_fields
-                            else f"Not allowed to modify fields: {sorted(disallowed)}. "
+                            else "Not allowed to modify fields: {sorted(disallowed)}. "
                             "Your permissions do not allow any config modifications."
                         )
             except ValueError:
@@ -422,11 +246,6 @@ class ConfigResolver:
                 logger.warning(f"Failed to check permissions for bank {bank_id}: {e}")
                 # Continue without permission check (fail open for backward compatibility)
 
-        # Validate every value against its declared field type before the
-        # field-specific checks below, so a wrong-shaped value is reported as such
-        # instead of tripping a structural validator with a confusing message.
-        _validate_config_value_types(normalized_updates)
-
         # Validate entity_labels structure
         if "entity_labels" in normalized_updates and normalized_updates["entity_labels"] is not None:
             from .engine.retain.entity_labels import parse_entity_labels
@@ -443,73 +262,16 @@ class ConfigResolver:
                 raise ValueError(
                     "Strategy names must not be empty strings. Remove entries with empty names before saving."
                 )
-            # A strategy's overrides are applied with dataclasses.replace() at retain
-            # time, so a wrong-shaped value there wedges the bank exactly as a
-            # top-level one would. Same contract, same door.
-            for strategy_name, strategy_overrides in normalized_updates["retain_strategies"].items():
-                if not isinstance(strategy_overrides, dict):
-                    raise ValueError(f"Invalid retain strategy {strategy_name!r}: must be an object")
-                try:
-                    _validate_config_value_types(normalize_config_dict(strategy_overrides))
-                except ValueError as e:
-                    raise ValueError(f"Invalid retain strategy {strategy_name!r}: {e}") from e
 
         # Validate recall budget fields
         _validate_recall_budget_updates(normalized_updates)
 
-        # Validate disposition trait fields (1-5 integer scale)
-        _validate_disposition_updates(normalized_updates)
-
-        chunking_fields_updated = (
-            "retain_chunk_size" in normalized_updates
-            or "retain_structured_chunk_size" in normalized_updates
-            or "retain_strategies" in normalized_updates
-        )
-        if chunking_fields_updated:
-            config_dict = await self._resolve_parent_config_dict(bank_id, context)
-            active_bank_overrides = (
-                await self._load_bank_config(bank_id)
-                if projected_bank_overrides is None
-                else dict(projected_bank_overrides)
-            )
-            for key, value in normalized_updates.items():
-                if key not in self._configurable_fields:
-                    continue
-                if value is None:
-                    active_bank_overrides.pop(key, None)
-                else:
-                    active_bank_overrides[key] = value
-            config_dict.update(active_bank_overrides)
-            base_config = HindsightConfig(**config_dict)
-            validate_retain_chunking_config(
-                base_config.retain_chunk_size,
-                base_config.retain_structured_chunk_size,
-            )
-            _validate_retain_strategy_chunking(base_config, base_config.retain_strategies)
-
-        return normalized_updates
-
-    async def update_bank_config(
-        self, bank_id: str, updates: dict[str, Any], context: RequestContext | None = None
-    ) -> None:
-        """Validate and persist bank configuration overrides for an existing bank.
-
-        Bank creation belongs to ``MemoryEngine``; this raises ``ValueError`` if
-        the bank does not exist rather than silently discarding the overrides.
-        """
-        normalized_updates = await self.validate_bank_config_updates(bank_id, updates, context)
-        await self._persist_bank_config(bank_id, normalized_updates)
-
-    async def _persist_bank_config(self, bank_id: str, normalized_updates: dict[str, Any]) -> None:
-        """Persist already-validated overrides without changing bank lifecycle state."""
-        # Bank lifecycle belongs to MemoryEngine. Callers must create the row
-        # before reaching this persistence step. COALESCE guards against a NULL
-        # config column (NULL || jsonb is NULL), which would drop the override.
+        # Merge with existing config (JSONB || operator)
         async with self._backend.acquire() as conn:
-            result = await conn.execute(
+            await conn.execute(
                 f"""
                 UPDATE {fq_table("banks")}
-                SET config = COALESCE(config, '{{}}'::jsonb) || $1::jsonb,
+                SET config = config || $1::jsonb,
                     updated_at = now()
                 WHERE bank_id = $2
                 """,
@@ -517,14 +279,6 @@ class ConfigResolver:
                 bank_id,
             )
 
-        # A missing bank row matches zero rows, which would otherwise persist
-        # nothing while reporting success. Fail loudly instead: reaching here
-        # without the row means a caller skipped the engine's provisioning step.
-        # (The Oracle wrapper reshapes rowcount into the same "UPDATE <n>" form.)
-        updated = int(result.split()[-1]) if isinstance(result, str) and result.startswith("UPDATE") else 0
-        if updated == 0:
-            raise BankConfigPersistenceConflictError(bank_id)
-
         logger.info(f"Updated bank config for {bank_id}: {list(normalized_updates.keys())}")
 
     async def reset_bank_config(self, bank_id: str) -> None:
@@ -548,147 +302,6 @@ class ConfigResolver:
         logger.info(f"Reset bank config for {bank_id} to defaults")
 
 
-# Fields whose accepted input shape is deliberately wider than the dataclass
-# annotation, because a dedicated structural validator normalizes them later.
-_WIDENED_FIELD_TYPES: dict[str, tuple[type, ...]] = {
-    # parse_entity_labels() accepts both the bare list of label groups and the
-    # {"attributes": [...]} envelope, though the field is annotated `list | None`.
-    "entity_labels": (list, dict),
-}
-
-
-def _runtime_types(declared: Any) -> tuple[type, ...]:
-    """Runtime-checkable base classes for a dataclass field annotation.
-
-    Unwraps unions (``str | None``) and generic aliases (``list[str]`` -> ``list``);
-    ``None`` is dropped because callers handle the tombstone separately. Returns an
-    empty tuple for anything not reducible to concrete classes, which the callers
-    read as "no type contract to enforce".
-    """
-    if declared is type(None):
-        return ()
-    origin = get_origin(declared)
-    if origin in (Union, UnionType):
-        return tuple(t for arg in get_args(declared) for t in _runtime_types(arg))
-    if origin is not None:
-        return (origin,) if isinstance(origin, type) else ()
-    return (declared,) if isinstance(declared, type) else ()
-
-
-@lru_cache(maxsize=1)
-def _configurable_field_types() -> dict[str, tuple[type, ...]]:
-    """Map each configurable field to the value types it accepts."""
-    configurable = HindsightConfig.get_configurable_fields()
-    field_types: dict[str, tuple[type, ...]] = {}
-    for field in fields(HindsightConfig):
-        if field.name not in configurable:
-            continue
-        allowed = _WIDENED_FIELD_TYPES.get(field.name) or _runtime_types(field.type)
-        if allowed:
-            field_types[field.name] = allowed
-    return field_types
-
-
-def _value_matches_type(value: Any, allowed: tuple[type, ...]) -> bool:
-    """Whether ``value`` satisfies a field's declared type contract."""
-    if isinstance(value, bool):
-        # bool is an int subclass; it must not slip into a numeric field.
-        return bool in allowed
-    if isinstance(value, int) and float in allowed:
-        # JSON draws no int/float distinction: 1 is a valid ratio.
-        return True
-    return isinstance(value, allowed)
-
-
-# Field types are reported to API clients, so name them the way the JSON payload
-# reads rather than by their Python class.
-_TYPE_DESCRIPTIONS: dict[type, str] = {
-    bool: "a boolean",
-    int: "an integer",
-    float: "a number",
-    str: "a string",
-    list: "a list",
-    dict: "an object",
-}
-
-
-def _describe_types(allowed: tuple[type, ...]) -> str:
-    return " or ".join(dict.fromkeys(_TYPE_DESCRIPTIONS.get(t, t.__name__) for t in allowed))
-
-
-def _validate_config_value_types(updates: dict[str, Any]) -> None:
-    """Reject values whose type contradicts the declared HindsightConfig type.
-
-    Without this, the bank-config API happily stores e.g. a JSON object in
-    ``observations_mission``; the write succeeds and the bank then fails every
-    consolidation with ``expected string or bytes-like object, got 'dict'`` from
-    deep inside prompt assembly (issue #3218). Reject at the door instead, naming
-    the field and the expected type.
-    """
-    field_types = _configurable_field_types()
-    for key, value in updates.items():
-        allowed = field_types.get(key)
-        # None is the "clear this override" tombstone; unknown keys are rejected
-        # elsewhere as non-configurable.
-        if allowed is None or value is None:
-            continue
-        if not _value_matches_type(value, allowed):
-            raise ValueError(f"{key} must be {_describe_types(allowed)}, got {type(value).__name__}")
-
-
-def _coerce_stored_bank_overrides(bank_id: str, overrides: dict[str, Any], where: str = "") -> dict[str, Any]:
-    """Make stored bank overrides safe to consume, tolerating pre-validation shapes.
-
-    ``_validate_config_value_types`` rejects bad types at write time, but banks
-    configured before that landed can still hold e.g. a JSON object in a
-    string-typed field. Every consumer that treats such a value as text blows up
-    identically on every run (``escape_for_prompt`` -> ``re.sub`` ->
-    "expected string or bytes-like object, got 'dict'"), so the bank's
-    consolidation never recovers on its own (issue #3218).
-
-    String fields are JSON-encoded, which preserves the author's intent — the
-    structure still reaches the prompt, as text. Anything else is dropped so the
-    bank falls back to the tenant/global value rather than wedging.
-
-    ``where`` labels the location in warnings; it is set when recursing into a
-    retain strategy, whose overrides reach the same fields via ``apply_strategy``.
-    """
-    field_types = _configurable_field_types()
-    coerced: dict[str, Any] = {}
-    for key, value in overrides.items():
-        allowed = field_types.get(key)
-        # None passes through: the caller has already dropped top-level tombstones,
-        # and inside a retain strategy a null is a deliberate override to None.
-        if allowed is None or value is None or _value_matches_type(value, allowed):
-            coerced[key] = value
-            continue
-        if str in allowed:
-            coerced[key] = json.dumps(value, ensure_ascii=False)
-            logger.warning(
-                f"Bank {bank_id} config field '{key}'{where} holds a {type(value).__name__} but is a string field; "
-                f"using its JSON encoding. Re-save this field as a string to silence this warning."
-            )
-        else:
-            logger.warning(
-                f"Bank {bank_id} config field '{key}'{where} holds a {type(value).__name__} but must be "
-                f"{_describe_types(allowed)}; ignoring the override and falling back to the server default."
-            )
-
-    # Strategy overrides are spliced onto the resolved config by apply_strategy(),
-    # so a bad value nested there wedges the bank just as a top-level one does.
-    strategies = coerced.get("retain_strategies")
-    if isinstance(strategies, dict):
-        coerced["retain_strategies"] = {
-            name: (
-                _coerce_stored_bank_overrides(bank_id, strategy, where=f" in retain strategy {name!r}")
-                if isinstance(strategy, dict)
-                else strategy
-            )
-            for name, strategy in strategies.items()
-        }
-    return coerced
-
-
 _RECALL_BUDGET_FIXED_KEYS = (
     "recall_budget_fixed_low",
     "recall_budget_fixed_mid",
@@ -736,31 +349,6 @@ def _validate_recall_budget_updates(updates: dict[str, Any]) -> None:
             )
 
 
-_DISPOSITION_KEYS = (
-    "disposition_skepticism",
-    "disposition_literalism",
-    "disposition_empathy",
-)
-
-
-def _validate_disposition_updates(updates: dict[str, Any]) -> None:
-    """Validate disposition trait config updates. Raises ValueError on invalid input.
-
-    Each trait is an integer on a 1-5 scale (or None to clear the per-bank
-    override). The read overlay injects the stored value verbatim into a strict
-    ``DispositionTraits(int, ge=1, le=5)``; an out-of-contract value (a float, a
-    0-1 scale, or an int outside 1-5) accepted here would later 500 the whole
-    bank list when any bank profile is serialized (issue #2348).
-    """
-    for key in _DISPOSITION_KEYS:
-        if key in updates:
-            value = updates[key]
-            if value is None:
-                continue
-            if not isinstance(value, int) or isinstance(value, bool) or not (1 <= value <= 5):
-                raise ValueError(f"{key} must be an integer between 1 and 5, got {value!r}")
-
-
 def apply_strategy(config: HindsightConfig, strategy_name: str) -> HindsightConfig:
     """
     Apply a named retain strategy's overrides on top of a resolved config.
@@ -768,8 +356,7 @@ def apply_strategy(config: HindsightConfig, strategy_name: str) -> HindsightConf
     A strategy is a named set of hierarchical field overrides stored in
     config.retain_strategies. Any field in _HIERARCHICAL_FIELDS can be
     overridden, including retain_extraction_mode, retain_chunk_size,
-    retain_structured_chunk_size, entity_labels,
-    entities_allow_free_form, etc.
+    entity_labels, entities_allow_free_form, etc.
 
     Unknown strategy names log a warning and return config unchanged.
     Unknown or non-hierarchical fields in the strategy are silently ignored.
@@ -791,17 +378,4 @@ def apply_strategy(config: HindsightConfig, strategy_name: str) -> HindsightConf
         return config
 
     logger.debug(f"Applying retain strategy '{strategy_name}': {list(filtered.keys())}")
-    resolved = replace(config, **filtered)
-    validate_retain_chunking_config(
-        resolved.retain_chunk_size,
-        resolved.retain_structured_chunk_size,
-    )
-    validate_retain_completion_token_budget(
-        llm_provider=resolved.llm_provider,
-        retain_max_completion_tokens=resolved.retain_max_completion_tokens,
-        retain_chunk_size=resolved.retain_chunk_size,
-        retain_llm_model=resolved.retain_llm_model,
-        llm_model=resolved.llm_model,
-        retain_llm_provider=resolved.retain_llm_provider,
-    )
-    return resolved
+    return replace(config, **filtered)

hindsight-api-slim/hindsight_api/daemon.py

@@ -14,7 +14,10 @@ import subprocess
 import sys
 import time
 from pathlib import Path
-from typing import IO
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from typing import IO
 
 logger = logging.getLogger(__name__)
 
@@ -39,28 +42,37 @@ class IdleTimeoutMiddleware:
         self.app = app
         self.idle_timeout = idle_timeout
         self.last_activity = time.time()
+        self._checker_task = None
 
     async def __call__(self, scope, receive, send):
+        # Update activity timestamp on each request
         self.last_activity = time.time()
         await self.app(scope, receive, send)
 
+    def start_idle_checker(self):
+        """Start the background task that checks for idle timeout."""
+        self._checker_task = asyncio.create_task(self._check_idle())
+
     async def _check_idle(self):
-        """Exit the daemon after the configured period without requests."""
+        """Background task that exits the process after idle timeout."""
+        # If idle_timeout is 0, don't auto-exit
         if self.idle_timeout <= 0:
             return
 
         while True:
-            await asyncio.sleep(30)
+            await asyncio.sleep(30)  # Check every 30 seconds
             idle_time = time.time() - self.last_activity
             if idle_time > self.idle_timeout:
                 logger.info(f"Idle timeout reached ({self.idle_timeout}s), shutting down daemon")
+                # Give a moment for any in-flight requests
                 await asyncio.sleep(1)
+                # Send SIGTERM to ourselves to trigger graceful shutdown
                 import signal
 
                 os.kill(os.getpid(), signal.SIGTERM)
 
 
-def _detach_popen_kwargs(log_handle: IO[bytes]) -> dict:
+def _detach_popen_kwargs(log_handle: "IO[bytes]") -> dict:
     """Cross-platform kwargs to spawn a subprocess detached from the caller.
 
     On POSIX, ``start_new_session=True`` calls ``setsid(2)`` so the child
@@ -157,3 +169,17 @@ def daemonize():
         subprocess.Popen(cmd, env=env, **_detach_popen_kwargs(log_handle))
 
     sys.exit(0)
+
+
+def check_daemon_running(port: int = DEFAULT_DAEMON_PORT) -> bool:
+    """Check if a daemon is running and responsive on the given port."""
+    import socket
+
+    try:
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        sock.settimeout(1)
+        result = sock.connect_ex(("127.0.0.1", port))
+        sock.close()
+        return result == 0
+    except Exception:
+        return False

hindsight-api-slim/hindsight_api/engine/__init__.py

@@ -3,7 +3,7 @@ Memory Engine - Core implementation of the memory system.
 
 This package contains all the implementation details of the memory engine:
 - MemoryEngine: Main class for memory operations
-- Utility modules: embedding_utils, link_utils, bank_utils
+- Utility modules: embedding_utils, link_utils, think_utils, bank_utils
 - Supporting modules: embeddings, cross_encoder, entity_resolver, etc.
 """
 

hindsight-api-slim/hindsight_api/engine/audit.py

@@ -10,67 +10,17 @@ import asyncio
 import json
 import logging
 import uuid
-from collections.abc import Awaitable, Callable
+from collections.abc import Callable
 from contextlib import asynccontextmanager
 from dataclasses import dataclass, field
 from datetime import datetime, timezone
 from typing import Any
 
-from pydantic import BaseModel, Field
-
 from ..engine.db_utils import acquire_with_retry
-from ..models import RequestContext
-from .schema import fq_table_explicit
 
 logger = logging.getLogger(__name__)
 
 
-class AuditLogEntry(BaseModel):
-    """A single audit log entry."""
-
-    id: str
-    action: str
-    transport: str
-    bank_id: str | None
-    started_at: str | None
-    ended_at: str | None
-    duration_ms: int | None = Field(
-        default=None,
-        description="Server-computed duration in milliseconds (started_at → ended_at). Null if not yet completed.",
-    )
-    request: dict[str, Any] | None
-    response: dict[str, Any] | None
-    metadata: dict[str, Any]
-
-
-class AuditLogListResponse(BaseModel):
-    """Response model for list audit logs endpoint."""
-
-    bank_id: str
-    total: int
-    limit: int
-    offset: int
-    items: list[AuditLogEntry]
-
-
-class AuditLogStatsBucket(BaseModel):
-    """A single time bucket in audit log stats."""
-
-    time: str
-    actions: dict[str, int]
-    total: int
-
-
-class AuditLogStatsResponse(BaseModel):
-    """Response model for audit log stats endpoint."""
-
-    bank_id: str
-    period: str
-    trunc: str
-    start: str
-    buckets: list[AuditLogStatsBucket]
-
-
 @dataclass
 class AuditEntry:
     """A single audit log entry."""
@@ -109,11 +59,11 @@ def _safe_json(data: Any) -> str | None:
         return None
 
 
-class AuditLogger:
-    """Fire-and-forget audit log writer.
+_SWEEP_INTERVAL_SECONDS = 3600  # Run retention sweep every hour
 
-    Retention of old rows is handled by the background :class:`MaintenanceLoop`.
-    """
+
+class AuditLogger:
+    """Fire-and-forget audit log writer with optional retention sweep."""
 
     def __init__(
         self,
@@ -121,60 +71,26 @@ class AuditLogger:
         schema_getter: Callable[[], str],
         enabled: bool,
         allowed_actions: list[str],
-        bank_enabled_resolver: Callable[[str, RequestContext | None], Awaitable[bool]] | None = None,
+        retention_days: int = -1,
     ) -> None:
         self._pool_getter = pool_getter
         self._schema_getter = schema_getter
         self._enabled = enabled
         self._allowed_actions: frozenset[str] | None = frozenset(allowed_actions) if allowed_actions else None
-        # Resolves the hierarchical ``audit_log_enabled`` for one bank
-        # (env -> tenant -> bank). None means "no per-bank resolution wired",
-        # in which case the global value alone decides.
-        self._bank_enabled_resolver = bank_enabled_resolver
+        self._retention_days = retention_days
+        self._sweep_task: asyncio.Task | None = None
 
-    def action_allowed(self, action: str) -> bool:
-        """Global action-allowlist check. Cheap, synchronous, bank-independent.
-
-        The allowlist is deployment-wide, so this is a valid pre-filter to skip
-        work for actions that can never be audited. It deliberately does NOT
-        consult the enabled flag: that is per-bank overridable, so a bank may
-        turn auditing ON even when the deployment default is off.
-        """
-        if self._allowed_actions is None:
-            return True
-        return action in self._allowed_actions
-
-    async def should_log(self, action: str, bank_id: str | None, context: RequestContext | None = None) -> bool:
-        """Full audit decision: action allowlist AND the bank's resolved switch.
-
-        ``audit_log_enabled`` is hierarchical (env -> tenant -> bank), so the
-        effective value depends on which bank the action targets. Falls back to
-        the global value when there is no bank in scope or no resolver wired.
-        """
-        if not self.action_allowed(action):
+    def is_enabled(self, action: str) -> bool:
+        """Check if audit logging is enabled for this action."""
+        if not self._enabled:
             return False
-        if bank_id is None or self._bank_enabled_resolver is None:
-            return self._enabled
-        try:
-            return await self._bank_enabled_resolver(bank_id, context)
-        except Exception as e:
-            # Never let a config-resolution failure break the request. Fall back
-            # to the deployment default: a transient DB blip must not silently
-            # create an audit gap for a bank meant to be audited. The tradeoff is
-            # the opt-out direction — a bank that overrode to false under a
-            # default-on deployment will be audited during the outage. We accept
-            # that: a few extra audit rows during a DB blip is the safer failure
-            # than dropping records that compliance may require.
-            logger.warning(f"Audit config resolution failed for bank={bank_id}: {e}; using global default")
-            return self._enabled
+        if self._allowed_actions is not None:
+            return action in self._allowed_actions
+        return True
 
     def log_fire_and_forget(self, entry: AuditEntry) -> None:
-        """Schedule an audit write as a background task.
-
-        Assumes the caller already made the audit decision via ``should_log``;
-        only the bank-independent allowlist is re-checked here.
-        """
-        if not self.action_allowed(entry.action):
+        """Schedule an audit write as a background task."""
+        if not self.is_enabled(entry.action):
             return
         try:
             asyncio.create_task(self._safe_log(entry))
@@ -189,12 +105,8 @@ class AuditLogger:
             logger.debug("Audit log skipped: pool not available")
             return
         try:
-            # fq_table_explicit qualifies per dialect: "schema".audit_log on
-            # PostgreSQL, bare audit_log on Oracle (where the schema is set at the
-            # session level). A raw f"{schema}.audit_log" produced public.audit_log
-            # on Oracle, where "public" is a reserved word — every write failed
-            # with ORA-00903 even though the table exists.
-            table = fq_table_explicit("audit_log", self._schema_getter())
+            schema = self._schema_getter()
+            table = f"{schema}.audit_log"
             async with acquire_with_retry(pool, max_retries=1) as conn:
                 await conn.execute(
                     f"""
@@ -216,6 +128,48 @@ class AuditLogger:
         except Exception as e:
             logger.warning(f"Audit log write failed for action={entry.action}: {e}")
 
+    def start_retention_sweep(self) -> None:
+        """Start the periodic retention sweep if retention is configured."""
+        if self._retention_days <= 0 or not self._enabled:
+            return
+        try:
+            self._sweep_task = asyncio.create_task(self._sweep_loop())
+        except RuntimeError:
+            logger.debug("Cannot start retention sweep: no running event loop")
+
+    async def stop_retention_sweep(self) -> None:
+        """Stop the periodic retention sweep."""
+        if self._sweep_task and not self._sweep_task.done():
+            self._sweep_task.cancel()
+            try:
+                await self._sweep_task
+            except asyncio.CancelledError:
+                pass
+            self._sweep_task = None
+
+    async def _sweep_loop(self) -> None:
+        """Periodically delete audit log entries older than retention_days."""
+        while True:
+            await self._run_sweep()
+            await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
+
+    async def _run_sweep(self) -> None:
+        """Delete expired audit log entries. Concurrent-safe via row-level deletes."""
+        pool = self._pool_getter()
+        if pool is None:
+            return
+        try:
+            schema = self._schema_getter()
+            table = f"{schema}.audit_log"
+            async with acquire_with_retry(pool, max_retries=1) as conn:
+                result = await conn.execute(
+                    f"DELETE FROM {table} WHERE started_at < NOW() - INTERVAL '{self._retention_days} days'"
+                )
+                if result and result != "DELETE 0":
+                    logger.info(f"Audit log retention sweep: {result}")
+        except Exception as e:
+            logger.warning(f"Audit log retention sweep failed: {e}")
+
 
 @asynccontextmanager
 async def audit_context(
@@ -225,7 +179,6 @@ async def audit_context(
     bank_id: str | None = None,
     request: dict[str, Any] | None = None,
     metadata: dict[str, Any] | None = None,
-    context: RequestContext | None = None,
 ):
     """Async context manager that times the operation and writes audit on exit.
 
@@ -234,7 +187,7 @@ async def audit_context(
             result = await do_work()
             entry.response = result_dict
     """
-    if audit_logger is None or not await audit_logger.should_log(action, bank_id, context):
+    if audit_logger is None or not audit_logger.is_enabled(action):
         entry = AuditEntry(action=action, transport=transport, bank_id=bank_id)
         yield entry
         return

hindsight-api-slim/hindsight_api/engine/bank_attribution.py

@@ -1,47 +0,0 @@
-"""Per-bank provider cost attribution via the OpenAI ``user`` field.
-
-Shared by the OpenAI-compatible LLM path and the OpenAI embeddings path so both
-tag outbound requests identically. Opt-in via ``HINDSIGHT_API_LLM_SEND_BANK_AS_USER``;
-downstream cost gateways (OpenRouter usage accounting, LiteLLM, Helicone) key spend
-on the OpenAI ``user`` field.
-
-Note: when enabled, the bank id is transmitted to the upstream provider as the
-end-user identifier. Banks that are themselves end-user identifiers are therefore
-forwarded to the provider — which is exactly what the OpenAI ``user`` field is for,
-but operators should opt in with that in mind.
-"""
-
-from typing import Any
-
-RERANKER_BANK_ID_HEADER = "X-Hindsight-Bank-Id"
-
-
-def apply_bank_attribution(request: dict[str, Any]) -> None:
-    """Tag ``request`` with ``user=<bank_id>`` for per-bank cost attribution.
-
-    Mutates ``request`` in place. No-op when the flag is off, no bank is in context,
-    or the caller already set ``user`` — we never override an explicit value.
-    """
-    if "user" in request:
-        return
-    # Lazy imports: memory_engine imports the embeddings/provider modules that call
-    # this, so a top-level import of memory_engine here would be circular.
-    from ..config import get_config
-    from .memory_engine import get_current_bank_id
-
-    if not get_config().llm_send_bank_as_user:
-        return
-    bank_id = get_current_bank_id()
-    if bank_id:
-        request["user"] = bank_id
-
-
-def reranker_bank_attribution_headers() -> dict[str, str]:
-    """Return the fixed per-bank header for trusted remote reranker endpoints."""
-    from ..config import get_config
-    from .memory_engine import get_current_bank_id
-
-    if not get_config().reranker_send_bank_as_header:
-        return {}
-    bank_id = get_current_bank_id()
-    return {RERANKER_BANK_ID_HEADER: bank_id} if bank_id else {}

hindsight-api-slim/hindsight_api/engine/bank_stats_cache.py

@@ -1,254 +0,0 @@
-"""TTL + coalescing cache for `get_bank_stats`.
-
-`get_bank_stats` aggregates over `memory_links` (and joins to `memory_units`),
-which can be a multi-second parallel sequential scan on banks with millions of
-rows. The result is intentionally approximate (it powers a UI widget and a
-freshness hint inside `reflect`), so caching it for a few tens of seconds is
-safe and dramatically reduces planner-driven thrash from clients that poll.
-
-The cache also coalesces concurrent misses on the same key onto a single
-in-flight task so that N concurrent callers produce one query rather than N.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import logging
-import time
-from collections import OrderedDict
-from typing import TYPE_CHECKING, Any, Awaitable, Callable
-
-from .db_utils import acquire_with_retry
-
-if TYPE_CHECKING:
-    from .db.base import DatabaseBackend
-
-logger = logging.getLogger(__name__)
-
-
-class BankStatsCache:
-    """Per-process TTL cache keyed on (schema, bank_id).
-
-    `ttl_seconds <= 0` disables caching: each call passes straight through to
-    the loader. `max_entries` bounds memory in environments with many banks.
-    """
-
-    def __init__(self, *, ttl_seconds: float, max_entries: int) -> None:
-        self._ttl = float(ttl_seconds)
-        self._max_entries = int(max_entries) if max_entries and max_entries > 0 else 0
-        self._entries: OrderedDict[tuple[str, str], tuple[float, dict[str, Any]]] = OrderedDict()
-        self._in_flight: dict[tuple[str, str], asyncio.Future[dict[str, Any]]] = {}
-        self._lock = asyncio.Lock()
-
-    @property
-    def enabled(self) -> bool:
-        return self._ttl > 0
-
-    def _now(self) -> float:
-        return time.monotonic()
-
-    def _get_fresh_unlocked(self, key: tuple[str, str]) -> dict[str, Any] | None:
-        entry = self._entries.get(key)
-        if entry is None:
-            return None
-        expires_at, value = entry
-        if expires_at <= self._now():
-            # Expired — drop so the loader runs again.
-            self._entries.pop(key, None)
-            return None
-        # Mark as recently used for LRU eviction.
-        self._entries.move_to_end(key)
-        return value
-
-    def _store_unlocked(self, key: tuple[str, str], value: dict[str, Any]) -> None:
-        if not self.enabled:
-            return
-        self._entries[key] = (self._now() + self._ttl, value)
-        self._entries.move_to_end(key)
-        if self._max_entries:
-            while len(self._entries) > self._max_entries:
-                self._entries.popitem(last=False)
-
-    async def get_or_load(
-        self,
-        schema: str,
-        bank_id: str,
-        loader: Callable[[], Awaitable[dict[str, Any]]],
-        *,
-        force_refresh: bool = False,
-    ) -> dict[str, Any]:
-        """Return cached stats for `(schema, bank_id)` or call `loader()`.
-
-        Concurrent misses on the same key are coalesced onto a single
-        in-flight loader. When ``force_refresh`` is set the cached value is
-        ignored: the loader runs and its result replaces the cached entry.
-        """
-        if not self.enabled:
-            return await loader()
-
-        key = (schema, bank_id)
-
-        if force_refresh:
-            value = await loader()
-            async with self._lock:
-                self._store_unlocked(key, value)
-                # Supersede any loader that was in flight for this key.
-                self._in_flight.pop(key, None)
-            return value
-
-        async with self._lock:
-            cached = self._get_fresh_unlocked(key)
-            if cached is not None:
-                return cached
-            in_flight = self._in_flight.get(key)
-            if in_flight is None:
-                in_flight = asyncio.get_running_loop().create_future()
-                self._in_flight[key] = in_flight
-                is_owner = True
-            else:
-                is_owner = False
-
-        if not is_owner:
-            return await asyncio.shield(in_flight)
-
-        try:
-            value = await loader()
-        except BaseException as exc:
-            async with self._lock:
-                # Invalidation may have detached this loader and allowed a new
-                # one to claim the key. Never remove that newer loader's slot.
-                if self._in_flight.get(key) is in_flight:
-                    self._in_flight.pop(key, None)
-            if not in_flight.done():
-                in_flight.set_exception(exc)
-            # Suppress "Future exception was never retrieved" when no other
-            # caller was waiting on this loader — we re-raise to the owner
-            # immediately and the future is a no-op in that case.
-            in_flight.exception()
-            raise
-
-        async with self._lock:
-            # Only the loader that still owns the key may populate the cache.
-            # An invalidated loader can finish for its original callers, but its
-            # pre-invalidation result must not overwrite a newer load.
-            if self._in_flight.get(key) is in_flight:
-                self._store_unlocked(key, value)
-                self._in_flight.pop(key, None)
-        if not in_flight.done():
-            in_flight.set_result(value)
-        return value
-
-    async def invalidate(self, schema: str, bank_id: str) -> None:
-        """Drop any cached stats for `(schema, bank_id)`."""
-        async with self._lock:
-            key = (schema, bank_id)
-            self._entries.pop(key, None)
-            # Detach rather than cancel: existing callers may finish with the
-            # snapshot they requested, while post-invalidation callers reload.
-            self._in_flight.pop(key, None)
-
-    async def clear(self) -> None:
-        async with self._lock:
-            self._entries.clear()
-            self._in_flight.clear()
-
-
-class DistributedBankStatsCache:
-    """Table-backed (cross-process) TTL cache for `get_bank_stats`.
-
-    Same ``get_or_load`` / ``invalidate`` / ``clear`` contract as
-    :class:`BankStatsCache`, but the store is the per-schema ``bank_stats_cache``
-    table instead of a per-process dict — so one worker's computation is shared
-    with every other worker, and no caller recomputes while a fresh row exists.
-
-    On a hit, a call is a single primary-key ``SELECT`` (sub-millisecond); only a
-    miss runs the (expensive) ``loader`` and writes the row back. Concurrent
-    misses are *not* coalesced across processes (that would need a lock): they
-    each compute and ``UPSERT``, last write wins — all results are correct, at the
-    cost of a brief redundant compute at expiry.
-
-    Every DB touch is best-effort: if the cache table is unreachable or missing
-    (e.g. a schema mid-migration), the call degrades to computing without caching
-    rather than failing ``get_bank_stats``. PostgreSQL only — the engine keeps the
-    in-process :class:`BankStatsCache` for Oracle.
-    """
-
-    def __init__(self, *, backend: "DatabaseBackend", ttl_seconds: float) -> None:
-        self._backend = backend
-        self._ttl = float(ttl_seconds)
-
-    @property
-    def enabled(self) -> bool:
-        return self._ttl > 0
-
-    @staticmethod
-    def _qualified(schema: str) -> str:
-        return f'"{schema}".bank_stats_cache' if schema else "bank_stats_cache"
-
-    async def get_or_load(
-        self,
-        schema: str,
-        bank_id: str,
-        loader: Callable[[], Awaitable[dict[str, Any]]],
-        *,
-        force_refresh: bool = False,
-    ) -> dict[str, Any]:
-        if not self.enabled:
-            return await loader()
-
-        table = self._qualified(schema)
-
-        # 1. Fresh row? Single PK lookup; ``payload::text`` sidesteps any
-        #    jsonb->object codec so we always decode the same way. Skipped when
-        #    the caller forces a refresh — then we recompute and overwrite below.
-        if not force_refresh:
-            try:
-                async with acquire_with_retry(self._backend) as conn:
-                    row = await conn.fetchrow(
-                        f"SELECT payload::text AS payload FROM {table} "
-                        f"WHERE bank_id = $1 AND computed_at > now() - make_interval(secs => $2::double precision)",
-                        bank_id,
-                        self._ttl,
-                    )
-                if row is not None:
-                    return json.loads(row["payload"])
-            except Exception as exc:  # noqa: BLE001 — cache read must never break the endpoint
-                logger.debug("bank_stats_cache read failed for %s.%s (%s); computing uncached", schema, bank_id, exc)
-                return await loader()
-
-        # 2. Miss — compute, then write the row back (best-effort).
-        value = await loader()
-        try:
-            async with acquire_with_retry(self._backend) as conn:
-                await conn.execute(
-                    f"INSERT INTO {table} (bank_id, payload, computed_at) VALUES ($1, $2::jsonb, now()) "
-                    f"ON CONFLICT (bank_id) DO UPDATE SET payload = EXCLUDED.payload, computed_at = now()",
-                    bank_id,
-                    json.dumps(value),
-                )
-        except Exception as exc:  # noqa: BLE001 — a failed write just means no caching this round
-            logger.warning("bank_stats_cache write failed for %s.%s (%s)", schema, bank_id, exc)
-        return value
-
-    async def invalidate(self, schema: str, bank_id: str) -> None:
-        """Drop the cached row so the next read recomputes."""
-        if not self.enabled:
-            return
-        try:
-            async with acquire_with_retry(self._backend) as conn:
-                await conn.execute(f"DELETE FROM {self._qualified(schema)} WHERE bank_id = $1", bank_id)
-        except Exception as exc:  # noqa: BLE001 — invalidation must never break the write path
-            logger.debug("bank_stats_cache invalidate failed for %s.%s (%s)", schema, bank_id, exc)
-
-    async def clear(self) -> None:
-        """Drop all cached rows in the current schema (best-effort)."""
-        if not self.enabled:
-            return
-        from .memory_engine import get_current_schema
-
-        try:
-            async with acquire_with_retry(self._backend) as conn:
-                await conn.execute(f"DELETE FROM {self._qualified(get_current_schema())}")
-        except Exception as exc:  # noqa: BLE001
-            logger.debug("bank_stats_cache clear failed (%s)", exc)

hindsight-api-slim/hindsight_api/engine/cache_affinity.py

@@ -1,185 +0,0 @@
-"""Server-side prompt-cache affinity hints for OpenAI-compatible providers.
-
-Prompt caching only pays off when the same conversation reaches the same backend
-cache, and providers expose different mechanisms for that:
-
-- xAI stores prompt-cache entries **per backend server** and routes requests
-  carrying the same ``x-grok-conv-id`` to one server (docs.x.ai, "Maximizing
-  Cache Hits"). Without it, consecutive calls of one agentic loop can each land
-  on a cache-cold replica.
-- OpenAI accepts a ``prompt_cache_key`` request field that improves its own
-  cache routing.
-
-Hindsight already does provider-specific cache work for its first-class
-providers (``anthropic_llm`` sets ``cache_control`` breakpoints; ``gemini_llm``
-runs an explicit ``CachedContent`` manager). This module is the equivalent for
-the OpenAI-compatible family — ``OpenAICompatibleLLM`` and its ``fireworks``
-and ``nous`` subclasses — which sent no affinity hint at all.
-
-Default ``auto`` per member (``cache_affinity``). ``auto`` is an allowlist, not a
-best-effort probe: it emits a hint only for hosts documented to accept one and
-resolves to ``none`` for everything else, so an unknown OpenAI-compatible backend
-never receives an unfamiliar field. Every helper here is fail-open — when no id
-can be derived the request goes out byte-identical to before. Set ``none`` to
-disable entirely.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import logging
-from enum import StrEnum
-from typing import Any
-from urllib.parse import urlparse
-
-logger = logging.getLogger(__name__)
-
-# xAI's documented cache-pinning header, and OpenAI's cache-routing field.
-XAI_CONV_ID_HEADER = "x-grok-conv-id"
-OPENAI_PROMPT_CACHE_KEY_PARAM = "prompt_cache_key"
-
-# Hosts (exact or parent domain) whose backends implement the xAI header.
-_XAI_DOMAINS = ("x.ai", "grok.com")
-# Hosts (exact or parent domain) that accept OpenAI's prompt_cache_key field.
-_OPENAI_DOMAINS = ("openai.com", "openai.azure.com")
-
-
-class CacheAffinityMode(StrEnum):
-    """How (and whether) to pin a request to a backend prompt cache."""
-
-    NONE = "none"
-    XAI_CONV_ID = "xai_conv_id"
-    OPENAI_PROMPT_CACHE_KEY = "openai_prompt_cache_key"
-    AUTO = "auto"
-
-
-def parse_cache_affinity(value: str | None) -> CacheAffinityMode:
-    """Validate a configured cache-affinity mode, defaulting to ``none``.
-
-    Raises ``ValueError`` on an unrecognized value so a typo fails loudly at
-    provider construction rather than silently disabling the feature — the whole
-    point of the setting is that its effect is invisible in the response.
-    """
-    if not value:
-        return CacheAffinityMode.NONE
-    try:
-        return CacheAffinityMode(value.strip().lower())
-    except ValueError as e:
-        valid = ", ".join(mode.value for mode in CacheAffinityMode)
-        raise ValueError(f"Invalid cache_affinity {value!r}. Must be one of: {valid}.") from e
-
-
-def _host_matches(hostname: str, domain: str) -> bool:
-    """True when ``hostname`` is ``domain`` itself or a subdomain of it.
-
-    Parsed-host suffix matching, never a substring test: a bare
-    ``"x.ai" in base_url`` also matches ``vertex.ai`` and
-    ``https://x.ai.evil.example``. The in-tree Azure check
-    (``".openai.azure.com" in self.base_url``) gets away with a substring only
-    because its needle is long and dotted; ``x.ai`` is four characters.
-    """
-    return hostname == domain or hostname.endswith(f".{domain}")
-
-
-def resolve_cache_affinity(mode: CacheAffinityMode, provider: str, base_url: str | None) -> CacheAffinityMode:
-    """Resolve ``auto`` to a concrete mode from the provider and base-URL host.
-
-    Non-``auto`` modes are returned unchanged. ``auto`` resolves to
-    ``xai_conv_id`` for an x.ai / grok.com host, ``openai_prompt_cache_key`` for
-    native OpenAI (no base URL) or an openai.com / Azure OpenAI host, and
-    ``none`` for everything else — an unknown backend gets no unfamiliar field.
-
-    The xAI check is host-only and deliberately provider-independent: the
-    documented setup for an xAI endpoint is ``provider=openai`` plus an x.ai base
-    URL, exactly like Azure OpenAI, so keying on the provider name would miss it.
-    """
-    if mode is not CacheAffinityMode.AUTO:
-        return mode
-
-    hostname = (urlparse(base_url).hostname or "") if base_url else ""
-    if hostname and any(_host_matches(hostname, domain) for domain in _XAI_DOMAINS):
-        return CacheAffinityMode.XAI_CONV_ID
-    if provider.lower() == "openai":
-        if not hostname:
-            return CacheAffinityMode.OPENAI_PROMPT_CACHE_KEY
-        if any(_host_matches(hostname, domain) for domain in _OPENAI_DOMAINS):
-            return CacheAffinityMode.OPENAI_PROMPT_CACHE_KEY
-    return CacheAffinityMode.NONE
-
-
-def _first_message_fingerprint(messages: Any) -> str | None:
-    """Hash the first message into a 32-hex id, or None if the shape is wrong.
-
-    Used only when no trace context is bound (direct provider use, tests). The
-    first message is the system prompt, so the id is stable as the message list
-    grows through an agent loop — which is the property cache pinning needs —
-    while differing across conversations whose first messages differ.
-
-    Shape-checked rather than truthiness-checked: a bare string ``messages``
-    would index to its first character and mint an id from garbage. Anything
-    unexpected returns None and the request goes out with no affinity hint.
-    """
-    if not isinstance(messages, list) or not messages or not isinstance(messages[0], dict):
-        return None
-    try:
-        canonical = json.dumps(messages[0], sort_keys=True, ensure_ascii=False, default=str)
-    except (TypeError, ValueError):
-        logger.debug("Cache affinity: first message not serializable; sending no hint", exc_info=True)
-        return None
-    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:32]
-
-
-def cache_affinity_id(messages: Any) -> str | None:
-    """Return the affinity id for the in-flight call, or None to send nothing.
-
-    Primary source is the operation's ``trace_id`` — one uuid per
-    retain/reflect/consolidation run, generated in ``LLMProvider.with_config``
-    and bound around every underlying provider call, so every LLM call of one
-    run shares it. That is engine identity rather than payload hashing: it stays
-    constant across a run even when the first message changes mid-run.
-
-    The value is always 32 lowercase hex characters, including for the trace_id
-    path (hashed rather than passed through) so the wire format is uniform and
-    carries no uuid semantics.
-    """
-    from .llm_trace import current_trace_context
-
-    trace_ctx = current_trace_context()
-    if trace_ctx is not None and trace_ctx.trace_id:
-        return hashlib.sha256(str(trace_ctx.trace_id).encode("utf-8")).hexdigest()[:32]
-    return _first_message_fingerprint(messages)
-
-
-def apply_cache_affinity(request: dict[str, Any], mode: CacheAffinityMode) -> None:
-    """Add this request's cache-affinity hint to ``request`` in place.
-
-    ``mode`` must already be resolved (see :func:`resolve_cache_affinity`);
-    ``none`` — and an unresolved ``auto`` — add nothing.
-
-    User-wins semantics throughout, matching the file's ``setdefault`` precedent
-    in ``_apply_provider_extra_body_defaults``: an ``x-grok-conv-id`` the caller
-    already placed in ``extra_headers`` is kept, and a ``prompt_cache_key`` in
-    the operator's configured ``extra_body`` (the escape hatch for a backend
-    that wants its own value) suppresses ours entirely.
-
-    Never raises: when no id can be derived the request is left byte-identical
-    to a pre-affinity one.
-    """
-    affinity_id = cache_affinity_id(request.get("messages"))
-    if affinity_id is None:
-        return
-
-    if mode is CacheAffinityMode.XAI_CONV_ID:
-        extra_headers = request.setdefault("extra_headers", {})
-        extra_headers.setdefault(XAI_CONV_ID_HEADER, affinity_id)
-    elif mode is CacheAffinityMode.OPENAI_PROMPT_CACHE_KEY:
-        # prompt_cache_key is a first-class named parameter on
-        # chat.completions.create() in the resolved openai SDK, so it goes at the
-        # top level rather than through extra_body. An operator value in
-        # extra_body would still reach the same wire field, so honour it and
-        # send nothing rather than sending both.
-        extra_body = request.get("extra_body")
-        if isinstance(extra_body, dict) and OPENAI_PROMPT_CACHE_KEY_PARAM in extra_body:
-            return
-        request.setdefault(OPENAI_PROMPT_CACHE_KEY_PARAM, affinity_id)

hindsight-api-slim/hindsight_api/engine/causal_links.py

@@ -1,70 +0,0 @@
-"""Shared causal-link taxonomy.
-
-Retain writes only the canonical relationship.  Transfer import/export also
-preserves historical relationship types so existing banks keep their graph
-semantics without allowing new retain output to create those types.
-"""
-
-from dataclasses import dataclass
-from typing import Any
-
-CANONICAL_CAUSAL_LINK_TYPE = "caused_by"
-LEGACY_CAUSAL_LINK_TYPE_NAMES = ("causes", "enables", "prevents")
-
-CANONICAL_CAUSAL_LINK_TYPES = frozenset({CANONICAL_CAUSAL_LINK_TYPE})
-LEGACY_CAUSAL_LINK_TYPES = frozenset(LEGACY_CAUSAL_LINK_TYPE_NAMES)
-CAUSAL_LINK_TYPES = (CANONICAL_CAUSAL_LINK_TYPE, *LEGACY_CAUSAL_LINK_TYPE_NAMES)
-
-DEFAULT_CAUSAL_LINK_WEIGHT = 1.0
-
-
-@dataclass(frozen=True)
-class CausalLinkDescriptor:
-    """One causal edge, parked on the curation archive while an endpoint is invalidated.
-
-    Invalidation moves a fact out of ``memory_units``, so the FK cascade deletes
-    its ``memory_links`` rows — and nothing could recreate a causal edge, which
-    is extraction output rather than derived data. The descriptor is what the
-    archive row stores so revert can rematerialize the edge (#2864).
-    """
-
-    from_unit_id: str
-    to_unit_id: str
-    link_type: str
-    weight: float = DEFAULT_CAUSAL_LINK_WEIGHT
-
-    def as_json_dict(self) -> dict[str, Any]:
-        """Serializable form written to ``invalidated_memory_units.causal_links``.
-
-        The key names double as the column list of the ``jsonb_to_recordset``
-        read in ``snapshot_causal_links`` — keep them in sync.
-        """
-        return {
-            "from_unit_id": self.from_unit_id,
-            "to_unit_id": self.to_unit_id,
-            "link_type": self.link_type,
-            "weight": self.weight,
-        }
-
-    @classmethod
-    def from_json_dict(cls, raw: Any) -> "CausalLinkDescriptor | None":
-        """Parse one stored descriptor, or None when it isn't a usable causal edge.
-
-        The archive column is plain JSON with no schema enforcement (a restore
-        from an older backup, or a hand-edited row, can put anything there), and
-        ``memory_links`` has a ``link_type`` CHECK constraint — so an unusable
-        entry is skipped rather than allowed to abort the whole revert.
-        """
-        if not isinstance(raw, dict):
-            return None
-        from_unit_id = raw.get("from_unit_id")
-        to_unit_id = raw.get("to_unit_id")
-        link_type = raw.get("link_type")
-        if not from_unit_id or not to_unit_id or link_type not in CAUSAL_LINK_TYPES:
-            return None
-        return cls(
-            from_unit_id=str(from_unit_id),
-            to_unit_id=str(to_unit_id),
-            link_type=str(link_type),
-            weight=float(raw.get("weight") or DEFAULT_CAUSAL_LINK_WEIGHT),
-        )

hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py

@@ -17,20 +17,6 @@ _MISSION_PRIORITY_NOTE = (
     "DECISION GUIDE, or OUTPUT FORMAT below, the MISSION takes priority."
 )
 
-# Default language rule — used only when HINDSIGHT_API_LLM_OUTPUT_LANGUAGE is
-# unset. Without it the whole prompt is English and multilingual models drift:
-# Chinese source facts intermittently produce English observations. Retain's
-# fact extraction carries the equivalent rule (see _BASE_FACT_EXTRACTION_PROMPT),
-# so this makes "preserve the source language" the pipeline-wide default. When an
-# output language IS configured, this section is omitted and
-# output_language_directive() takes over — the two must never both be present or
-# they contradict each other.
-_DEFAULT_LANGUAGE_RULE = """## LANGUAGE
-
-Write every observation in the language of its own source facts — never translate them. Per observation, not per batch: when one merges facts of several languages, the majority wins. Proper nouns, identifiers, and units stay verbatim.
-
-When an existing observation is written in a different language from the new facts updating it, do NOT edit its wording in place — that is what produces an English sentence with a Chinese detail bolted on. Discard the old phrasing and compose the merged observation from scratch in the new facts' language."""
-
 _PROCESSING_RULES = """## PROCESSING RULES
 
 1. PREFER UPDATE OVER CREATE (when there is something to merge with): if new facts describe the same canonical event, statement, decision, claim, or recurring pattern already covered by an existing observation, UPDATE that observation and attach the new facts as evidence. Do NOT create a near-duplicate sibling. One canonical observation with many source facts is always better than many siblings with one source fact each. Merge aggressively on: same named event, same diagnostic finding, same architectural decision, same recurring claim. **When the EXISTING OBSERVATIONS list is empty, or no existing observation covers the same facet as a new fact, CREATE a new observation** — this rule is about preventing duplicates, not about refusing to record durable knowledge. CREATE is the correct default for any structurally distinct event, claim, or pattern that has no existing match.
@@ -51,41 +37,8 @@ _PROCESSING_RULES = """## PROCESSING RULES
 
 9. KEEP DISTINCT TOPICS DISTINCT: do not merge observations about different people, entities, or unrelated topics. Merging is for the same canonical fact recurring — not for related-but-distinct claims."""
 
-# Field-by-field definitions of the input shape used by the cached system
-# prefix. The call site runs .format(), so these strings must contain no braces.
-_FACT_FIELDS = """One per line, formatted as `[uuid] fact text (temporal fields)`:
-- `[uuid]`: the fact's identifier — copy it verbatim into `source_fact_ids`
-- `occurred_start` / `occurred_end`: when the described event happened. This can be long before the fact was stated — a fact recorded today may describe a 2019 event.
-- `mentioned_at`: when the source material that states this fact was written. This is the fact's recency: how up to date the statement is, NOT when it was added to memory. A fact taken from an old document keeps its old `mentioned_at` even if it was only just processed."""
-
-_OBSERVATION_FIELDS = """- `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
-- `text`: the observation content
-- `proof_count`: how many source facts this observation has already merged
-- `occurred_start` / `occurred_end`: the span of the events behind the observation — earliest start and latest end across its source facts
-- `mentioned_at`: the latest of the `mentioned_at` values of its source facts — the most recent point at which this observation was stated
-- `source_memories`: the supporting facts behind this observation. May be partial or absent for large observations — the count above remains the true total. Each entry carries the same `text` and temporal fields as a new fact, plus:
-  - `context`: optional surrounding context for that fact"""
-
-# Stable description of the input shape. For the cached split path this lives in
-# the system prefix (build_consolidation_system_prompt) so it is not re-sent on
-# every batch; the per-batch user message then carries only the actual data.
-_INPUT_FORMAT_NOTE = f"""## INPUT FORMAT
-
-Each request provides new facts and existing observations. Every temporal field is optional and is omitted when unknown.
-
-### New facts
-
-{_FACT_FIELDS}
-
-### Existing observations
-
-A JSON array pooled from recalls across the new facts. Each entry has:
-{_OBSERVATION_FIELDS}"""
-
-# Per-batch data section for the cached split path — the stable format
-# explanation above is omitted here (it lives in the cached prefix); only the
-# variable facts/observations remain. Placeholders substituted at call time.
-_SPLIT_INPUT_SECTION = """## INPUT
+# Data section — format placeholders {facts_text} and {observations_text} are substituted at call time
+_INPUT_SECTION = """## INPUT
 
 ### New facts
 
@@ -93,6 +46,13 @@ _SPLIT_INPUT_SECTION = """## INPUT
 
 ### Existing observations
 
+JSON array, pooled from recalls across all new facts above. Each entry has:
+- `id`: unique identifier — copy this exactly when issuing an UPDATE or DELETE
+- `text`: the observation content
+- `proof_count`: number of supporting memories
+- `occurred_start` / `occurred_end`: temporal range of source facts
+- `source_memories`: array of supporting facts with their text and dates
+
 {observations_text}"""
 
 _DECISION_GUIDE = """## DECISION GUIDE
@@ -105,7 +65,7 @@ _DECISION_GUIDE = """## DECISION GUIDE
 # Output format — JSON braces escaped as {{ }} so .format() leaves them literal
 _OUTPUT_SECTION = """## OUTPUT FORMAT
 
-Return a JSON object with three arrays: `creates`, `updates`, `deletes`. Every entry must include a `reason`.
+Return a JSON object with three arrays: `creates`, `updates`, `deletes`.
 
 ### Example 1 — Merging recurring claims into an existing observation
 
@@ -119,7 +79,7 @@ Existing observation:
 Expected output (one UPDATE, no creates — both new facts are additional evidence for the same canonical decision):
 
 {{"creates": [],
-  "updates": [{{"text": "Donald named Athena's sovereignty as a foundational principle of the Janus architecture.", "observation_id": "11111111-1111-1111-1111-111111111111", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890", "b2c3d4e5-f6a7-8901-bcde-f12345678901"], "reason": "Both new facts restate the same sovereignty decision already captured by obs 1111 — merged as evidence rather than creating siblings."}}],
+  "updates": [{{"text": "Donald named Athena's sovereignty as a foundational principle of the Janus architecture.", "observation_id": "11111111-1111-1111-1111-111111111111", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890", "b2c3d4e5-f6a7-8901-bcde-f12345678901"]}}],
   "deletes": []}}
 
 ### Example 2 — State change updates one observation; unrelated fact creates a new one
@@ -133,8 +93,8 @@ Existing observation:
 
 Expected output (UPDATE for the state change; CREATE for the unrelated work-hours facet):
 
-{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"], "reason": "Work-hours is a distinct facet; no existing observation covers it, so CREATE."}}],
-  "updates": [{{"text": "Alice owned a 2019 Honda Civic; sold it on March 15, 2025.", "observation_id": "22222222-2222-2222-2222-222222222222", "source_fact_ids": ["c3d4e5f6-a7b8-9012-cdef-123456789012"], "reason": "State change to the existing Honda Civic observation 2222 — UPDATE, not a new sibling."}}],
+{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"]}}],
+  "updates": [{{"text": "Alice owned a 2019 Honda Civic; sold it on March 15, 2025.", "observation_id": "22222222-2222-2222-2222-222222222222", "source_fact_ids": ["c3d4e5f6-a7b8-9012-cdef-123456789012"]}}],
   "deletes": []}}
 
 ### Observation text rules
@@ -150,64 +110,38 @@ Expected output (UPDATE for the state change; CREATE for the unrelated work-hour
 - One create or update may reference multiple facts when they jointly support the observation.
 - **AT MOST ONE UPDATE PER `observation_id`**: if several new facts all update the same existing observation, emit a single `updates` entry that lists all contributing `source_fact_ids` and a single consolidated `text`. Never emit two `updates` entries with the same `observation_id` in one response — they would silently overwrite each other.
 - `deletes`: only when an observation is directly superseded or contradicted by new facts.
-- `reason`: REQUIRED on every create/update/delete — one sentence explaining the choice. For a CREATE, state which existing observation(s) you considered and why none matched (a near-identical existing observation means you should UPDATE, not CREATE). This is audited to catch duplicate creates.
 - Do NOT include `tags` — handled automatically.
 - Return `{{"creates": [], "updates": [], "deletes": []}}` if nothing durable is found."""
 
 
-def build_consolidation_system_prompt(
+def build_batch_consolidation_prompt(
+    observations_mission: str | None = None,
+    observation_capacity_note: str | None = None,
     llm_output_language: str | None = None,
 ) -> str:
-    """Bank-agnostic, cacheable system instruction for batch consolidation.
-
-    Holds only what is constant across banks: processing rules, input format,
-    decision guide, and output format. The bank's MISSION is deliberately NOT
-    here — baking it in would make the prefix bank-specific and force a separate
-    Gemini context cache per mission. The mission, the per-batch INPUT, and any
-    capacity constraint all ride in the user message (see
-    :func:`build_consolidation_input`), so this prefix is identical for every
-    bank and a single CachedContent serves them all. Returns final text
-    (brace-escaped examples already unescaped) for verbatim use as system message
-    and cached prefix.
-
-    ``llm_output_language`` picks between two mutually exclusive language rules:
-    unset keeps each observation in the language of its own source facts (the
-    default), set forces every observation into that one configured language.
     """
-    language_section = "" if llm_output_language else f"{_DEFAULT_LANGUAGE_RULE}\n\n"
-    template = (
+    Build the consolidation prompt for batch mode (multiple facts per LLM call).
+
+    The mission defines *what* to track (customisable per bank) and takes
+    priority over the built-in processing rules when the two conflict.
+    Processing rules, decision guide, and output format are always present.
+    When ``llm_output_language`` is set, observations are emitted in that
+    language.
+    """
+    mission = escape_for_prompt(observations_mission or _DEFAULT_MISSION)
+
+    capacity_section = ""
+    if observation_capacity_note:
+        capacity_section = f"\n\n## CAPACITY CONSTRAINT\n\n{escape_for_prompt(observation_capacity_note)}"
+
+    return (
         "You are a memory consolidation system. Synthesize new facts into "
         "observations, merging with existing observations when appropriate.\n\n"
-        f"{_MISSION_PRIORITY_NOTE}\n\n"
-        f"{language_section}"
+        f"## MISSION\n\n{mission}\n\n"
+        f"{_MISSION_PRIORITY_NOTE}"
+        f"{capacity_section}\n\n"
         f"{_PROCESSING_RULES}\n\n"
-        f"{_INPUT_FORMAT_NOTE}\n\n"
+        f"{_INPUT_SECTION}\n\n"
         f"{_DECISION_GUIDE}\n\n"
         f"{_OUTPUT_SECTION}" + output_language_directive(llm_output_language)
     )
-    # No {facts_text}/{observations_text} placeholders here — the only braces are
-    # the doubled {{ }} in the OUTPUT examples, which .format() unescapes.
-    return template.format()
-
-
-def build_consolidation_input(
-    facts_text: str,
-    observations_text: str,
-    observations_mission: str | None = None,
-    observation_capacity_note: str | None = None,
-) -> str:
-    """Per-batch user message: MISSION + INPUT data + any capacity constraint.
-
-    The MISSION lives here (not in the cached system prefix) so the prefix stays
-    bank-agnostic and one CachedContent serves every bank. The capacity note also
-    lives here since it varies as observation slots fill.
-    """
-    mission = escape_for_prompt(observations_mission or _DEFAULT_MISSION)
-    mission_section = f"## MISSION\n\n{mission}\n\n"
-    capacity_section = ""
-    if observation_capacity_note:
-        capacity_section = f"## CAPACITY CONSTRAINT\n\n{escape_for_prompt(observation_capacity_note)}\n\n"
-    # _SPLIT_INPUT_SECTION omits the stable observation-format explanation (now in
-    # the cached system prefix) — only the variable facts/observations remain.
-    template = mission_section + capacity_section + _SPLIT_INPUT_SECTION
-    return template.format(facts_text=facts_text, observations_text=observations_text)

hindsight-api-slim/hindsight_api/engine/cross_encoder.py

@@ -8,10 +8,10 @@ Configuration via environment variables - see hindsight_api.config for all env v
 
 import asyncio
 import logging
+import os
 import warnings
 from abc import ABC, abstractmethod
 from concurrent.futures import ThreadPoolExecutor
-from typing import Any
 
 import httpx
 
@@ -20,32 +20,86 @@ from ..config import (
     DEFAULT_RERANKER_ALIBABA_MODEL,
     DEFAULT_RERANKER_COHERE_MODEL,
     DEFAULT_RERANKER_FLASHRANK_CACHE_DIR,
+    DEFAULT_RERANKER_FLASHRANK_CPU_MEM_ARENA,
     DEFAULT_RERANKER_FLASHRANK_MODEL,
     DEFAULT_RERANKER_GOOGLE_MODEL,
     DEFAULT_RERANKER_LITELLM_MAX_TOKENS_PER_DOC,
     DEFAULT_RERANKER_LITELLM_MODEL,
     DEFAULT_RERANKER_LITELLM_SDK_MODEL,
     DEFAULT_RERANKER_LOCAL_BATCH_SIZE,
+    DEFAULT_RERANKER_LOCAL_FORCE_CPU,
+    DEFAULT_RERANKER_LOCAL_MAX_CONCURRENT,
     DEFAULT_RERANKER_LOCAL_MODEL,
+    DEFAULT_RERANKER_LOCAL_TRUST_REMOTE_CODE,
+    DEFAULT_RERANKER_PROVIDER,
     DEFAULT_RERANKER_SILICONFLOW_BASE_URL,
     DEFAULT_RERANKER_SILICONFLOW_MODEL,
     DEFAULT_RERANKER_TEI_BATCH_SIZE,
+    DEFAULT_RERANKER_TEI_HTTP_TIMEOUT,
     DEFAULT_RERANKER_TEI_MAX_CONCURRENT,
     DEFAULT_RERANKER_ZEROENTROPY_MODEL,
     DEFAULT_ZEROENTROPY_BASE_URL,
-    RerankerMemberConfig,
+    ENV_RERANKER_ALIBABA_API_KEY,
+    ENV_RERANKER_COHERE_API_KEY,
+    ENV_RERANKER_COHERE_MODEL,
+    ENV_RERANKER_FLASHRANK_CACHE_DIR,
+    ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA,
+    ENV_RERANKER_FLASHRANK_MODEL,
+    ENV_RERANKER_GOOGLE_PROJECT_ID,
+    ENV_RERANKER_LITELLM_SDK_API_KEY,
+    ENV_RERANKER_LOCAL_FORCE_CPU,
+    ENV_RERANKER_LOCAL_MAX_CONCURRENT,
+    ENV_RERANKER_LOCAL_MODEL,
+    ENV_RERANKER_LOCAL_TRUST_REMOTE_CODE,
+    ENV_RERANKER_PROVIDER,
+    ENV_RERANKER_SILICONFLOW_API_KEY,
+    ENV_RERANKER_TEI_BATCH_SIZE,
+    ENV_RERANKER_TEI_HTTP_TIMEOUT,
+    ENV_RERANKER_TEI_MAX_CONCURRENT,
+    ENV_RERANKER_TEI_URL,
+    ENV_RERANKER_ZEROENTROPY_API_KEY,
 )
-from .bank_attribution import reranker_bank_attribution_headers
-from .local_device import (
-    release_local_inference_memory,
-    resolve_model_device_type,
-    select_local_device,
-)
-from .tei_retry import tei_retry_delay
 
 logger = logging.getLogger(__name__)
 
 
+def _resolve_malloc_trim():
+    """Return a callable that asks glibc to release freed heap pages to the OS.
+
+    Local CPU rerankers (FlashRank/ONNX, SentenceTransformers/torch) allocate
+    large transient numpy/tensor buffers per call. On Linux glibc, those pages
+    are freed at the Python level but kept by the allocator as a high-water
+    mark — RSS grows monotonically across many recalls (see issue #1717).
+    Calling `malloc_trim(0)` after each batch returns those pages to the OS.
+
+    Resolved once at import; returns a no-op on non-glibc platforms (macOS,
+    musl, Windows) where the call is unavailable or unnecessary.
+    """
+    import sys
+
+    if sys.platform != "linux":
+        return lambda: None
+
+    import ctypes
+    import ctypes.util
+
+    libc_path = ctypes.util.find_library("c")
+    if libc_path is None:
+        return lambda: None
+    try:
+        libc = ctypes.CDLL(libc_path)
+        trim = libc.malloc_trim
+    except (OSError, AttributeError):
+        # Not glibc (musl has no malloc_trim) or libc lookup failed.
+        return lambda: None
+    trim.argtypes = [ctypes.c_size_t]
+    trim.restype = ctypes.c_int
+    return lambda: trim(0)
+
+
+_malloc_trim = _resolve_malloc_trim()
+
+
 class CrossEncoderModel(ABC):
     """
     Abstract base class for cross-encoder reranking.
@@ -59,15 +113,6 @@ class CrossEncoderModel(ABC):
         """Return a human-readable name for this provider (e.g., 'local', 'tei')."""
         pass
 
-    @property
-    def blocking_init(self) -> bool:
-        """Whether ``initialize()`` blocks the event loop (loads a model in-process).
-
-        Callers run those in a thread pool. Remote providers leave this False, and
-        so does :class:`MultiCrossEncoder` — it offloads its own members.
-        """
-        return False
-
     @abstractmethod
     async def initialize(self) -> None:
         """
@@ -119,7 +164,6 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         fp16: bool = False,
         bucket_batching: bool = False,
         batch_size: int = DEFAULT_RERANKER_LOCAL_BATCH_SIZE,
-        allow_mps: bool = False,
     ):
         """
         Initialize local SentenceTransformers cross-encoder.
@@ -141,9 +185,6 @@ class LocalSTCrossEncoder(CrossEncoderModel):
                             Default: False (opt-in via env var).
             batch_size: Batch size for predict() calls. Optimal values vary by
                        hardware and model (MPS: 32, CUDA: 128+). Default: 32.
-            allow_mps: Opt in to the Apple Silicon MPS GPU. Disabled by default
-                      because MPS leaks memory under variable-length workloads
-                      (see engine/local_device.py). Default: False
         """
         self.model_name = model_name or DEFAULT_RERANKER_LOCAL_MODEL
         self.force_cpu = force_cpu
@@ -151,19 +192,13 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         self.fp16 = fp16
         self.bucket_batching = bucket_batching
         self.batch_size = batch_size
-        self.allow_mps = allow_mps
         self._model = None
-        self._device_type: str = "cpu"
         LocalSTCrossEncoder._max_concurrent = max_concurrent
 
     @property
     def provider_name(self) -> str:
         return "local"
 
-    @property
-    def blocking_init(self) -> bool:
-        return True
-
     async def initialize(self) -> None:
         """Load the cross-encoder model and initialize the executor."""
         if self._model is not None:
@@ -179,13 +214,30 @@ class LocalSTCrossEncoder(CrossEncoderModel):
 
         logger.info(f"Reranker: initializing local provider with model {self.model_name}")
 
-        # Determine device based on hardware availability. We always set
-        # low_cpu_mem_usage=False to prevent lazy loading (meta tensors) which can
-        # cause issues when accelerate is installed but no GPU is available.
+        # Determine device based on hardware availability.
+        # We always set low_cpu_mem_usage=False to prevent lazy loading (meta tensors)
+        # which can cause issues when accelerate is installed but no GPU is available.
         # Note: We do NOT use device_map because CrossEncoder internally calls .to(device)
         # after loading, which conflicts with accelerate's device_map handling.
-        # MPS is opt-in (allow_mps) — see engine/local_device.py for why.
-        device = select_local_device(self.force_cpu, self.allow_mps)
+        import torch
+
+        # Force CPU mode if configured (used in daemon mode to avoid MPS/XPC issues on macOS)
+        if self.force_cpu:
+            device = "cpu"
+            logger.info("Reranker: forcing CPU mode (HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1)")
+        else:
+            # Check for GPU (CUDA) or Apple Silicon (MPS)
+            # Wrap in try-except to gracefully handle any device detection issues
+            # (e.g., in CI environments or when PyTorch is built without GPU support)
+            device = "cpu"  # Default to CPU
+            try:
+                has_gpu = torch.cuda.is_available() or (
+                    hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+                )
+                if has_gpu:
+                    device = None  # Let sentence-transformers auto-detect GPU/MPS
+            except Exception as e:
+                logger.warning(f"Failed to detect GPU/MPS, falling back to CPU: {e}")
 
         # Patch transformers 5.x compatibility for models using XLM-RoBERTa
         # (e.g., jina-reranker-v2-base-multilingual). transformers 5.x removed
@@ -229,11 +281,9 @@ class LocalSTCrossEncoder(CrossEncoderModel):
                 # Restore original logging level
                 transformers_logger.setLevel(original_level)
 
-        self._device_type = resolve_model_device_type(self._model)
-
         # FP16 inference: convert model weights to half precision.
         # Empirically validated: 27-36% faster on MPS, quality-identical (20/20 overlap).
-        if self.fp16 and self._device_type != "cpu":
+        if self.fp16 and device != "cpu":
             self._model.model.half()
             logger.info("Reranker: FP16 inference enabled")
 
@@ -254,6 +304,7 @@ class LocalSTCrossEncoder(CrossEncoderModel):
         - bucket_batching: sort pairs by token length to reduce padding waste (36-54% speedup)
         - batch_size: explicit batch size for predict() calls (MPS optimal: 32)
         """
+        import numpy as np
 
         try:
             if self.bucket_batching and len(pairs) > 1:
@@ -276,7 +327,7 @@ class LocalSTCrossEncoder(CrossEncoderModel):
             scores = self._model.predict(pairs, batch_size=self.batch_size, show_progress_bar=False)
             return scores.tolist() if hasattr(scores, "tolist") else list(scores)
         finally:
-            release_local_inference_memory(self._device_type)
+            _malloc_trim()
 
     async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
         """
@@ -392,20 +443,14 @@ class RemoteTEICrossEncoder(CrossEncoderModel):
                         await asyncio.sleep(delay)
                         delay *= 2  # Exponential backoff
                 except httpx.HTTPStatusError as e:
-                    # TEI uses 429 as normal overload backpressure. Retry it with
-                    # the same bounded budget as transient server errors.
-                    if (e.response.status_code == 429 or e.response.status_code >= 500) and attempt < self.max_retries:
+                    # Retry on 5xx server errors
+                    if e.response.status_code >= 500 and attempt < self.max_retries:
                         last_error = e
-                        sleep_delay = tei_retry_delay(
-                            e.response,
-                            delay,
-                            request_timeout=self.timeout,
-                        )
                         logger.warning(
-                            f"TEI transient error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
-                            f"Retrying in {sleep_delay:.2f}s..."
+                            f"TEI server error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
+                            f"Retrying in {delay}s..."
                         )
-                        await asyncio.sleep(sleep_delay)
+                        await asyncio.sleep(delay)
                         delay *= 2
                     else:
                         raise
@@ -451,7 +496,6 @@ class RemoteTEICrossEncoder(CrossEncoderModel):
                 semaphore,
                 "POST",
                 f"{self.base_url}/rerank",
-                headers=reranker_bank_attribution_headers(),
                 json={
                     "query": query,
                     "texts": texts,
@@ -592,11 +636,7 @@ class _CohereCompatibleRerankClient:
             if self.include_top_n:
                 body["top_n"] = len(texts)
 
-            response = await self._async_client.post(
-                self.rerank_url,
-                headers=reranker_bank_attribution_headers(),
-                json=body,
-            )
+            response = await self._async_client.post(self.rerank_url, json=body)
             response.raise_for_status()
             result = response.json()
 
@@ -891,7 +931,6 @@ class FlashRankCrossEncoder(CrossEncoderModel):
         self.max_length = max_length
         self.cpu_mem_arena = cpu_mem_arena
         self._ranker = None
-        self._device_type: str = "cpu"  # FlashRank runs on CPU via ONNX Runtime
         FlashRankCrossEncoder._max_concurrent = max_concurrent
 
     @property
@@ -963,11 +1002,11 @@ class FlashRankCrossEncoder(CrossEncoderModel):
 
     def _predict_sync(self, pairs: list[tuple[str, str]]) -> list[float]:
         """Synchronous predict - processes each query group."""
+        from flashrank import RerankRequest
+
         if not pairs:
             return []
 
-        from flashrank import RerankRequest
-
         try:
             # Group pairs by query
             query_groups: dict[str, list[tuple[int, str]]] = {}
@@ -996,7 +1035,7 @@ class FlashRankCrossEncoder(CrossEncoderModel):
 
             return all_scores
         finally:
-            release_local_inference_memory(self._device_type)
+            _malloc_trim()
 
     async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
         """
@@ -1124,7 +1163,6 @@ class LiteLLMCrossEncoder(CrossEncoderModel):
             # LiteLLM /rerank follows Cohere API format
             response = await self._async_client.post(
                 f"{self.api_base}/rerank",
-                headers=reranker_bank_attribution_headers(),
                 json={
                     "model": self.model,
                     "query": query,
@@ -1161,7 +1199,7 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):
 
     def __init__(
         self,
-        api_key: str | None = None,
+        api_key: str,
         model: str = DEFAULT_RERANKER_LITELLM_SDK_MODEL,
         api_base: str | None = None,
         timeout: float = 60.0,
@@ -1171,8 +1209,7 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):
         Initialize LiteLLM SDK cross-encoder client.
 
         Args:
-            api_key: API key for the reranking provider (optional — omit for
-                     providers that use ambient credentials, e.g. AWS Bedrock with IAM)
+            api_key: API key for the reranking provider
             model: Model name with provider prefix (e.g., "deepinfra/Qwen3-reranker-8B")
             api_base: Custom base URL for API (optional)
             timeout: Request timeout in seconds (default: 60.0)
@@ -1243,22 +1280,32 @@ class LiteLLMSDKCrossEncoder(CrossEncoderModel):
             indices = [idx for idx, _ in indexed_texts]
 
             # Build kwargs for rerank call
-            rerank_kwargs: dict[str, Any] = {
+            rerank_kwargs = {
                 "model": self.model,
                 "query": query,
                 "documents": texts,
-                "headers": reranker_bank_attribution_headers(),
+                "api_key": self.api_key,
             }
-            if self.api_key:
-                rerank_kwargs["api_key"] = self.api_key
             if self.api_base:
                 rerank_kwargs["api_base"] = self.api_base
 
             response = await self._litellm.arerank(**rerank_kwargs)
 
-            for result in response.results:
-                original_idx = result["index"]
-                all_scores[indices[original_idx]] = result["relevance_score"]
+            # Map scores back to original positions
+            # Response format: RerankResponse with results list
+            # Each result is a TypedDict with "index" and "relevance_score"
+            if hasattr(response, "results") and response.results:
+                for result in response.results:
+                    # Results are TypedDicts, use dict-style access
+                    original_idx = result["index"]
+                    score = result.get("relevance_score", result.get("score", 0.0))
+                    all_scores[indices[original_idx]] = score
+            elif isinstance(response, list):
+                # Direct list of scores (unlikely but defensive)
+                for i, score in enumerate(response):
+                    all_scores[indices[i]] = score
+            else:
+                logger.warning(f"Unexpected response format from LiteLLM rerank: {type(response)}")
 
         return all_scores
 
@@ -1577,246 +1624,136 @@ class AlibabaCloudCrossEncoder(CrossEncoderModel):
         return await self._client.predict(pairs)
 
 
-class MultiCrossEncoder(CrossEncoderModel):
-    """Failover across an ordered chain of cross-encoders.
-
-    Member 0 is the primary (the unindexed ``HINDSIGHT_API_RERANKER_*`` config);
-    members 1..N are the indexed fallbacks. Each ``predict`` tries members in order
-    and returns the first usable set of scores, so an unreachable reranker costs
-    ranking quality (whatever the next member gives) instead of the whole recall.
-    Put ``rrf`` last to degrade to the fusion order rather than failing.
-
-    Each member keeps its own retry budget, so we only advance after a member has
-    exhausted its retries and raised. A member that fails to initialize is not
-    fatal — that is the point of the chain — it is retried lazily on the next
-    request that reaches it.
+def create_cross_encoder_from_env() -> CrossEncoderModel:
     """
+    Create a CrossEncoderModel instance based on configuration.
 
-    def __init__(self, members: list[CrossEncoderModel]) -> None:
-        if len(members) < 2:
-            raise ValueError("MultiCrossEncoder requires at least two members")
-        self._members = members
-        self._ready = [False] * len(members)
-        self._locks = [asyncio.Lock() for _ in members]
-        self._active = 0
-
-    @property
-    def provider_name(self) -> str:
-        """The provider of the member that last served a request (primary before any).
-
-        Callers use this to detect a passthrough reranker, so it has to track the
-        member actually serving rather than name the chain: a chain that has
-        degraded to its ``rrf`` member is passthrough. Concurrent requests share it,
-        so a request that fails over can briefly mislabel a neighbour — this only
-        tunes downstream scoring, never correctness.
-        """
-        return self._members[self._active].provider_name
-
-    async def _initialize_member(self, index: int) -> None:
-        """Initialize one member, off the event loop when it loads a model in-process."""
-        member = self._members[index]
-        if member.blocking_init:
-            loop = asyncio.get_running_loop()
-            await loop.run_in_executor(None, lambda: asyncio.run(member.initialize()))
-        else:
-            await member.initialize()
-        self._ready[index] = True
-
-    async def _ensure_member_ready(self, index: int) -> None:
-        async with self._locks[index]:
-            if not self._ready[index]:
-                await self._initialize_member(index)
-
-    async def initialize(self) -> None:
-        """Initialize every member, tolerating members that are down.
-
-        Members initialize concurrently so one unreachable member cannot eat the
-        startup budget the others need. Failures are logged and retried on use.
-        """
-        results = await asyncio.gather(
-            *(self._ensure_member_ready(i) for i in range(len(self._members))),
-            return_exceptions=True,
-        )
-        for index, result in enumerate(results):
-            if isinstance(result, BaseException):
-                logger.warning(
-                    "Reranker member %d (%s) failed to initialize: %s; it will be retried on use",
-                    index,
-                    self._members[index].provider_name,
-                    result,
-                )
-        if not any(self._ready):
-            logger.error("Reranker: no member of the failover chain initialized; recall will retry them per request")
-
-    async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
-        """Score ``pairs`` with the first member that answers usably."""
-        last_exc: BaseException | None = None
-        for index, member in enumerate(self._members):
-            try:
-                if not self._ready[index]:
-                    await self._ensure_member_ready(index)
-                scores = await member.predict(pairs)
-                if len(scores) != len(pairs):
-                    raise RuntimeError(f"returned {len(scores)} scores for {len(pairs)} pairs")
-            except Exception as e:  # noqa: BLE001 - re-raised below if no member answers
-                last_exc = e
-                remaining = len(self._members) - index - 1
-                logger.warning(
-                    "Reranker member %d (%s) failed: %s%s",
-                    index,
-                    member.provider_name,
-                    e,
-                    f"; trying next member ({remaining} left)" if remaining else "; no members left",
-                )
-                continue
-            if index != self._active:
-                logger.info(
-                    "Reranker: now serving from member %d (%s)",
-                    index,
-                    member.provider_name,
-                )
-            self._active = index
-            return scores
-        # All members failed; surface the last error (loop ran at least once).
-        assert last_exc is not None
-        raise last_exc
-
-
-def create_cross_encoder(member: RerankerMemberConfig) -> CrossEncoderModel:
-    """
-    Create a CrossEncoderModel for one member of the reranker chain.
-
-    ``member`` is the primary (index 0, the unindexed ``HINDSIGHT_API_RERANKER_*``
-    config) or an indexed fallback. Missing-setting errors name the member's own
-    env var, so a chain misconfiguration points at the exact indexed variable.
-
-    Args:
-        member: Resolved settings for this member
+    Reads configuration via get_config() to ensure consistency across the codebase.
 
     Returns:
         Configured CrossEncoderModel instance
     """
-    provider = member.provider.lower()
+    from ..config import get_config
+
+    config = get_config()
+    provider = config.reranker_provider.lower()
 
     if provider == "tei":
-        url = member.tei_url
+        url = config.reranker_tei_url
         if not url:
-            raise ValueError(f"{member.env_name('TEI_URL')} is required when {member.env_name('PROVIDER')} is 'tei'")
+            raise ValueError(f"{ENV_RERANKER_TEI_URL} is required when {ENV_RERANKER_PROVIDER} is 'tei'")
         return RemoteTEICrossEncoder(
             base_url=url,
-            timeout=member.tei_http_timeout,
-            batch_size=member.tei_batch_size,
-            max_concurrent=member.tei_max_concurrent,
+            timeout=config.reranker_tei_http_timeout,
+            batch_size=config.reranker_tei_batch_size,
+            max_concurrent=config.reranker_tei_max_concurrent,
         )
     elif provider == "local":
         return LocalSTCrossEncoder(
-            model_name=member.local_model,
-            max_concurrent=member.local_max_concurrent,
-            force_cpu=member.local_force_cpu,
-            trust_remote_code=member.local_trust_remote_code,
-            fp16=member.local_fp16,
-            bucket_batching=member.local_bucket_batching,
-            batch_size=member.local_batch_size,
-            allow_mps=member.local_allow_mps,
+            model_name=config.reranker_local_model,
+            max_concurrent=config.reranker_local_max_concurrent,
+            force_cpu=config.reranker_local_force_cpu,
+            trust_remote_code=config.reranker_local_trust_remote_code,
+            fp16=config.reranker_local_fp16,
+            bucket_batching=config.reranker_local_bucket_batching,
+            batch_size=config.reranker_local_batch_size,
         )
     elif provider == "cohere":
-        api_key = member.cohere_api_key
+        api_key = config.reranker_cohere_api_key
         if not api_key:
-            raise ValueError(
-                f"{member.env_name('COHERE_API_KEY')} is required when {member.env_name('PROVIDER')} is 'cohere'"
-            )
+            raise ValueError(f"{ENV_RERANKER_COHERE_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'cohere'")
         return CohereCrossEncoder(
             api_key=api_key,
-            model=member.cohere_model,
-            base_url=member.cohere_base_url,
-            timeout=member.cohere_timeout,
+            model=config.reranker_cohere_model,
+            base_url=config.reranker_cohere_base_url,
+            timeout=config.reranker_cohere_timeout,
         )
     elif provider == "openrouter":
-        api_key = member.openrouter_api_key
+        api_key = config.reranker_openrouter_api_key
         if not api_key:
-            shared = ", HINDSIGHT_API_OPENROUTER_API_KEY, or HINDSIGHT_API_LLM_API_KEY" if member.index == 0 else ""
             raise ValueError(
-                f"{member.env_name('OPENROUTER_API_KEY')}{shared} is required "
-                f"when {member.env_name('PROVIDER')} is 'openrouter'"
+                "HINDSIGHT_API_RERANKER_OPENROUTER_API_KEY, HINDSIGHT_API_OPENROUTER_API_KEY, "
+                f"or HINDSIGHT_API_LLM_API_KEY is required when {ENV_RERANKER_PROVIDER} is 'openrouter'"
             )
         return CohereCrossEncoder(
             api_key=api_key,
-            model=member.openrouter_model,
-            base_url=member.openrouter_base_url,
-            timeout=member.openrouter_timeout,
+            model=config.reranker_openrouter_model,
+            base_url="https://openrouter.ai/api/v1/rerank",
+            timeout=config.reranker_openrouter_timeout,
         )
     elif provider == "flashrank":
-        return FlashRankCrossEncoder(
-            model_name=member.flashrank_model,
-            cache_dir=member.flashrank_cache_dir,
-            cpu_mem_arena=member.flashrank_cpu_mem_arena,
-        )
+        model = os.environ.get(ENV_RERANKER_FLASHRANK_MODEL, DEFAULT_RERANKER_FLASHRANK_MODEL)
+        cache_dir = os.environ.get(ENV_RERANKER_FLASHRANK_CACHE_DIR, DEFAULT_RERANKER_FLASHRANK_CACHE_DIR)
+        cpu_mem_arena = os.environ.get(
+            ENV_RERANKER_FLASHRANK_CPU_MEM_ARENA, str(DEFAULT_RERANKER_FLASHRANK_CPU_MEM_ARENA)
+        ).lower() in ("true", "1", "yes")
+        return FlashRankCrossEncoder(model_name=model, cache_dir=cache_dir, cpu_mem_arena=cpu_mem_arena)
     elif provider == "litellm":
         return LiteLLMCrossEncoder(
-            api_base=member.litellm_api_base,
-            api_key=member.litellm_api_key,
-            model=member.litellm_model,
-            max_tokens_per_doc=member.litellm_max_tokens_per_doc,
-            timeout=member.litellm_timeout,
+            api_base=config.reranker_litellm_api_base,
+            api_key=config.reranker_litellm_api_key,
+            model=config.reranker_litellm_model,
+            max_tokens_per_doc=config.reranker_litellm_max_tokens_per_doc,
+            timeout=config.reranker_litellm_timeout,
         )
     elif provider == "litellm-sdk":
-        return LiteLLMSDKCrossEncoder(
-            api_key=member.litellm_sdk_api_key or None,
-            model=member.litellm_sdk_model,
-            api_base=member.litellm_sdk_api_base,
-            max_tokens_per_doc=member.litellm_max_tokens_per_doc,
-            timeout=member.litellm_sdk_timeout,
-        )
-    elif provider == "zeroentropy":
-        api_key = member.zeroentropy_api_key
+        api_key = config.reranker_litellm_sdk_api_key
         if not api_key:
             raise ValueError(
-                f"{member.env_name('ZEROENTROPY_API_KEY')} is required "
-                f"when {member.env_name('PROVIDER')} is 'zeroentropy'"
+                f"{ENV_RERANKER_LITELLM_SDK_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'litellm-sdk'"
+            )
+        return LiteLLMSDKCrossEncoder(
+            api_key=api_key,
+            model=config.reranker_litellm_sdk_model,
+            api_base=config.reranker_litellm_sdk_api_base,
+            max_tokens_per_doc=config.reranker_litellm_max_tokens_per_doc,
+            timeout=config.reranker_litellm_sdk_timeout,
+        )
+    elif provider == "zeroentropy":
+        api_key = config.reranker_zeroentropy_api_key
+        if not api_key:
+            raise ValueError(
+                f"{ENV_RERANKER_ZEROENTROPY_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'zeroentropy'"
             )
         return ZeroEntropyCrossEncoder(
             api_key=api_key,
-            model=member.zeroentropy_model,
-            base_url=member.zeroentropy_base_url,
-            timeout=member.zeroentropy_timeout,
+            model=config.reranker_zeroentropy_model,
+            base_url=config.reranker_zeroentropy_base_url,
+            timeout=config.reranker_zeroentropy_timeout,
         )
     elif provider == "siliconflow":
-        api_key = member.siliconflow_api_key
+        api_key = config.reranker_siliconflow_api_key
         if not api_key:
             raise ValueError(
-                f"{member.env_name('SILICONFLOW_API_KEY')} is required "
-                f"when {member.env_name('PROVIDER')} is 'siliconflow'"
+                f"{ENV_RERANKER_SILICONFLOW_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'siliconflow'"
             )
         return SiliconFlowCrossEncoder(
             api_key=api_key,
-            model=member.siliconflow_model,
-            base_url=member.siliconflow_base_url,
-            timeout=member.siliconflow_timeout,
+            model=config.reranker_siliconflow_model,
+            base_url=config.reranker_siliconflow_base_url,
+            timeout=config.reranker_siliconflow_timeout,
         )
     elif provider == "google":
-        project_id = member.google_project_id
+        project_id = config.reranker_google_project_id
         if not project_id:
-            shared = " (or HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID)" if member.index == 0 else ""
             raise ValueError(
-                f"{member.env_name('GOOGLE_PROJECT_ID')}{shared} "
-                f"is required when {member.env_name('PROVIDER')} is 'google'"
+                f"{ENV_RERANKER_GOOGLE_PROJECT_ID} (or HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID) "
+                f"is required when {ENV_RERANKER_PROVIDER} is 'google'"
             )
         return GoogleCrossEncoder(
             project_id=project_id,
-            model=member.google_model,
-            service_account_key=member.google_service_account_key,
-            timeout=member.google_timeout,
+            model=config.reranker_google_model,
+            service_account_key=config.reranker_google_service_account_key,
+            timeout=config.reranker_google_timeout,
         )
     elif provider == "alibaba":
-        api_key = member.alibaba_api_key
+        api_key = config.reranker_alibaba_api_key
         if not api_key:
-            raise ValueError(
-                f"{member.env_name('ALIBABA_API_KEY')} is required when {member.env_name('PROVIDER')} is 'alibaba'"
-            )
+            raise ValueError(f"{ENV_RERANKER_ALIBABA_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'alibaba'")
         return AlibabaCloudCrossEncoder(
             api_key=api_key,
-            model=member.alibaba_model,
-            timeout=member.alibaba_timeout,
+            model=config.reranker_alibaba_model,
+            timeout=config.reranker_alibaba_timeout,
         )
     elif provider == "rrf":
         return RRFPassthroughCrossEncoder()
@@ -1826,23 +1763,3 @@ def create_cross_encoder(member: RerankerMemberConfig) -> CrossEncoderModel:
         raise ValueError(
             f"Unknown reranker provider: {provider}. Supported: 'local', 'tei', 'cohere', 'zeroentropy', 'siliconflow', 'alibaba', 'google', 'flashrank', 'litellm', 'litellm-sdk', 'rrf', 'jina-mlx'"
         )
-
-
-def create_cross_encoder_from_env() -> CrossEncoderModel:
-    """
-    Create the configured reranker, based on configuration.
-
-    Reads configuration via get_config() to ensure consistency across the codebase.
-    With no ``HINDSIGHT_API_RERANKER_<n>_*`` members configured (the default) this
-    is the single configured reranker; otherwise the chain is wrapped in a
-    :class:`MultiCrossEncoder` that fails over across members in order.
-
-    Returns:
-        Configured CrossEncoderModel instance
-    """
-    from ..config import get_config
-
-    chain = get_config().reranker_chain()
-    if len(chain) == 1:
-        return create_cross_encoder(chain[0])
-    return MultiCrossEncoder([create_cross_encoder(member) for member in chain])

hindsight-api-slim/hindsight_api/engine/db/__init__.py

@@ -67,28 +67,16 @@ def create_database_backend(backend_type: str) -> DatabaseBackend:
     return _get_backend_class(backend_type)()
 
 
-_OPS_CACHE: dict[str, DataAccessOps] = {}
-
-
 def create_data_access_ops(backend_type: str) -> DataAccessOps:
-    """Factory: the DataAccessOps for a backend name.
-
-    Returns a per-dialect SINGLETON: ``DataAccessOps`` is stateless (it only builds and runs SQL),
-    so one shared instance per dialect is correct — and it means the database backend and the
-    memories store hold the *same* ops object, so a test that patches a method on it (e.g.
-    ``enqueue_graph_maintenance``) observes every caller regardless of which layer issued it.
+    """Factory: create a DataAccessOps by backend name.
 
     Args:
         backend_type: One of "postgresql" or "oracle".
 
     Returns:
-        The shared DataAccessOps instance for that backend.
+        A DataAccessOps instance.
 
     Raises:
         ValueError: If backend_type is not recognized.
     """
-    ops = _OPS_CACHE.get(backend_type)
-    if ops is None:
-        ops = _get_ops_class(backend_type)()
-        _OPS_CACHE[backend_type] = ops
-    return ops
+    return _get_ops_class(backend_type)()

hindsight-api-slim/hindsight_api/engine/db/base.py

@@ -112,23 +112,6 @@ class DatabaseConnection(ABC):
         """
         ...
 
-    async def execute_rows_affected(self, query: str, *args: Any, timeout: float | None = None) -> int:
-        """Execute a DML statement and return the number of rows it affected.
-
-        Normalizes the dialect-specific execute result into a plain int so callers
-        never hand-parse an ``"UPDATE <n>"`` / ``"DELETE <n>"`` command tag in
-        business logic (mirrors ``parse_json`` above, which normalizes the other
-        dialect-divergent result shape). asyncpg returns the tag directly; the
-        Oracle connection reshapes ``cursor.rowcount`` into the same trailing-count
-        form, so parsing the last token is dialect-safe. Returns 0 when the status
-        has no trailing count (e.g. a non-DML statement).
-        """
-        status = await self.execute(query, *args, timeout=timeout)
-        if not isinstance(status, str):
-            return 0
-        parts = status.split()
-        return int(parts[-1]) if parts and parts[-1].isdigit() else 0
-
     @abstractmethod
     async def executemany(self, query: str, args: list[tuple[Any, ...]], *, timeout: float | None = None) -> None:
         """Execute a query for each set of arguments.
@@ -324,17 +307,6 @@ class DatabaseBackend(ABC):
         """Close the connection pool and release all resources."""
         ...
 
-    @property
-    @abstractmethod
-    def is_ready(self) -> bool:
-        """Whether the pool exists and can serve connections.
-
-        False before :meth:`initialize` and after :meth:`shutdown`. Best-effort
-        callers (tracing, auditing) check this to skip work during those windows
-        instead of acquiring and interpreting the resulting error.
-        """
-        ...
-
     @abstractmethod
     @asynccontextmanager
     async def acquire(self) -> AsyncIterator[DatabaseConnection]:

hindsight-api-slim/hindsight_api/engine/db/ops.py

@@ -18,75 +18,13 @@ and mirrors Django's ``DatabaseOperations`` architecture.
 
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
-from datetime import datetime
 from typing import Any
+from uuid import UUID
 
 from .base import DatabaseConnection
 from .result import ResultRow
 
 
-def graph_maintenance_bank_serialization_sql(table: str, alias: str) -> str:
-    """SQL predicate serialising ``graph_maintenance`` claims per bank (#3230).
-
-    Every graph_maintenance run is the same bank-wide sweep — the payload carries
-    only ``bank_id``, and ``run_graph_maintenance_job`` drains the whole queue —
-    so a second concurrent run for one bank adds no work. It is worse than
-    useless: ``claim_graph_maintenance_batch`` locks queue rows ``FOR UPDATE``
-    *without* ``SKIP LOCKED`` (it is written assuming a single runner per bank),
-    so the runs convoy on each other's row locks while each holds a worker slot.
-
-    Same guarantee ``consolidation`` already gets from its ``bank_id != ALL(busy)``
-    exclusion, and the same caveat: a row wedged in 'processing' holds its bank
-    until something releases it (``hindsight-admin recover``, or a restart with a
-    stable ``HINDSIGHT_API_WORKER_ID`` so ``recover_own_tasks`` matches it). That
-    is a general gap in claim recovery, not specific to graph_maintenance.
-
-    Two differences from the consolidation form, both forced by the shape of this
-    problem:
-
-    * It is a **predicate**, not a separate claim phase. Pulling graph_maintenance
-      into its own phase after the generic shared-pool query would drop it below
-      every other operation type: it has no reserved-slot floor
-      (``WORKER_SLOT_TYPE_DEFAULTS`` gives consolidation 2 and graph_maintenance
-      0), and the poller's fairness pass calls ``claim_tasks`` with
-      ``shared_limit=1``, so a single pending retain would starve it indefinitely.
-      As a predicate it keeps competing by ``created_at``.
-    * It also suppresses every same-bank row but the oldest **within one batch**.
-      Excluding busy banks alone does not: with several pending rows and nothing
-      yet processing, one batch claims them all — the convoy, unchanged. Several
-      pending rows per bank are reachable through the recovery paths
-      (``_reclaim_own_processing_tasks`` resets *all* of a worker's processing
-      rows in one statement, from ``recover_own_tasks`` at startup and
-      ``release_own_tasks`` at shutdown, plus ``_schedule_retry`` /
-      ``_defer_operation`` / ``hindsight-admin recover``).
-
-    The candidate row is always 'pending' and the 'pending' branch is
-    strictly-older, so the subquery can never match the candidate itself. The
-    fragment carries no SQL comments on purpose — it is rewritten for Oracle by
-    regex (``db/oracle.py``).
-
-    Args:
-        table: Fully-qualified async_operations table.
-        alias: Alias of the outer candidate row in the calling query.
-    """
-    return f"""
-        ({alias}.operation_type <> 'graph_maintenance' OR NOT EXISTS (
-            SELECT 1 FROM {table} gm_peer
-            WHERE gm_peer.bank_id = {alias}.bank_id
-              AND gm_peer.operation_type = 'graph_maintenance'
-              AND (
-                  gm_peer.status = 'processing'
-                  OR (gm_peer.status = 'pending'
-                      AND gm_peer.task_payload IS NOT NULL
-                      AND (gm_peer.next_retry_at IS NULL OR gm_peer.next_retry_at <= NOW())
-                      AND (gm_peer.created_at < {alias}.created_at
-                           OR (gm_peer.created_at = {alias}.created_at
-                               AND gm_peer.operation_id < {alias}.operation_id)))
-              )
-        ))
-    """
-
-
 @dataclass
 class TagListingParts:
     """Backend-specific SQL fragments for the tag listing query."""
@@ -97,57 +35,6 @@ class TagListingParts:
     bank_prefix: str
 
 
-@dataclass(frozen=True)
-class UpdatedWindow:
-    """Recall's ``created_after``/``created_before`` bounds, as SQL for graph expansion.
-
-    Recall applies the window to ``updated_at`` — a consolidation touch makes a
-    fact current again — so link expansion has to bound the same column its seed
-    query does.  Filtering only the seeds is not enough: a single in-window seed
-    would otherwise drag its whole neighbourhood (shared entities, semantic kNN
-    links, causal links) into the results no matter how old those neighbours are.
-
-    ``first_param_index`` is where the bounds land in the owning query's param
-    list, so each call site keeps the placeholder numbering next to the params it
-    binds.  Rendering is per-alias because the same window is applied to several
-    correlation names within one query.
-    """
-
-    after: datetime | None
-    before: datetime | None
-    first_param_index: int
-
-    def clause(self, alias: str) -> str:
-        """``AND <alias>.updated_at > $n ...`` — empty when the window is unbounded."""
-        parts: list[str] = []
-        index = self.first_param_index
-        if self.after is not None:
-            parts.append(f" AND {alias}.updated_at > ${index}")
-            index += 1
-        if self.before is not None:
-            parts.append(f" AND {alias}.updated_at < ${index}")
-        return "".join(parts)
-
-    @property
-    def params(self) -> list[datetime]:
-        """The bound values, in placeholder order. Append to the owning param list."""
-        return [bound for bound in (self.after, self.before) if bound is not None]
-
-
-@dataclass(frozen=True)
-class LinkExpansionRows:
-    """The three link-expansion signals, kept apart until they are scored.
-
-    They cannot be concatenated at the SQL layer: each carries a different score
-    scale (shared-entity count, kNN weight, causal weight) and the caller applies
-    a different transformation to each before summing them.
-    """
-
-    entity: list[ResultRow]
-    semantic: list[ResultRow]
-    causal: list[ResultRow]
-
-
 class DataAccessOps(ABC):
     """Backend-specific multi-statement data access operations.
 
@@ -185,30 +72,6 @@ class DataAccessOps(ABC):
         """
         ...
 
-    @abstractmethod
-    async def lock_document_for_write(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        doc_id: str,
-        bank_id: str,
-    ) -> str | None:
-        """Ensure the document row exists, take a row lock on it, and return its
-        pre-existing ``content_hash``.
-
-        This serializes all concurrent writers for ``doc_id`` at the DB level
-        (so interleaved same-document retains can't corrupt each other), while
-        creating the row on first write. The returned hash is ``'__pending__'``
-        for a freshly inserted row, the stored hash for an existing one, or
-        ``None`` if the row could not be read back.
-
-        PG does this in a single statement (``INSERT ... ON CONFLICT DO UPDATE
-        ... RETURNING``), which always takes the row lock as part of the upsert.
-        Oracle can't (``MERGE`` doesn't support ``RETURNING``), so it splits the
-        work into an idempotent insert plus a ``SELECT ... FOR UPDATE``.
-        """
-        ...
-
     @abstractmethod
     async def insert_facts_batch(
         self,
@@ -263,14 +126,9 @@ class DataAccessOps(ABC):
         bank_id: str,
         entity_names: list[str],
         entity_dates: list,
-        entity_kinds: list[str],
     ) -> dict[str, str]:
         """Bulk insert entities with ON CONFLICT DO NOTHING, returning id-by-lowercase-name.
 
-        ``entity_kinds`` ("regular"/"label", parallel to ``entity_names``) is
-        stored on the row so label entities stay out of the partial trigram
-        index (#3208).
-
         PG uses INSERT ... SELECT FROM unnest() with RETURNING.
         Non-PG inserts row-by-row then SELECTs.
         """
@@ -291,26 +149,6 @@ class DataAccessOps(ABC):
         """
         ...
 
-    @abstractmethod
-    async def bulk_reassert_entities(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        bank_id: str,
-        entity_ids: list[str],
-        canonical_names: list[str],
-        entity_kinds: list[str],
-    ) -> None:
-        """Lock resolved parents and re-create any pruned since Phase-1 resolution.
-
-        Closes the retain Phase-1/prune race (#2662): existing rows are locked
-        (PG ``FOR KEY SHARE`` / Oracle ``FOR UPDATE``) so a concurrent
-        ``prune_orphan_entities`` blocks until the caller's transaction commits,
-        while rows already deleted are re-inserted idempotently. ``entity_ids``
-        must be sorted by the caller for a stable lock order.
-        """
-        ...
-
     @abstractmethod
     async def bulk_insert_unit_entities(
         self,
@@ -369,16 +207,12 @@ class DataAccessOps(ABC):
         mu_table: str,
         ue_table: str,
         per_entity_limit: int,
-        window: UpdatedWindow,
     ) -> str:
         """Build entity expansion CTE for link expansion retrieval.
 
         PG uses DISTINCT ON with CROSS JOIN LATERAL and GROUP BY.
         Non-PG splits into entity_scores subquery then JOINs for full columns
         (can't GROUP BY CLOB).
-
-        ``window`` narrows candidates *before* the per-entity cap, so out-of-window
-        neighbours don't consume an entity's bounded fan-out.
         """
         ...
 
@@ -387,7 +221,6 @@ class DataAccessOps(ABC):
         self,
         ml_table: str,
         mu_table: str,
-        window: UpdatedWindow,
     ) -> str:
         """Build semantic + causal expansion CTEs.
 
@@ -406,8 +239,7 @@ class DataAccessOps(ABC):
         seed_ids: list,
         budget: int,
         per_entity_limit: int,
-        window: UpdatedWindow,
-    ) -> LinkExpansionRows:
+    ) -> tuple[list[ResultRow], list[ResultRow], list[ResultRow]]:
         """Observation-specific graph expansion.
 
         PG uses native array ops (source_memory_ids column) for performance.
@@ -629,23 +461,6 @@ class DataAccessOps(ABC):
 
     # -- Task claiming operations ------------------------------------------
 
-    @abstractmethod
-    async def prune_terminal_operations(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        cutoff: datetime,
-        *,
-        batch_size: int,
-    ) -> int:
-        """Delete one deterministic batch of terminal operations older than ``cutoff``.
-
-        Implementations must lock candidates without waiting on rows another
-        worker is pruning, never select pending/processing rows, and return the
-        number deleted. The caller provides a transaction around this method.
-        """
-        ...
-
     @abstractmethod
     async def claim_tasks(
         self,
@@ -663,10 +478,6 @@ class DataAccessOps(ABC):
         Oracle implementation uses two-step claims (query busy banks first, then
         claim excluding them) to avoid ORA-02014.
 
-        Implementations must apply :func:`graph_maintenance_bank_serialization_sql`
-        to every query that can return a ``graph_maintenance`` row, so at most one
-        such row per bank is ever in flight.
-
         Args:
             consolidation_bank_priority: Per-bank priority for consolidation scheduling.
                 Maps bank name patterns to integer priorities (higher = claimed first).

hindsight-api-slim/hindsight_api/engine/db/ops_oracle.py

@@ -8,19 +8,13 @@ columns can't appear in GROUP BY).
 import json
 import uuid as uuid_mod
 from datetime import UTC, datetime
+from typing import Any
+from uuid import UUID
 
 from .base import DatabaseConnection
-from .ops import (
-    DataAccessOps,
-    LinkExpansionRows,
-    TagListingParts,
-    UpdatedWindow,
-    graph_maintenance_bank_serialization_sql,
-)
+from .ops import DataAccessOps, TagListingParts
 from .result import DictResultRow as ResultRow
 
-ORACLE_IN_LIST_LIMIT = 1000
-
 
 class OracleOps(DataAccessOps):
     """Oracle-specific data access operations."""
@@ -53,37 +47,6 @@ class OracleOps(DataAccessOps):
             column_types=["text[]", "text[]", "text[]", "text[]", "integer[]", "text[]"],
         )
 
-    async def lock_document_for_write(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        doc_id: str,
-        bank_id: str,
-    ) -> str | None:
-        # Oracle can't express the PG "INSERT ... ON CONFLICT DO UPDATE ...
-        # RETURNING" upsert in one statement — MERGE doesn't support RETURNING,
-        # so the single-statement form rewrites to a MERGE that returns no rows
-        # (DPY-1003). Split it into two statements instead:
-        #   1. Idempotent insert that silently skips an existing row. The
-        #      IGNORE_ROW_ON_DUPKEY_INDEX hint suppresses ORA-00001 server-side;
-        #      a concurrent uncommitted insert of the same key blocks here until
-        #      the other writer commits, so writers still serialize.
-        #   2. SELECT ... FOR UPDATE to take the row lock and read the hash
-        #      ('__pending__' for a row we just inserted, the stored hash for an
-        #      existing one).
-        await conn.execute(
-            f"INSERT /*+ IGNORE_ROW_ON_DUPKEY_INDEX({table}, pk_documents) */ "
-            f"INTO {table} (id, bank_id, original_text, content_hash) "
-            f"VALUES ($1, $2, '', '__pending__')",
-            doc_id,
-            bank_id,
-        )
-        return await conn.fetchval(
-            f"SELECT content_hash FROM {table} WHERE id = $1 AND bank_id = $2 FOR UPDATE",
-            doc_id,
-            bank_id,
-        )
-
     async def insert_facts_batch(
         self,
         conn: DatabaseConnection,
@@ -180,24 +143,22 @@ class OracleOps(DataAccessOps):
         bank_id: str,
         entity_names: list[str],
         entity_dates: list,
-        entity_kinds: list[str],
     ) -> dict[str, str]:
         # Row-by-row insert with duplicate suppression.
         # Can't use RETURNING with ON CONFLICT DO NOTHING reliably,
         # so INSERT (ignoring dups) then SELECT all IDs at the end.
         id_by_name: dict[str, str] = {}
-        for name, event_date, kind in zip(entity_names, entity_dates, entity_kinds):
+        for name, event_date in zip(entity_names, entity_dates):
             ts = event_date if event_date else datetime.now(UTC)
             await conn.execute(
                 f"""
-                INSERT INTO {table} (bank_id, canonical_name, first_seen, last_seen, mention_count, entity_kind)
-                VALUES ($1, $2, $3, $3, 0, $4)
+                INSERT INTO {table} (bank_id, canonical_name, first_seen, last_seen, mention_count)
+                VALUES ($1, $2, $3, $3, 0)
                 ON CONFLICT (bank_id, LOWER(canonical_name)) DO NOTHING
                 """,
                 bank_id,
                 name,
                 ts,
-                kind,
             )
         # Now SELECT all the entities we just inserted (or that already existed)
         for name in entity_names:
@@ -226,7 +187,7 @@ class OracleOps(DataAccessOps):
         for orig_name in missing_names:
             row = await conn.fetchrow(
                 f"""
-                SELECT id, canonical_name, LOWER(canonical_name) AS name_lower
+                SELECT id, LOWER(canonical_name) AS name_lower
                 FROM {table}
                 WHERE bank_id = $1 AND LOWER(canonical_name) = LOWER($2)
                 """,
@@ -234,41 +195,10 @@ class OracleOps(DataAccessOps):
                 orig_name,
             )
             if row:
+                # Wrap in a dict-like to include input_name for downstream compat
                 results.append(row)
         return results
 
-    async def bulk_reassert_entities(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        bank_id: str,
-        entity_ids: list[str],
-        canonical_names: list[str],
-        entity_kinds: list[str],
-    ) -> None:
-        # Oracle has no FOR KEY SHARE; FOR UPDATE is the row-lock equivalent that
-        # blocks a concurrent prune DELETE until this transaction commits. Lock
-        # each surviving parent in the caller's stable id order (pruned ids are
-        # simply absent here), then re-insert any that vanished. The translation
-        # layer rewrites ON CONFLICT DO NOTHING to strip-and-catch ORA-00001, so
-        # a name recreated under a new id is suppressed rather than raising.
-        for entity_id in entity_ids:
-            await conn.fetchrow(
-                f"SELECT id FROM {table} WHERE id = $1 FOR UPDATE",
-                entity_id,
-            )
-        await conn.executemany(
-            f"""
-            INSERT INTO {table} (id, bank_id, canonical_name, entity_kind)
-            VALUES ($1, $2, $3, $4)
-            ON CONFLICT DO NOTHING
-            """,
-            [
-                (entity_id, bank_id, canonical_name, kind)
-                for entity_id, canonical_name, kind in zip(entity_ids, canonical_names, entity_kinds)
-            ],
-        )
-
     async def bulk_insert_unit_entities(
         self,
         conn: DatabaseConnection,
@@ -294,31 +224,16 @@ class OracleOps(DataAccessOps):
     ) -> None:
         if not unit_ids:
             return
-        # Locking upsert (#3034), the Oracle analogue of the PG
-        # ``ON CONFLICT DO UPDATE``. The old IGNORE_ROW_ON_DUPKEY_INDEX insert
-        # skipped duplicates WITHOUT locking the existing row, so a mutation
-        # re-enqueueing an already-queued unit could not block a worker from
-        # concurrently claiming (deleting) that row and processing the unit's
-        # pre-mutation state — the re-enqueue signal was silently lost. MERGE
-        # WHEN MATCHED takes an exclusive row lock on the existing queue row
-        # (the SET is a deliberate no-op that preserves enqueued_at); WHEN NOT
-        # MATCHED inserts a fresh row. That serialises the mutation against the
-        # worker's claim for the same (bank_id, unit_id).
-        #
-        # Sort to enforce a global (bank_id, unit_id) lock-acquisition order,
-        # matching claim_graph_maintenance_batch's delete order, so overlapping
-        # mutation/worker sets acquire the shared row locks ascending and cannot
-        # cycle.
-        sorted_unit_ids = sorted(unit_ids)
+        # Oracle doesn't support ON CONFLICT; rely on the PK and the
+        # IGNORE_ROW_ON_DUPKEY_INDEX hint to skip duplicates server-side.
+        # The hint name must match the PK constraint exactly.
         await conn.executemany(
             f"""
-            MERGE INTO {table} q
-            USING (SELECT $1 AS bank_id, $2 AS unit_id FROM dual) s
-            ON (q.bank_id = s.bank_id AND q.unit_id = s.unit_id)
-            WHEN MATCHED THEN UPDATE SET q.enqueued_at = q.enqueued_at
-            WHEN NOT MATCHED THEN INSERT (bank_id, unit_id) VALUES (s.bank_id, s.unit_id)
+            INSERT /*+ IGNORE_ROW_ON_DUPKEY_INDEX({table}, pk_graph_maintenance_queue) */
+            INTO {table} (bank_id, unit_id)
+            VALUES ($1, $2)
             """,
-            [(bank_id, uid) for uid in sorted_unit_ids],
+            [(bank_id, uid) for uid in unit_ids],
         )
 
     async def claim_graph_maintenance_batch(
@@ -341,15 +256,7 @@ class OracleOps(DataAccessOps):
             bank_id,
             limit,
         )
-        # Ordered locking (#3034): the per-row DELETE takes the queue rows'
-        # exclusive locks in executemany array order. Sort the claimed keys by
-        # unit_id so those locks are acquired in the same (bank_id, unit_id)
-        # order the enqueue MERGE uses — overlapping mutation/worker sets then
-        # lock the shared rows ascending and cannot cycle. (The batch is still
-        # *chosen* oldest-first by enqueued_at above; only the lock/delete order
-        # is normalised.) The Pass 1 retry wrap in run_graph_maintenance_job is
-        # the ORA-00060 backstop for any residual interleaving.
-        claimed = sorted(str(row["unit_id"]) for row in rows)
+        claimed = [str(row["unit_id"]) for row in rows]
         if claimed:
             await conn.executemany(
                 f"DELETE FROM {table} WHERE bank_id = $1 AND unit_id = $2",
@@ -385,12 +292,6 @@ class OracleOps(DataAccessOps):
         entities_table: str,
         bank_id: str,
     ) -> int:
-        # NB: the Postgres path additionally selects victims FOR UPDATE in sorted
-        # (entity_id_1, entity_id_2) order to prevent the #2529 deadlock against
-        # retain's sorted cooccurrence upsert. Oracle's DELETE can't carry that
-        # ordered-lock CTE the same way, so here we rely on the Pass 2/3 retry
-        # wrap in run_graph_maintenance_job (retry_with_backoff is ORA-00060
-        # deadlock-aware) to recover instead. Deliberate dialect asymmetry.
         deleted = await conn.execute(
             f"""
             DELETE FROM {ec_table}
@@ -492,7 +393,6 @@ class OracleOps(DataAccessOps):
         mu_table: str,
         ue_table: str,
         per_entity_limit: int,
-        window: UpdatedWindow,
     ) -> str:
         # Oracle: can't GROUP BY CLOB columns (text, context).
         # Restructure: count entities per unit_id in a subquery, then join to get full columns.
@@ -510,16 +410,6 @@ class OracleOps(DataAccessOps):
                     FROM {ue_table} ue_target
                     WHERE ue_target.entity_id = se.entity_id
                       AND ue_target.unit_id != ALL($1::uuid[])
-                      -- Filter before applying the cap: candidates from other fact
-                      -- types, or outside the recall window, must not consume this
-                      -- entity's bounded fan-out.
-                      AND EXISTS (
-                          SELECT 1
-                          FROM {mu_table} mu_target
-                          WHERE mu_target.id = ue_target.unit_id
-                            AND mu_target.fact_type = $2
-                            {window.clause("mu_target")}
-                      )
                     ORDER BY ue_target.unit_id DESC
                     FETCH FIRST {per_entity_limit} ROWS ONLY
                 ) t
@@ -532,6 +422,7 @@ class OracleOps(DataAccessOps):
                        es.score, 'entity' AS source
                 FROM entity_scores es
                 JOIN {mu_table} mu ON mu.id = es.unit_id
+                WHERE mu.fact_type = $2
                 ORDER BY es.score DESC
                 FETCH FIRST $3 ROWS ONLY
             )"""
@@ -540,7 +431,6 @@ class OracleOps(DataAccessOps):
         self,
         ml_table: str,
         mu_table: str,
-        window: UpdatedWindow,
     ) -> str:
         # Non-PG: can't GROUP BY CLOB columns, no DISTINCT ON.
         # Restructure semantic: compute max weight per id, then join for full columns.
@@ -555,7 +445,6 @@ class OracleOps(DataAccessOps):
                       AND ml.link_type = 'semantic'
                       AND mu.fact_type = $2
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                     UNION ALL
                     SELECT mu.id, ml.weight
                     FROM {ml_table} ml
@@ -564,7 +453,6 @@ class OracleOps(DataAccessOps):
                       AND ml.link_type = 'semantic'
                       AND mu.fact_type = $2
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                 ) sem_raw
                 GROUP BY id
             ),
@@ -591,7 +479,6 @@ class OracleOps(DataAccessOps):
                 WHERE ml.from_unit_id = ANY($1::uuid[])
                   AND ml.link_type IN ('causes', 'caused_by', 'enables', 'prevents')
                   AND mu.fact_type = $2
-                  {window.clause("mu")}
             ),
             causal_expanded AS (
                 SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at,
@@ -610,8 +497,7 @@ class OracleOps(DataAccessOps):
         seed_ids: list,
         budget: int,
         per_entity_limit: int,
-        window: UpdatedWindow,
-    ) -> LinkExpansionRows:
+    ) -> tuple[list[ResultRow], list[ResultRow], list[ResultRow]]:
         import logging
 
         logger = logging.getLogger(__name__)
@@ -665,13 +551,11 @@ class OracleOps(DataAccessOps):
                   WHERE os3.observation_id = mu.id
                     AND os3.source_id IN (SELECT source_id FROM connected_sources)
               )
-              {window.clause("mu")}
             ORDER BY score DESC
             FETCH FIRST $2 ROWS ONLY
             """,
             seed_ids,
             budget,
-            *window.params,
         )
         logger.debug(f"[LinkExpansion] observation graph (Oracle): found {len(entity_rows)} connected observations")
 
@@ -687,14 +571,12 @@ class OracleOps(DataAccessOps):
                     WHERE ml.from_unit_id = ANY($1::uuid[])
                       AND ml.link_type = 'semantic' AND mu.fact_type = 'observation'
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                     UNION ALL
                     SELECT mu.id, ml.weight
                     FROM {ml_table} ml JOIN {mu_table} mu ON mu.id = ml.from_unit_id
                     WHERE ml.to_unit_id = ANY($1::uuid[])
                       AND ml.link_type = 'semantic' AND mu.fact_type = 'observation'
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                 ) sem_raw
                 GROUP BY id
             ),
@@ -720,7 +602,6 @@ class OracleOps(DataAccessOps):
                 WHERE ml.from_unit_id = ANY($1::uuid[])
                   AND ml.link_type IN ('causes', 'caused_by', 'enables', 'prevents')
                   AND mu.fact_type = 'observation'
-                  {window.clause("mu")}
             ),
             causal_expanded AS (
                 SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at,
@@ -735,12 +616,11 @@ class OracleOps(DataAccessOps):
             """,
             seed_ids,
             budget,
-            *window.params,
         )
 
         semantic_rows = [r for r in sem_causal_rows if r["source"] == "semantic"]
         causal_rows = [r for r in sem_causal_rows if r["source"] == "causal"]
-        return LinkExpansionRows(entity=list(entity_rows), semantic=semantic_rows, causal=causal_rows)
+        return list(entity_rows), semantic_rows, causal_rows
 
     def build_tag_listing_parts(self, mu_table: str) -> TagListingParts:
         return TagListingParts(
@@ -907,157 +787,6 @@ class OracleOps(DataAccessOps):
 
     # -- Task claiming operations ------------------------------------------
 
-    async def prune_terminal_operations(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        cutoff: datetime,
-        *,
-        batch_size: int,
-    ) -> int:
-        # Oracle rejects a row-limited SELECT ... FOR UPDATE (ORA-02014). Pick
-        # the deterministic bounded IDs first, then lock only that candidate
-        # set and re-check eligibility before deleting in the same transaction.
-        # Clamp to Oracle's 1000-expression IN-list limit because the adapter
-        # expands the candidate UUID list into individual bind variables.
-        # Cancelled children cannot complete parent aggregation, so retain the
-        # parent guard only for completed/failed children. Before removing a
-        # cancelled child, preserve its signal by cancelling a pending parent
-        # in this transaction and refreshing the parent's retention window.
-        # Validate metadata before HEXTORAW: CASE makes malformed UUIDs yield
-        # NULL while keeping the indexed RAW parent.operation_id key unwrapped.
-        effective_batch_size = min(batch_size, ORACLE_IN_LIST_LIMIT)
-        candidates = await conn.fetch(
-            f"""
-            SELECT candidate_operation.operation_id
-            FROM {table} candidate_operation
-            WHERE candidate_operation.status IN ('completed', 'failed', 'cancelled')
-              AND candidate_operation.updated_at < $1
-              AND (
-                  candidate_operation.status = 'cancelled'
-                  OR NOT EXISTS (
-                      SELECT 1
-                      FROM {table} parent
-                      WHERE parent.operation_id = CASE
-                          WHEN REGEXP_LIKE(
-                              JSON_VALUE(
-                                  candidate_operation.result_metadata,
-                                  '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                              ),
-                              '^[0-9A-Fa-f]{{8}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{12}}$'
-                          )
-                          THEN HEXTORAW(REPLACE(
-                              JSON_VALUE(
-                                  candidate_operation.result_metadata,
-                                  '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                              ),
-                              '-',
-                              ''
-                          ))
-                          ELSE NULL
-                      END
-                        AND parent.bank_id = candidate_operation.bank_id
-                  )
-              )
-            ORDER BY candidate_operation.updated_at, candidate_operation.operation_id
-            LIMIT $2
-            """,
-            cutoff,
-            effective_batch_size,
-        )
-        if not candidates:
-            return 0
-
-        candidate_ids = [row["operation_id"] for row in candidates]
-        locked = await conn.fetch(
-            f"""
-            SELECT candidate_operation.operation_id
-            FROM {table} candidate_operation
-            WHERE candidate_operation.operation_id = ANY($1)
-              AND candidate_operation.status IN ('completed', 'failed', 'cancelled')
-              AND candidate_operation.updated_at < $2
-              AND (
-                  candidate_operation.status = 'cancelled'
-                  OR NOT EXISTS (
-                      SELECT 1
-                      FROM {table} parent
-                      WHERE parent.operation_id = CASE
-                          WHEN REGEXP_LIKE(
-                              JSON_VALUE(
-                                  candidate_operation.result_metadata,
-                                  '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                              ),
-                              '^[0-9A-Fa-f]{{8}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{12}}$'
-                          )
-                          THEN HEXTORAW(REPLACE(
-                              JSON_VALUE(
-                                  candidate_operation.result_metadata,
-                                  '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                              ),
-                              '-',
-                              ''
-                          ))
-                          ELSE NULL
-                      END
-                        AND parent.bank_id = candidate_operation.bank_id
-                  )
-              )
-            ORDER BY candidate_operation.updated_at, candidate_operation.operation_id
-            FOR UPDATE OF candidate_operation.operation_id SKIP LOCKED
-            """,
-            candidate_ids,
-            cutoff,
-        )
-        if not locked:
-            return 0
-        operation_ids = [row["operation_id"] for row in locked]
-        await conn.execute(
-            f"""
-            UPDATE {table} parent
-            SET status = 'cancelled',
-                updated_at = now(),
-                completed_at = COALESCE(parent.completed_at, now()),
-                error_message = COALESCE(
-                    parent.error_message,
-                    'Cancelled because a child operation was cancelled'
-                )
-            WHERE parent.status = 'pending'
-              AND EXISTS (
-                  SELECT 1
-                  FROM {table} candidate_operation
-                  WHERE candidate_operation.operation_id = ANY($1)
-                    AND candidate_operation.status = 'cancelled'
-                    AND candidate_operation.updated_at < $2
-                    AND candidate_operation.bank_id = parent.bank_id
-                    AND parent.operation_id = CASE
-                        WHEN REGEXP_LIKE(
-                            JSON_VALUE(
-                                candidate_operation.result_metadata,
-                                '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                            ),
-                            '^[0-9A-Fa-f]{{8}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{4}}-[0-9A-Fa-f]{{12}}$'
-                        )
-                        THEN HEXTORAW(REPLACE(
-                            JSON_VALUE(
-                                candidate_operation.result_metadata,
-                                '$.parent_operation_id' RETURNING VARCHAR2(36) NULL ON ERROR
-                            ),
-                            '-',
-                            ''
-                        ))
-                        ELSE NULL
-                    END
-              )
-            """,
-            operation_ids,
-            cutoff,
-        )
-        await conn.execute(
-            f"DELETE FROM {table} WHERE operation_id = ANY($1)",
-            operation_ids,
-        )
-        return len(operation_ids)
-
     async def _claim_consolidation_tasks(
         self,
         conn,
@@ -1338,14 +1067,13 @@ class OracleOps(DataAccessOps):
             else:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type = $1
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type = $1
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                    ORDER BY created_at
                     LIMIT $2
                     FOR UPDATE SKIP LOCKED
                     """,
@@ -1360,21 +1088,18 @@ class OracleOps(DataAccessOps):
         # --- Phase 2: claim from shared pool ---
         remaining_shared = shared_limit
         if remaining_shared > 0:
-            # 2a. Non-consolidation tasks. graph_maintenance stays in this
-            # created_at-ordered query — see graph_maintenance_bank_serialization_sql
-            # for why it is a predicate rather than a phase of its own.
+            # 2a. Non-consolidation tasks
             if claimed_ids:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type != 'consolidation'
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND o.operation_id != ALL($1::uuid[])
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type != 'consolidation'
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                      AND operation_id != ALL($1::uuid[])
+                    ORDER BY created_at
                     LIMIT $2
                     FOR UPDATE SKIP LOCKED
                     """,
@@ -1384,14 +1109,13 @@ class OracleOps(DataAccessOps):
             else:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type != 'consolidation'
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type != 'consolidation'
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                    ORDER BY created_at
                     LIMIT $1
                     FOR UPDATE SKIP LOCKED
                     """,

hindsight-api-slim/hindsight_api/engine/db/ops_postgresql.py

@@ -4,77 +4,19 @@ Uses unnest(), LATERAL, DISTINCT ON, and native array operations for
 efficient batch operations.
 """
 
-import asyncio
-from datetime import datetime
+import json
+from datetime import UTC, datetime
+from typing import Any
+from uuid import UUID
 
 from .base import DatabaseConnection
-from .ops import (
-    DataAccessOps,
-    LinkExpansionRows,
-    TagListingParts,
-    UpdatedWindow,
-    graph_maintenance_bank_serialization_sql,
-)
+from .ops import DataAccessOps, TagListingParts
 from .result import ResultRow
 
 
-def pg_search_vector_expr(
-    config,
-    *,
-    text_col: str = "text",
-    context_col: str = "context",
-    signals_col: str | None = "text_signals",
-    native_inline: bool = True,
-) -> str | None:
-    """SQL expression that builds ``search_vector`` for the configured PG text-search backend.
-
-    Single source of truth shared by ``memory_units`` (the batch insert over the
-    ``input_data`` CTE columns and the curation-revert recompute) and
-    ``mental_models`` (the knowledge-page writes), so the per-backend tokenization
-    can never drift between the two tables. Returns ``None`` for backends that
-    leave ``search_vector`` unpopulated — pgroonga / pg_textsearch / pg_search
-    index the base text columns directly and keep only a dummy column, so there is
-    nothing to build.
-
-    The ``*_col`` arguments are the SQL for each text source (a column name or a
-    bind placeholder); pass ``signals_col=None`` for a two-column table like
-    ``mental_models`` (name + content). Pass ``native_inline=False`` when the
-    table's native ``search_vector`` is a GENERATED column that populates itself
-    (``mental_models``) — writing it inline would fail; only vchord's plain
-    bm25vector column then needs an explicit value.
-
-    ``text_search_extension_native_language`` is validated as a PG identifier in
-    ``HindsightConfig.validate()``, so embedding it as a SQL literal is safe.
-    """
-    cols = [text_col, context_col] + ([signals_col] if signals_col is not None else [])
-    combined = " || ' ' || ".join(f"COALESCE({c}, '')" for c in cols)
-    if config.text_search_extension == "vchord":
-        return f"tokenize({combined}, 'llmlingua2')::bm25_catalog.bm25vector"
-    if config.text_search_extension == "native" and native_inline:
-        return f"to_tsvector('{config.text_search_extension_native_language}'::regconfig, {combined})"
-    return None
-
-
 class PostgreSQLOps(DataAccessOps):
     """PostgreSQL-specific data access operations using unnest and LATERAL."""
 
-    def __init__(self) -> None:
-        # Per-table serialization of per-bank vector-index DDL within this
-        # process. Concurrent index DDL on one relation deadlocks by design:
-        # DROP INDEX CONCURRENTLY holds ShareUpdateExclusive while it waits out
-        # every transaction whose snapshot could still see the index — including
-        # other sessions' index DDL queued on that same lock — so many banks
-        # deleted at once form a wait cycle Postgres resolves by killing one.
-        # A session advisory lock would serialize this across processes too, but
-        # advisory locks are banned here (poolers hand sessions around; see the
-        # Database Locking standard). In-process the asyncio lock removes the
-        # cycle outright; across processes the callers' retry-with-backoff
-        # absorbs the (now much rarer) collisions.
-        self._index_ddl_locks: dict[str, asyncio.Lock] = {}
-
-    def _index_ddl_lock(self, table: str) -> asyncio.Lock:
-        return self._index_ddl_locks.setdefault(table, asyncio.Lock())
-
     @property
     def uses_observation_sources_table(self) -> bool:
         return False  # PG uses native array ops on source_memory_ids
@@ -107,30 +49,6 @@ class PostgreSQLOps(DataAccessOps):
             content_hashes,
         )
 
-    async def lock_document_for_write(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        doc_id: str,
-        bank_id: str,
-    ) -> str | None:
-        # Single upsert that both creates the row (if absent) and locks it (if
-        # present) atomically. ON CONFLICT DO UPDATE always takes the row lock as
-        # part of the statement, so all concurrent same-document writers serialize
-        # on the document row in one consistent step (the earlier two-step form —
-        # DO NOTHING + a separate SELECT FOR UPDATE — could deadlock because
-        # DO NOTHING takes no lock on an existing row). The SET is a no-op
-        # self-assignment used only to acquire the lock; RETURNING yields the
-        # pre-existing hash (or '__pending__' for a freshly inserted row).
-        return await conn.fetchval(
-            f"INSERT INTO {table} (id, bank_id, original_text, content_hash) "
-            f"VALUES ($1, $2, '', '__pending__') "
-            f"ON CONFLICT (id, bank_id) DO UPDATE SET content_hash = {table}.content_hash "
-            f"RETURNING content_hash",
-            doc_id,
-            bank_id,
-        )
-
     async def insert_facts_batch(
         self,
         conn: DatabaseConnection,
@@ -156,39 +74,101 @@ class PostgreSQLOps(DataAccessOps):
         config = get_config()
         table = self._get_mu_table()
 
-        # search_vector is populated inline for backends that store a real vector
-        # (native tsvector, vchord bm25vector). pgroonga / pg_textsearch / pg_search
-        # index the base text columns directly and keep only a dummy column, so the
-        # expression is None and the column is left out of the insert entirely.
-        # Same expression is reused by curation revert (see pg_search_vector_expr).
-        sv_expr = pg_search_vector_expr(config)
-        sv_insert_col = ", search_vector" if sv_expr else ""
-        sv_select_val = f",\n                    {sv_expr}" if sv_expr else ""
-        query = f"""
-            WITH input_data AS (
-                SELECT * FROM unnest(
-                    $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-                    $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
-                ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                       context, fact_type, metadata, chunk_id, document_id, tags_json,
-                       observation_scopes_json, text_signals)
-            )
-            INSERT INTO {table} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                 context, fact_type, metadata, chunk_id, document_id, tags,
-                                 observation_scopes, text_signals{sv_insert_col})
-            SELECT
-                $1,
-                text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                context, fact_type, metadata, chunk_id, document_id,
-                COALESCE(
-                    (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
-                    '{{}}'::varchar[]
-                ),
-                observation_scopes_json,
-                text_signals{sv_select_val}
-            FROM input_data
-            RETURNING id
-        """
+        if config.text_search_extension == "vchord":
+            query = f"""
+                WITH input_data AS (
+                    SELECT * FROM unnest(
+                        $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
+                        $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
+                    ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                           context, fact_type, metadata, chunk_id, document_id, tags_json,
+                           observation_scopes_json, text_signals)
+                )
+                INSERT INTO {table} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                                     context, fact_type, metadata, chunk_id, document_id, tags,
+                                     observation_scopes, text_signals, search_vector)
+                SELECT
+                    $1,
+                    text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                    context, fact_type, metadata, chunk_id, document_id,
+                    COALESCE(
+                        (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
+                        '{{}}'::varchar[]
+                    ),
+                    observation_scopes_json,
+                    text_signals,
+                    tokenize(
+                        COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, ''),
+                        'llmlingua2'
+                    )::bm25_catalog.bm25vector
+                FROM input_data
+                RETURNING id
+            """
+        elif config.text_search_extension == "native":
+            # search_vector is a regular tsvector column populated here using the
+            # configured native dictionary. It used to be GENERATED ALWAYS with
+            # a hardcoded 'english', which prevented per-deployment language
+            # configuration. text_search_extension_native_language is validated
+            # in HindsightConfig.validate() as a PG identifier, so embedding it
+            # as a SQL literal is safe.
+            query = f"""
+                WITH input_data AS (
+                    SELECT * FROM unnest(
+                        $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
+                        $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
+                    ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                           context, fact_type, metadata, chunk_id, document_id, tags_json,
+                           observation_scopes_json, text_signals)
+                )
+                INSERT INTO {table} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                                     context, fact_type, metadata, chunk_id, document_id, tags,
+                                     observation_scopes, text_signals, search_vector)
+                SELECT
+                    $1,
+                    text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                    context, fact_type, metadata, chunk_id, document_id,
+                    COALESCE(
+                        (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
+                        '{{}}'::varchar[]
+                    ),
+                    observation_scopes_json,
+                    text_signals,
+                    to_tsvector(
+                        '{config.text_search_extension_native_language}'::regconfig,
+                        COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, '')
+                    )
+                FROM input_data
+                RETURNING id
+            """
+        else:
+            # pg_textsearch, pgroonga, and pg_search: search_vector is a dummy
+            # TEXT column; the actual full-text index operates on the base text
+            # columns directly, so we don't populate search_vector at insert time.
+            query = f"""
+                WITH input_data AS (
+                    SELECT * FROM unnest(
+                        $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
+                        $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
+                    ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                           context, fact_type, metadata, chunk_id, document_id, tags_json,
+                           observation_scopes_json, text_signals)
+                )
+                INSERT INTO {table} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                                     context, fact_type, metadata, chunk_id, document_id, tags,
+                                     observation_scopes, text_signals)
+                SELECT
+                    $1,
+                    text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                    context, fact_type, metadata, chunk_id, document_id,
+                    COALESCE(
+                        (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
+                        '{{}}'::varchar[]
+                    ),
+                    observation_scopes_json,
+                    text_signals
+                FROM input_data
+                RETURNING id
+            """
 
         results = await conn.fetch(
             query,
@@ -286,22 +266,12 @@ class PostgreSQLOps(DataAccessOps):
         bank_id: str,
         entity_names: list[str],
         entity_dates: list,
-        entity_kinds: list[str],
     ) -> dict[str, str]:
-        # ORDER BY LOWER(name) so every concurrent batch inserts in the same order
-        # as the conflict target (bank_id, LOWER(canonical_name)). ON CONFLICT DO
-        # NOTHING takes a ShareLock on the inserting transaction of any speculative
-        # row it collides with, so two batches with overlapping names inserting in
-        # different orders deadlock. The caller already sorts by Python's
-        # ``str.lower()``, which agrees with the index for ASCII but not for every
-        # locale (see the Turkish-İ note in entity_resolver) — ordering in SQL makes
-        # the database's own collation the single arbiter for all writers.
         inserted_rows = await conn.fetch(
             f"""
-            INSERT INTO {table} (bank_id, canonical_name, first_seen, last_seen, mention_count, entity_kind)
-            SELECT $1, name, COALESCE(event_date, now()), COALESCE(event_date, now()), 0, kind
-            FROM unnest($2::text[], $3::timestamptz[], $4::text[]) AS t(name, event_date, kind)
-            ORDER BY LOWER(name)
+            INSERT INTO {table} (bank_id, canonical_name, first_seen, last_seen, mention_count)
+            SELECT $1, name, COALESCE(event_date, now()), COALESCE(event_date, now()), 0
+            FROM unnest($2::text[], $3::timestamptz[]) AS t(name, event_date)
             ON CONFLICT (bank_id, LOWER(canonical_name))
             DO NOTHING
             RETURNING id, LOWER(canonical_name) AS name_lower
@@ -309,7 +279,6 @@ class PostgreSQLOps(DataAccessOps):
             bank_id,
             entity_names,
             entity_dates,
-            entity_kinds,
         )
         return {row["name_lower"]: row["id"] for row in inserted_rows}
 
@@ -322,7 +291,7 @@ class PostgreSQLOps(DataAccessOps):
     ) -> list[ResultRow]:
         return await conn.fetch(
             f"""
-            SELECT e.id, e.canonical_name, LOWER(e.canonical_name) AS name_lower, inputs.input_name
+            SELECT e.id, LOWER(e.canonical_name) AS name_lower, inputs.input_name
             FROM {table} e
             JOIN (
                 SELECT LOWER(n) AS input_name_lower, n AS input_name
@@ -334,44 +303,6 @@ class PostgreSQLOps(DataAccessOps):
             missing_names,
         )
 
-    async def bulk_reassert_entities(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        bank_id: str,
-        entity_ids: list[str],
-        canonical_names: list[str],
-        entity_kinds: list[str],
-    ) -> None:
-        # One statement, one round-trip (same shape as bulk_insert_links):
-        #   * the CTE takes FOR KEY SHARE on every parent that still exists,
-        #     held to COMMIT, so a concurrent prune_orphan_entities DELETE blocks
-        #     until the caller's unit_entities insert has committed;
-        #   * the INSERT re-creates only the parents that were already pruned
-        #     (NOT IN locked), carrying the canonical_name resolved in Phase 1.
-        # ON CONFLICT DO NOTHING (no target) keeps the rare case where another
-        # worker recreated the name under a new id from raising — that row stays
-        # absent and its unit link is the sole casualty, never the whole batch.
-        await conn.execute(
-            f"""
-            WITH locked AS (
-                SELECT id FROM {table}
-                WHERE id = ANY($2::uuid[])
-                ORDER BY id
-                FOR KEY SHARE
-            )
-            INSERT INTO {table} (id, bank_id, canonical_name, entity_kind)
-            SELECT t.entity_id, $1, t.canonical_name, t.entity_kind
-            FROM unnest($2::uuid[], $3::text[], $4::text[]) AS t(entity_id, canonical_name, entity_kind)
-            WHERE t.entity_id NOT IN (SELECT id FROM locked)
-            ON CONFLICT DO NOTHING
-            """,
-            bank_id,
-            entity_ids,
-            canonical_names,
-            entity_kinds,
-        )
-
     async def bulk_insert_unit_entities(
         self,
         conn: DatabaseConnection,
@@ -398,36 +329,14 @@ class PostgreSQLOps(DataAccessOps):
     ) -> None:
         if not unit_ids:
             return
-        # Sort to enforce a global lock-acquisition order on the
-        # (bank_id, unit_id) unique-key. Without this, two concurrent
-        # transactions inserting overlapping unit_id sets in different
-        # orders can deadlock on the ON CONFLICT row locks — Postgres
-        # acquires a short-lived lock per row being checked, and cycle
-        # detection then aborts one transaction. Sorting gives every
-        # concurrent caller the same lock order, so conflicting inserts
-        # queue cleanly instead of cycling.
-        sorted_unit_ids = sorted(unit_ids)
-        # DO UPDATE (not DO NOTHING) on a duplicate enqueue — #3034. The SET is a
-        # deliberate no-op that preserves enqueued_at; its only purpose is to take
-        # the existing row's lock. DO NOTHING does NOT lock the conflicting row, so
-        # a mutation that re-enqueues an already-queued unit could not block a
-        # worker from concurrently claiming (deleting) that row and processing the
-        # unit's pre-mutation state; the re-enqueue signal was then silently lost
-        # and the unit's derived links stayed stale with an empty queue. Locking
-        # the row serialises the mutation against the worker's claim for that
-        # (bank_id, unit_id): the worker either waits for the committed post-mutation
-        # state, or (if it claimed first) this INSERT lands a fresh row after the
-        # worker's delete commits. Row locks are acquired in sorted unit_id order,
-        # matching claim_graph_maintenance_batch, so the two never cycle.
         await conn.execute(
             f"""
             INSERT INTO {table} (bank_id, unit_id)
             SELECT $1, v FROM unnest($2::uuid[]) AS t(v)
-            ON CONFLICT (bank_id, unit_id)
-                DO UPDATE SET enqueued_at = {table}.enqueued_at
+            ON CONFLICT (bank_id, unit_id) DO NOTHING
             """,
             bank_id,
-            sorted_unit_ids,
+            unit_ids,
         )
 
     async def claim_graph_maintenance_batch(
@@ -437,35 +346,16 @@ class PostgreSQLOps(DataAccessOps):
         bank_id: str,
         limit: int,
     ) -> list[str]:
-        # Ordered locking (#3034). Choose the oldest batch by enqueued_at, but
-        # acquire the row locks in (bank_id, unit_id) order — the same order the
-        # enqueue upsert takes them — so a foreground mutation re-enqueueing an
-        # overlapping unit set can never cycle against a worker draining it. The
-        # `chosen` CTE is MATERIALIZED so the enqueued_at pick is fenced from the
-        # locking clause; `FOR UPDATE OF q ... ORDER BY q.unit_id` then puts
-        # LockRows above the Sort, so locks are taken ascending by unit_id (same
-        # idiom as prune_stale_cooccurrences' #2529 ordered lock). A concurrent
-        # enqueue holding one of these rows blocks this claim until it commits, at
-        # which point the worker deletes and processes the committed state.
         rows = await conn.fetch(
             f"""
-            WITH chosen AS MATERIALIZED (
+            DELETE FROM {table}
+            WHERE (bank_id, unit_id) IN (
                 SELECT bank_id, unit_id FROM {table}
                 WHERE bank_id = $1
                 ORDER BY enqueued_at
                 LIMIT $2
-            ),
-            locked AS (
-                SELECT q.bank_id, q.unit_id
-                FROM {table} q
-                JOIN chosen c ON c.bank_id = q.bank_id AND c.unit_id = q.unit_id
-                ORDER BY q.unit_id
-                FOR UPDATE OF q
             )
-            DELETE FROM {table} q
-            USING locked l
-            WHERE q.bank_id = l.bank_id AND q.unit_id = l.unit_id
-            RETURNING q.unit_id
+            RETURNING unit_id
             """,
             bank_id,
             limit,
@@ -507,48 +397,19 @@ class PostgreSQLOps(DataAccessOps):
         # Scope by joining through entities.bank_id (entity_cooccurrences itself
         # has no bank_id column — entities don't span banks, so scoping via
         # entity_id_1 is sufficient).
-        #
-        # Ordered locking (deadlock avoidance, #2529): retain's concurrent
-        # cooccurrence upsert (entity_resolver._flush_pending) locks rows in
-        # sorted (entity_id_1, entity_id_2) order — sorted specifically to give
-        # every writer one consistent lock-acquisition order. A plain
-        # `DELETE ... USING` scans/locks in whatever order the join plan picks,
-        # so it could lock the same rows in the opposite order and cycle. We
-        # instead select the victims in that same sorted order `FOR UPDATE`
-        # first — the locking clause materialises the CTE and places LockRows
-        # above the Sort, so locks are acquired ascending, matching the upsert —
-        # then delete the already-locked rows. Same order on both sides ⇒ no
-        # cycle (the deadlock is prevented, not merely retried). The Pass 2/3
-        # retry wrap in run_graph_maintenance_job stays as a backstop for the
-        # residual paths (FK cascade from prune_orphan_entities, Oracle).
-        #
-        # The staleness predicate is an INTERSECT of the two entities' unit sets
-        # rather than the equivalent `unit_entities u1 JOIN u2 ON u1.unit_id =
-        # u2.unit_id` self-join (#2473): both INTERSECT branches resolve as Index
-        # Only Scans on idx_unit_entities_entity_unit (entity_id, unit_id), so the
-        # per-pair cost is bounded by the two entities' degrees. The self-join let
-        # the planner pick an anti-join that rescanned a high-degree hub entity's
-        # membership set for every pair — 28-30min on a bank with a ~100K-membership
-        # hub, even when zero rows were stale. Don't "simplify" it back.
         result = await conn.execute(
             f"""
-            WITH victims AS (
-                SELECT c.entity_id_1, c.entity_id_2
-                FROM {ec_table} c
-                JOIN {entities_table} e ON e.id = c.entity_id_1
-                WHERE e.bank_id = $1
-                  AND NOT EXISTS (
-                      SELECT unit_id FROM {ue_table} WHERE entity_id = c.entity_id_1
-                      INTERSECT
-                      SELECT unit_id FROM {ue_table} WHERE entity_id = c.entity_id_2
-                  )
-                ORDER BY c.entity_id_1, c.entity_id_2
-                FOR UPDATE OF c
-            )
             DELETE FROM {ec_table} c
-            USING victims v
-            WHERE c.entity_id_1 = v.entity_id_1
-              AND c.entity_id_2 = v.entity_id_2
+            USING {entities_table} e
+            WHERE e.id = c.entity_id_1
+              AND e.bank_id = $1
+              AND NOT EXISTS (
+                  SELECT 1
+                  FROM {ue_table} u1
+                  JOIN {ue_table} u2 ON u1.unit_id = u2.unit_id
+                  WHERE u1.entity_id = c.entity_id_1
+                    AND u2.entity_id = c.entity_id_2
+              )
             """,
             bank_id,
         )
@@ -560,21 +421,11 @@ class PostgreSQLOps(DataAccessOps):
         mu_table: str,
         unit_ids: list[str],
     ) -> list[ResultRow]:
-        # Cast only canonical UUID text inputs, never the indexed column. The old
-        # ``id::text`` predicate silently ignored malformed, uppercase, braced,
-        # and unhyphenated inputs; filtering before the cast preserves that
-        # behavior while allowing the primary-key index to serve the lookup.
         return await conn.fetch(
             f"""
             SELECT id, event_date, fact_type
             FROM {mu_table}
-            WHERE id = ANY(
-                ARRAY(
-                    SELECT input.unit_id::uuid
-                    FROM unnest($1::text[]) AS input(unit_id)
-                    WHERE input.unit_id ~ '^[0-9a-f]{{8}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{12}}$'
-                )
-            )
+            WHERE id::text = ANY($1)
             """,
             unit_ids,
         )
@@ -643,7 +494,6 @@ class PostgreSQLOps(DataAccessOps):
         mu_table: str,
         ue_table: str,
         per_entity_limit: int,
-        window: UpdatedWindow,
     ) -> str:
         return f"""
             seed_entities AS (
@@ -663,20 +513,11 @@ class PostgreSQLOps(DataAccessOps):
                     FROM {ue_table} ue_target
                     WHERE ue_target.entity_id = se.entity_id
                       AND ue_target.unit_id != ALL($1::uuid[])
-                      -- Filter before applying the cap: candidates from other fact
-                      -- types, or outside the recall window, must not consume this
-                      -- entity's bounded fan-out.
-                      AND EXISTS (
-                          SELECT 1
-                          FROM {mu_table} mu_target
-                          WHERE mu_target.id = ue_target.unit_id
-                            AND mu_target.fact_type = $2
-                            {window.clause("mu_target")}
-                      )
                     ORDER BY ue_target.unit_id DESC
                     LIMIT {per_entity_limit}
                 ) t
                 JOIN {mu_table} mu ON mu.id = t.unit_id
+                WHERE mu.fact_type = $2
                 GROUP BY mu.id
                 ORDER BY score DESC
                 LIMIT $3
@@ -686,7 +527,6 @@ class PostgreSQLOps(DataAccessOps):
         self,
         ml_table: str,
         mu_table: str,
-        window: UpdatedWindow,
     ) -> str:
         # Exact v0.5.6 query shape: GROUP BY + MAX(weight) for semantic,
         # DISTINCT ON for causal.
@@ -710,7 +550,6 @@ class PostgreSQLOps(DataAccessOps):
                       AND ml.link_type = 'semantic'
                       AND mu.fact_type = $2
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                     UNION ALL
                     SELECT
                         mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start,
@@ -723,7 +562,6 @@ class PostgreSQLOps(DataAccessOps):
                       AND ml.link_type = 'semantic'
                       AND mu.fact_type = $2
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                 ) sem_raw
                 GROUP BY id, text, context, event_date, occurred_start,
                          occurred_end, mentioned_at,
@@ -743,7 +581,6 @@ class PostgreSQLOps(DataAccessOps):
                 WHERE ml.from_unit_id = ANY($1::uuid[])
                   AND ml.link_type IN ('causes', 'caused_by', 'enables', 'prevents')
                   AND mu.fact_type = $2
-                  {window.clause("mu")}
                 ORDER BY mu.id, ml.weight DESC
                 LIMIT $3
             )"""
@@ -757,13 +594,9 @@ class PostgreSQLOps(DataAccessOps):
         seed_ids: list,
         budget: int,
         per_entity_limit: int,
-        window: UpdatedWindow,
-    ) -> LinkExpansionRows:
+    ) -> tuple[list[ResultRow], list[ResultRow], list[ResultRow]]:
         # v0.5.6 array ops: unnest, &&, COUNT(DISTINCT) on source_memory_ids.
-        #
-        # The window bounds the observations that come *back*, not the source facts
-        # traversed to reach them: an observation is in the window when it was itself
-        # written or refreshed there, regardless of how old the facts underneath it are.
+        from ..schema import fq_table
 
         entity_rows = await conn.fetch(
             f"""
@@ -805,13 +638,11 @@ class PostgreSQLOps(DataAccessOps):
               AND mu.id != ALL($1::uuid[])
               AND ca.source_ids IS NOT NULL
               AND mu.source_memory_ids && ca.source_ids
-              {window.clause("mu")}
             ORDER BY score DESC
             LIMIT $2
             """,
             seed_ids,
             budget,
-            *window.params,
         )
 
         # Exact v0.5.6 query shape: GROUP BY + MAX(weight) for semantic,
@@ -833,7 +664,6 @@ class PostgreSQLOps(DataAccessOps):
                     WHERE ml.from_unit_id = ANY($1::uuid[])
                       AND ml.link_type = 'semantic' AND mu.fact_type = 'observation'
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                     UNION ALL
                     SELECT mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start,
                            mu.occurred_end, mu.mentioned_at, mu.fact_type, mu.document_id,
@@ -842,7 +672,6 @@ class PostgreSQLOps(DataAccessOps):
                     WHERE ml.to_unit_id = ANY($1::uuid[])
                       AND ml.link_type = 'semantic' AND mu.fact_type = 'observation'
                       AND mu.id != ALL($1::uuid[])
-                      {window.clause("mu")}
                 ) sem_raw
                 GROUP BY id, text, context, event_date, occurred_start, occurred_end,
                          mentioned_at, fact_type, document_id, chunk_id, tags, proof_count
@@ -857,7 +686,6 @@ class PostgreSQLOps(DataAccessOps):
                 WHERE ml.from_unit_id = ANY($1::uuid[])
                   AND ml.link_type IN ('causes', 'caused_by', 'enables', 'prevents')
                   AND mu.fact_type = 'observation'
-                  {window.clause("mu")}
                 ORDER BY mu.id, ml.weight DESC LIMIT $2
             )
             SELECT * FROM semantic_expanded
@@ -866,12 +694,11 @@ class PostgreSQLOps(DataAccessOps):
             """,
             seed_ids,
             budget,
-            *window.params,
         )
 
         semantic_rows = [r for r in sem_causal_rows if r["source"] == "semantic"]
         causal_rows = [r for r in sem_causal_rows if r["source"] == "causal"]
-        return LinkExpansionRows(entity=list(entity_rows), semantic=semantic_rows, causal=causal_rows)
+        return list(entity_rows), semantic_rows, causal_rows
 
     def build_tag_listing_parts(self, mu_table: str) -> TagListingParts:
         return TagListingParts(
@@ -891,15 +718,14 @@ class PostgreSQLOps(DataAccessOps):
         fact_types: dict[str, str],
     ) -> None:
         escaped = bank_id.replace("'", "''")
-        async with self._index_ddl_lock(table):
-            for ft, suffix in fact_types.items():
-                uid = str(internal_id).replace("-", "")[:16]
-                idx = f"idx_mu_emb_{suffix}_{uid}"
-                await conn.execute(
-                    f"CREATE INDEX IF NOT EXISTS {idx} "
-                    f"ON {table} {index_clause} "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped}'"
-                )
+        for ft, suffix in fact_types.items():
+            uid = str(internal_id).replace("-", "")[:16]
+            idx = f"idx_mu_emb_{suffix}_{uid}"
+            await conn.execute(
+                f"CREATE INDEX IF NOT EXISTS {idx} "
+                f"ON {table} {index_clause} "
+                f"WHERE fact_type = '{ft}' AND bank_id = '{escaped}'"
+            )
 
     async def drop_bank_vector_indexes(
         self,
@@ -908,19 +734,10 @@ class PostgreSQLOps(DataAccessOps):
         internal_id: str,
         fact_types: dict[str, str],
     ) -> None:
-        # CONCURRENTLY so the drop takes ShareUpdateExclusive, not ACCESS
-        # EXCLUSIVE, on the shared memory_units table. A plain DROP INDEX blocks
-        # (and deadlocks with) every other bank's concurrent reads/writes on the
-        # table; CONCURRENTLY does not conflict with DML. The caller
-        # (delete_bank) runs this on an autocommit connection after its delete
-        # transaction has committed — CONCURRENTLY cannot run inside a tx.
-        # The lock key must match create_bank_vector_indexes', whose `table`
-        # is the fq name this reconstructs from `schema`.
-        async with self._index_ddl_lock(f"{schema}.memory_units"):
-            for ft, suffix in fact_types.items():
-                uid = str(internal_id).replace("-", "")[:16]
-                idx = f"idx_mu_emb_{suffix}_{uid}"
-                await conn.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}.{idx}")
+        for ft, suffix in fact_types.items():
+            uid = str(internal_id).replace("-", "")[:16]
+            idx = f"idx_mu_emb_{suffix}_{uid}"
+            await conn.execute(f"DROP INDEX IF EXISTS {schema}.{idx}")
 
     def get_entity_resolution_strategy(self) -> str:
         return "trigram"
@@ -1050,93 +867,6 @@ class PostgreSQLOps(DataAccessOps):
 
     # -- Task claiming operations ------------------------------------------
 
-    async def prune_terminal_operations(
-        self,
-        conn: DatabaseConnection,
-        table: str,
-        cutoff: datetime,
-        *,
-        batch_size: int,
-    ) -> int:
-        # Lock only the bounded candidate set. SKIP LOCKED lets multiple
-        # workers prune disjoint batches without waiting or double-deleting.
-        # Cancelled children cannot complete parent aggregation, so retain the
-        # parent guard only for completed/failed children. Before removing a
-        # cancelled child, preserve its signal by cancelling a pending parent
-        # in this transaction and refreshing the parent's retention window.
-        candidates = await conn.fetch(
-            f"""
-            SELECT candidate_operation.operation_id
-            FROM {table} candidate_operation
-            WHERE candidate_operation.status IN ('completed', 'failed', 'cancelled')
-              AND candidate_operation.updated_at < $1
-              AND (
-                  candidate_operation.status = 'cancelled'
-                  OR NOT EXISTS (
-                      SELECT 1
-                      FROM {table} parent
-                      WHERE parent.operation_id = CASE
-                          WHEN candidate_operation.result_metadata->>'parent_operation_id'
-                              ~* '^[0-9a-f]{{8}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{12}}$'
-                          THEN (candidate_operation.result_metadata->>'parent_operation_id')::uuid
-                          ELSE NULL
-                      END
-                        AND parent.bank_id = candidate_operation.bank_id
-                  )
-              )
-            ORDER BY candidate_operation.updated_at, candidate_operation.operation_id
-            LIMIT $2
-            FOR UPDATE OF candidate_operation SKIP LOCKED
-            """,
-            cutoff,
-            batch_size,
-        )
-        if not candidates:
-            return 0
-
-        candidate_ids = [row["operation_id"] for row in candidates]
-        await conn.execute(
-            f"""
-            UPDATE {table} parent
-            SET status = 'cancelled',
-                updated_at = now(),
-                completed_at = COALESCE(parent.completed_at, now()),
-                error_message = COALESCE(
-                    parent.error_message,
-                    'Cancelled because a child operation was cancelled'
-                )
-            WHERE parent.status = 'pending'
-              AND EXISTS (
-                  SELECT 1
-                  FROM {table} candidate_operation
-                  WHERE candidate_operation.operation_id = ANY($1)
-                    AND candidate_operation.status = 'cancelled'
-                    AND candidate_operation.updated_at < $2
-                    AND candidate_operation.bank_id = parent.bank_id
-                    AND parent.operation_id = CASE
-                        WHEN candidate_operation.result_metadata->>'parent_operation_id'
-                            ~* '^[0-9a-f]{{8}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{4}}-[0-9a-f]{{12}}$'
-                        THEN (candidate_operation.result_metadata->>'parent_operation_id')::uuid
-                        ELSE NULL
-                    END
-              )
-            """,
-            candidate_ids,
-            cutoff,
-        )
-        rows = await conn.fetch(
-            f"""
-            DELETE FROM {table}
-            WHERE operation_id = ANY($1)
-              AND status IN ('completed', 'failed', 'cancelled')
-              AND updated_at < $2
-            RETURNING operation_id
-            """,
-            candidate_ids,
-            cutoff,
-        )
-        return len(rows)
-
     async def _claim_consolidation_tasks(
         self,
         conn,
@@ -1427,14 +1157,13 @@ class PostgreSQLOps(DataAccessOps):
             else:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type = $1
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type = $1
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                    ORDER BY created_at
                     LIMIT $2
                     FOR UPDATE SKIP LOCKED
                     """,
@@ -1449,21 +1178,18 @@ class PostgreSQLOps(DataAccessOps):
         # --- Phase 2: claim from shared pool ---
         remaining_shared = shared_limit
         if remaining_shared > 0:
-            # 2a. Non-consolidation tasks. graph_maintenance stays in this
-            # created_at-ordered query — see graph_maintenance_bank_serialization_sql
-            # for why it is a predicate rather than a phase of its own.
+            # 2a. Non-consolidation tasks
             if claimed_ids:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type != 'consolidation'
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND o.operation_id != ALL($1::uuid[])
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type != 'consolidation'
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                      AND operation_id != ALL($1::uuid[])
+                    ORDER BY created_at
                     LIMIT $2
                     FOR UPDATE SKIP LOCKED
                     """,
@@ -1473,14 +1199,13 @@ class PostgreSQLOps(DataAccessOps):
             else:
                 rows = await conn.fetch(
                     f"""
-                    SELECT o.operation_id, o.operation_type, o.task_payload, o.retry_count
-                    FROM {table} o
-                    WHERE o.status = 'pending'
-                      AND o.task_payload IS NOT NULL
-                      AND o.operation_type != 'consolidation'
-                      AND (o.next_retry_at IS NULL OR o.next_retry_at <= NOW())
-                      AND {graph_maintenance_bank_serialization_sql(table, "o")}
-                    ORDER BY o.created_at
+                    SELECT operation_id, operation_type, task_payload, retry_count
+                    FROM {table}
+                    WHERE status = 'pending'
+                      AND task_payload IS NOT NULL
+                      AND operation_type != 'consolidation'
+                      AND (next_retry_at IS NULL OR next_retry_at <= NOW())
+                    ORDER BY created_at
                     LIMIT $1
                     FOR UPDATE SKIP LOCKED
                     """,

hindsight-api-slim/hindsight_api/engine/db/optional_routines.py

@@ -106,15 +106,6 @@ SCHEMAS_WITH_PENDING_WORK = OptionalRoutine(
         deployment.
       * Should be cheap and idempotent — called every poll cycle (~30s).
 
-    The poller trusts the result wholesale: any schema the routine does
-    not return is treated as having no work this cycle. It does NOT
-    second-guess omissions with a per-schema scan — that would re-run the
-    exact queries this routine exists to avoid. Consequently the routine
-    is *only* appropriate for multi-tenant deployments. Single-schema
-    (default/public only) installs should NOT create it: the per-schema
-    fallback below is a single cheap EXISTS check that covers ``public``
-    correctly and cannot starve.
-
     Fallback when the routine is absent: per-schema ``EXISTS`` queries
     from Python (~4ms per schema). The server-side path is a single-
     round-trip optimisation worth ~200ms in deployments with thousands

hindsight-api-slim/hindsight_api/engine/db/oracle.py

@@ -14,7 +14,6 @@ Supports multi-tenant schema isolation via ALTER SESSION SET CURRENT_SCHEMA.
 """
 
 import datetime
-import inspect
 import json
 import logging
 import re
@@ -23,8 +22,6 @@ from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 from typing import Any, NamedTuple
 
-from .pool_instrumentation import PoolStats, acquire_conn
-
 
 class _OracleJSONEncoder(json.JSONEncoder):
     """JSON encoder that handles datetime and UUID objects."""
@@ -80,9 +77,7 @@ _LIKE_ANY_RE = re.compile(r"(\w+)\s+LIKE\s+ANY\s*\(\s*:(\d+)\s*\)", re.IGNORECAS
 _NOT_LIKE_ALL_RE = re.compile(r"(\w+)\s+NOT\s+LIKE\s+ALL\s*\(\s*:(\d+)\s*\)", re.IGNORECASE)
 
 _JSON_ARROW_TEXT_RE = re.compile(r'("?\w+"?)\s*->>\s*\'(\w+)\'')  # handles both col and "col"
-# Reserved-word columns ("trigger") are already quoted by the time this runs, so the
-# column group must accept the quoted form too — same shape as the arrow regex above.
-_JSON_HAS_KEY_RE = re.compile(r"(\"?\w+\"?)\s*\?\s*'(\w+)'")
+_JSON_HAS_KEY_RE = re.compile(r"(\w+)\s*\?\s*'(\w+)'")
 _JSONB_CONTAINS_RE = re.compile(r"(\w+)\s*@>\s*:(\d+)")
 
 # ---------------------------------------------------------------------------
@@ -150,7 +145,6 @@ _JSON_COL_NAMES = {
     "config",
     "observation_scopes",
     "source_memory_ids",
-    "causal_links",
     "trigger",
     "http_config",
     "event_types",
@@ -161,23 +155,6 @@ _JSON_COL_NAMES = {
     "task_payload",
     "history",
 }
-# NOTE: the history tables' JSON payload column is named ``content`` — deliberately
-# NOT added here, because ``mental_models.content`` is plain text (adding "content"
-# would corrupt those reads). The history read paths json.loads ``content`` directly.
-
-# Columns backed by CLOB in Oracle (large text or JSON). When such a column is
-# returned via a ``RETURNING`` clause it must be bound as DB_TYPE_CLOB; binding
-# it as VARCHAR raises ORA-22835 ("buffer too small for CLOB to CHAR") once the
-# value exceeds 4000 bytes. Union of the JSON-CLOB columns above and the
-# large-text CLOB columns.
-_CLOB_RETURNING_COLS = _JSON_COL_NAMES | {
-    "content",
-    "text",
-    "context",
-    "structured_content",
-    "text_signals",
-    "search_vector",
-}
 
 
 def _is_uuid_column(col: str) -> bool:
@@ -449,6 +426,9 @@ def _rewrite_pg_to_oracle(query: str) -> RewriteResult:
     if has_for_update:
         # FOR UPDATE path: use ROWNUM instead of FETCH FIRST.
         # Extract and remove LIMIT clause, inject ROWNUM into WHERE.
+        def _limit_to_rownum(m):
+            return ""  # Remove the LIMIT clause; we'll add ROWNUM below
+
         limit_val = None
         limit_match = re.search(r"\bLIMIT\s+(\d+|:\w+)\b", query, re.IGNORECASE)
         if limit_match:
@@ -689,6 +669,7 @@ class OracleConnection(DatabaseConnection):
                 "max_tokens",
                 "priority",
                 "proof_count",
+                "access_count",
                 "importance_score",
                 "decay_factor",
                 "chunk_index",
@@ -704,11 +685,6 @@ class OracleConnection(DatabaseConnection):
                     params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_TIMESTAMP_TZ, arraysize=1)
                 elif clean in _NUMERIC_COLS:
                     params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_NUMBER, arraysize=1)
-                elif clean in _CLOB_RETURNING_COLS:
-                    # CLOB-backed column: a VARCHAR out-bind caps at 4000 bytes and
-                    # raises ORA-22835 for larger values. Read back as a LOB in
-                    # _read_returning_values.
-                    params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_CLOB, arraysize=1)
                 else:
                     params[f"ret_{i}"] = cursor.var(oracledb.DB_TYPE_VARCHAR, arraysize=1)
 
@@ -886,7 +862,7 @@ class OracleConnection(DatabaseConnection):
 
         return query, params
 
-    async def _read_returning_values(self, returning_cols: list[str], params: dict[str, Any]) -> dict[str, Any] | None:
+    def _read_returning_values(self, returning_cols: list[str], params: dict[str, Any]) -> dict[str, Any] | None:
         """Read values from RETURNING INTO output variables after execute."""
         row: dict[str, Any] = {}
         for i, col in enumerate(returning_cols):
@@ -896,14 +872,6 @@ class OracleConnection(DatabaseConnection):
                 return None
             val = values[0] if isinstance(values, list) else values
 
-            # CLOB-bound columns return a LOB handle; read it to a string. The
-            # async pool yields AsyncLOB whose read() is a coroutine.
-            if val is not None and not isinstance(val, (str, bytes, int, float)) and hasattr(val, "read"):
-                data = val.read()
-                if inspect.isawaitable(data):
-                    data = await data
-                val = data
-
             # Clean alias: "LOWER(canonical_name) AS name_lower" → "name_lower"
             clean_col = col.strip()
             upper = clean_col.upper()
@@ -1091,7 +1059,7 @@ class OracleConnection(DatabaseConnection):
                 raise
 
             if ret_cols is not None:
-                row_dict = await self._read_returning_values(ret_cols, params)
+                row_dict = self._read_returning_values(ret_cols, params)
                 return [ResultRow(row_dict)] if row_dict else []
 
             columns = [col[0].lower() for col in cursor.description or []]
@@ -1129,7 +1097,7 @@ class OracleConnection(DatabaseConnection):
                 raise
 
             if ret_cols is not None:
-                row_dict = await self._read_returning_values(ret_cols, params)
+                row_dict = self._read_returning_values(ret_cols, params)
                 return ResultRow(row_dict) if row_dict else None
 
             columns = [col[0].lower() for col in cursor.description or []]
@@ -1162,7 +1130,7 @@ class OracleConnection(DatabaseConnection):
             await cursor.execute(query, params)
 
             if ret_cols is not None:
-                row_dict = await self._read_returning_values(ret_cols, params)
+                row_dict = self._read_returning_values(ret_cols, params)
                 if row_dict is None:
                     return None
                 vals = list(row_dict.values())
@@ -1243,11 +1211,6 @@ class OracleBackend(DatabaseBackend):
     def __init__(self) -> None:
         self._pool: Any = None
         self._oracledb: Any = None
-        # Oracle pooled sessions retain CURRENT_SCHEMA across checkouts. Cache
-        # SESSION_USER so default-schema acquisitions can explicitly reset a
-        # connection that was previously used for a tenant schema.
-        self._default_schema: str | None = None
-        self._acquire_warn_threshold_s: float = 1.0
 
     async def initialize(
         self,
@@ -1263,10 +1226,6 @@ class OracleBackend(DatabaseBackend):
         oracledb = _import_oracledb()
         self._oracledb = oracledb
 
-        from ...config import get_config
-
-        self._acquire_warn_threshold_s = get_config().db_acquire_warn_threshold_ms / 1000.0
-
         # Parse URL-format DSN (oracle://user:pass@host:port/service)
         from urllib.parse import urlparse
 
@@ -1287,17 +1246,11 @@ class OracleBackend(DatabaseBackend):
         logger.info(f"Oracle pool created (min={min_size}, max={max_size})")
 
     async def shutdown(self) -> None:
-        # Drop the reference before awaiting close() so is_ready flips False for
-        # the whole teardown, not just after it completes (see PostgreSQLBackend).
-        pool, self._pool = self._pool, None
-        if pool is not None:
-            await pool.close(force=True)
+        if self._pool is not None:
+            await self._pool.close(force=True)
+            self._pool = None
             logger.info("Oracle pool closed")
 
-    @property
-    def is_ready(self) -> bool:
-        return self._pool is not None
-
     async def _set_session_schema(self, conn: Any) -> None:
         """Set the session schema on an Oracle connection.
 
@@ -1310,41 +1263,15 @@ class OracleBackend(DatabaseBackend):
         from ..memory_engine import get_current_schema
 
         schema = get_current_schema()
-        cursor = conn.cursor()
-        try:
-            if self._default_schema is None:
-                await cursor.execute("SELECT SYS_CONTEXT('USERENV', 'SESSION_USER') FROM DUAL")
-                row = await cursor.fetchone()
-                if not row or not row[0]:
-                    raise RuntimeError("Oracle did not return SESSION_USER while resetting CURRENT_SCHEMA")
-                self._default_schema = str(row[0])
-
-            target_schema = self._default_schema if not schema or schema == "public" else schema
-            safe_schema = target_schema.replace('"', '""')
-            await cursor.execute(f'ALTER SESSION SET CURRENT_SCHEMA = "{safe_schema}"')
-        finally:
-            # oracledb's AsyncCursor.close() is synchronous (not a coroutine);
-            # awaiting it raises "object NoneType can't be used in 'await'
-            # expression" and aborts every acquire().
-            cursor.close()
-
-    def _pool_stats(self) -> PoolStats | None:
-        """Snapshot for slow-acquire logs, from oracledb pool attributes."""
-        pool = self._pool
-        if pool is None:
-            return None
-        try:
-            busy = pool.busy
-            return PoolStats(in_use=busy, max=pool.max, idle=pool.opened - busy)
-        except Exception:
-            return None
+        if schema and schema != "public":
+            cursor = conn.cursor()
+            await cursor.execute(f'ALTER SESSION SET CURRENT_SCHEMA = "{schema}"')
+            await cursor.close()
 
     @asynccontextmanager
     async def acquire(self) -> AsyncIterator[OracleConnection]:
         pool = self._ensure_pool()
-        conn = await acquire_conn(
-            pool.acquire(), pool_stats=self._pool_stats, warn_threshold_s=self._acquire_warn_threshold_s
-        )
+        conn = await pool.acquire()
         try:
             await self._set_session_schema(conn)
             yield OracleConnection(conn)
@@ -1360,9 +1287,7 @@ class OracleBackend(DatabaseBackend):
     @asynccontextmanager
     async def transaction(self) -> AsyncIterator[OracleConnection]:
         pool = self._ensure_pool()
-        conn = await acquire_conn(
-            pool.acquire(), pool_stats=self._pool_stats, warn_threshold_s=self._acquire_warn_threshold_s
-        )
+        conn = await pool.acquire()
         try:
             await self._set_session_schema(conn)
             yield OracleConnection(conn)

hindsight-api-slim/hindsight_api/engine/db/pool_instrumentation.py

@@ -1,137 +0,0 @@
-"""Instrumentation for database connection-pool acquisition.
-
-asyncpg exposes pool *size* and *idle* counts, but not how many callers are
-currently **queued waiting** for a connection — and that queue depth is the
-signal that actually distinguishes a saturated pool from a healthy one. When the
-pool is exhausted, ``/health`` (which itself acquires a connection to run
-``SELECT 1``) blocks in ``pool.acquire()`` until a connection frees or the acquire
-times out, so a liveness probe can fail **with the event loop completely idle**.
-
-This module tracks the process-wide count of in-flight acquisitions that have not
-yet obtained a connection, and times each acquire so a slow one logs with full
-pool stats. It is the DB-side counterpart to ``loop_watchdog`` (which covers loop
-stalls); together, a stuck ``/health`` can be attributed to either a blocked loop
-or pool exhaustion from the logs alone.
-
-The counter is a plain int mutated only from the event-loop thread (asyncpg
-acquisitions are awaited on the loop), so no lock is needed.
-"""
-
-from __future__ import annotations
-
-import logging
-import time
-from collections.abc import AsyncIterator, Callable
-from contextlib import asynccontextmanager
-from dataclasses import dataclass
-from typing import Any
-
-logger = logging.getLogger("hindsight.db.pool")
-
-_waiting = 0  # callers currently blocked in pool.acquire(), process-wide
-
-
-@dataclass(frozen=True, slots=True)
-class PoolStats:
-    """Point-in-time connection-pool utilization snapshot."""
-
-    in_use: int
-    max: int
-    idle: int
-
-
-def waiting_count() -> int:
-    """Number of callers currently blocked waiting to acquire a pooled connection."""
-    return _waiting
-
-
-@asynccontextmanager
-async def instrument_acquire(
-    acquire_cm: Any,
-    *,
-    pool_stats: Callable[[], PoolStats | None] | None = None,
-    warn_threshold_s: float,
-) -> AsyncIterator[Any]:
-    """Wrap a pool's ``acquire()`` context manager with wait tracking + slow-acquire logging.
-
-    Args:
-        acquire_cm: an async context manager yielding a connection (e.g. the object
-            returned by ``asyncpg.Pool.acquire()``).
-        pool_stats: optional zero-arg callable returning a ``PoolStats`` snapshot for
-            the slow-acquire log line.
-        warn_threshold_s: log a warning when the acquire itself takes at least this long.
-
-    Yields:
-        The acquired connection.
-    """
-    global _waiting
-    _waiting += 1
-    start = time.monotonic()
-    acquired = False
-    try:
-        async with acquire_cm as conn:
-            acquired = True
-            _waiting -= 1
-            _record_acquire_wait(time.monotonic() - start, pool_stats, warn_threshold_s)
-            yield conn
-    finally:
-        # If __aenter__ raised (acquire timeout / cancellation), we never
-        # decremented above — do it here so the waiter count can't leak.
-        if not acquired:
-            _waiting -= 1
-
-
-async def acquire_conn(
-    acquire_awaitable: Any,
-    *,
-    pool_stats: Callable[[], PoolStats | None] | None = None,
-    warn_threshold_s: float,
-) -> Any:
-    """Await a pool acquire that returns a connection, with wait tracking + slow log.
-
-    For pools whose acquire is ``conn = await pool.acquire()`` (oracledb) rather than
-    an async context manager (asyncpg — use ``instrument_acquire`` for those). The
-    caller is responsible for releasing the returned connection.
-    """
-    global _waiting
-    _waiting += 1
-    start = time.monotonic()
-    try:
-        conn = await acquire_awaitable
-    finally:
-        _waiting -= 1
-    _record_acquire_wait(time.monotonic() - start, pool_stats, warn_threshold_s)
-    return conn
-
-
-def _record_acquire_wait(
-    wait_s: float,
-    pool_stats: Callable[[], PoolStats | None] | None,
-    warn_threshold_s: float,
-) -> None:
-    try:
-        from ...metrics import get_metrics_collector
-
-        get_metrics_collector().record_db_acquire_wait(wait_s)
-    except Exception:
-        pass
-
-    if wait_s < warn_threshold_s:
-        return
-
-    stats: PoolStats | None = None
-    if pool_stats is not None:
-        try:
-            stats = pool_stats()
-        except Exception:
-            stats = None
-    logger.warning(
-        "slow DB pool acquire: waited %.3fs for a connection "
-        "(in_use=%s max=%s idle=%s waiting=%s). The pool is likely saturated; "
-        "/health can stall on connection acquisition while the event loop is free.",
-        wait_s,
-        stats.in_use if stats else None,
-        stats.max if stats else None,
-        stats.idle if stats else None,
-        _waiting,
-    )

hindsight-api-slim/hindsight_api/engine/db/postgresql.py

@@ -15,7 +15,6 @@ from typing import Any
 import asyncpg  # noqa: F401
 
 from .base import DatabaseBackend, DatabaseConnection
-from .pool_instrumentation import PoolStats, instrument_acquire
 
 logger = logging.getLogger(__name__)
 
@@ -77,8 +76,6 @@ class PostgreSQLBackend(DatabaseBackend):
 
     def __init__(self) -> None:
         self._pool: asyncpg.Pool | None = None
-        self._acquire_warn_threshold_s: float = 1.0
-        self._acquire_timeout_s: float | None = None
 
     async def initialize(
         self,
@@ -91,16 +88,6 @@ class PostgreSQLBackend(DatabaseBackend):
         statement_cache_size: int = 0,
         init_callback: Any | None = None,
     ) -> None:
-        from ...config import get_config
-
-        self._acquire_warn_threshold_s = get_config().db_acquire_warn_threshold_ms / 1000.0
-        # Kept for acquire() below: asyncpg's ``timeout`` create_pool kwarg is a
-        # *connect* kwarg (how long establishing a new connection may take), and
-        # ``Pool.acquire()`` defaults to waiting for a free connection forever.
-        # Passing it here alone made HINDSIGHT_API_DB_ACQUIRE_TIMEOUT a no-op for
-        # the wait it names: a pool-exhaustion stall never surfaced as an error,
-        # it just hung (#3002). 0 restores the unbounded behaviour.
-        self._acquire_timeout_s = acquire_timeout if acquire_timeout > 0 else None
         self._pool = await asyncpg.create_pool(
             dsn,
             min_size=min_size,
@@ -108,12 +95,7 @@ class PostgreSQLBackend(DatabaseBackend):
             command_timeout=command_timeout,
             statement_cache_size=statement_cache_size,
             timeout=acquire_timeout,
-            # init runs once per new connection; setup runs on every acquire,
-            # after asyncpg's release-time RESET ALL. Passing init_callback as
-            # both keeps the per-connection session GUCs (hnsw.ef_search, etc.)
-            # applied after a connection is reused, not just on first creation.
             init=init_callback,
-            setup=init_callback,
         )
         logger.info(
             f"PostgreSQL pool created (min={min_size}, max={max_size}, "
@@ -121,45 +103,21 @@ class PostgreSQLBackend(DatabaseBackend):
         )
 
     async def shutdown(self) -> None:
-        # Drop the reference *before* awaiting close(): closing is not
-        # instantaneous, and anything acquiring during that window would
-        # otherwise get an asyncpg "pool is closing" error rather than seeing
-        # is_ready False.
-        pool, self._pool = self._pool, None
-        if pool is not None:
-            await pool.close()
+        if self._pool is not None:
+            await self._pool.close()
+            self._pool = None
             logger.info("PostgreSQL pool closed")
 
-    @property
-    def is_ready(self) -> bool:
-        return self._pool is not None
-
-    def _pool_stats(self) -> PoolStats | None:
-        """Snapshot for slow-acquire logs. in_use = live connections minus idle ones."""
-        pool = self._pool
-        if pool is None:
-            return None
-        idle = pool.get_idle_size()
-        return PoolStats(in_use=pool.get_size() - idle, max=pool.get_max_size(), idle=idle)
-
     @asynccontextmanager
     async def acquire(self) -> AsyncIterator[PostgresConnection]:
         pool = self._ensure_pool()
-        async with instrument_acquire(
-            pool.acquire(timeout=self._acquire_timeout_s),
-            pool_stats=self._pool_stats,
-            warn_threshold_s=self._acquire_warn_threshold_s,
-        ) as conn:
+        async with pool.acquire() as conn:
             yield PostgresConnection(conn)
 
     @asynccontextmanager
     async def transaction(self) -> AsyncIterator[PostgresConnection]:
         pool = self._ensure_pool()
-        async with instrument_acquire(
-            pool.acquire(timeout=self._acquire_timeout_s),
-            pool_stats=self._pool_stats,
-            warn_threshold_s=self._acquire_warn_threshold_s,
-        ) as conn:
+        async with pool.acquire() as conn:
             async with conn.transaction():
                 yield PostgresConnection(conn)
 

hindsight-api-slim/hindsight_api/engine/db_budget.py

@@ -164,6 +164,35 @@ class BudgetedOperation:
         """
         return BudgetedPool(pool, self)
 
+    async def acquire_many(
+        self,
+        pool: Any,
+        count: int,
+    ) -> AsyncIterator[list[Any]]:
+        """
+        Acquire multiple connections within the budget.
+
+        Note: This acquires connections sequentially to respect the budget.
+        For parallel acquisition, use multiple acquire() calls with asyncio.gather().
+        This method is intended for use with raw asyncpg pools only, not DatabaseBackend.
+
+        Args:
+            pool: asyncpg connection pool (raw pool only)
+            count: Number of connections to acquire
+
+        Yields:
+            List of database connections
+        """
+        connections = []
+        try:
+            for _ in range(count):
+                conn = await pool.acquire()
+                connections.append(conn)
+            yield connections
+        finally:
+            for conn in connections:
+                await pool.release(conn)
+
 
 # Global default manager instance
 _default_manager: ConnectionBudgetManager | None = None

hindsight-api-slim/hindsight_api/engine/db_utils.py

@@ -4,7 +4,6 @@ Database utility functions for connection management with retry logic.
 
 import asyncio
 import logging
-import random
 import time
 from collections.abc import AsyncIterator
 from contextlib import AsyncExitStack, asynccontextmanager
@@ -17,20 +16,6 @@ DEFAULT_MAX_RETRIES = 3
 DEFAULT_BASE_DELAY = 0.5  # seconds
 DEFAULT_MAX_DELAY = 5.0  # seconds
 
-
-def _backoff_delay(attempt: int, base_delay: float, max_delay: float) -> float:
-    """Exponential backoff with equal jitter.
-
-    Deterministic backoff makes concurrent retriers wake in lock-step and
-    re-collide on the very same rows, re-triggering the deadlock they just
-    backed off from. "Equal jitter" — half the window fixed, half random —
-    keeps a floor (so we don't hot-spin) while decorrelating the wake-ups, so
-    two contenders that deadlocked together are very unlikely to retry in sync.
-    """
-    ceil = min(base_delay * (2**attempt), max_delay)
-    return ceil / 2 + random.uniform(0, ceil / 2)
-
-
 # Retryable exception types (checked by class name to avoid hard imports)
 _RETRYABLE_EXCEPTION_NAMES = frozenset(
     {
@@ -93,7 +78,7 @@ async def retry_with_backoff(
                 raise
             last_exception = e
             if attempt < max_retries:
-                delay = _backoff_delay(attempt, base_delay, max_delay)
+                delay = min(base_delay * (2**attempt), max_delay)
                 if type(e).__name__ == "DeadlockDetectedError" or _is_oracle_deadlock(e):
                     logger.warning(
                         "Deadlock detected during parallel document processing — "
@@ -151,7 +136,7 @@ async def acquire_with_retry(backend_or_pool: Any, max_retries: int = DEFAULT_MA
                     if not _is_retryable(e):
                         raise
                     if attempt < max_retries:
-                        delay = _backoff_delay(attempt, DEFAULT_BASE_DELAY, DEFAULT_MAX_DELAY)
+                        delay = min(DEFAULT_BASE_DELAY * (2**attempt), DEFAULT_MAX_DELAY)
                         logger.warning(
                             f"Database acquire failed (attempt {attempt + 1}/{max_retries + 1}): {e}. "
                             f"Retrying in {delay:.1f}s..."

hindsight-api-slim/hindsight_api/engine/embeddings.py

@@ -26,8 +26,11 @@ from ..config import (
     DEFAULT_EMBEDDINGS_GEMINI_MODEL,
     DEFAULT_EMBEDDINGS_LITELLM_MODEL,
     DEFAULT_EMBEDDINGS_LITELLM_SDK_MODEL,
+    DEFAULT_EMBEDDINGS_LOCAL_FORCE_CPU,
     DEFAULT_EMBEDDINGS_LOCAL_MODEL,
+    DEFAULT_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE,
     DEFAULT_EMBEDDINGS_OPENAI_MODEL,
+    DEFAULT_EMBEDDINGS_PROVIDER,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_BATCH_SIZE,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_DIMENSIONS,
     DEFAULT_EMBEDDINGS_ZEROENTROPY_ENCODING_FORMAT,
@@ -37,6 +40,9 @@ from ..config import (
     DEFAULT_ZEROENTROPY_BASE_URL,
     ENV_EMBEDDINGS_COHERE_API_KEY,
     ENV_EMBEDDINGS_GEMINI_API_KEY,
+    ENV_EMBEDDINGS_LOCAL_FORCE_CPU,
+    ENV_EMBEDDINGS_LOCAL_MODEL,
+    ENV_EMBEDDINGS_LOCAL_TRUST_REMOTE_CODE,
     ENV_EMBEDDINGS_OPENAI_API_KEY,
     ENV_EMBEDDINGS_OPENAI_BASE_URL,
     ENV_EMBEDDINGS_OPENAI_MODEL,
@@ -47,13 +53,6 @@ from ..config import (
     ENV_EMBEDDINGS_ZEROENTROPY_ENCODING_FORMAT,
     ENV_LLM_API_KEY,
 )
-from .bank_attribution import apply_bank_attribution
-from .local_device import (
-    release_local_inference_memory,
-    resolve_model_device_type,
-    select_local_device,
-)
-from .tei_retry import tei_retry_delay
 
 logger = logging.getLogger(__name__)
 
@@ -142,13 +141,7 @@ class LocalSTEmbeddings(Embeddings):
     The embedding dimension is auto-detected from the model.
     """
 
-    def __init__(
-        self,
-        model_name: str | None = None,
-        force_cpu: bool = False,
-        trust_remote_code: bool = False,
-        allow_mps: bool = False,
-    ):
+    def __init__(self, model_name: str | None = None, force_cpu: bool = False, trust_remote_code: bool = False):
         """
         Initialize local SentenceTransformers embeddings.
 
@@ -160,17 +153,12 @@ class LocalSTEmbeddings(Embeddings):
             trust_remote_code: Allow loading models with custom code (security risk).
                               Required for some models with custom architectures.
                               Default: False (disabled for security)
-            allow_mps: Opt in to the Apple Silicon MPS GPU. Disabled by default
-                      because MPS leaks memory under variable-length workloads
-                      (see engine/local_device.py). Default: False
         """
         self.model_name = model_name or DEFAULT_EMBEDDINGS_LOCAL_MODEL
         self.force_cpu = force_cpu
         self.trust_remote_code = trust_remote_code
-        self.allow_mps = allow_mps
         self._model = None
         self._dimension: int | None = None
-        self._device_type: str = "cpu"
 
     @property
     def provider_name(self) -> str:
@@ -197,11 +185,28 @@ class LocalSTEmbeddings(Embeddings):
 
         logger.info(f"Embeddings: initializing local provider with model {self.model_name}")
 
-        # Determine device based on hardware availability. We always set
-        # low_cpu_mem_usage=False to prevent lazy loading (meta tensors) which can
-        # cause issues when accelerate is installed but no GPU is available.
-        # MPS is opt-in (allow_mps) — see engine/local_device.py for why.
-        device = select_local_device(self.force_cpu, self.allow_mps)
+        # Determine device based on hardware availability.
+        # We always set low_cpu_mem_usage=False to prevent lazy loading (meta tensors)
+        # which can cause issues when accelerate is installed but no GPU is available.
+        import torch
+
+        # Force CPU mode if configured (used in daemon mode to avoid MPS/XPC issues on macOS)
+        if self.force_cpu:
+            device = "cpu"
+            logger.info("Embeddings: forcing CPU mode")
+        else:
+            # Check for GPU (CUDA) or Apple Silicon (MPS)
+            # Wrap in try-except to gracefully handle any device detection issues
+            # (e.g., in CI environments or when PyTorch is built without GPU support)
+            device = "cpu"  # Default to CPU
+            try:
+                has_gpu = torch.cuda.is_available() or (
+                    hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+                )
+                if has_gpu:
+                    device = None  # Let sentence-transformers auto-detect GPU/MPS
+            except Exception as e:
+                logger.warning(f"Failed to detect GPU/MPS, falling back to CPU: {e}")
 
         # Suppress verbose transformers warnings during model loading
         # This suppresses the "UNEXPECTED" warnings from BertModel which are harmless
@@ -228,8 +233,7 @@ class LocalSTEmbeddings(Embeddings):
                 transformers_logger.setLevel(original_level)
 
         self._dimension = self._model.get_sentence_embedding_dimension()
-        self._device_type = resolve_model_device_type(self._model)
-        logger.info(f"Embeddings: local provider initialized (dim: {self._dimension}, device: {self._device_type})")
+        logger.info(f"Embeddings: local provider initialized (dim: {self._dimension})")
 
     def encode(self, texts: list[str]) -> list[list[float]]:
         """
@@ -241,215 +245,11 @@ class LocalSTEmbeddings(Embeddings):
         Returns:
             List of embedding vectors
         """
-        return self._encode_local(texts)
-
-    def encode_query(self, texts: list[str]) -> list[list[float]]:
-        return self._encode_local(texts, input_type="query")
-
-    def encode_documents(self, texts: list[str]) -> list[list[float]]:
-        return self._encode_local(texts, input_type="document")
-
-    def _encode_local(
-        self, texts: list[str], input_type: Literal["query", "document"] | None = None
-    ) -> list[list[float]]:
         if self._model is None:
             raise RuntimeError("Embeddings not initialized. Call initialize() first.")
 
-        try:
-            # Delegate to SentenceTransformers' own asymmetric entry points rather than
-            # prefixing here: they apply whatever prompts the model ships with (and route
-            # the task for models exposing a Router module), so asymmetric models such as
-            # Qwen3-Embedding get their configured query prompt without Hindsight carrying
-            # per-model prefix config the way the ONNX provider has to. Models that declare
-            # no prompts are unaffected — SentenceTransformers defaults them to empty
-            # strings and skips prompt handling entirely, so this is byte-identical to
-            # encode() for e.g. the default BAAI/bge-small-en-v1.5.
-            # encode_query/encode_document exist only in sentence-transformers >= 5.0,
-            # which is why local-ml pins that floor.
-            if input_type == "query":
-                encode = self._model.encode_query
-            elif input_type == "document":
-                encode = self._model.encode_document
-            else:
-                encode = self._model.encode
-            embeddings = encode(texts, convert_to_numpy=True, show_progress_bar=False)
-            return [emb.tolist() for emb in embeddings]
-        finally:
-            # Only reclaim the GPU allocator pool here, and only when actually on a
-            # GPU (opt-in MPS/CUDA/XPU). encode() runs in tight retain loops, so a
-            # gc.collect()/malloc_trim on every call is too costly on the CPU default
-            # — and unnecessary: refcounting frees the small transient buffers
-            # immediately and the allocator reuses them for the next batch. (The
-            # reranker keeps its per-batch heap trim for the #1717 CPU case; it runs
-            # on the lighter recall path.) See engine/local_device.py.
-            if self._device_type != "cpu":
-                release_local_inference_memory(self._device_type)
-
-
-class OnnxEmbeddings(Embeddings):
-    """Local ONNX Runtime embeddings provider.
-
-    This provider runs transformer embedding models in-process with ONNX Runtime,
-    avoiding a sidecar Ollama/TEI server or a remote embeddings API. It supports
-    sentence-transformer style mean pooling and E5-style asymmetric prefixes.
-    """
-
-    def __init__(
-        self,
-        model_id: str,
-        model_path: str | None = None,
-        tokenizer_name_or_path: str | None = None,
-        onnx_file: str = "onnx/model.onnx",
-        dimensions: int | None = None,
-        max_tokens: int = 512,
-        pooling: str = "mean",
-        normalize: bool = True,
-        query_prefix: str = "query: ",
-        passage_prefix: str = "passage: ",
-        output_name: str | None = None,
-    ):
-        self.model_id = model_id
-        self.model_path = model_path
-        if model_path and tokenizer_name_or_path is None:
-            logger.warning(
-                "Embeddings: ONNX model_path is set without tokenizer_name_or_path; "
-                "falling back to tokenizer from model_id %s. Set "
-                "HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH when using local ONNX artifacts.",
-                model_id,
-            )
-        self.tokenizer_name_or_path = tokenizer_name_or_path or model_id
-        self.onnx_file = onnx_file
-        self.configured_dimensions = dimensions
-        self.max_tokens = max_tokens
-        self.pooling = pooling.lower()
-        if self.pooling not in {"mean", "cls"}:
-            raise ValueError("ONNX embeddings pooling must be 'mean' or 'cls'")
-        self.normalize = normalize
-        self.query_prefix = query_prefix
-        self.passage_prefix = passage_prefix
-        self.output_name = output_name
-        self._session = None
-        self._tokenizer = None
-        self._dimension: int | None = dimensions
-
-    @property
-    def provider_name(self) -> str:
-        return "onnx"
-
-    @property
-    def dimension(self) -> int:
-        if self._dimension is None:
-            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
-        return self._dimension
-
-    async def initialize(self) -> None:
-        if self._session is not None and self._tokenizer is not None:
-            return
-
-        try:
-            import onnxruntime as ort
-            from transformers import AutoTokenizer
-        except ImportError as exc:
-            raise ImportError(
-                "onnxruntime and transformers are required for OnnxEmbeddings. "
-                "Install with: pip install 'hindsight-api-slim[local-onnx]'"
-            ) from exc
-
-        model_path = self.model_path
-        if not model_path:
-            try:
-                from huggingface_hub import snapshot_download
-            except ImportError as exc:
-                raise ImportError(
-                    "huggingface-hub is required to download ONNX embedding models. "
-                    "Set HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH or install local-onnx."
-                ) from exc
-            # Some large ONNX exports, for example BAAI/bge-m3, store weights in
-            # an external sidecar file next to model.onnx. Download both the
-            # requested graph and its conventional *_data sidecar when present.
-            snapshot_dir = snapshot_download(
-                repo_id=self.model_id,
-                allow_patterns=[self.onnx_file, f"{self.onnx_file}_data"],
-            )
-            model_path = os.path.join(snapshot_dir, self.onnx_file)
-
-        logger.info(
-            "Embeddings: initializing ONNX provider with model %s (%s)",
-            self.model_id,
-            model_path,
-        )
-        logger.info(
-            "Embeddings: ONNX query_prefix=%r passage_prefix=%r pooling=%s normalize=%s",
-            self.query_prefix,
-            self.passage_prefix,
-            self.pooling,
-            self.normalize,
-        )
-        self._tokenizer = AutoTokenizer.from_pretrained(self.tokenizer_name_or_path)
-        self._session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
-
-        detected = len(self.encode(["test"])[0])
-        if self.configured_dimensions is not None and detected != self.configured_dimensions:
-            raise ValueError(
-                f"Configured ONNX embedding dimension {self.configured_dimensions} does not match model output {detected}"
-            )
-        self._dimension = detected
-        logger.info("Embeddings: ONNX provider initialized (dim: %s)", self._dimension)
-
-    def _encode_prefixed(self, texts: list[str], prefix: str) -> list[list[float]]:
-        if prefix:
-            return self.encode([f"{prefix}{text}" for text in texts])
-        return self.encode(texts)
-
-    def encode_query(self, texts: list[str]) -> list[list[float]]:
-        return self._encode_prefixed(texts, self.query_prefix)
-
-    def encode_documents(self, texts: list[str]) -> list[list[float]]:
-        return self._encode_prefixed(texts, self.passage_prefix)
-
-    def encode(self, texts: list[str]) -> list[list[float]]:
-        if self._session is None or self._tokenizer is None:
-            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
-        if not texts:
-            return []
-
-        import numpy as np
-
-        encoded = self._tokenizer(
-            texts,
-            padding=True,
-            truncation=True,
-            max_length=self.max_tokens,
-            return_tensors="np",
-        )
-        input_names = {inp.name for inp in self._session.get_inputs()}
-        ort_inputs = {name: value for name, value in encoded.items() if name in input_names}
-        if "token_type_ids" in input_names and "token_type_ids" not in ort_inputs:
-            ort_inputs["token_type_ids"] = np.zeros_like(encoded["input_ids"])
-
-        outputs = self._session.run([self.output_name] if self.output_name else None, ort_inputs)
-        token_embeddings = outputs[0]
-
-        # Some exported models expose a pooled 2-D embedding as their first output.
-        if getattr(token_embeddings, "ndim", 0) == 2:
-            embeddings = token_embeddings
-        elif self.pooling == "cls":
-            embeddings = token_embeddings[:, 0]
-        else:
-            attention_mask = encoded.get("attention_mask")
-            if attention_mask is None:
-                attention_mask = np.ones(token_embeddings.shape[:2], dtype=np.float32)
-            mask = attention_mask[..., None].astype(np.float32)
-            summed = (token_embeddings * mask).sum(axis=1)
-            counts = np.clip(mask.sum(axis=1), a_min=1e-9, a_max=None)
-            embeddings = summed / counts
-
-        if self.normalize:
-            norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
-            norms[norms == 0] = 1
-            embeddings = embeddings / norms
-
-        return embeddings.astype(float).tolist()
+        embeddings = self._model.encode(texts, convert_to_numpy=True, show_progress_bar=False)
+        return [emb.tolist() for emb in embeddings]
 
 
 class RemoteTEIEmbeddings(Embeddings):
@@ -514,7 +314,7 @@ class RemoteTEIEmbeddings(Embeddings):
                     response = self._client.post(url, **kwargs)
                 response.raise_for_status()
                 return response
-            except (httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout, httpx.WriteTimeout) as e:
+            except (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout) as e:
                 last_error = e
                 if attempt < self.max_retries:
                     logger.warning(
@@ -523,20 +323,13 @@ class RemoteTEIEmbeddings(Embeddings):
                     time.sleep(delay)
                     delay *= 2  # Exponential backoff
             except httpx.HTTPStatusError as e:
-                # TEI uses 429 as normal overload backpressure. Retry it with
-                # the same bounded budget as transient server errors.
-                if (e.response.status_code == 429 or e.response.status_code >= 500) and attempt < self.max_retries:
+                # Retry on 5xx server errors
+                if e.response.status_code >= 500 and attempt < self.max_retries:
                     last_error = e
-                    sleep_delay = tei_retry_delay(
-                        e.response,
-                        delay,
-                        request_timeout=self.timeout,
-                    )
                     logger.warning(
-                        f"TEI transient error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. "
-                        f"Retrying in {sleep_delay:.2f}s..."
+                        f"TEI server error (attempt {attempt + 1}/{self.max_retries + 1}): {e}. Retrying in {delay}s..."
                     )
-                    time.sleep(sleep_delay)
+                    time.sleep(delay)
                     delay *= 2
                 else:
                     raise
@@ -742,7 +535,6 @@ class OpenAIEmbeddings(Embeddings):
             }
             if self.dimensions is not None:
                 request["dimensions"] = self.dimensions
-            apply_bank_attribution(request)
 
             response = self._client.embeddings.create(**request)
 
@@ -755,8 +547,7 @@ class OpenAIEmbeddings(Embeddings):
 
 class CodexOAuthEmbeddings(OpenAIEmbeddings):
     """
-    OpenAI embeddings using the Codex/ChatGPT OAuth token from the Codex
-    ``auth.json`` (``$CODEX_HOME/auth.json``, or ``~/.codex/auth.json`` when unset).
+    OpenAI embeddings using the Codex/ChatGPT OAuth token from ``~/.codex/auth.json``.
 
     Codex OAuth is an LLM-provider auth path in Hindsight, but the same bearer token
     can also authenticate against the standard OpenAI embeddings endpoint. This keeps
@@ -1386,21 +1177,6 @@ class LiteLLMSDKEmbeddings(Embeddings):
         return all_embeddings
 
 
-# Gemini Embedding 2+ multimodal models return a SINGLE aggregated embedding
-# for a multi-input request instead of one vector per input (see
-# https://ai.google.dev/gemini-api/docs/embeddings#embedding-aggregation). For
-# these models we must embed one input per call to preserve the 1:1 input→vector
-# alignment the rest of the pipeline relies on. The marker matches preview and GA
-# names (e.g. "gemini-embedding-2-preview", "gemini-embedding-2"), with or
-# without a "google/" or "models/" prefix.
-_GEMINI_AGGREGATING_MODEL_MARKER = "gemini-embedding-2"
-
-
-def _gemini_model_aggregates_inputs(model: str) -> bool:
-    """Whether the model aggregates a multi-input request into one embedding."""
-    return _GEMINI_AGGREGATING_MODEL_MARKER in model.lower()
-
-
 class GeminiEmbeddings(Embeddings):
     """
     Google embeddings via the google.genai SDK.
@@ -1410,10 +1186,6 @@ class GeminiEmbeddings(Embeddings):
     2. Vertex AI with service account or Application Default Credentials (ADC)
 
     Uses the embed_content API: client.models.embed_content(model, contents)
-
-    Gemini Embedding 2+ multimodal models aggregate a multi-input request into a
-    single embedding, so for those the batch size is forced to 1 (one input per
-    call) to keep one vector per input.
     """
 
     def __init__(
@@ -1568,13 +1340,9 @@ class GeminiEmbeddings(Embeddings):
 
         all_embeddings = []
 
-        # Gemini Embedding 2+ multimodal models return one aggregated vector for a
-        # multi-input request, so embed one input per call to keep 1:1 alignment.
-        batch_size = 1 if _gemini_model_aggregates_inputs(self.model) else self.batch_size
-
         # Process in batches
-        for i in range(0, len(texts), batch_size):
-            batch = texts[i : i + batch_size]
+        for i in range(0, len(texts), self.batch_size):
+            batch = texts[i : i + self.batch_size]
 
             embed_kwargs = {"model": self.model, "contents": batch}
             if self._embed_config is not None:
@@ -1582,13 +1350,7 @@ class GeminiEmbeddings(Embeddings):
 
             result = self._client.models.embed_content(**embed_kwargs)
 
-            embeddings = result.embeddings or []
-            if len(embeddings) != len(batch):
-                raise RuntimeError(
-                    f"Gemini embeddings backend returned {len(embeddings)} vectors for "
-                    f"{len(batch)} input texts (model {self.model}); expected exact 1:1 alignment"
-                )
-            all_embeddings.extend([emb.values for emb in embeddings])
+            all_embeddings.extend([emb.values for emb in result.embeddings])
 
         # L2-normalize when output_dimensionality is set — Gemini only returns
         # normalized vectors at full 3072 dims; truncated dims need re-normalization
@@ -1628,21 +1390,6 @@ def create_embeddings_from_env() -> Embeddings:
             model_name=config.embeddings_local_model,
             force_cpu=config.embeddings_local_force_cpu,
             trust_remote_code=config.embeddings_local_trust_remote_code,
-            allow_mps=config.embeddings_local_allow_mps,
-        )
-    elif provider == "onnx":
-        return OnnxEmbeddings(
-            model_id=config.embeddings_onnx_model_id,
-            model_path=config.embeddings_onnx_model_path,
-            tokenizer_name_or_path=config.embeddings_onnx_tokenizer_name_or_path,
-            onnx_file=config.embeddings_onnx_file,
-            dimensions=config.embeddings_onnx_dimensions,
-            max_tokens=config.embeddings_onnx_max_tokens,
-            pooling=config.embeddings_onnx_pooling,
-            normalize=config.embeddings_onnx_normalize,
-            query_prefix=config.embeddings_onnx_query_prefix,
-            passage_prefix=config.embeddings_onnx_passage_prefix,
-            output_name=config.embeddings_onnx_output_name,
         )
     elif provider == "openai":
         # Use dedicated embeddings API key, or fall back to LLM API key
@@ -1682,20 +1429,6 @@ def create_embeddings_from_env() -> Embeddings:
             batch_size=config.embeddings_openai_batch_size,
             dimensions=config.embeddings_openai_dimensions,
         )
-    elif provider == "requesty":
-        api_key = config.embeddings_requesty_api_key
-        if not api_key:
-            raise ValueError(
-                "HINDSIGHT_API_EMBEDDINGS_REQUESTY_API_KEY, HINDSIGHT_API_REQUESTY_API_KEY, "
-                f"or {ENV_LLM_API_KEY} is required when {ENV_EMBEDDINGS_PROVIDER} is 'requesty'"
-            )
-        return OpenAIEmbeddings(
-            api_key=api_key,
-            model=config.embeddings_requesty_model,
-            base_url="https://router.requesty.ai/v1",
-            batch_size=config.embeddings_openai_batch_size,
-            dimensions=config.embeddings_openai_dimensions,
-        )
     elif provider == "zeroentropy":
         api_key = config.embeddings_zeroentropy_api_key
         if not api_key:
@@ -1759,6 +1492,6 @@ def create_embeddings_from_env() -> Embeddings:
     else:
         raise ValueError(
             f"Unknown embeddings provider: {provider}. "
-            f"Supported: 'local', 'onnx', 'tei', 'openai', 'openai-codex', 'openrouter', 'requesty', 'cohere', 'google', "
+            f"Supported: 'local', 'tei', 'openai', 'openai-codex', 'openrouter', 'cohere', 'google', "
             f"'zeroentropy', 'litellm', 'litellm-sdk'"
         )