Merge remote-tracking branch 'origin/main' into fix/opencode-cloud-default-and-e2e

Branch carried an older snapshot of hindsight-docs/static/openapi.json
that pre-dated main's addition of OperationProgress. Re-checkout main's
openapi.json so check-openapi-compatibility passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

.claude-plugin/marketplace.json

@@ -0,0 +1,15 @@
+{
+  "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
+  "name": "hindsight",
+  "description": "Official Hindsight integrations for Claude Code",
+  "owner": {
+    "name": "vectorize-io"
+  },
+  "plugins": [
+    {
+      "name": "hindsight-memory",
+      "description": "Automatic long-term memory for Claude Code via Hindsight",
+      "source": "./hindsight-integrations/claude-code"
+    }
+  ]
+}

.claude/skills/code-review/SKILL.md

@@ -0,0 +1,233 @@
+---
+name: code-review
+description: Review changed code against project standards. Checks for missing tests, dead code, type safety, lint issues, and coding conventions. Run after completing any implementation work.
+user_invocable: true
+---
+
+# Code Review
+
+Review all changed code against the project's quality standards and coding conventions.
+
+## Code Standards
+
+Read and internalize these standards before writing code. The review steps below verify compliance.
+
+### Python Style
+- Python 3.11+, type hints required
+- Async throughout (asyncpg, async FastAPI)
+- Pydantic models for request/response
+- Ruff for linting (line-length 120)
+- No Python files at project root - maintain clean directory structure
+- **Never use multi-item tuple return values** — not even for internal/private functions. Always use a dataclass or Pydantic model. No exceptions, no "it's just two values" shortcuts. If a function returns more than one value, define a named type for it.
+
+### Type Safety with Pydantic Models
+**NEVER use raw `dict` types for structured data** — this applies to all code, including internal helpers and private functions. If the dict has known keys, it must be a dataclass or Pydantic model:
+- Use Pydantic `BaseModel` for all data structures passed between functions
+- Use `@dataclass` for lightweight internal data containers when Pydantic validation isn't needed
+- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
+- Avoid `dict.get()` patterns - use typed model attributes instead
+- Parse external data (JSON, API responses) into Pydantic models at the boundary
+- This catches type errors at parse time, not deep in business logic
+- The only acceptable `dict` usage is for truly dynamic/unknown keys (e.g., arbitrary metadata, JSON blobs with no fixed schema)
+
+```python
+# BAD - error-prone dict access
+def process(data: dict) -> str:
+    return data.get("name", "")  # No validation, silent failures
+
+# GOOD - typed and validated
+class UserData(BaseModel):
+    name: str
+    created_at: datetime
+
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def ensure_tz_aware(cls, v):
+        if isinstance(v, str):
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if v.tzinfo is None:
+            return v.replace(tzinfo=timezone.utc)
+        return v
+
+def process(data: UserData) -> str:
+    return data.name  # Type-safe, validated at construction
+```
+
+### TypeScript Style
+- Next.js App Router for control plane
+- Tailwind CSS with shadcn/ui components
+
+### Code Comments
+- **Always comment non-trivial technical decisions** with the reasoning behind the choice. If someone would ask "why is it done this way?", there should be a comment.
+- **Keep comments up to date with history** — when changing an approach, update the comment to explain what was tried before and why it was changed. Comments serve as a tracker of previous implementations that likely had problems.
+- Don't comment obvious code — only where the "why" isn't self-evident from the code itself.
+
+```python
+# BAD - no context for future readers
+results = await asyncio.gather(*tasks, return_exceptions=True)
+
+# GOOD - explains the non-obvious choice
+# Use return_exceptions=True to avoid cancelling sibling tasks on failure.
+# Previously we used TaskGroup but it cancelled all tasks when one failed,
+# causing partial writes that left orphaned entity links (see #412).
+results = await asyncio.gather(*tasks, return_exceptions=True)
+```
+
+### API Layer & Data Access
+- **No direct database access in `api/http.py`** (or any API router). HTTP handlers must not build SQL, call `acquire_with_retry` / `conn.fetch` / `conn.fetchrow` / `conn.execute`, or reference `fq_table(...)`. All persistence and queries live in `MemoryEngine` (the engine layer). A handler parses/validates the request, calls an engine method, shapes the HTTP response, and maps domain results to status codes (e.g. a `None` return → 404).
+- **Authentication/tenancy is enforced inside each engine method, not assumed by the handler.** Every engine method that touches bank-scoped data must authenticate via `request_context` — typically `await self._authenticate_tenant(request_context)` (often indirectly through `get_bank_profile(...)`) — so the correct tenant schema is resolved before any query runs. Handlers must thread `request_context` through to the engine method; never query a tenant-scoped table assuming the schema is already set.
+- Engine methods return typed models (Pydantic/dataclass), not raw dicts (see Type Safety).
+
+### Branch Hygiene
+- **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
+- **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
+
+### General Principles
+- Don't add features, refactor code, or make "improvements" beyond what was asked
+- Don't add unnecessary error handling for impossible scenarios
+- Don't create helpers or abstractions for one-time operations
+- No backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
+- Three similar lines of code is better than a premature abstraction
+
+## Review Steps
+
+### 1. Check branch hygiene
+
+- Run `git log --oneline main..HEAD` to list all commits on the branch.
+- Verify every commit is relevant to the feature/PR. Flag any unrelated commits.
+- Check the branch is based on a recent `origin/main` (no stale base).
+
+### 2. Identify changed files
+
+Run `git diff --name-only HEAD` (unstaged) and `git diff --cached --name-only` (staged) to get all changed files. If there are no local changes, diff against the base branch using `git diff main...HEAD --name-only` and `git diff main...HEAD` to review all commits on the current branch.
+
+### 3. Run linters
+
+```bash
+./scripts/hooks/lint.sh
+```
+
+Report any failures. Do NOT fix them yourself — just report.
+
+### 4. Check for dead code
+
+For each changed Python file, check for:
+- Unused imports (Ruff should catch these, but verify)
+- Functions/methods/classes that were added but are never called from anywhere
+- Variables assigned but never read
+- Commented-out code blocks that should be removed
+
+For each changed TypeScript file, check for:
+- Unused imports
+- Unused variables or functions
+- Commented-out code
+
+### 5. Check type safety (Python)
+
+For each changed Python file, check for violations:
+- **No raw `dict` for structured data** — must use Pydantic model or dataclass, even for internal/private functions (only exception: truly dynamic/unknown keys)
+- **No multi-item tuple returns** — must use dataclass or Pydantic model, even for internal/private functions (no exceptions)
+- **Missing type hints** on function parameters and return types
+- **Missing `@field_validator`** for datetime fields that should be timezone-aware
+
+### 6. Check for missing tests
+
+For each new or significantly changed function/endpoint/class:
+- Check if there is a corresponding test addition or update
+- New API endpoints MUST have integration tests
+- New utility functions MUST have unit tests
+- Bug fixes SHOULD have a regression test
+
+Flag any new logic that lacks test coverage.
+
+**LLM-behaviour changes need a real-LLM judge test, not MockLLM.** If the change alters how the model interprets a prompt — fact/observation extraction, `fact_type` (world/experience) classification, speaker attribution, instruction-following, prompt wording — there MUST be a test marked `pytest.mark.hs_llm_core` that runs the real pipeline and asserts via `tests.llm_judge.assert_meets_criteria` (not string/enum matching). Flag these as findings:
+- A prompt/classification change verified only by MockLLM or string assertions (MockLLM echoes input — such tests pass spuriously). **Should fix.**
+- A test that hard-asserts `fact_type == "world"/"experience"` (or other model-decided output) instead of judging it — non-deterministic, will flake across providers/runs. **Should fix** (move the classification check into the judge `criteria`; keep only genuinely deterministic structural asserts direct).
+- Deterministic mechanics (prompt assembly, suppression/branching logic) that are covered *only* by a slow LLM test — these should also have fast non-LLM unit tests. **Note.**
+
+See CLAUDE.md → Key Conventions → Testing for the full pattern.
+
+### 7. Check API consistency
+
+If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
+- Were the OpenAPI specs regenerated? (`./scripts/generate-openapi.sh`)
+- Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
+- Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
+
+### 7b. Check API-layer data-access boundary
+
+For each changed handler in `hindsight-api-slim/hindsight_api/api/` (e.g. `http.py`, `mcp.py`):
+- **Flag any direct DB access in the handler** — `acquire_with_retry`, `conn.fetch` / `fetchrow` / `execute`, raw SQL strings, or `fq_table(...)`. These are a **must fix**: the query must be moved into a `MemoryEngine` method that returns a typed model, and the handler must call that method.
+- **Verify authentication is enforced in the engine** — the handler must delegate to an engine method that authenticates via `request_context` (`_authenticate_tenant`, typically through `get_bank_profile`). A handler that reads/writes tenant-scoped data without an engine method enforcing auth is a **must fix** (tenant data could leak across schemas).
+
+### 8. Check code comments
+
+For each non-trivial change:
+- **New non-obvious logic** — is there a comment explaining the reasoning?
+- **Changed approach** — does the comment include what was done before and why it changed?
+- **Stale comments** — do existing comments near the changed code still accurately describe the behavior?
+
+### 9. Check integration completeness
+
+If any files in `hindsight-integrations/` were added or changed, verify:
+- **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
+- **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
+- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
+- **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
+
+### 10. Check MCP tool registration completeness
+
+If any new MCP tools were added or existing tools renamed in `hindsight-api-slim/hindsight_api/mcp_tools.py`:
+- **`_ALL_TOOLS` set** in `mcp_tools.py` — must include the new tool name
+- **`tools_to_register` default set** in `register_mcp_tools()` in `mcp_tools.py` — must include the new tool name
+- **`_SINGLE_BANK_TOOLS` set** in `hindsight-api-slim/hindsight_api/api/mcp.py` — must include the new tool if it is bank-scoped (not a bank-management tool like `list_banks`/`create_bank`)
+- **`MCP_TOOL_GROUPS`** in `hindsight-control-plane/src/components/bank-config-view.tsx` — must include the new tool in the appropriate group for the UI tool selector
+- **Tool count assertions** in tests (e.g., `test_mcp_tools.py`) — must be updated to reflect the new count
+
+### 11. Check backup/restore table coverage
+
+If a migration adds a new PostgreSQL table (look for `CREATE TABLE` / `op.create_table` in `hindsight-api-slim/hindsight_api/alembic/versions/`):
+- **`BACKUP_TABLES`** in `hindsight-api-slim/hindsight_api/admin/cli.py` — must include the new table, placed after any table it references via foreign key (parents before children). A missing entry is silent data loss: the table is never backed up, and restore's `TRUNCATE banks CASCADE` wipes any FK-to-banks child (e.g. `mental_models`, `directives`) on restore even though it was never saved.
+- The guard test `test_backup_tables_covers_entire_schema` in `tests/test_admin_backup_restore.py` enforces this — flag it as a **must fix** if a new table is absent from `BACKUP_TABLES`.
+- Oracle-only tables (e.g. `observation_sources`) are intentionally excluded — admin backup/restore is PostgreSQL-only.
+
+### 12. Review against other coding standards
+
+Check the diff for violations of the standards listed above:
+- Python files at project root (not allowed)
+- Missing async patterns (should be async throughout)
+- Pydantic models for request/response
+- Line length > 120 chars
+- New features/code beyond what was asked (over-engineering)
+- Unnecessary error handling for impossible scenarios
+- Premature abstractions or speculative helpers
+- Backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
+
+### 13. Report findings
+
+Present a clear summary organized by severity:
+
+**Must fix** — issues that will break CI or violate hard project rules:
+- Unrelated commits on the branch
+- Lint failures
+- Missing type hints on public functions
+- Raw dict usage for structured data (including internal code)
+- Multi-item tuple returns (including internal code)
+- Missing tests for new endpoints
+- Direct DB access (raw SQL / `acquire_with_retry` / `fq_table`) in an `api/` handler instead of a `MemoryEngine` method
+- Tenant-scoped data accessed without authentication enforced in the engine (`_authenticate_tenant` / `get_bank_profile`)
+- New integration missing tests, CI job, or release-integration.sh entry
+- New PostgreSQL table missing from `BACKUP_TABLES` in `admin/cli.py` (silent data loss on restore)
+
+**Should fix** — issues that hurt code quality:
+- Dead code / unused imports missed by linter
+- Missing tests for non-trivial utility functions
+- Over-engineering beyond the task scope
+
+**Note** — observations that may or may not need action:
+- API changes that might need client regeneration
+- Patterns that deviate from nearby code style
+
+For each finding, include the file path, line number, and a brief explanation.
+
+Do NOT auto-fix any issues. Report all findings and let the user decide what to address. If there are no findings, confirm the code looks good.

.env.example

@@ -2,17 +2,41 @@
 # Copy this file to .env and fill in your values
 
 # LLM Configuration (Required)
-# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio
+# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, volcano
 HINDSIGHT_API_LLM_PROVIDER=openai
 HINDSIGHT_API_LLM_API_KEY=your-api-key-here
-HINDSIGHT_API_LLM_MODEL=o3-mini
+HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
 HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
+# Reasoning effort for providers/models that support it. Examples: low, medium, high, xhigh.
+# HINDSIGHT_API_LLM_REASONING_EFFORT=low
 
 # Example: Anthropic Claude configuration
 # HINDSIGHT_API_LLM_PROVIDER=anthropic
 # HINDSIGHT_API_LLM_API_KEY=your-anthropic-api-key
 # HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-20250514
 
+# Example: Google Vertex AI configuration
+# HINDSIGHT_API_LLM_PROVIDER=vertexai
+# HINDSIGHT_API_LLM_MODEL=google/gemini-2.0-flash-001
+# HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
+# HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
+# HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/service-account-key.json  # Optional, uses ADC if not set
+
+# Example: MiniMax configuration (1M context window)
+# HINDSIGHT_API_LLM_PROVIDER=minimax
+# HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
+# HINDSIGHT_API_LLM_MODEL=MiniMax-M3  # or MiniMax-M2.7 for the previous generation
+
+# Example: DeepSeek configuration (https://api.deepseek.com)
+# HINDSIGHT_API_LLM_PROVIDER=deepseek
+# HINDSIGHT_API_LLM_API_KEY=your-deepseek-api-key
+# HINDSIGHT_API_LLM_MODEL=deepseek-v4-flash  # or deepseek-v4-pro / deepseek-chat / deepseek-reasoner
+
+# Example: z.ai configuration (Zhipu GLM series, https://z.ai)
+# HINDSIGHT_API_LLM_PROVIDER=zai
+# HINDSIGHT_API_LLM_API_KEY=your-zai-api-key
+# HINDSIGHT_API_LLM_MODEL=glm-4.5-flash  # or glm-4.5-air for the paid tier
+
 # Example: LM Studio local configuration (Qwen 2.5 32B recommended)
 # HINDSIGHT_API_LLM_PROVIDER=lmstudio
 # HINDSIGHT_API_LLM_API_KEY=lmstudio
@@ -24,16 +48,77 @@ HINDSIGHT_API_HOST=0.0.0.0
 HINDSIGHT_API_PORT=8888
 HINDSIGHT_API_LOG_LEVEL=info
 
+# Base Path / Reverse Proxy Support (Optional)
+# Set these when deploying behind a reverse proxy with path-based routing
+# Example: To deploy at example.com/hindsight/, set both to "/hindsight"
+# HINDSIGHT_API_BASE_PATH=/hindsight
+# NEXT_PUBLIC_BASE_PATH=/hindsight
+
 # Database (Optional - uses embedded pg0 by default)
 # HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
+# HINDSIGHT_API_READ_DATABASE_URL=  # Optional read-replica URL. When set, recall queries (semantic, BM25, graph, temporal) flow through a separate pool against this URL, offloading the primary. Typically points to a read-only endpoint (CNPG's <cluster>-ro service or Aurora reader endpoint).
+# HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
+# HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
+
+# Vector Extension (Optional - uses pgvector by default)
+# Options: "pgvector" (default), "vchord", "pgvectorscale" (DiskANN)
+# HINDSIGHT_API_VECTOR_EXTENSION=pgvector
+# For Azure PostgreSQL with DiskANN:
+# HINDSIGHT_API_VECTOR_EXTENSION=pgvectorscale  # Auto-detects pg_diskann on Azure
+
+# Text Search Extension (Optional - uses native PostgreSQL full-text search by default)
+# Backend options: "native" (default), "vchord", "pg_textsearch", "pgroonga", "pg_search"
+# HINDSIGHT_API_TEXT_SEARCH_EXTENSION=native
+# Native backend dictionary (only used by HINDSIGHT_API_TEXT_SEARCH_EXTENSION=native)
+# HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE=english
+# ParadeDB pg_search tokenizer (only used when creating pg_search BM25 indexes).
+# Empty uses ParadeDB's default tokenizer: unicode_words.
+# Supported values: unicode_words, simple, whitespace, literal, literal_normalized,
+# chinese_compatible, icu, jieba, source_code,
+# chinese_lindera/lindera(chinese), japanese_lindera/lindera(japanese),
+# korean_lindera/lindera(korean), ngram(min,max), edge_ngram(min,max)
+# HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER=
 
 # Embeddings Configuration (Optional - uses local by default)
-# Provider: "local" (default) or "tei" (HuggingFace Text Embeddings Inference)
+# Provider: "local" (default), "onnx", "tei", "openai", "cohere", "google", "openrouter", "zeroentropy", "litellm", or "litellm-sdk"
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
 # For local provider:
 # HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=BAAI/bge-small-en-v1.5
+# For ONNX provider (local CPU embeddings without an Ollama/TEI sidecar):
+# HINDSIGHT_API_EMBEDDINGS_PROVIDER=onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_ID=intfloat/multilingual-e5-small
+# HINDSIGHT_API_EMBEDDINGS_ONNX_FILE=onnx/model.onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_DIMENSIONS=384
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MAX_TOKENS=512
+# HINDSIGHT_API_EMBEDDINGS_ONNX_POOLING=mean
+# HINDSIGHT_API_EMBEDDINGS_ONNX_NORMALIZE=true
+# HINDSIGHT_API_EMBEDDINGS_ONNX_QUERY_PREFIX="query: "
+# HINDSIGHT_API_EMBEDDINGS_ONNX_PASSAGE_PREFIX="passage: "
+# Optional for local model paths or pre-downloaded artifacts:
+# HINDSIGHT_API_EMBEDDINGS_ONNX_MODEL_PATH=/models/multilingual-e5-small/onnx/model.onnx
+# HINDSIGHT_API_EMBEDDINGS_ONNX_TOKENIZER_NAME_OR_PATH=/models/multilingual-e5-small
+# Optional for China network / restricted HF access:
+# HF_ENDPOINT=https://hf-mirror.com
 # For TEI provider:
 # HINDSIGHT_API_EMBEDDINGS_TEI_URL=http://localhost:8080
+# For OpenAI-compatible embeddings:
+# HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=sk-xxxx
+# HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-small
+# HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL=https://api.openai.com/v1
+# For ZeroEntropy zembed-1:
+# HINDSIGHT_API_EMBEDDINGS_PROVIDER=zeroentropy
+# HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_API_KEY=ze-xxxx
+# HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_MODEL=zembed-1
+# HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_DIMENSIONS=1280
+# HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_ENCODING_FORMAT=float
+# HINDSIGHT_API_EMBEDDINGS_ZEROENTROPY_LATENCY=fast
+#
+# IMPORTANT: Embedding keys require provider-specific names:
+# HINDSIGHT_API_EMBEDDINGS_{PROVIDER}_{PARAMETER}
+# (for example, HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL).
+#
+# DeepSeek note: DeepSeek is supported for LLM calls, but not for embeddings.
+# If using DeepSeek as LLM provider, keep embeddings on local/openai/cohere/google/etc.
 
 # Reranker Configuration (Optional - uses local by default)
 # Provider: "local" (default) or "tei" (HuggingFace Text Embeddings Inference)
@@ -42,3 +127,35 @@ HINDSIGHT_API_LOG_LEVEL=info
 # HINDSIGHT_API_RERANKER_LOCAL_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
 # For TEI provider:
 # HINDSIGHT_API_RERANKER_TEI_URL=http://localhost:8081
+
+# Observability & Tracing (Optional - disabled by default)
+# Enable OpenTelemetry tracing for LLM calls (GenAI semantic conventions)
+# HINDSIGHT_API_OTEL_TRACES_ENABLED=true
+#
+# Local development with Grafana LGTM stack (recommended - see scripts/dev/grafana/README.md)
+# HINDSIGHT_API_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+#
+# Cloud backends (Grafana Cloud, Langfuse, DataDog, etc.)
+# HINDSIGHT_API_OTEL_EXPORTER_OTLP_ENDPOINT=https://your-backend-url
+# HINDSIGHT_API_OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-token"
+#
+# Custom service name and environment (optional, defaults: hindsight-api, development)
+# HINDSIGHT_API_OTEL_SERVICE_NAME=hindsight-production
+# HINDSIGHT_API_OTEL_DEPLOYMENT_ENVIRONMENT=production
+
+# -----------------------------------------------------------------------------
+# Control Plane (Optional)
+# -----------------------------------------------------------------------------
+
+# Dataplane API URL - where the CP proxies requests to
+# HINDSIGHT_CP_DATAPLANE_API_URL=http://localhost:8888
+
+# Optional: Bearer token the CP sends as `Authorization: Bearer <key>` to the
+# dataplane API. Required when the API service is auth-protected; omit for a
+# public/unauthenticated API.
+# HINDSIGHT_CP_DATAPLANE_API_KEY=your-dataplane-bearer-token
+
+# Optional: Require a shared access key to view the Control Plane UI.
+# When set, visitors see a login page and must enter the key before
+# accessing the dashboard or any /api/* routes (except /api/health).
+# HINDSIGHT_CP_ACCESS_KEY=your-shared-secret-key

.github/dependabot.yml

@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

.github/workflows/deploy-docs.yml

@@ -21,17 +21,20 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
+      - uses: actions/checkout@v6
+      - uses: actions/setup-node@v6
         with:
           node-version: 20
           cache: npm
           cache-dependency-path: package-lock.json
-      - uses: astral-sh/setup-uv@v4
+      - uses: astral-sh/setup-uv@v7
       - run: npm ci --workspace=hindsight-docs
       - run: uv run generate-llms-full
       - run: npm run build --workspace=hindsight-docs
-      - uses: actions/upload-pages-artifact@v3
+        env:
+          UMAMI_URL: https://analytics.hindsight.vectorize.io
+          UMAMI_WEBSITE_ID: ${{ secrets.UMAMI_WEBSITE_ID }}
+      - uses: actions/upload-pages-artifact@v5
         with:
           path: hindsight-docs/build
   deploy:
@@ -41,5 +44,5 @@ jobs:
     runs-on: ubuntu-latest
     needs: build
     steps:
-      - uses: actions/deploy-pages@v4
+      - uses: actions/deploy-pages@v5
         id: deployment

.github/workflows/perf-test.yml

@@ -0,0 +1,296 @@
+name: Performance Tests
+
+on:
+  schedule:
+    # Run daily at 06:00 UTC
+    - cron: "0 6 * * *"
+  workflow_dispatch:
+    inputs:
+      scale:
+        description: "Test scale (perf-test)"
+        type: choice
+        options:
+          - tiny
+          - small
+          - medium
+          - large
+        default: large
+      suite:
+        description: "Perf-test suite to run (blank = all)"
+        type: choice
+        options:
+          - ""
+          - retain
+          - recall
+          - recall-with-observations
+          - consolidation
+          - graph-maintenance
+        default: ""
+      locomo_conversations:
+        description: "LoComo conversation IDs (space-separated). Blank = curated set (conv-26 conv-30 conv-43)."
+        type: string
+        default: ""
+      locomo_skip:
+        description: "Skip LoComo job"
+        type: boolean
+        default: false
+      obs_skip:
+        description: "Skip observation-dedup benchmark job"
+        type: boolean
+        default: false
+      obs_dataset:
+        description: "Obs benchmark dataset substring (blank = English hermes transcript)."
+        type: string
+        default: ""
+      obs_fraction:
+        description: "Obs benchmark fraction (0-1] of each document to run."
+        type: string
+        default: "1.0"
+      ref:
+        description: "Git ref to test (branch, tag, or SHA). Defaults to main."
+        type: string
+        default: ""
+
+concurrency:
+  group: perf-test
+  cancel-in-progress: true
+
+jobs:
+  perf-test:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ inputs.ref || github.ref }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Cache HuggingFace models
+      uses: actions/cache@v5
+      with:
+        path: ~/.cache/huggingface
+        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-huggingface-
+
+    - name: Pre-download models
+      working-directory: ./hindsight-api-slim
+      run: |
+        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
+        from sentence_transformers import SentenceTransformer
+        print('Downloading embedding model...')
+        SentenceTransformer('BAAI/bge-small-en-v1.5')
+        print('Model downloaded successfully')
+        "
+
+    - name: Install hindsight-dev dependencies
+      run: |
+        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
+
+    - name: Run perf-test
+      run: |
+        SUITE_ARG=""
+        if [ -n "${{ inputs.suite }}" ]; then
+          SUITE_ARG="--suite ${{ inputs.suite }}"
+        fi
+        ./scripts/benchmarks/run-perf-test.sh \
+          --scale ${{ inputs.scale || 'large' }} \
+          $SUITE_ARG \
+          --output perf-results.json
+
+    - name: Upload perf results
+      if: always()
+      uses: actions/upload-artifact@v7
+      with:
+        name: perf-results-${{ github.sha }}
+        path: hindsight-dev/perf-results.json
+        retention-days: 90
+
+    # Publish enriched results (perf JSON + commit metadata) to the dashboard
+    # repo's gh-pages branch. The static site at
+    #   https://vectorize-io.github.io/hindsight-continuous-performance-monitor/
+    # reads data/index.json + data/<run>.json and renders charts client-side.
+    - name: Publish to dashboard
+      if: github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch'
+      env:
+        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: ./scripts/benchmarks/publish-perf-results.sh hindsight-dev/perf-results.json
+
+  locomo:
+    if: inputs.locomo_skip != true
+    runs-on: ubuntu-latest
+    env:
+      HINDSIGHT_API_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
+      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
+      HINDSIGHT_API_JUDGE_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_JUDGE_LLM_MODEL: google/gemini-2.5-flash-lite
+      HINDSIGHT_API_ANSWER_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_ANSWER_LLM_MODEL: google/gemini-2.5-flash
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ inputs.ref || github.ref }}
+
+    - name: Setup GCP credentials
+      run: |
+        printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
+        PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
+        echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Cache HuggingFace models
+      uses: actions/cache@v5
+      with:
+        path: ~/.cache/huggingface
+        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-huggingface-
+
+    - name: Pre-download models
+      working-directory: ./hindsight-api-slim
+      run: |
+        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
+        from sentence_transformers import SentenceTransformer
+        print('Downloading embedding model...')
+        SentenceTransformer('BAAI/bge-small-en-v1.5')
+        print('Model downloaded successfully')
+        "
+
+    - name: Install hindsight-dev dependencies
+      run: |
+        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
+
+    - name: Run LoComo benchmark
+      # Curated 3-conversation subset (best/middle/worst by accuracy on the
+      # last successful full run): conv-26 (best), conv-30 (middle), conv-43
+      # (worst). Excludes conv-44, the bank with the largest unconsolidated
+      # set that has been pushing scheduled runs over the per-bank
+      # _wait_for_consolidation timeout. Override via workflow_dispatch with
+      # the locomo_conversations input.
+      run: |
+        CONVERSATIONS="${{ inputs.locomo_conversations }}"
+        if [ -z "$CONVERSATIONS" ]; then
+          CONVERSATIONS="conv-26 conv-30 conv-43"
+        fi
+        uv run python hindsight-dev/benchmarks/locomo/locomo_benchmark.py \
+          --wait-consolidation \
+          --conversation $CONVERSATIONS
+
+    - name: Upload LoComo results
+      if: always()
+      uses: actions/upload-artifact@v7
+      with:
+        name: locomo-results-${{ github.sha }}
+        path: hindsight-dev/benchmarks/locomo/results/
+        retention-days: 90
+
+    - name: Publish LoComo to dashboard
+      if: success() && (github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch')
+      env:
+        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: ./scripts/benchmarks/publish-locomo-results.sh hindsight-dev/benchmarks/locomo/results/benchmark_results.json
+
+  obs:
+    # Observation-dedup quality benchmark: ingests a transcript, drains consolidation
+    # (serial SyncTaskBackend + embedded pg0 — no external DB / worker), and reports the
+    # near-duplicate observation rate. Real LLM via VertexAI, mirroring the LoComo job.
+    if: inputs.obs_skip != true
+    runs-on: ubuntu-latest
+    env:
+      HINDSIGHT_API_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
+      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
+      HINDSIGHT_API_ENABLE_OBSERVATIONS: "true"
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ inputs.ref || github.ref }}
+
+    - name: Setup GCP credentials
+      run: |
+        printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
+        PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
+        echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Cache HuggingFace models
+      uses: actions/cache@v5
+      with:
+        path: ~/.cache/huggingface
+        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
+        restore-keys: |
+          ${{ runner.os }}-huggingface-
+
+    - name: Pre-download models
+      working-directory: ./hindsight-api-slim
+      run: |
+        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
+        from sentence_transformers import SentenceTransformer
+        print('Downloading embedding model...')
+        SentenceTransformer('BAAI/bge-small-en-v1.5')
+        print('Model downloaded successfully')
+        "
+
+    - name: Install hindsight-dev dependencies
+      run: |
+        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
+
+    - name: Run obs benchmark
+      # Default to the English hermes transcript at full fraction — a clean, deterministic
+      # consolidation-dedup signal (the Chinese variant adds a cross-lingual embedding
+      # confound). Override dataset/fraction via workflow_dispatch.
+      run: |
+        DATASET="${{ inputs.obs_dataset }}"
+        if [ -z "$DATASET" ]; then DATASET="hermes_session_2026-05-15_en"; fi
+        FRACTION="${{ inputs.obs_fraction }}"
+        if [ -z "$FRACTION" ]; then FRACTION="1.0"; fi
+        cd hindsight-dev
+        uv run python -m benchmarks.obs.obs_benchmark \
+          --dataset "$DATASET" --fraction "$FRACTION" --wipe-bank --output obs-results.json
+
+    - name: Upload obs results
+      if: always()
+      uses: actions/upload-artifact@v7
+      with:
+        name: obs-results-${{ github.sha }}
+        path: hindsight-dev/obs-results.json
+        retention-days: 90
+
+    - name: Publish obs to dashboard
+      if: success() && (github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'workflow_dispatch')
+      env:
+        PERF_DASHBOARD_TOKEN: ${{ secrets.PERF_DASHBOARD_TOKEN }}
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: ./scripts/benchmarks/publish-obs-results.sh hindsight-dev/obs-results.json

.github/workflows/release-integration.yml

@@ -0,0 +1,131 @@
+name: Release Integration
+
+on:
+  push:
+    tags:
+      - 'integrations/**'
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write  # for PyPI trusted publishing
+
+    steps:
+    - uses: actions/checkout@v6
+
+    - name: Extract integration info
+      id: info
+      run: |
+        # refs/tags/integrations/litellm/v0.1.0 → integration=litellm, version=0.1.0
+        TAG="${GITHUB_REF#refs/tags/}"
+        INTEGRATION=$(echo "$TAG" | cut -d'/' -f2)
+        VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
+        echo "integration=$INTEGRATION" >> $GITHUB_OUTPUT
+        echo "version=$VERSION" >> $GITHUB_OUTPUT
+        echo "tag=$TAG" >> $GITHUB_OUTPUT
+        echo "Integration: $INTEGRATION, Version: $VERSION"
+
+    - name: Detect integration type
+      id: type
+      run: |
+        if [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/pyproject.toml" ]; then
+          echo "type=python" >> $GITHUB_OUTPUT
+        elif [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/package.json" ]; then
+          echo "type=typescript" >> $GITHUB_OUTPUT
+        else
+          echo "type=plugin" >> $GITHUB_OUTPUT
+        fi
+
+    # ── Python integrations (litellm, pydantic-ai, crewai) ──────────────────
+
+    - name: Install uv
+      if: steps.type.outputs.type == 'python'
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+
+    - name: Set up Python
+      if: steps.type.outputs.type == 'python'
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build Python package
+      if: steps.type.outputs.type == 'python'
+      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
+      run: uv build --out-dir dist
+
+    - name: Publish Python package to PyPI
+      if: steps.type.outputs.type == 'python'
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        packages-dir: ./hindsight-integrations/${{ steps.info.outputs.integration }}/dist
+        skip-existing: true
+
+    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
+
+    # ── Plugin integrations (claude-code) — no package to publish ───────────
+
+    - name: Plugin release
+      if: steps.type.outputs.type == 'plugin'
+      run: |
+        echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
+        echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
+
+    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
+
+    - name: Set up Node.js
+      if: steps.type.outputs.type == 'typescript'
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+        cache: 'npm'
+        cache-dependency-path: package-lock.json
+
+    - name: Check integration lockfile
+      if: steps.type.outputs.type == 'typescript'
+      run: ./scripts/check-integration-lockfiles.sh
+
+    # Some integrations depend on workspace packages (hindsight-client,
+    # hindsight-all, hindsight-agent-sdk) via file: refs. Install from root
+    # so npm resolves them, then build the workspace deps before the integration.
+    - name: Install root workspace dependencies
+      if: steps.type.outputs.type == 'typescript'
+      run: npm ci
+
+    - name: Build workspace deps (hindsight-client, hindsight-all, hindsight-agent-sdk)
+      if: steps.type.outputs.type == 'typescript'
+      run: |
+        npm run build --workspace=hindsight-clients/typescript
+        npm run build --workspace=hindsight-all-npm
+        npm run build --workspace=hindsight-tools/hindsight-agent-sdk
+
+    - name: Install integration dependencies
+      if: steps.type.outputs.type == 'typescript'
+      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
+      run: npm ci
+
+    - name: Build TypeScript package
+      if: steps.type.outputs.type == 'typescript'
+      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
+      run: npm run build
+
+    - name: Publish TypeScript package to npm
+      if: steps.type.outputs.type == 'typescript'
+      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public --provenance 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/release-tool.yml

@@ -0,0 +1,65 @@
+name: Release Tool
+
+on:
+  push:
+    tags:
+      - 'tools/**'
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+
+    - name: Extract tool info
+      id: info
+      run: |
+        # refs/tags/tools/self-driving-agents/v0.0.1 → tool=self-driving-agents, version=0.0.1
+        TAG="${GITHUB_REF#refs/tags/}"
+        TOOL=$(echo "$TAG" | cut -d'/' -f2)
+        VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
+        echo "tool=$TOOL" >> $GITHUB_OUTPUT
+        echo "version=$VERSION" >> $GITHUB_OUTPUT
+        echo "tag=$TAG" >> $GITHUB_OUTPUT
+        echo "Tool: $TOOL, Version: $VERSION"
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+        cache: 'npm'
+        cache-dependency-path: package-lock.json
+
+    # Tools live under hindsight-tools/ and may depend on workspace packages
+    # (e.g. @vectorize-io/hindsight-client). Install from root so npm resolves
+    # workspace deps, then build any required workspace packages first.
+    - name: Install root workspace dependencies
+      run: npm ci
+
+    - name: Build hindsight-client (workspace dep)
+      run: npm run build --workspace=hindsight-clients/typescript
+
+    - name: Build hindsight-agent-sdk (workspace dep)
+      run: npm run build --workspace=hindsight-tools/hindsight-agent-sdk
+
+    - name: Build tool
+      run: npm run build --workspace=hindsight-tools/${{ steps.info.outputs.tool }}
+
+    - name: Publish to npm
+      working-directory: ./hindsight-tools/${{ steps.info.outputs.tool }}
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/release.yml

@@ -13,15 +13,15 @@ jobs:
       id-token: write
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Install uv
-      uses: astral-sh/setup-uv@v5
+      uses: astral-sh/setup-uv@v7
       with:
         enable-cache: true
 
     - name: Set up Python
-      uses: actions/setup-python@v5
+      uses: actions/setup-python@v6
       with:
         python-version-file: ".python-version"
 
@@ -30,29 +30,39 @@ jobs:
       working-directory: ./hindsight-clients/python
       run: uv build --out-dir dist
 
+    - name: Build hindsight-api-slim
+      working-directory: ./hindsight-api-slim
+      run: uv build --out-dir dist
+
     - name: Build hindsight-api
       working-directory: ./hindsight-api
       run: uv build --out-dir dist
 
     - name: Build hindsight-all
-      working-directory: ./hindsight
+      working-directory: ./hindsight-all
       run: uv build --out-dir dist
 
-    - name: Build hindsight-litellm
-      working-directory: ./hindsight-integrations/litellm
+    - name: Build hindsight-all-slim
+      working-directory: ./hindsight-all-slim
       run: uv build --out-dir dist
 
     - name: Build hindsight-embed
       working-directory: ./hindsight-embed
       run: uv build --out-dir dist
 
-    # Publish in order (client and api first, then hindsight-all which depends on them)
+    # Publish in order (client and api-slim first, then api/all wrappers which depend on them)
     - name: Publish hindsight-client to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         packages-dir: ./hindsight-clients/python/dist
         skip-existing: true
 
+    - name: Publish hindsight-api-slim to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        packages-dir: ./hindsight-api-slim/dist
+        skip-existing: true
+
     - name: Publish hindsight-api to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
@@ -62,13 +72,13 @@ jobs:
     - name: Publish hindsight-all to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight/dist
+        packages-dir: ./hindsight-all/dist
         skip-existing: true
 
-    - name: Publish hindsight-litellm to PyPI
+    - name: Publish hindsight-all-slim to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight-integrations/litellm/dist
+        packages-dir: ./hindsight-all-slim/dist
         skip-existing: true
 
     - name: Publish hindsight-embed to PyPI
@@ -79,14 +89,15 @@ jobs:
 
     # Upload artifacts for GitHub release
     - name: Upload artifacts
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: python-packages
         path: |
           hindsight-clients/python/dist/*
+          hindsight-api-slim/dist/*
           hindsight-api/dist/*
-          hindsight/dist/*
-          hindsight-integrations/litellm/dist/*
+          hindsight-all/dist/*
+          hindsight-all-slim/dist/*
           hindsight-embed/dist/*
         retention-days: 1
 
@@ -95,10 +106,10 @@ jobs:
     environment: npm
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Set up Node.js
-      uses: actions/setup-node@v4
+      uses: actions/setup-node@v6
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -133,21 +144,70 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: typescript-client
         path: hindsight-clients/typescript/*.tgz
         retention-days: 1
 
+  release-hindsight-all-npm:
+    runs-on: ubuntu-latest
+    environment: npm
+
+    steps:
+    - uses: actions/checkout@v6
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v6
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+        cache: 'npm'
+        cache-dependency-path: package-lock.json
+
+    - name: Install dependencies
+      run: npm ci --workspace=hindsight-all-npm
+
+    - name: Build
+      run: npm run build --workspace=hindsight-all-npm
+
+    - name: Publish to npm
+      working-directory: ./hindsight-all-npm
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+    - name: Pack for GitHub release
+      working-directory: ./hindsight-all-npm
+      run: npm pack
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v7
+      with:
+        name: hindsight-all-npm
+        path: hindsight-all-npm/*.tgz
+        retention-days: 1
+
   release-control-plane:
     runs-on: ubuntu-latest
     environment: npm
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Set up Node.js
-      uses: actions/setup-node@v4
+      uses: actions/setup-node@v6
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -170,11 +230,14 @@ jobs:
     - name: Build
       run: npm run build --workspace=hindsight-control-plane
 
+    - name: Verify standalone build
+      run: test -f hindsight-control-plane/standalone/server.js || (echo 'standalone/server.js missing - build failed' && exit 1)
+
     - name: Publish to npm
       working-directory: ./hindsight-control-plane
       run: |
         set +e
-        OUTPUT=$(npm publish --access public 2>&1)
+        OUTPUT=$(npm publish --access public --ignore-scripts 2>&1)
         EXIT_CODE=$?
         echo "$OUTPUT"
         if [ $EXIT_CODE -ne 0 ]; then
@@ -192,7 +255,7 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: control-plane
         path: hindsight-control-plane/*.tgz
@@ -215,9 +278,13 @@ jobs:
             target: aarch64-apple-darwin
             artifact_name: hindsight
             asset_name: hindsight-darwin-arm64
+          - os: ubuntu-24.04-arm
+            target: aarch64-unknown-linux-gnu
+            artifact_name: hindsight
+            asset_name: hindsight-linux-arm64
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Install Rust
       uses: dtolnay/rust-toolchain@stable
@@ -235,29 +302,49 @@ jobs:
         chmod +x artifacts/${{ matrix.asset_name }}
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: rust-cli-${{ matrix.asset_name }}
         path: artifacts/${{ matrix.asset_name }}
         retention-days: 1
 
   release-docker-images:
+    name: Release Docker (${{ matrix.image_name }}${{ matrix.tag_suffix }})
     runs-on: ubuntu-latest
     permissions:
       contents: read
       packages: write
+      id-token: write
     strategy:
       matrix:
         include:
           - target: api-only
             image_name: hindsight-api
+            tag_suffix: ""
+            build_args: ""
+          - target: api-only
+            image_name: hindsight-api
+            tag_suffix: "-slim"
+            build_args: |
+              INCLUDE_LOCAL_MODELS=false
+              PRELOAD_ML_MODELS=false
           - target: cp-only
             image_name: hindsight-control-plane
+            tag_suffix: ""
+            build_args: ""
           - target: standalone
             image_name: hindsight
+            tag_suffix: ""
+            build_args: ""
+          - target: standalone
+            image_name: hindsight
+            tag_suffix: "-slim"
+            build_args: |
+              INCLUDE_LOCAL_MODELS=false
+              PRELOAD_ML_MODELS=false
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Free Disk Space
       uses: jlumbroso/free-disk-space@main
@@ -271,13 +358,13 @@ jobs:
         swap-storage: true
 
     - name: Set up QEMU
-      uses: docker/setup-qemu-action@v3
+      uses: docker/setup-qemu-action@v4
 
     - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@v3
+      uses: docker/setup-buildx-action@v4
 
     - name: Log in to GitHub Container Registry
-      uses: docker/login-action@v3
+      uses: docker/login-action@v4
       with:
         registry: ghcr.io
         username: ${{ github.actor }}
@@ -289,9 +376,12 @@ jobs:
 
     - name: Extract metadata for release tags
       id: meta
-      uses: docker/metadata-action@v5
+      uses: docker/metadata-action@v6
       with:
         images: ghcr.io/${{ github.repository_owner }}/${{ matrix.image_name }}
+        flavor: |
+          latest=auto
+          suffix=${{ matrix.tag_suffix }}
         tags: |
           type=semver,pattern={{version}},value=${{ steps.get_version.outputs.VERSION }}
           type=semver,pattern={{major}}.{{minor}},value=${{ steps.get_version.outputs.VERSION }}
@@ -302,7 +392,7 @@ jobs:
     # # Step 1: Build for local testing (single platform, no push)
     # # This creates an identical image to what will be released, just for one platform
     # - name: Build image for testing
-    #   uses: docker/build-push-action@v6
+    #   uses: docker/build-push-action@v7
     #   with:
     #     context: .
     #     file: docker/standalone/Dockerfile
@@ -317,20 +407,47 @@ jobs:
     # - name: Smoke test - verify container starts
     #   env:
     #     GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
-    #   run: ./scripts/docker-smoke-test.sh "${{ matrix.image_name }}:test" "${{ matrix.target }}"
+    #   run: ./docker/test-image.sh "${{ matrix.image_name }}:test" "${{ matrix.target }}"
 
     # Build multi-platform and push to release tags
     - name: Build and push release images
-      uses: docker/build-push-action@v6
+      id: build
+      uses: docker/build-push-action@v7
       with:
         context: .
         file: docker/standalone/Dockerfile
         target: ${{ matrix.target }}
+        build-args: ${{ matrix.build_args }}
         push: true
         platforms: linux/amd64,linux/arm64
         tags: ${{ steps.meta.outputs.tags }}
         labels: ${{ steps.meta.outputs.labels }}
 
+    - name: Install cosign
+      uses: sigstore/cosign-installer@v3
+
+    - name: Sign published images
+      env:
+        TAGS: ${{ steps.meta.outputs.tags }}
+        DIGEST: ${{ steps.build.outputs.digest }}
+      run: |
+        set -euo pipefail
+        refs=()
+        while IFS= read -r tag; do
+          [[ -z "${tag}" ]] && continue
+          refs+=("${tag}@${DIGEST}")
+        done <<< "${TAGS}"
+        cosign sign --yes "${refs[@]}"
+
+    - name: Verify signature on primary tag
+      env:
+        IMAGE: ghcr.io/${{ github.repository_owner }}/${{ matrix.image_name }}
+        DIGEST: ${{ steps.build.outputs.digest }}
+      run: |
+        cosign verify "${IMAGE}@${DIGEST}" \
+          --certificate-identity-regexp "^https://github\.com/${{ github.repository }}/\.github/workflows/release\.yml@.*" \
+          --certificate-oidc-issuer https://token.actions.githubusercontent.com
+
   release-helm-chart:
     runs-on: ubuntu-latest
     permissions:
@@ -338,10 +455,10 @@ jobs:
       packages: write
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Install Helm
-      uses: azure/setup-helm@v4
+      uses: azure/setup-helm@v5
       with:
         version: 'latest'
 
@@ -358,7 +475,7 @@ jobs:
       run: helm push helm-packages/*.tgz oci://ghcr.io/${{ github.repository_owner }}/charts
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v4
+      uses: actions/upload-artifact@v7
       with:
         name: helm-chart
         path: helm-packages/*.tgz
@@ -366,55 +483,67 @@ jobs:
 
   create-github-release:
     runs-on: ubuntu-latest
-    needs: [release-python-packages, release-typescript-client, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
+    needs: [release-python-packages, release-typescript-client, release-hindsight-all-npm, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
     permissions:
       contents: write
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Extract version from tag
       id: get_version
       run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
 
     - name: Download Python packages
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: python-packages
         path: ./artifacts/python-packages
 
     - name: Download TypeScript client
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: typescript-client
         path: ./artifacts/typescript-client
 
     - name: Download Control Plane
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: control-plane
         path: ./artifacts/control-plane
 
+    - name: Download hindsight-embed npm wrapper
+      uses: actions/download-artifact@v8
+      with:
+        name: hindsight-all-npm
+        path: ./artifacts/hindsight-all-npm
+
     - name: Download Rust CLI (Linux)
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: rust-cli-hindsight-linux-amd64
         path: ./artifacts/rust-cli-linux
 
+    - name: Download Rust CLI (Linux ARM)
+      uses: actions/download-artifact@v8
+      with:
+        name: rust-cli-hindsight-linux-arm64
+        path: ./artifacts/rust-cli-linux-arm64
+
     - name: Download Rust CLI (macOS Intel)
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: rust-cli-hindsight-darwin-amd64
         path: ./artifacts/rust-cli-darwin-amd64
 
     - name: Download Rust CLI (macOS ARM)
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: rust-cli-hindsight-darwin-arm64
         path: ./artifacts/rust-cli-darwin-arm64
 
     - name: Download Helm chart
-      uses: actions/download-artifact@v4
+      uses: actions/download-artifact@v8
       with:
         name: helm-chart
         path: ./artifacts/helm-chart
@@ -424,16 +553,20 @@ jobs:
         mkdir -p release-assets
         # Python packages
         cp artifacts/python-packages/hindsight-clients/python/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-api-slim/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-api/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-integrations/litellm/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-all/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-all-slim/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-embed/dist/* release-assets/ || true
         # TypeScript client
         cp artifacts/typescript-client/*.tgz release-assets/ || true
+        # hindsight-embed npm wrapper
+        cp artifacts/hindsight-all-npm/*.tgz release-assets/ || true
         # Control Plane
         cp artifacts/control-plane/*.tgz release-assets/ || true
         # Rust CLI binaries
         cp artifacts/rust-cli-linux/hindsight-linux-amd64 release-assets/ || true
+        cp artifacts/rust-cli-linux-arm64/hindsight-linux-arm64 release-assets/ || true
         cp artifacts/rust-cli-darwin-amd64/hindsight-darwin-amd64 release-assets/ || true
         cp artifacts/rust-cli-darwin-arm64/hindsight-darwin-arm64 release-assets/ || true
         # Helm chart
@@ -441,7 +574,7 @@ jobs:
         ls -la release-assets/
 
     - name: Create GitHub Release
-      uses: softprops/action-gh-release@v2
+      uses: softprops/action-gh-release@v3
       with:
         files: release-assets/*
         generate_release_notes: true

.github/workflows/sign-images.yml

@@ -0,0 +1,71 @@
+name: Sign published images
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version to sign (without leading v, e.g. 0.6.0)'
+        required: true
+        type: string
+        default: '0.6.0'
+
+permissions:
+  contents: read
+  packages: write
+  id-token: write
+
+jobs:
+  sign:
+    name: Sign ${{ matrix.image }}:${{ inputs.version }}${{ matrix.suffix }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - { image: hindsight-api, suffix: '' }
+          - { image: hindsight-api, suffix: '-slim' }
+          - { image: hindsight-control-plane, suffix: '' }
+          - { image: hindsight, suffix: '' }
+          - { image: hindsight, suffix: '-slim' }
+
+    steps:
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v4
+
+      - name: Log in to GHCR
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Install cosign
+        uses: sigstore/cosign-installer@v3
+
+      - name: Resolve image digest
+        id: resolve
+        env:
+          IMAGE: ghcr.io/${{ github.repository_owner }}/${{ matrix.image }}
+          TAG: ${{ inputs.version }}${{ matrix.suffix }}
+        run: |
+          set -euo pipefail
+          DIGEST=$(docker buildx imagetools inspect "${IMAGE}:${TAG}" --format '{{json .Manifest.Digest}}' | tr -d '"')
+          if [[ -z "${DIGEST}" || "${DIGEST}" != sha256:* ]]; then
+            echo "Failed to resolve digest for ${IMAGE}:${TAG} (got: ${DIGEST})" >&2
+            exit 1
+          fi
+          echo "Resolved ${IMAGE}:${TAG} -> ${DIGEST}"
+          echo "ref=${IMAGE}@${DIGEST}" >> "$GITHUB_OUTPUT"
+
+      - name: Sign image
+        env:
+          REF: ${{ steps.resolve.outputs.ref }}
+        run: cosign sign --yes "${REF}"
+
+      - name: Verify signature
+        env:
+          REF: ${{ steps.resolve.outputs.ref }}
+        run: |
+          cosign verify "${REF}" \
+            --certificate-identity-regexp "^https://github\.com/${{ github.repository }}/\.github/workflows/sign-images\.yml@.*" \
+            --certificate-oidc-issuer https://token.actions.githubusercontent.com

.github/workflows/windows-smoke.yml

@@ -0,0 +1,115 @@
+name: Windows Smoke Test
+
+# Daily smoke test that installs the API on Windows and runs the Python client
+# integration tests against a live server. Windows is only exercised by the
+# hindsight-embed jobs in test.yml on PRs; this catches Windows-specific
+# regressions in the API server + client path (e.g. process spawning, console
+# subsystem / ConPTY behaviour, see #1885) that the Linux client jobs miss.
+on:
+  schedule:
+    # 06:00 UTC daily.
+    - cron: "0 6 * * *"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  windows-client-smoke:
+    # Don't run on forks: the job needs the org's Vertex AI credentials.
+    if: github.repository == 'vectorize-io/hindsight'
+    runs-on: windows-latest
+    timeout-minutes: 45
+    env:
+      HINDSIGHT_API_LLM_PROVIDER: vertexai
+      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: ${{ github.workspace }}/gcp-credentials.json
+      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
+      HINDSIGHT_API_URL: http://localhost:8888
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      # Force UTF-8 I/O so the API/CLI's ✓/box-drawing output doesn't crash the
+      # default Windows cp1252 codec (matches test-embed-windows in test.yml).
+      PYTHONIOENCODING: utf-8
+      PYTHONUTF8: "1"
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Setup GCP credentials
+        shell: bash
+        run: |
+          printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > gcp-credentials.json
+          PROJECT_ID=$(jq -r '.project_id' gcp-credentials.json)
+          echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> "$GITHUB_ENV"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+          prune-cache: false
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version-file: ".python-version"
+
+      - name: Install API dependencies (all extras - local-ml + embedded pg0)
+        working-directory: ./hindsight-api-slim
+        run: uv sync --frozen --all-extras --index-strategy unsafe-best-match
+
+      - name: Install Python client test dependencies
+        working-directory: ./hindsight-clients/python
+        run: uv sync --frozen --extra test --index-strategy unsafe-best-match
+
+      - name: Cache HuggingFace models
+        uses: actions/cache@v5
+        with:
+          path: ~/.cache/huggingface
+          key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
+          restore-keys: |
+            ${{ runner.os }}-huggingface-
+
+      # `uv run` re-syncs the project env to its default (no-extras) state before
+      # running, which drops sentence-transformers / pg0. Pass --all-extras on
+      # every `uv run` so the local-ml + embedded-db deps stay installed (this is
+      # the same reason hindsight-embed launches the daemon with `--extra all`).
+      - name: Pre-download models
+        working-directory: ./hindsight-api-slim
+        run: |
+          uv run --all-extras python -c "from sentence_transformers import SentenceTransformer, CrossEncoder; SentenceTransformer('BAAI/bge-small-en-v1.5'); CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2'); print('Models downloaded')"
+
+      # Start the server and run the client tests in a SINGLE step. On Windows
+      # runners a process backgrounded with `&` in one step is not reliably kept
+      # alive for later steps (unlike Linux, where it reparents to init), so the
+      # server must live in the same shell that runs pytest.
+      - name: Start API server and run Python client tests
+        shell: bash
+        run: |
+          # Config is read straight from the environment (job-level env + the
+          # PROJECT_ID exported to GITHUB_ENV above), so no .env file is needed.
+          # Embedded pg0 is the default when HINDSIGHT_API_DATABASE_URL is unset.
+          ( cd hindsight-api-slim && uv run --all-extras hindsight-api --port 8888 ) > "$RUNNER_TEMP/api-server.log" 2>&1 &
+          server_pid=$!
+          echo "Waiting for API server to be ready (pid $server_pid)..."
+          # pg0 unpacks Postgres + runs initdb on first boot, which is slow on a
+          # cold Windows runner — give it a generous budget before failing.
+          ready=false
+          for i in $(seq 1 300); do
+            if curl -sf http://localhost:8888/health > /dev/null 2>&1; then
+              echo "API server is ready after ${i}s"
+              ready=true
+              break
+            fi
+            sleep 1
+          done
+          if [ "$ready" != true ]; then
+            echo "API server failed to start after 300s"
+            cat "$RUNNER_TEMP/api-server.log"
+            exit 1
+          fi
+
+          cd hindsight-clients/python && uv run --extra test pytest tests -v
+
+      - name: Show API server logs
+        if: always()
+        shell: bash
+        run: cat "$RUNNER_TEMP/api-server.log" || echo "No API server log found"

.gitignore

@@ -45,9 +45,18 @@ hindsight-docs/static/llms-full.txt
 
 hindsight-dev/benchmarks/locomo/results/
 hindsight-dev/benchmarks/longmemeval/results/
+hindsight-dev/benchmarks/consolidation/results/
+hindsight-dev/benchmarks/perf/results/
+benchmarks/results/
 hindsight-cli/target
 hindsight-clients/rust/target
-.claude
+.claude/*
+!.claude/skills/
 whats-next.md
 TASK.md
-CHANGELOG.md
+# Parked / draft integrations that aren't ready to ship
+hindsight-integrations/_drafts/
+# Changelog is now tracked in hindsight-docs/src/pages/changelog.md
+# CHANGELOG.md
+
+blog-post*

.prettierrc.json

@@ -0,0 +1,7 @@
+{
+  "semi": true,
+  "singleQuote": false,
+  "tabWidth": 2,
+  "trailingComma": "es5",
+  "printWidth": 100
+}

CLAUDE.md

@@ -11,26 +11,32 @@ Hindsight is an agent memory system that provides long-term memory for AI agents
 
 ## Development Commands
 
+### Local Development (API + UI)
+```bash
+# Start both API server and control plane UI
+./scripts/dev/start.sh
+```
+
 ### API Server (Python/FastAPI)
 ```bash
-# Start API server (loads .env automatically)
+# Start API server only (loads .env automatically)
 ./scripts/dev/start-api.sh
 
 # Run all tests (parallelized with pytest-xdist)
-cd hindsight-api && uv run pytest tests/
+cd hindsight-api-slim && uv run pytest tests/
 
 # Run specific test file
-cd hindsight-api && uv run pytest tests/test_http_api_integration.py -v
+cd hindsight-api-slim && uv run pytest tests/test_http_api_integration.py -v
 
 # Run single test function
-cd hindsight-api && uv run pytest tests/test_retain.py::test_retain_simple -v
+cd hindsight-api-slim && uv run pytest tests/test_retain.py::test_retain_simple -v
 
 # Lint and format
-cd hindsight-api && uv run ruff check .
-cd hindsight-api && uv run ruff format .
+cd hindsight-api-slim && uv run ruff check .
+cd hindsight-api-slim && uv run ruff format .
 
 # Type checking (uses ty - extremely fast type checker from Astral)
-cd hindsight-api && uv run ty check hindsight_api/
+cd hindsight-api-slim && uv run ty check hindsight_api/
 ```
 
 ### Control Plane (Next.js)
@@ -45,6 +51,7 @@ cd hindsight-control-plane && npm run dev
 ./scripts/dev/start-docs.sh
 ```
 
+
 ### Generating Clients/OpenAPI
 ```bash
 # Regenerate OpenAPI spec after API changes (REQUIRED after changing endpoints)
@@ -56,26 +63,33 @@ cd hindsight-control-plane && npm run dev
 
 ### Benchmarks
 ```bash
+# Accuracy benchmarks
 ./scripts/benchmarks/run-longmemeval.sh
 ./scripts/benchmarks/run-locomo.sh
+
+# Performance benchmarks
+./scripts/benchmarks/run-perf-test.sh                      # System perf (mock LLM + pg0)
+./scripts/benchmarks/run-perf-test.sh --scale tiny          # Quick smoke test
+./scripts/benchmarks/run-consolidation.sh
+
+# Results viewer
 ./scripts/benchmarks/start-visualizer.sh  # View results at localhost:8001
 ```
 
 ## Architecture
 
 ### Monorepo Structure
-- **hindsight-api/**: Core FastAPI server with memory engine (Python, uv)
-- **hindsight/**: Embedded Python bundle (hindsight-all package)
+- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)
 - **hindsight-control-plane/**: Admin UI (Next.js, npm)
 - **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)
 - **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)
 - **hindsight-docs/**: Docusaurus documentation site
-- **hindsight-integrations/**: Framework integrations (LiteLLM, OpenAI)
+- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)
 - **hindsight-dev/**: Development tools and benchmarks
 
-### Core Engine (hindsight-api/hindsight_api/engine/)
-- `memory_engine.py`: Main orchestrator (~170KB) for retain/recall/reflect operations
-- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, Groq, Ollama, LM Studio
+### Core Engine (hindsight-api-slim/hindsight_api/engine/)
+- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
+- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
 - `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
 - `cross_encoder.py`: Reranking (local or TEI)
 - `entity_resolver.py`: Entity extraction and normalization
@@ -88,13 +102,13 @@ cd hindsight-control-plane && npm run dev
 
 **search/**: Multi-strategy retrieval
 - `retrieval.py`: Main retrieval orchestrator
-- `graph_retrieval.py`: Entity/relationship graph traversal
-- `mpfp_retrieval.py`: Multi-Path Fact Propagation retrieval
+- `graph_retrieval.py`: Graph retrieval abstract base class
+- `link_expansion_retrieval.py`: Link expansion graph retrieval
 - `fusion.py`: Reciprocal rank fusion for combining results
 - `reranking.py`: Cross-encoder reranking
 
-### API Layer (hindsight-api/hindsight_api/api/)
-- `http.py`: FastAPI HTTP routers (~80KB) for all REST endpoints
+### API Layer (hindsight-api-slim/hindsight_api/api/)
+- `http.py`: FastAPI HTTP routers for all REST endpoints
 - `mcp.py`: Model Context Protocol server implementation
 
 Main operations:
@@ -103,18 +117,23 @@ Main operations:
 - **Reflect**: Disposition-aware reasoning using memories and mental models.
 
 ### Database
-PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api/hindsight_api/alembic/`. Migrations run automatically on API startup.
+PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api-slim/hindsight_api/alembic/`. Migrations run automatically on API startup.
 
 Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 
 ### Adding Database Migrations
 
-1. **Create a new migration file** in `hindsight-api/hindsight_api/alembic/versions/`:
+Hindsight runs the same Alembic tree against PostgreSQL and Oracle 23ai. Each
+migration file dispatches through `run_for_dialect`, which calls either
+`_pg_upgrade` or `_oracle_upgrade` based on the live connection. A pytest lint
+(`tests/test_migration_shape.py`) fails CI if a migration omits the dispatcher.
+
+1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:
    - File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)
    - Use a unique hex revision ID (12 chars)
    - Set `down_revision` to the previous migration's revision ID
 
-2. **Migration template**:
+2. **Migration template** (the `script.py.mako` template scaffolds this; fill in the bodies):
    ```python
    """Description of the migration
 
@@ -125,28 +144,61 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
    from collections.abc import Sequence
    from alembic import context, op
 
+   from hindsight_api.alembic._dialect import run_for_dialect
+
    revision: str = "f1a2b3c4d5e6"
    down_revision: str | Sequence[str] | None = "<previous_revision_id>"
    branch_labels: str | Sequence[str] | None = None
    depends_on: str | Sequence[str] | None = None
 
-   def _get_schema_prefix() -> str:
-       """Get schema prefix for table names (required for multi-tenant support)."""
+
+   def _pg_schema_prefix() -> str:
+       """Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
        schema = context.config.get_main_option("target_schema")
        return f'"{schema}".' if schema else ""
 
-   def upgrade() -> None:
-       schema = _get_schema_prefix()
+
+   def _pg_upgrade() -> None:
+       schema = _pg_schema_prefix()
        op.execute(f"CREATE INDEX ... ON {schema}table_name(...)")
 
-   def downgrade() -> None:
-       schema = _get_schema_prefix()
+
+   def _pg_downgrade() -> None:
+       schema = _pg_schema_prefix()
        op.execute(f"DROP INDEX IF EXISTS {schema}index_name")
+
+
+   def _oracle_upgrade() -> None:
+       # Oracle 23ai equivalent. Use op.get_bind().exec_driver_sql for forms
+       # that Alembic core does not model (vector/text indexes, partitions).
+       op.execute("CREATE INDEX ... ON table_name(...)")
+
+
+   def _oracle_downgrade() -> None:
+       op.execute("DROP INDEX IF EXISTS index_name")
+
+
+   def upgrade() -> None:
+       run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
+
+
+   def downgrade() -> None:
+       run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
    ```
 
+   **Dialect-only migrations.** If a change genuinely doesn't apply to one
+   dialect (e.g. enabling `pg_trgm` is PG-only), omit the unused slot:
+   ```python
+   def upgrade() -> None:
+       run_for_dialect(pg=_pg_upgrade)  # oracle slot intentionally absent → no-op
+   ```
+   Make the asymmetry deliberate. Don't leave an Oracle slot empty just because
+   you didn't think about it — copy-pasting a PG migration without the Oracle
+   half is exactly how schemas drift.
+
 3. **Run migrations locally**:
    ```bash
-   # Set database URL and run migrations
+   # Set database URL and run migrations for the base schema plus all tenants
    uv run hindsight-admin run-db-migration
 
    # Run on a specific tenant schema
@@ -156,11 +208,41 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 ## Key Conventions
 
 ### Code Quality
+
+**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).
+
 **Always run the lint script after making Python or TypeScript/Node changes:**
 ```bash
 ./scripts/hooks/lint.sh
 ```
-This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Prettier for TypeScript).
+
+**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
+
+**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
+
+### Testing
+
+Most tests are deterministic (MockLLM, pure functions) — assert directly.
+
+**Tests that verify LLM behaviour use a real LLM + an LLM-as-judge.** When the thing under test is *how the model interprets a prompt* (classification, attribution, dimension preservation, instruction-following), MockLLM can't simulate it and exact string/enum asserts flake across providers and runs. Use this pattern instead:
+
+1. Mark the test module `pytestmark = pytest.mark.hs_llm_core` (single-provider; CI runs it in the core-LLM job). Use `hs_llm_mat` only for provider-matrix acceptance tests.
+2. Call the real pipeline (`LLMConfig.from_env()`, `_get_raw_config()`), e.g. `extract_facts_from_text(...)`.
+3. Assert with the judge, not string matching:
+   ```python
+   from tests.llm_judge import assert_meets_criteria
+   facts_summary = "\n".join(f"- [{f.fact_type}] {f.fact}" for f in facts)
+   await assert_meets_criteria(
+       response=facts_summary,
+       criteria="The first-person user statements are classified 'world' and attributed to the user, not the agent.",
+       context="What the input said and who was speaking.",
+   )
+   ```
+
+Rules of thumb:
+- **Judge anything non-deterministic** — including `fact_type` classification and speaker attribution. Do NOT hard-assert `fact_type == "..."`; pass a `[fact_type] fact` summary to the judge instead. Structural facts that ARE deterministic (counts, presence of a field, that a substring was injected into a prompt) stay as direct asserts in fast unit tests.
+- **Split the test surface**: cover the deterministic mechanics (prompt assembly, suppression logic) with fast non-LLM unit tests, and the model-following behaviour with one `hs_llm_core` judge test. (Example pair: `test_narrator_resolution.py` + `test_narrator_context_override.py`.)
+- The judge model is independent of the test provider (defaults to Gemini); never judge with the same call you're testing.
 
 ### Memory Banks
 - Each bank is an isolated memory store (like a "brain" for one user/agent)
@@ -192,71 +274,78 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
    - Update the client type definition in `lib/api.ts`
    - Update any UI components that need to use the new parameter
 
-### Python Style
-- Python 3.11+, type hints required
-- Async throughout (asyncpg, async FastAPI)
-- Pydantic models for request/response
-- Ruff for linting (line-length 120)
-- No Python files at project root - maintain clean directory structure
-- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
+### Adding New Integrations
 
-### Type Safety with Pydantic Models
-**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
-- Use Pydantic `BaseModel` for all data structures passed between functions
-- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
-- Avoid `dict.get()` patterns - use typed model attributes instead
-- Parse external data (JSON, API responses) into Pydantic models at the boundary
-- This catches type errors at parse time, not deep in business logic
+Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
 
-```python
-# BAD - error-prone dict access
-def process(data: dict) -> str:
-    return data.get("name", "")  # No validation, silent failures
+1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.
+2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.
+3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.
+4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).
 
-# GOOD - typed and validated
-class UserData(BaseModel):
-    name: str
-    created_at: datetime
+If any of these are missing, the integration is incomplete and must not be pushed or merged.
 
-    @field_validator("created_at", mode="before")
-    @classmethod
-    def ensure_tz_aware(cls, v):
-        if isinstance(v, str):
-            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
-        if v.tzinfo is None:
-            return v.replace(tzinfo=timezone.utc)
-        return v
+### Changelogs
 
-def process(data: UserData) -> str:
-    return data.name  # Type-safe, validated at construction
-```
-
-### TypeScript Style
-- Next.js App Router for control plane
-- Tailwind CSS with shadcn/ui components
+Never add "Unreleased" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.
 
 ### Adding New API Configuration Flags
 
-When adding a new environment variable configuration:
+Configuration follows a hierarchical system: **Global (env vars) → Tenant (via extension) → Bank (database)**.
 
-1. **config.py** (`hindsight-api/hindsight_api/config.py`):
-   - Add `ENV_*` constant for the environment variable name
+Fields must be categorized as either **hierarchical** (can be overridden per-tenant/bank) or **static** (server-level only).
+
+#### Adding a New Configuration Field
+
+1. **config.py** (`hindsight-api-slim/hindsight_api/config.py`):
+   - Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
    - Add `DEFAULT_*` constant for the default value
-   - Add field to `HindsightConfig` dataclass
+   - Add field to `HindsightConfig` dataclass with type annotation
+   - **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API
    - Add initialization in `from_env()` method
 
-2. **main.py** (`hindsight-api/hindsight_api/main.py`):
+   ```python
+   # Configurable field (can be overridden per-tenant/bank via API)
+   _CONFIGURABLE_FIELDS = {
+       ...,
+       "my_setting",  # Add here for configurable
+   }
+
+   # Static field - just don't add to _CONFIGURABLE_FIELDS
+   ```
+
+2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
    - Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
 
-3. **Use the config** in code:
+3. **Use hierarchical config in MemoryEngine**:
+   ```python
+   # Config is resolved automatically per bank via ConfigResolver
+   config_dict = await self._config_resolver.get_bank_config(bank_id, context)
+   value = config_dict["my_setting"]
+   ```
+
+4. **Use static config** (non-hierarchical):
    ```python
    from ...config import get_config
    config = get_config()
-   value = config.your_new_field
+   value = config.my_static_field
    ```
 
-4. **Documentation** (`hindsight-docs/docs/developer/configuration.md`):
+5. **Documentation** (`hindsight-docs/docs/developer/configuration.md`):
    - Add to appropriate section table with Variable, Description, Default
+   - Mark if it's hierarchical (can be overridden per-bank)
+
+#### Hierarchical vs Static Guidelines
+
+**Hierarchical** (per-bank overridable):
+- LLM settings (provider, model, API key, base URL)
+- Operation-specific settings (retain mode, chunk size, etc.)
+- Feature flags that vary by customer/bank
+
+**Static** (server-level only):
+- Infrastructure settings (database URL, port, host)
+- Global limits (max concurrent operations)
+- System-wide feature flags
 
 ## Environment Setup
 
@@ -265,18 +354,19 @@ cp .env.example .env
 # Edit .env with LLM API key
 
 # Python deps
-uv sync --directory hindsight-api/
+uv sync --directory hindsight-api-slim/
 
 # Node deps (uses npm workspaces)
 npm install
 ```
 
 Required env vars:
-- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, ollama, lmstudio
+- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio
 - `HINDSIGHT_API_LLM_API_KEY`: Your API key
-- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., o3-mini, claude-sonnet-4-20250514)
+- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., gpt-4o-mini, claude-sonnet-4-20250514)
 
 Optional (uses local models by default):
 - `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei
 - `HINDSIGHT_API_RERANKER_PROVIDER`: local (default) or tei
 - `HINDSIGHT_API_DATABASE_URL`: External PostgreSQL (uses embedded pg0 by default)
+- `HINDSIGHT_API_ENABLE_BANK_CONFIG_API`: Enable per-bank config API (default: true)

CONTRIBUTING.md

@@ -9,13 +9,36 @@ Thanks for your interest in contributing to Hindsight!
    git clone [email protected]:vectorize-io/hindsight.git
    cd hindsight
    ```
-2. Set up your environment:
+
+2. Bootstrap your dev environment in one shot:
+   ```bash
+   ./scripts/dev/setup.sh
+   ```
+   This is idempotent (safe to re-run) and gets you ready to develop, including
+   offline. It:
+   - installs the required toolchains if missing (uv/Python, Node/npm, Rust/cargo),
+   - creates `.env` from `.env.example` (remember to add your LLM API key),
+   - configures git hooks,
+   - installs all Python and Node workspace dependencies,
+   - pre-downloads the local ML models + tokenizer so the API runs offline,
+   - builds the TypeScript SDK and the Rust CLI.
+
+   Useful flags: `--skip-build` (deps only), `--skip-models` (skip ML model
+   download), `--with-docs` (also build the docs site), `--force` (rebuild
+   artifacts). Docker image builds are out of scope. Run
+   `./scripts/dev/setup.sh --help` for details.
+
+### Manual setup
+
+If you'd rather set things up by hand instead of running the script above:
+
+1. Set up your environment:
    ```bash
    cp .env.example .env
    ```
    Edit the .env to add LLM API key and config as required
 
-3. Install dependencies:
+2. Install dependencies:
    ```bash
    # Python dependencies
    uv sync --directory hindsight-api/
@@ -93,6 +116,34 @@ uv run ty check hindsight_api  # Type check
 3. Run tests to ensure nothing breaks
 4. Submit a PR with a clear description of changes
 
+## Release Process
+
+The project uses `scripts/release.sh` for creating releases. This script automates the entire release workflow:
+
+1. Bumps version in all components (API, clients, CLI, control plane, Helm)
+2. **Regenerates OpenAPI spec and client SDKs** (Python, TypeScript, Rust)
+3. Updates documentation versioning
+4. Creates a commit and git tag
+5. Pushes to GitHub (triggers CI/CD to publish packages)
+
+### Usage
+
+```bash
+./scripts/release.sh <version>
+```
+
+**Example:**
+```bash
+./scripts/release.sh 0.5.0
+```
+
+### Important for Developers
+
+- During development, version bumps in `__init__.py` do NOT require client regeneration
+- Clients are only regenerated during releases
+- Do not manually run `./scripts/generate-clients.sh` unless testing generation changes
+- Client version comments will reflect the API version from the latest release
+
 ## Reporting Issues
 
 Open an issue on GitHub with:

README.md

@@ -1,92 +1,96 @@
 <div align="center">
 
-![Hindsight Banner](./hindsight-docs/static/img/banner.svg)
+![Hindsight Banner](./hindsight-docs/static/img/hindsight-github-banner.png)
 
-[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://vectorize.io/hindsight/cloud)
+[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)
 
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
 [![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
+<br/>
 
-
+<a href="https://trendshift.io/repositories/15603" target="_blank"><img src="https://trendshift.io/api/badge/repositories/15603" alt="vectorize-io%2Fhindsight | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </div>
 
 ---
 
 ## What is Hindsight?
 
-Hindsight™ is an agent memory system built to create smarter agents that learn over time. It eliminates the shortcomings of alternative techniques such as RAG and knowledge graph and delivers state-of-the-art performance on long term memory tasks.
+Hindsight™ is an agent memory system built to create smarter agents that learn over time. Most agent memory systems focus on recalling conversation history. Hindsight is focused on making agents that learn, not just remember.
 
-Hindsight addresses common challenges that have frustrated AI engineers building agents to automate tasks and assist users with conversational interfaces. Many of these challenges stem directly from a lack of memory.
 
-- **Inconsistency:** Agents complete tasks successfully one time, then fail when asked to complete the same task again. Memory gives the agent a mechanism to remember what worked and what didn't and to use that information to reduce errors and improve consistency.
-- **Hallucinations:** Long term memory can be seeded with external knowledge to ground agent behavior in reliable sources to augment training data.
-- **Cognitive Overload:** As workflows get complex, retrievals, tool calls, user messages and agent responses can grow to fill the context window leading to context rot. Short term memory optimization allows agents to reduce tokens and focus context by removing irrelevant details.
+<video src="https://github.com/user-attachments/assets/923b798d-3581-4897-bb62-9cfa5a931682" controls></video>
 
-## How is Hindsight Different From Other Memory Systems?
-
-![Overview](./hindsight-docs/static/img/hindsight-overview.webp)
-
-Most agent memory implementation rely on basic vector search or sometimes use a knowledge graph. Hindsight uses biomimetic data structures to organize agent memories in a way that is more like how human memory works:
-
-- **World:** Facts about the world ("The stove gets hot")
-- **Experiences:** Agent's own experiences ("I touched the stove and it really hurt")
-- **Opinion:** Beliefs with confidence scores ("I shouldn't touch the stove again" - .99 confidence)
-- **Observation:** Complex mental models derived by reflecting on facts and experiences ("Curling irons, ovens, and fire are also hot. I shouldn't touch those either.")
-
-Memories in Hindsight are stored in banks (i.e. memory banks). When memories are added to Hindsight, they are pushed into either the world facts or experiences memory pathway. They are then represented as a combination of entities, relationships, and time series with sparse/dense vector representations to aid in later recall.
-
-Hindsight provides three simple methods to interact with the system:
-
-- **Retain:** Provide information to Hindsight that you want it to remember
-- **Recall:** Retrieve memories from Hindsight
-- **Reflect:** Reflect on memories and experiences to generate new observations and insights from existing memories.
-
-### Agent Memory That Learns
-
-A key goal of Hindsight is to build agent memory that enables agents to learn and improve over time. This is the role of the `reflect` operation which provides the agent to form broader opinions and observations over time.
-
-For example, imagine a product support agent that is helping a user troubleshoot a problem. It uses a `search-documentation` tool it found on an MCP server. Later in the conversation, the agent discovers that the documentation returned from the tool wasn't for the product the user was asking about. The agent now has an experience in its memory bank. And just like humans, we want that agent to learn from its experience.
-
-As the agent gains more experiences, `reflect` allows the agent to form observations about what worked, what didn't, and what to do differently the next time it encounters a similar task.
-
----
+It eliminates the shortcomings of alternative techniques such as RAG and knowledge graph and delivers state-of-the-art performance on long term memory tasks.
 
 ## Memory Performance & Accuracy
 
-Hindsight has achieved state-of-the-art performance on the LongMemEval benchmark, widely used to assess memory system performance across a variety of conversational
-AI scenarios. The current reported performance of Hindsight and other agent memory solutions as of December 2025 is shown here:
+Hindsight is the most accurate agent memory system ever tested according to benchmark performance. It has achieved state-of-the-art performance on the LongMemEval benchmark, widely used to assess memory system performance across a variety of conversational AI scenarios. The current reported performance of Hindsight and other agent memory solutions as of January 2026 is shown here:
 
-![Overview](./hindsight-docs/static/img/hindsight-bench.jpg)
+![Overview](./hindsight-docs/static/img/hindsight-benchmarks.png)
 
-The benchmark performance data for Hindsight and GPT-4o (full context) have been reproduced by research collaborators at the Virginia Tech [Sanghani Center for Artificial Intelligence and Data Analytics](https://sanghani.cs.vt.edu/) and The Washington Post. Other scores are self-reported by software vendors.
+The benchmark performance data for Hindsight has been independently reproduced by research collaborators at the Virginia Tech [Sanghani Center for Artificial Intelligence and Data Analytics](https://sanghani.cs.vt.edu/) and The Washington Post. Other scores are self-reported by software vendors.
 
-A thorough examination of the techniques implemented in Hindsight and detailed breakdowns of benchmark performance are [available on arXiv](https://arxiv.org/abs/2512.12818). This research is currently being prepared for conference submission and the wider peer review process.
+Hindsight is being used in production at Fortune 500 enterprises and by a growing number of AI startups. 
+
+## Adding Hindsight to Your AI Agents
+
+The easiest way to use Hindsight with an existing agent is with the LLM Wrapper. You can add memory to your agent with 2 lines of code. That will swap your current LLM client out with the Hindsight wrapper. After that, memories will be stored and retrieved automatically as you make LLM calls.
+
+If you need more control over how and when your agent stores and recalls memories, there's also a simple API you can integrate with using the SDKs or directly via HTTP.
+
+![Hindsight Banner](./hindsight-docs/static/img/migration-code.png)
+
+---
+
+> 🤖 **Using a coding agent?** Install the Hindsight documentation skill for instant access to docs while you code:
+> ```bash
+> npx skills add https://github.com/vectorize-io/hindsight --skill hindsight-docs
+> ```
+> Works with Claude Code, Cursor, and other AI coding assistants.
+
+---
 
-The benchmark results from this research can be inspected in our [visual benchmark explorer](https://hindsight-benchmarks.vercel.app). As additional improvements are made to Hindsight, new benchmark data will be available for review using this same tool.
 
 ## Quick Start
 
 ### Docker (recommended)
 
 ```bash
-export OPENAI_API_KEY=your-key
+export OPENAI_API_KEY=sk-xxx
 
-docker run --rm -it --pull always -p 8888:8888 -p 9999:9999 \
+docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8888 -p 9999:9999 \
   -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
-  -e HINDSIGHT_API_LLM_MODEL=o3-mini \
-  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
+  -v hindsight-data:/home/hindsight/.pg0 \
   ghcr.io/vectorize-io/hindsight:latest
 ```
 
-You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, and `lmstudio`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
+>API: http://localhost:8888
+>UI: http://localhost:9999
 
-API: http://localhost:8888  
-UI: http://localhost:9999
+You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `minimax`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
 
-Install client:
+
+
+### Docker (external PostgreSQL)
+
+```bash
+export OPENAI_API_KEY=sk-xxx
+export HINDSIGHT_DB_PASSWORD=choose-a-password
+cd docker/docker-compose
+docker compose up 
+```
+
+> Oracle AI Database is also supported for enterprise deployments with full feature parity. See the [storage documentation](https://hindsight.vectorize.io/developer/storage) for details.
+
+
+>API: http://localhost:8888
+>UI: http://localhost:9999
+
+### Client
 
 ```bash
 pip install hindsight-client -U
@@ -94,7 +98,7 @@ pip install hindsight-client -U
 npm install @vectorize-io/hindsight-client
 ```
 
-Python example:
+#### Python
 
 ```python
 from hindsight_client import Hindsight
@@ -111,7 +115,29 @@ client.recall(bank_id="my-bank", query="What does Alice do?")
 client.reflect(bank_id="my-bank", query="Tell me about Alice")
 ```
 
-### Python (embedded, no Docker)
+#### Node.js / TypeScript
+
+```bash
+npm install @vectorize-io/hindsight-client
+```
+
+```javascript
+const { HindsightClient } = require('@vectorize-io/hindsight-client');
+
+const main = async () => {
+  const client = new HindsightClient({ baseUrl: 'http://localhost:8888' });
+
+  await client.retain('my-bank', 'Alice loves hiking in Yosemite');
+
+  const results = await client.recall('my-bank', 'What does Alice like?');
+  console.log(results);
+}
+
+main();
+```
+
+
+### Python Embedded (no server required)
 
 ```bash
 pip install hindsight-all -U
@@ -131,25 +157,48 @@ with HindsightServer(
     results = client.recall(bank_id="my-bank", query="Where does Alice work?")
 ```
 
-### Node.js / TypeScript
 
-```bash
-npm install @vectorize-io/hindsight-client
-```
+---
 
-```javascript
-const { HindsightClient } = require('@vectorize-io/hindsight-client');
+## Use Cases
 
-const client = new HindsightClient({ baseUrl: 'http://localhost:8888' });
 
-await client.retain('my-bank', 'Alice loves hiking in Yosemite');
-await client.recall('my-bank', 'What does Alice like?');
-```
+Hindsight is built to support conversational AI agents as well as agents that are intended to perform tasks autonomously. The ideal use case for Hindsight are agents that require a blend of these features such as AI employees that need to handle open-ended tasks, change behavior based on user feedback, and learn to perform complex tasks to automate work at a level that approximates a human work. Hindsight can be used with simple AI workflows like those built with n8n and other similar tools, but may be overkill for such applications.
+
+### Per-User Memories and Chat History
+
+One of the simpler use cases you can use Hindsight for is to personalize AI chatbots and other conversational agents by storing and recalling memories associated with individual users.
+
+The requirements for this use case usually look something like this:
+
+![Per-User Memories](./hindsight-docs/static/img/per-user-memory-requirements.png)
+
+<video src="https://github.com/user-attachments/assets/4805e8e1-e7d1-47c6-a4f8-2344a5ec8906" controls></video>
+
+Satisfying these requirements in Hindsight is straightforward. When new user inputs and tool calls are ingested into Hindsight using the retain operation, custom metadata can be used to enrich the new memories. Metadata provides a convenient way to isolate memories that need to be restricted to a given user. Once these are fed into the retain operation, any raw memories and mental models that get created can be filtered when retrieving relevant memories. 
+
+![Per-User Memories](./hindsight-docs/static/img/per-user-memory-howto.png)
 
 ---
 
 ## Architecture & Operations
 
+![Overview](./hindsight-docs/static/img/hindsight-overview.webp)
+
+Most agent memory implementations rely on basic vector search or sometimes use a knowledge graph. Hindsight uses biomimetic data structures to organize agent memories in a way that is more like how human memory works:
+
+- **World:** Facts about the world ("The stove gets hot")
+- **Experiences:** Agent's own experiences ("I touched the stove and it really hurt")
+- **Mental Models:** Learned understanding of the agent's world formed by reflecting on raw memories and experiences.
+
+Memories in Hindsight are stored in banks (i.e. memory banks). When memories are added to Hindsight, they are pushed into either the world facts or experiences memory pathway. They are then represented as a combination of entities, relationships, and time series with sparse/dense vector representations to aid in later recall.
+
+Hindsight provides three simple methods to interact with the system:
+
+- **Retain:** Provide information to Hindsight that you want it to remember
+- **Recall:** Retrieve memories from Hindsight
+- **Reflect:** Reflect on memories and experiences to generate new observations and insights from existing memories.
+
 ### Retain
 
 The `retain` operation is used to push new memories into Hindsight. It tells Hindsight to _retain_ the information you pass in as an input.
@@ -208,7 +257,7 @@ The final output is trimmed as needed to fit within the token limit.
 
 ### Reflect
 
-The reflect operation is used to perform a more thorough analysis of existing memories. This allows the agent to form new connections between memories which are then persisted as opinions and/or observations. When building agents, the reflect operation is a key capability to enable the agent to learn from its experiences. 
+The reflect operation is used to perform a more thorough analysis of existing memories. This allows the agent to form new connections between memories and build a more thorough understanding of its world.
 
 For example, the `reflect` operation can be used to support use cases such as:
 
@@ -262,3 +311,5 @@ MIT — see [LICENSE](./LICENSE)
 ---
 
 Built by [Vectorize.io](https://vectorize.io)
+
+<img src="https://umami-pixel.chris-latimer.workers.dev/?id=a8b043e6-6964-454d-80df-69b69d3f0d50&host=github.com&url=/vectorize-io/hindsight" width="1" height="1" alt="" />

deno.lock

@@ -0,0 +1,171 @@
+{
+  "version": "5",
+  "specifiers": {
+    "jsr:@std/assert@^1.0.17": "1.0.19",
+    "jsr:@std/assert@^1.0.19": "1.0.19",
+    "jsr:@std/expect@*": "1.0.18",
+    "jsr:@std/internal@^1.0.12": "1.0.12",
+    "jsr:@std/path@^1.1.4": "1.1.4",
+    "jsr:@std/testing@*": "1.0.17"
+  },
+  "jsr": {
+    "@std/[email protected]": {
+      "integrity": "eaada96ee120cb980bc47e040f82814d786fe8162ecc53c91d8df60b8755991e",
+      "dependencies": [
+        "jsr:@std/internal"
+      ]
+    },
+    "@std/[email protected]": {
+      "integrity": "8566eab35200466f8609eb7e7aed062ed0db314e9a258d5d201b1b8997ce801a",
+      "dependencies": [
+        "jsr:@std/assert@^1.0.19",
+        "jsr:@std/internal",
+        "jsr:@std/path"
+      ]
+    },
+    "@std/[email protected]": {
+      "integrity": "972a634fd5bc34b242024402972cd5143eac68d8dffaca5eaa4dba30ce17b027"
+    },
+    "@std/[email protected]": {
+      "integrity": "1d2d43f39efb1b42f0b1882a25486647cb851481862dc7313390b2bb044314b5",
+      "dependencies": [
+        "jsr:@std/internal"
+      ]
+    },
+    "@std/[email protected]": {
+      "integrity": "87bdc2700fa98249d48a17cd72413352d3d3680dcfbdb64947fd0982d6bbf681",
+      "dependencies": [
+        "jsr:@std/assert@^1.0.17",
+        "jsr:@std/internal"
+      ]
+    }
+  },
+  "workspace": {
+    "members": {
+      "hindsight-all-npm": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@types/node@22",
+            "npm:tsup@^8.5.1",
+            "npm:typescript@^5.7.0",
+            "npm:vitest@^4.1.2"
+          ]
+        }
+      },
+      "hindsight-clients/typescript": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@hey-api/[email protected]",
+            "npm:@types/jest@29",
+            "npm:@types/node@20",
+            "npm:jest@29",
+            "npm:ts-jest@29",
+            "npm:tsup@^8.5.1",
+            "npm:typescript@5"
+          ]
+        }
+      },
+      "hindsight-control-plane": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@chenglou/pretext@^0.0.3",
+            "npm:@eslint/eslintrc@^3.3.3",
+            "npm:@eslint/js@^9.39.2",
+            "npm:@radix-ui/react-alert-dialog@^1.1.15",
+            "npm:@radix-ui/react-checkbox@^1.3.3",
+            "npm:@radix-ui/react-dialog@^1.1.15",
+            "npm:@radix-ui/react-dropdown-menu@^2.1.16",
+            "npm:@radix-ui/react-label@^2.1.8",
+            "npm:@radix-ui/react-popover@^1.1.15",
+            "npm:@radix-ui/react-radio-group@^1.3.8",
+            "npm:@radix-ui/react-select@^2.2.6",
+            "npm:@radix-ui/react-slider@^1.3.6",
+            "npm:@radix-ui/react-slot@^1.2.4",
+            "npm:@radix-ui/react-switch@^1.2.6",
+            "npm:@radix-ui/react-tabs@^1.1.13",
+            "npm:@radix-ui/react-tooltip@^1.2.8",
+            "npm:@tailwindcss/postcss@^4.1.17",
+            "npm:@tailwindcss/typography@~0.5.19",
+            "npm:@types/cytoscape@^3.21.9",
+            "npm:@types/node@^24.10.0",
+            "npm:@types/react-dom@^19.2.2",
+            "npm:@types/react@^19.2.2",
+            "npm:autoprefixer@^10.4.21",
+            "npm:class-variance-authority@~0.7.1",
+            "npm:clsx@^2.1.1",
+            "npm:cmdk@^1.1.1",
+            "npm:cytoscape-fcose@^2.2.0",
+            "npm:cytoscape@^3.33.1",
+            "npm:eslint-config-next@^16.0.1",
+            "npm:eslint-plugin-react-hooks@^7.0.1",
+            "npm:eslint-plugin-react@^7.37.5",
+            "npm:eslint@^9.39.1",
+            "npm:[email protected]",
+            "npm:next-themes@~0.4.6",
+            "npm:next@^16.1.7",
+            "npm:postcss@^8.5.6",
+            "npm:prettier@^3.7.4",
+            "npm:react-chrono@^2.9.1",
+            "npm:react-dom@^19.2.0",
+            "npm:react-is@^19.2.4",
+            "npm:react-markdown@^10.1.0",
+            "npm:react18-json-view@~0.2.9",
+            "npm:react@^19.2.0",
+            "npm:recharts@^3.5.1",
+            "npm:remark-gfm@^4.0.1",
+            "npm:sonner@^2.0.7",
+            "npm:tailwind-merge@^3.4.0",
+            "npm:tailwindcss-animate@^1.0.7",
+            "npm:tailwindcss@^4.1.17",
+            "npm:[email protected]",
+            "npm:typescript-eslint@^8.50.0",
+            "npm:typescript@^5.9.3"
+          ]
+        }
+      },
+      "hindsight-docs": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@docusaurus/[email protected]",
+            "npm:@docusaurus/[email protected]",
+            "npm:@docusaurus/[email protected]",
+            "npm:@docusaurus/theme-common@^3.9.2",
+            "npm:@docusaurus/theme-mermaid@^3.9.2",
+            "npm:@docusaurus/[email protected]",
+            "npm:@docusaurus/[email protected]",
+            "npm:@easyops-cn/docusaurus-search-local@~0.52.2",
+            "npm:@mdx-js/react@3",
+            "npm:clsx@2",
+            "npm:prism-react-renderer@^2.3.0",
+            "npm:raw-loader@^4.0.2",
+            "npm:react-dom@19",
+            "npm:react-icons@^5.6.0",
+            "npm:react@19",
+            "npm:redocusaurus@^2.5.0",
+            "npm:typescript@~5.6.2"
+          ]
+        }
+      },
+      "hindsight-tools/hindsight-agent-sdk": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@vectorize-io/hindsight-client@~0.5.6",
+            "npm:typescript@^5.4.0",
+            "npm:vitest@^4.1.2"
+          ]
+        }
+      },
+      "hindsight-tools/self-driving-agents": {
+        "packageJson": {
+          "dependencies": [
+            "npm:@clack/prompts@^1.2.0",
+            "npm:@vectorize-io/hindsight-client@~0.5.6",
+            "npm:picocolors@^1.1.0",
+            "npm:typescript@^5.4.0",
+            "npm:vitest@^4.1.2"
+          ]
+        }
+      }
+    }
+  }
+}

docker/docker-compose/alloydb/docker-compose.yaml

@@ -0,0 +1,90 @@
+name: hindsight
+# Docker Compose file for Hindsight with AlloyDB Omni and ScaNN
+# Uses Google's free AlloyDB Omni container image: https://hub.docker.com/r/google/alloydbomni
+#
+# Usage:
+# docker compose -f docker/docker-compose/alloydb/docker-compose.yaml up -d
+#
+# Make sure to set the required environment variables before running:
+# - HINDSIGHT_DB_PASSWORD: password for the AlloyDB Omni/PostgreSQL user
+# - Configure LLM provider variables as needed (see the hindsight service below)
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_VERSION: AlloyDB Omni image tag (default: 17)
+# - HINDSIGHT_DB_USER: database user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: database name (default: hindsight_db)
+
+services:
+  db:
+    image: google/alloydbomni:${HINDSIGHT_DB_VERSION:-17}
+    container_name: hindsight-db-alloydb
+    restart: always
+    ports:
+      - "5438:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - alloydb_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+
+  alloydb-init:
+    image: google/alloydbomni:${HINDSIGHT_DB_VERSION:-17}
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command:
+      - bash
+      - -c
+      - |
+        echo 'Waiting for AlloyDB Omni to be ready...'
+        until pg_isready -h hindsight-db-alloydb -p 5432 -U ${HINDSIGHT_DB_USER:-hindsight_user}; do
+          echo 'AlloyDB Omni is unavailable - sleeping'
+          sleep 2
+        done
+        echo 'AlloyDB Omni is ready - creating ${HINDSIGHT_DB_NAME:-hindsight_db} database'
+        psql -h hindsight-db-alloydb -p 5432 -U ${HINDSIGHT_DB_USER:-hindsight_user} -c 'CREATE DATABASE ${HINDSIGHT_DB_NAME:-hindsight_db};' 2>/dev/null || echo 'Database already exists'
+        echo 'Creating vector and alloydb_scann extensions'
+        psql -h hindsight-db-alloydb -p 5432 -U ${HINDSIGHT_DB_USER:-hindsight_user} -d ${HINDSIGHT_DB_NAME:-hindsight_db} -c 'CREATE EXTENSION IF NOT EXISTS vector;'
+        psql -h hindsight-db-alloydb -p 5432 -U ${HINDSIGHT_DB_USER:-hindsight_user} -d ${HINDSIGHT_DB_NAME:-hindsight_db} -c 'CREATE EXTENSION IF NOT EXISTS alloydb_scann CASCADE;'
+        echo 'Database and extensions created successfully'
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: scann
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: native
+
+    depends_on:
+      db:
+        condition: service_started
+      alloydb-init:
+        condition: service_completed_successfully
+    networks:
+      - hindsight-net
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  alloydb_data:

docker/docker-compose/claude-code/README.md

@@ -0,0 +1,113 @@
+# Hindsight with Claude Code (Claude Pro/Max subscription)
+
+Run Hindsight inside Docker using the `claude-code` LLM provider, backed by
+your host machine's Claude Pro or Max subscription credentials.
+
+The standalone Hindsight Docker image ships `claude-agent-sdk` but does **not**
+bundle the host `claude` CLI binary or any Claude credentials. This Compose
+file bind-mounts the host's CLI install and credentials into the container so
+the `claude-code` provider works without an API key.
+
+## When to use this
+
+- You have an active Claude Pro or Max subscription and want to use it for
+  Hindsight without paying separate Anthropic API costs.
+- You want a one-command `docker compose up` instead of a long `docker run`
+  invocation with many flags.
+- You are running on **Linux/amd64** — macOS Docker Desktop and Windows host
+  paths differ and are not yet covered (please open an issue if you'd like to
+  contribute a verified recipe for either).
+
+> **Personal-use only.**  Anthropic's
+> [Agent SDK documentation](https://docs.claude.com/en/api/agent-sdk/overview)
+> states that third-party developers should not offer claude.ai login or rate
+> limits for their products. Hindsight does **not** perform any login on your
+> behalf — it uses credentials you've already authenticated via
+> `claude auth login`. In January 2026, Anthropic
+> [enforced restrictions](https://paddo.dev/blog/anthropic-walled-garden-crackdown/)
+> against tools that spoofed the Claude Code client identity; Hindsight uses
+> the official Claude Agent SDK instead.
+>
+> Do not deploy this configuration to shared environments or production. For
+> that, use the `anthropic` provider with an API key from the
+> [Anthropic Console](https://console.anthropic.com/). Usage counts against
+> your Claude Pro/Max subscription limits.
+
+## Prerequisites
+
+- Host has `claude` CLI installed (e.g., `npm install -g @anthropics/claude-code`)
+  and `claude auth login` has been run successfully.
+- `~/.claude.json` and `~/.claude/.credentials.json` exist on the host.
+- Host `claude` CLI version is **2.1.128 or newer** — the version bundled with
+  `claude-agent-sdk` 0.5.x has a protocol incompatibility in containers, so
+  the recipe overrides it with the host binary.
+
+## Quick start
+
+```bash
+# Set your host UID/GID (defaults to 1000:1000 if unset)
+export HOST_UID=$(id -u)
+export HOST_GID=$(id -g)
+
+docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d
+```
+
+- API: http://localhost:8888
+- Control Plane: http://localhost:9999
+
+## Post-setup (one-time)
+
+After the container starts for the first time, run these commands to fix
+permissions and symlink the host `claude` binary into `$PATH`:
+
+```bash
+# Make ~/.claude writable by your UID (the CLI writes session/project state)
+docker exec --user 0:0 hindsight-claude-code chown $(id -u):$(id -g) /home/hindsight/.claude
+docker exec --user 0:0 hindsight-claude-code chmod 755 /home/hindsight/.claude
+
+# Symlink the host claude binary into PATH
+docker exec --user 0:0 hindsight-claude-code \
+  ln -sf /home/hindsight/.local/share/claude/versions/2.1.128 /usr/local/bin/claude
+```
+
+If you set `CLAUDE_CLI_VERSION` to a version other than `2.1.128`, update the
+symlink path accordingly.
+
+## Notes on the bind-mount surface (every flag is load-bearing)
+
+- **Host `claude` binary required** — the image ships only `claude-agent-sdk`,
+  not the CLI itself.
+- **SDK bundled-binary override** — the override of
+  `claude_agent_sdk/_bundled/claude` works around a protocol issue in the
+  bundled v2.1.121 binary inside containers. Once `claude-agent-sdk` ships
+  with v2.1.128+ this override can be dropped. Set `CLAUDE_CLI_VERSION` to
+  match your installed version.
+- **Single-file credential mounts** — credentials are mounted as individual
+  `:ro` files rather than a whole-directory `:ro` mount of `~/.claude`,
+  because the CLI writes session/project state at runtime and a read-only
+  directory mount silently breaks it.
+- **`--user` / `user:`** — the `user: ${HOST_UID}:${HOST_GID}` pattern
+  requires `chmod 755 /home/hindsight`, which is built into the image since
+  v0.6.0 (see [#1481](https://github.com/vectorize-io/hindsight/issues/1481)).
+- **`~/.hindsight-docker` data directory** — the pg0 data bind mount must be
+  writable by your host UID (see
+  [#1483](https://github.com/vectorize-io/hindsight/issues/1483)).
+- **Verified** on `linux/amd64` against `ghcr.io/vectorize-io/hindsight:latest`
+  v0.5.6+.
+
+## Using a different Claude CLI version
+
+If your host has a `claude` version other than 2.1.128, set
+`CLAUDE_CLI_VERSION` before starting:
+
+```bash
+export CLAUDE_CLI_VERSION=2.2.0
+docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d
+```
+
+Then update the post-setup symlink to match:
+
+```bash
+docker exec --user 0:0 hindsight-claude-code \
+  ln -sf /home/hindsight/.local/share/claude/versions/2.2.0 /usr/local/bin/claude
+```

docker/docker-compose/claude-code/docker-compose.yaml

@@ -0,0 +1,44 @@
+name: hindsight-claude-code
+# Run Hindsight with the claude-code LLM provider, using your host machine's
+# Claude Pro/Max subscription credentials.  Linux/amd64 only for now.
+#
+# Quick start:
+#   docker compose -f docker/docker-compose/claude-code/docker-compose.yaml up -d
+#
+# See README.md for prerequisites, post-setup steps, and important caveats.
+
+services:
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:latest
+    container_name: hindsight-claude-code
+    user: "${HOST_UID:-1000}:${HOST_GID:-1000}"
+    ports:
+      - "127.0.0.1:8888:8888"
+      - "127.0.0.1:9999:9999"
+    environment:
+      HOME: /home/hindsight
+      USER: hindsight
+      LOGNAME: hindsight
+      PATH: /usr/local/bin:/usr/bin:/bin:/app/api/.venv/bin
+      HINDSIGHT_API_LLM_PROVIDER: claude-code
+    volumes:
+      # ── Persistent data ────────────────────────────────────────────
+      # Writable pg0 data directory.  Must be writable by HOST_UID.
+      - ${HOME:-.}/.hindsight-docker:/home/hindsight/.pg0
+
+      # ── Claude credentials (read-only, single-file mounts) ────────
+      # A whole-directory :ro mount of ~/.claude silently breaks the
+      # CLI, which writes session/project state at runtime — so we
+      # mount only the two credential files.
+      - ${HOME}/.claude/.credentials.json:/home/hindsight/.claude/.credentials.json:ro
+      - ${HOME}/.claude.json:/home/hindsight/.claude.json:ro
+
+      # ── Claude CLI install (read-only) ─────────────────────────────
+      - ${HOME}/.local/share/claude:/home/hindsight/.local/share/claude:ro
+
+      # ── SDK bundled-binary override ────────────────────────────────
+      # The claude-agent-sdk 0.5.x image bundles v2.1.121 which has a
+      # protocol incompatibility in containers.  Override it with the
+      # host's v2.1.128+ binary.  Drop this mount once claude-agent-sdk
+      # ships with v2.1.128+.
+      - ${HOME}/.local/share/claude/versions/${CLAUDE_CLI_VERSION:-2.1.128}:/app/api/.venv/lib/python3.11/site-packages/claude_agent_sdk/_bundled/claude:ro

docker/docker-compose/custom-models/Dockerfile

@@ -0,0 +1,34 @@
+# Example: custom Hindsight image with non-default local models baked in.
+#
+# Use this pattern in production when you run a non-default embedder or
+# reranker. Baking models into the image removes the runtime dependency on
+# HuggingFace and lets the container registry handle caching per node, so
+# you don't need a model-cache PVC.
+#
+# Built on top of the slim image so only the deps and models you actually
+# use end up in the final image.
+FROM ghcr.io/vectorize-io/hindsight:latest-slim
+
+# Install the local-ml deps required to load sentence-transformers /
+# cross-encoder models at runtime. Pinned ranges mirror hindsight-api-slim's
+# `local-ml` extra in hindsight-api-slim/pyproject.toml. Use `uv pip
+# install` against the image's venv explicitly: the slim image's venv was
+# created by `uv sync` and does not ship its own `pip`, so a bare
+# `pip install` would fall back to user site-packages and not be visible
+# to the runtime python.
+RUN uv pip install --python /app/api/.venv/bin/python --no-cache \
+    'sentence-transformers>=3.3.0' \
+    'transformers>=4.53.0' \
+    'torch>=2.6.0'
+
+# Pre-download the models you want to use. Replace these with your own.
+# The defaults bundled in the full image are BAAI/bge-small-en-v1.5 and
+# cross-encoder/ms-marco-MiniLM-L-6-v2; here we pick multilingual variants
+# as a concrete non-default example.
+ARG EMBEDDER=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
+ARG RERANKER=cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
+ENV HF_HUB_DOWNLOAD_TIMEOUT=600
+RUN python -c "\
+from sentence_transformers import SentenceTransformer, CrossEncoder; \
+SentenceTransformer('${EMBEDDER}'); \
+CrossEncoder('${RERANKER}')"

docker/docker-compose/custom-models/README.md

@@ -0,0 +1,81 @@
+# Hindsight with Custom Local Models
+
+Example Docker Compose setup that builds a Hindsight image with **non-default
+local embedder and reranker models baked in at build time**.
+
+This is the recommended pattern for production when you use a non-default
+local model: the container registry caches model layers per node, pod
+startup is deterministic, and you don't need a model-cache PVC (or any
+runtime dependency on HuggingFace).
+
+## When to use this
+
+- You override `HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL` or
+  `HINDSIGHT_API_RERANKER_LOCAL_MODEL` to a non-default model.
+- You want pod startup to be deterministic and offline-capable.
+- You'd otherwise reach for a Helm `modelCache` PVC just to avoid
+  re-downloading models.
+
+If you're using the **default** local models, the published full image
+(`ghcr.io/vectorize-io/hindsight:latest`) already bakes them in — you don't
+need this example.
+
+If you're using **external** providers (TEI, OpenAI, Cohere, ...) for
+embeddings and reranking, use the slim image directly — no models are
+needed in the image.
+
+## Quick start
+
+```bash
+export OPENAI_API_KEY=sk-xxx
+
+docker compose -f docker/docker-compose/custom-models/docker-compose.yaml up --build
+```
+
+- API: http://localhost:8888
+- Control Plane: http://localhost:9999
+
+## Using your own models
+
+Override the build args to bake different models:
+
+```bash
+docker compose -f docker/docker-compose/custom-models/docker-compose.yaml build \
+  --build-arg EMBEDDER=your-org/your-embedder \
+  --build-arg RERANKER=your-org/your-reranker
+```
+
+Then update the matching `HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL` and
+`HINDSIGHT_API_RERANKER_LOCAL_MODEL` values in `docker-compose.yaml` so the
+runtime points at the same model IDs.
+
+## Verifying the models are baked in
+
+`docker-compose.yaml` sets `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`
+so that any attempt to download a model at runtime fails loudly instead of
+silently re-downloading. If the container starts and serves recall queries
+with these set, the models are correctly baked in.
+
+You can also inspect the image directly:
+
+```bash
+docker run --rm --entrypoint sh hindsight-custom-models-hindsight \
+  -c 'ls ~/.cache/huggingface/hub/'
+```
+
+## Why not a model-cache PVC?
+
+The Helm chart exposes an optional `api.persistence.modelCache` PVC for
+caching downloaded models across pod restarts. Compared to baking models
+into the image:
+
+- A PVC adds storage cost — one PVC per worker replica with
+  `volumeClaimTemplates`.
+- `ReadWriteOnce` (the default) pins pods to a node.
+- The PVC needs lifecycle management on `helm uninstall` / `helm upgrade`
+  — without `helm.sh/resource-policy: keep` it is deleted on uninstall;
+  with it, storage keeps billing forever until manually cleaned up.
+- Pod startup still depends on HuggingFace being reachable on first run.
+
+Image layers, by contrast, are pulled once per node and cached for free by
+the container runtime, with no orphaned-storage cleanup story.

docker/docker-compose/custom-models/docker-compose.yaml

@@ -0,0 +1,44 @@
+name: hindsight-custom-models
+# Example: run a custom Hindsight image with non-default local models baked
+# in at build time, so pod startup does not depend on HuggingFace at runtime.
+#
+# Quick start:
+#   export OPENAI_API_KEY=sk-xxx
+#   docker compose -f docker/docker-compose/custom-models/docker-compose.yaml up --build
+#
+# Required environment variables:
+# - OPENAI_API_KEY (or configure another LLM provider via HINDSIGHT_API_LLM_*)
+
+services:
+  hindsight:
+    build:
+      context: .
+      dockerfile: Dockerfile
+      # Override at build time to bake different models:
+      #   docker compose build --build-arg EMBEDDER=your-org/your-embedder
+      args:
+        EMBEDDER: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
+        RERANKER: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
+    container_name: hindsight-custom-models
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Point Hindsight at the models baked into the image above.
+      HINDSIGHT_API_EMBEDDINGS_PROVIDER: local
+      HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
+      HINDSIGHT_API_RERANKER_PROVIDER: local
+      HINDSIGHT_API_RERANKER_LOCAL_MODEL: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
+
+      # Fail fast if a model is missing from the image instead of silently
+      # falling back to a HuggingFace download at runtime.
+      HF_HUB_OFFLINE: "1"
+      TRANSFORMERS_OFFLINE: "1"
+    volumes:
+      - pg_data:/home/hindsight/.pg0
+
+volumes:
+  pg_data:

docker/docker-compose/external-pg/docker-compose.yaml

@@ -0,0 +1,54 @@
+# Docker Compose file for Hindsight with PostgreSQL and pgvector
+#
+# Make sure to set the required environment variables before running:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - Configure LLM provider variables as needed (see below in the hindsight service)
+#
+# Usage:
+# docker compose up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
+
+services:
+  db:
+    # Use a PostgreSQL-Image with pgvector extension pre-installed
+    # see https://hub.docker.com/r/pgvector/pgvector 
+    image: pgvector/pgvector:pg${HINDSIGHT_DB_VERSION:-18}
+    container_name: hindsight-db
+    restart: always
+    # Expose PostgreSQL port
+    # ports:
+      # - "5432:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      - HINDSIGHT_API_LLM_API_KEY=${OPENAI_API_KEY?Please set the OPENAI_API_KEY env variable}
+      - HINDSIGHT_API_DATABASE_URL=postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/docker-compose/local-llm/README.md

@@ -0,0 +1,103 @@
+# Hindsight with a local llama.cpp server sidecar
+
+Example Docker Compose setup that runs Hindsight against a **local
+llama.cpp server**, fully offline, with no external API key required.
+
+## Architecture
+
+```
+┌────────────┐  HTTP /v1/chat/completions  ┌──────────────────────────────┐
+│ hindsight  │ ──────────────────────────▶ │ llama.cpp server (sidecar)   │
+│ (API + CP) │                             │ ghcr.io/ggml-org/llama.cpp   │
+└────────────┘                             └──────────────────────────────┘
+```
+
+`llama.cpp` runs as its own container and exposes an OpenAI-compatible
+HTTP API. Hindsight talks to it via the standard `openai` LLM provider
+with `HINDSIGHT_API_LLM_BASE_URL` pointed at the sidecar.
+
+This pattern follows
+[*Hosting llama-server with Docker* (ServiceStack)](https://servicestack.net/posts/hosting-llama-server).
+
+### Why a sidecar and not the in-process `llamacpp` provider?
+
+Hindsight does ship an in-process `llamacpp` provider that spawns
+`llama-cpp-python`, but the **published `ghcr.io/vectorize-io/hindsight`
+image deliberately omits `llama-cpp-python`** to keep the image small and
+avoid bundling native inference libraries that most users don't need.
+Trying to set `HINDSIGHT_API_LLM_PROVIDER=llamacpp` against the published
+image fails with `ModuleNotFoundError: No module named 'llama_cpp'`.
+
+The sidecar approach side-steps that entirely: the official llama.cpp
+image is used as-is for inference, Hindsight is used as-is for memory.
+Clean separation, no derived images.
+
+## Quick start
+
+```bash
+docker compose -f docker/docker-compose/local-llm/docker-compose.yaml up
+```
+
+- API: http://localhost:8888
+- Control Plane: http://localhost:9999
+
+**First boot downloads ~3.5 GB** (Gemma 4 E2B Q4_K_M GGUF) into the
+`llama_models` named volume. Subsequent boots reuse it.
+
+Hindsight only starts after llama.cpp's `/health` endpoint reports
+healthy, so the API will appear "stuck" for a few minutes on the first
+run while the model downloads.
+
+## Using a different model
+
+Override the HuggingFace repo / file in `docker-compose.yaml`:
+
+```yaml
+environment:
+  LLAMA_ARG_HF_REPO: bartowski/Qwen2.5-7B-Instruct-GGUF
+  LLAMA_ARG_HF_FILE: Qwen2.5-7B-Instruct-Q4_K_M.gguf
+```
+
+Also update `HINDSIGHT_API_LLM_MODEL` on the `hindsight` service to a
+matching alias (the value is sent to llama-server as the OpenAI `model`
+field — llama-server is lenient about this but it shows up in logs).
+
+## GPU acceleration
+
+The default compose file targets CPU because not everyone has a GPU. On
+CPU, Gemma 4 E2B runs at ~2-3 tokens/sec — fine for a smoke test, but the
+retain pipeline (which makes several multi-hundred-token LLM calls per
+memory) will time out against Hindsight's default LLM timeout. **For any
+real use, run on a GPU.**
+
+### NVIDIA
+
+1. Switch the `llama` service image from `:server` to `:server-cuda`.
+2. Uncomment the `LLAMA_ARG_N_GPU_LAYERS: "999"` env var (offload all
+   layers to GPU).
+3. Uncomment the `deploy.resources.reservations.devices` block.
+4. Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
+   on the host.
+
+The compose file has all four spots marked with inline comments.
+
+### Apple Silicon / ROCm / Vulkan
+
+The official `ghcr.io/ggml-org/llama.cpp` image only ships CPU and CUDA
+variants. For Metal (Apple Silicon), ROCm (AMD), or Vulkan backends,
+build llama.cpp yourself with the appropriate flags and reference the
+image you build instead. Docker Desktop on macOS cannot pass through the
+host GPU to a Linux container in any case — for Apple Silicon, run
+llama-server directly on the host and only put Hindsight in Docker.
+
+## Caveats
+
+- llama.cpp's HTTP API is OpenAI-compatible but not 100% feature-parity.
+  Function/tool calling support depends on the chat template baked into
+  the GGUF; some retain/reflect flows may behave differently than against
+  a hosted OpenAI model.
+- Small GGUFs (~3 B params) are useful for smoke testing but will
+  underperform a hosted frontier model on retain quality. Use a larger
+  GGUF (7-13 B params) for production-quality memory.
+- The `llama_models` named volume persists the GGUF across `docker
+  compose down`/`up` so the model is downloaded once, not every restart.

docker/docker-compose/local-llm/docker-compose.yaml

@@ -0,0 +1,74 @@
+name: hindsight-local-llm
+# Example: run Hindsight against a local llama.cpp server sidecar — fully
+# offline, no external API key needed.
+#
+# Pattern follows https://servicestack.net/posts/hosting-llama-server :
+# llama.cpp runs as its own container exposing an OpenAI-compatible HTTP
+# API, and Hindsight talks to it via the `openai` LLM provider with a
+# custom `base_url`. This means we can use the published Hindsight image
+# unchanged — no derived Dockerfile, no `llama-cpp-python` install on top.
+#
+# Quick start:
+#   docker compose -f docker/docker-compose/local-llm/docker-compose.yaml up
+#
+# First boot downloads the default Gemma 4 E2B GGUF (~3.5 GB) into the
+# `llama_models` volume; subsequent boots reuse it.
+
+services:
+  llama:
+    image: ghcr.io/ggml-org/llama.cpp:server
+    container_name: hindsight-local-llm-llama
+    environment:
+      LLAMA_ARG_HOST: 0.0.0.0
+      LLAMA_ARG_PORT: "8080"
+      # Auto-download a small GGUF from HuggingFace on first start.
+      # Override these to use a different model.
+      LLAMA_ARG_HF_REPO: bartowski/google_gemma-4-E2B-it-GGUF
+      LLAMA_ARG_HF_FILE: google_gemma-4-E2B-it-Q4_K_M.gguf
+      LLAMA_ARG_CTX_SIZE: "8192"
+      # Uncomment for NVIDIA GPU (and switch image to :server-cuda):
+      # LLAMA_ARG_N_GPU_LAYERS: "999"
+    volumes:
+      # llama-server stores HuggingFace downloads under ~/.cache/huggingface
+      # (not ~/.cache/llama.cpp), so mount the named volume there to avoid
+      # re-downloading the GGUF on every recreate.
+      - llama_models:/root/.cache/huggingface
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 60
+      start_period: 30s
+    # For NVIDIA GPU acceleration, swap the image above to
+    # `ghcr.io/ggml-org/llama.cpp:server-cuda` and uncomment:
+    # deploy:
+    #   resources:
+    #     reservations:
+    #       devices:
+    #         - driver: nvidia
+    #           count: all
+    #           capabilities: [gpu]
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:latest
+    container_name: hindsight-local-llm
+    depends_on:
+      llama:
+        condition: service_healthy
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # llama-server is OpenAI-compatible, so use the `openai` provider and
+      # point base_url at the sidecar. The API key is unused by llama-server
+      # but Hindsight requires the env var to be set.
+      HINDSIGHT_API_LLM_PROVIDER: openai
+      HINDSIGHT_API_LLM_BASE_URL: http://llama:8080/v1
+      HINDSIGHT_API_LLM_API_KEY: not-needed
+      HINDSIGHT_API_LLM_MODEL: gemma-4-e2b-it
+    volumes:
+      - pg_data:/home/hindsight/.pg0
+
+volumes:
+  pg_data:
+  llama_models:

docker/docker-compose/nginx/README.md

@@ -0,0 +1,96 @@
+# Nginx Reverse Proxy with Custom Base Path
+
+Deploy Hindsight API under `/hindsight` (or any custom path) using Nginx reverse proxy.
+
+## Quick Start (Published Image - API Only)
+
+```bash
+docker-compose up
+```
+
+- **API:** http://localhost:8080/hindsight/docs
+- **Control Plane:** http://localhost:9999 (direct access, not proxied)
+
+## Full Stack with Custom Base Path (Requires Build)
+
+**Important:** You cannot rebuild from the published image with build args. You must build from source.
+
+### Build from Source with Custom Base Path
+
+1. **Clone the repository** (if you haven't):
+```bash
+git clone https://github.com/vectorize-io/hindsight.git
+cd hindsight
+```
+
+2. **Build with base path**:
+```bash
+docker build \
+  --build-arg NEXT_PUBLIC_BASE_PATH=/hindsight \
+  -f docker/standalone/Dockerfile \
+  -t hindsight:custom \
+  .
+```
+
+3. **Update docker-compose.yml** to use your built image:
+```yaml
+services:
+  hindsight:
+    image: hindsight:custom  # ← Change this
+    environment:
+      HINDSIGHT_API_BASE_PATH: /hindsight
+      NEXT_PUBLIC_BASE_PATH: /hindsight
+```
+
+4. **Update nginx.conf** to handle Control Plane routes (see below)
+
+5. **Run**:
+```bash
+docker-compose up
+```
+
+### Required nginx.conf for Full Stack
+
+Replace the current `nginx.conf` with this to proxy both API and Control Plane:
+
+```nginx
+events { worker_connections 1024; }
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    upstream hindsight_api { server hindsight:8888; }
+    upstream hindsight_cp { server hindsight:9999; }
+
+    server {
+        listen 80;
+
+        # API
+        location ~ ^/hindsight/(docs|openapi\.json|health|metrics|v1|mcp) {
+            proxy_pass http://hindsight_api;
+            proxy_set_header Host $http_host;
+        }
+
+        # Control Plane static files
+        location ~ ^/hindsight/_next/ {
+            proxy_pass http://hindsight_cp;
+            proxy_set_header Host $http_host;
+        }
+
+        # Control Plane UI
+        location /hindsight {
+            proxy_pass http://hindsight_cp;
+            proxy_set_header Host $http_host;
+        }
+
+        location = / { return 301 /hindsight; }
+    }
+}
+```
+
+### Why Build is Required
+
+Next.js requires `basePath` at **build time**. The published image was built without a custom base path, so you must rebuild from source with the `NEXT_PUBLIC_BASE_PATH` build arg to deploy the Control Plane under a subpath.
+
+The API works without rebuild because `HINDSIGHT_API_BASE_PATH` is a runtime environment variable.

docker/docker-compose/nginx/docker-compose.yml

@@ -0,0 +1,90 @@
+# Hindsight API deployment with Nginx reverse proxy (API-only)
+#
+# This example deploys Hindsight API under the path /hindsight with:
+# - Hindsight standalone image (API + Control Plane + embedded pg0)
+# - Nginx reverse proxy (API only)
+#
+# Quick Start:
+#   docker-compose -f docker/docker-compose/nginx/docker-compose.yml up
+#
+# Access:
+#   API (via nginx):         http://localhost:8080/hindsight/docs
+#   Control Plane (direct):  http://localhost:9999
+#
+# For full stack deployment (API + Control Plane both under /hindsight):
+#   See README.md in this directory for instructions on building with basePath.
+#
+# Note: This configuration uses the published image (no build required).
+#       Control Plane is served directly because Next.js basePath requires
+#       build-time configuration. See README.md for the full stack option.
+
+services:
+  # Hindsight (API + Control Plane + embedded pg0)
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:latest
+    ports:
+      - "9999:9999"  # Control Plane (direct access, not proxied)
+    environment:
+      # API base path for reverse proxy
+      HINDSIGHT_API_BASE_PATH: /hindsight
+
+      # LLM configuration
+      # Using mock provider for testing (no API key needed)
+      # For production, set OPENAI_API_KEY or ANTHROPIC_API_KEY and use a real provider
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-mock}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-not-needed-for-mock}
+      HINDSIGHT_API_LLM_MODEL: ${HINDSIGHT_API_LLM_MODEL:-mock-model}
+
+      # Production examples (uncomment and set appropriate API key):
+      # HINDSIGHT_API_LLM_PROVIDER: openai
+      # HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY}
+      # HINDSIGHT_API_LLM_MODEL: gpt-4o-mini
+
+      # HINDSIGHT_API_LLM_PROVIDER: anthropic
+      # HINDSIGHT_API_LLM_API_KEY: ${ANTHROPIC_API_KEY}
+      # HINDSIGHT_API_LLM_MODEL: claude-sonnet-4-20250514
+
+      # Server config
+      HINDSIGHT_API_HOST: 0.0.0.0
+      HINDSIGHT_API_PORT: 8888
+      HINDSIGHT_API_LOG_LEVEL: info
+
+      # Control Plane config
+      HINDSIGHT_CP_DATAPLANE_API_URL: http://localhost:8888
+      # Optional: Require a shared access key for Control Plane UI access
+      # HINDSIGHT_CP_ACCESS_KEY: your-secret-key
+    volumes:
+      # Persist embedded pg0 database
+      - hindsight_data:/app/data
+    # Note: Ports not exposed - access via Nginx at localhost:8080/hindsight/
+    # To debug directly, uncomment these ports:
+    # ports:
+    #   - "8888:8888"  # API
+    #   - "9999:9999"  # Control Plane
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8888/hindsight/health"]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+    networks:
+      - hindsight
+
+  # Nginx reverse proxy
+  nginx:
+    image: nginx:alpine
+    ports:
+      - "8080:80"
+    volumes:
+      - ./nginx.conf:/etc/nginx/nginx.conf:ro
+    depends_on:
+      hindsight:
+        condition: service_healthy
+    networks:
+      - hindsight
+
+volumes:
+  hindsight_data:
+
+networks:
+  hindsight:

docker/docker-compose/nginx/nginx.conf

@@ -0,0 +1,40 @@
+# Nginx configuration for API-only reverse proxy
+# Control Plane accessed directly (not through nginx)
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+
+    # Logging
+    access_log /var/log/nginx/access.log;
+    error_log /var/log/nginx/error.log;
+
+    # Upstream - Hindsight API
+    upstream hindsight_api {
+        server hindsight:8888;
+    }
+
+    server {
+        listen 80;
+        server_name _;
+
+        # API endpoints - forward with /hindsight prefix
+        location /hindsight/ {
+            proxy_pass http://hindsight_api;
+
+            proxy_set_header Host $http_host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+        }
+
+        # Redirect root to API docs
+        location = / {
+            return 301 /hindsight/docs;
+        }
+    }
+}

docker/docker-compose/pg_search/Dockerfile

@@ -0,0 +1,7 @@
+# PostgreSQL with pgvector and ParadeDB pg_search extensions.
+#
+# The official ParadeDB image ships PostgreSQL with pg_search and pgvector
+# already installed, so no build steps are required. We pin to the PG17
+# variant for parity with the other Hindsight docker-compose examples
+# (vchord, pg_textsearch).
+FROM paradedb/paradedb:latest-pg17

docker/docker-compose/pg_search/docker-compose.yaml

@@ -0,0 +1,96 @@
+name: hindsight
+# Docker Compose file for Hindsight with PostgreSQL and ParadeDB pg_search.
+#
+# pg_search is the only BM25 backend supported by Hindsight that works with
+# Citus, so this is the recommended setup for horizontally scaled deployments.
+#
+# Usage:
+#   docker compose -f docker/docker-compose/pg_search/docker-compose.yaml up -d
+#
+# Required environment variables:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - Configure LLM provider variables as needed (see the hindsight service)
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER: ParadeDB pg_search
+#   tokenizer for new BM25 indexes (default: empty, uses ParadeDB default)
+
+services:
+  db:
+    # Use ParadeDB image which bundles pgvector + pg_search
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: hindsight-db
+    restart: always
+    ports:
+      - "5437:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+
+  pg-search-init:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'Waiting for PostgreSQL to be ready...';
+        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
+          echo 'PostgreSQL is unavailable - sleeping';
+          sleep 2;
+        done;
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Creating extensions in hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_search CASCADE;';
+        echo 'Database and extensions created successfully';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: pgvector
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pg_search
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER: ${HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER:-}
+
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/docker-compose/pg_textsearch/Dockerfile

@@ -0,0 +1,32 @@
+# PostgreSQL with pgvector and pg_textsearch extensions
+# Note: pg_textsearch requires PostgreSQL 17+
+FROM postgres:17
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    git \
+    postgresql-server-dev-17 \
+    libpq-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install pgvector
+RUN cd /tmp && \
+    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
+    cd pgvector && \
+    make && \
+    make install
+
+# Install pg_textsearch
+RUN cd /tmp && \
+    git clone https://github.com/timescale/pg_textsearch.git && \
+    cd pg_textsearch && \
+    make && \
+    make install
+
+# Clean up source files and build dependencies
+RUN rm -rf /tmp/pgvector /tmp/pg_textsearch && \
+    apt-get purge -y --auto-remove build-essential git postgresql-server-dev-17
+
+# Ensure extensions are preloaded
+RUN echo "shared_preload_libraries = 'pg_textsearch'" >> /usr/share/postgresql/postgresql.conf.sample

docker/docker-compose/pg_textsearch/docker-compose.yaml

@@ -0,0 +1,91 @@
+name: hindsight
+# Docker Compose file for Hindsight with PostgreSQL and Timescale pg_textsearch
+# docker compose -f docker/docker-compose/pg_textsearch/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/pg_textsearch/docker-compose.yaml up -d
+# Make sure to set the required environment variables before running:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - Configure LLM provider variables as needed (see below in the hindsight service)
+#
+# Usage:
+# docker compose up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+
+services:
+  db:
+    # Use custom PostgreSQL image with pgvector and pg_textsearch extensions
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: hindsight-db
+    restart: always
+    # Expose PostgreSQL port
+    ports:
+      - "5437:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+
+  pg-textsearch-init:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'Waiting for PostgreSQL to be ready...';
+        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
+          echo 'PostgreSQL is unavailable - sleeping';
+          sleep 2;
+        done;
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Creating extensions in hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_textsearch CASCADE;';
+        echo 'Database and extensions created successfully';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: pgvector
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pg_textsearch
+
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/docker-compose/pgroonga/Dockerfile

@@ -0,0 +1,23 @@
+# PostgreSQL with pgvector and pgroonga extensions.
+#
+# pgroonga is a multilingual full-text search extension built on Groonga.
+# It works out of the box for CJK (Chinese, Japanese, Korean) and other
+# non-whitespace-segmented languages via the TokenBigram tokenizer.
+FROM groonga/pgroonga:latest-debian-pg17
+
+# Install pgvector on top of the pgroonga base image (which already provides
+# pgroonga and the Groonga library).
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    git \
+    postgresql-server-dev-17 \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN cd /tmp && \
+    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
+    cd pgvector && \
+    make && \
+    make install
+
+RUN rm -rf /tmp/pgvector && \
+    apt-get purge -y --auto-remove build-essential git postgresql-server-dev-17

docker/docker-compose/pgroonga/docker-compose.yaml

@@ -0,0 +1,91 @@
+name: hindsight
+# Docker Compose file for Hindsight with PostgreSQL and pgroonga
+#
+# pgroonga provides multilingual BM25 indexing that works out of the box for
+# CJK (Chinese, Japanese, Korean) and other non-whitespace-segmented languages.
+# Use this recipe if your bank content is not English/European.
+#
+# docker compose -f docker/docker-compose/pgroonga/docker-compose.yaml down && \
+#   sleep 2 && \
+#   docker compose -f docker/docker-compose/pgroonga/docker-compose.yaml up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_DB_PASSWORD: PostgreSQL password (default: hindsight_password)
+
+services:
+  db:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: hindsight-db
+    restart: always
+    ports:
+      - "5439:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+
+  pgroonga-init:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'Waiting for PostgreSQL to be ready...';
+        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
+          echo 'PostgreSQL is unavailable - sleeping';
+          sleep 2;
+        done;
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Creating extensions in hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pgroonga CASCADE;';
+        echo 'Database and extensions created successfully';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: pgvector
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pgroonga
+
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/docker-compose/s3-file-storage/docker-compose.yaml

@@ -0,0 +1,83 @@
+# Docker Compose file for Hindsight with S3 file storage (SeaweedFS)
+#
+# SeaweedFS (Apache 2.0) provides an S3-compatible object storage backend
+# for storing uploaded files instead of PostgreSQL BYTEA storage.
+#
+# Make sure to set the required environment variables before running:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - Configure LLM provider variables as needed (see below in the hindsight service)
+#
+# Usage:
+# docker compose up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
+# - SEAWEEDFS_S3_ACCESS_KEY: S3 access key (default: hindsight_s3_key)
+# - SEAWEEDFS_S3_SECRET_KEY: S3 secret key (default: hindsight_s3_secret)
+
+services:
+  db:
+    image: pgvector/pgvector:pg${HINDSIGHT_DB_VERSION:-18}
+    container_name: hindsight-db
+    restart: always
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
+    networks:
+      - hindsight-net
+
+  seaweedfs:
+    image: chrislusf/seaweedfs:latest
+    container_name: hindsight-seaweedfs
+    restart: always
+    # Single-node mode: master + volume + filer + S3 gateway all in one process
+    command: >
+      server
+      -s3
+      -s3.port=8333
+      -s3.config=/etc/seaweedfs/s3.json
+      -ip.bind=0.0.0.0
+    volumes:
+      - seaweedfs_data:/data
+      - ./s3.json:/etc/seaweedfs/s3.json:ro
+    # Expose S3 API port (uncomment to access from host)
+    # ports:
+    #   - "8333:8333"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      - HINDSIGHT_API_LLM_API_KEY=${OPENAI_API_KEY?Please set the OPENAI_API_KEY env variable}
+      - HINDSIGHT_API_DATABASE_URL=postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+      # S3 file storage configuration (SeaweedFS)
+      - HINDSIGHT_API_FILE_STORAGE_TYPE=s3
+      - HINDSIGHT_API_FILE_STORAGE_S3_BUCKET=hindsight
+      - HINDSIGHT_API_FILE_STORAGE_S3_ENDPOINT=http://seaweedfs:8333
+      - HINDSIGHT_API_FILE_STORAGE_S3_REGION=us-east-1
+      - HINDSIGHT_API_FILE_STORAGE_S3_ACCESS_KEY_ID=${SEAWEEDFS_S3_ACCESS_KEY:-hindsight_s3_key}
+      - HINDSIGHT_API_FILE_STORAGE_S3_SECRET_ACCESS_KEY=${SEAWEEDFS_S3_SECRET_KEY:-hindsight_s3_secret}
+    depends_on:
+      - db
+      - seaweedfs
+    networks:
+      - hindsight-net
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:
+  seaweedfs_data:

docker/docker-compose/s3-file-storage/s3.json

@@ -0,0 +1,19 @@
+{
+  "identities": [
+    {
+      "name": "hindsight",
+      "credentials": [
+        {
+          "accessKey": "hindsight_s3_key",
+          "secretKey": "hindsight_s3_secret"
+        }
+      ],
+      "actions": [
+        "Admin",
+        "Read",
+        "Write",
+        "List"
+      ]
+    }
+  ]
+}

docker/docker-compose/timescale/.dockerignore

@@ -0,0 +1,16 @@
+# Git
+.git
+.gitignore
+.gitattributes
+
+# Docker
+docker-compose.yaml
+.dockerignore
+
+# Documentation
+README.md
+*.md
+
+# Environment
+.env
+.env.example

docker/docker-compose/timescale/.env.example

@@ -0,0 +1,25 @@
+# PostgreSQL Configuration
+HINDSIGHT_DB_USER=hindsight_user
+HINDSIGHT_DB_PASSWORD=change-me-to-secure-password
+HINDSIGHT_DB_NAME=hindsight_db
+
+# Hindsight Version
+HINDSIGHT_VERSION=latest
+
+# LLM Configuration
+HINDSIGHT_API_LLM_PROVIDER=openai
+OPENAI_API_KEY=your-openai-api-key-here
+
+# Alternative LLM providers (uncomment and configure as needed):
+# HINDSIGHT_API_LLM_PROVIDER=anthropic
+# ANTHROPIC_API_KEY=your-anthropic-api-key
+
+# HINDSIGHT_API_LLM_PROVIDER=gemini
+# GEMINI_API_KEY=your-gemini-api-key
+
+# HINDSIGHT_API_LLM_PROVIDER=groq
+# GROQ_API_KEY=your-groq-api-key
+
+# Vector and Text Search (already configured in docker-compose.yaml)
+# HINDSIGHT_API_VECTOR_EXTENSION=pgvectorscale
+# HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pg_textsearch

docker/docker-compose/timescale/Dockerfile

@@ -0,0 +1,55 @@
+# PostgreSQL with pgvector, pgvectorscale, and pg_textsearch extensions
+# All three extensions from Timescale/pgvector for high-performance vector and text search
+# Note: Requires PostgreSQL 16+
+FROM postgres:17
+
+# Install build dependencies and Rust toolchain
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    git \
+    postgresql-server-dev-17 \
+    libpq-dev \
+    cmake \
+    curl \
+    pkg-config \
+    libssl-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Rust toolchain (required for pgvectorscale)
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+ENV PATH="/root/.cargo/bin:${PATH}"
+
+# Install pgvector (required by pgvectorscale)
+RUN cd /tmp && \
+    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
+    cd pgvector && \
+    make && \
+    make install && \
+    rm -rf /tmp/pgvector
+
+# Install cargo-pgrx (PostgreSQL extension framework for Rust)
+RUN cargo install cargo-pgrx --version 0.12.5 --locked && \
+    cargo pgrx init --pg17 /usr/bin/pg_config
+
+# Install pgvectorscale (DiskANN index support)
+RUN cd /tmp && \
+    git clone --branch 0.5.1 https://github.com/timescale/pgvectorscale.git && \
+    cd pgvectorscale/pgvectorscale && \
+    cargo pgrx install --release && \
+    rm -rf /tmp/pgvectorscale
+
+# Install pg_textsearch (BM25 text search)
+RUN cd /tmp && \
+    git clone https://github.com/timescale/pg_textsearch.git && \
+    cd pg_textsearch && \
+    make && \
+    make install && \
+    rm -rf /tmp/pg_textsearch
+
+# Clean up build dependencies (keep runtime dependencies)
+RUN apt-get purge -y --auto-remove git cmake curl && \
+    rm -rf /root/.cargo/registry /root/.cargo/git
+
+# Ensure extensions are preloaded (pg_textsearch requires preloading)
+RUN echo "shared_preload_libraries = 'pg_textsearch'" >> /usr/share/postgresql/postgresql.conf.sample
+

docker/docker-compose/timescale/README.md

@@ -0,0 +1,101 @@
+# Hindsight with Timescale Extensions
+
+This Docker Compose setup provides a complete Hindsight deployment with **Timescale extensions**:
+- **pgvectorscale** - DiskANN algorithm for disk-based scalable vector search  
+- **pg_textsearch** - High-performance BM25 text search
+
+Both extensions are from [Timescale](https://github.com/timescale) and provide production-grade performance.
+
+## Prerequisites
+
+- Docker and Docker Compose installed
+- OpenAI API key (or another LLM provider)
+
+## Quick Start
+
+```bash
+# Set environment variables
+export HINDSIGHT_DB_PASSWORD="your-secure-password"
+export OPENAI_API_KEY="your-openai-api-key"
+
+# Build and start
+docker compose -f docker/docker-compose/timescale/docker-compose.yaml up -d --build
+
+# Check logs
+
+docker compose -f docker/docker-compose/timescale/docker-compose.yaml logs -f
+```
+
+**Access:**
+- API: http://localhost:8888
+- Control Plane: http://localhost:9999
+
+## Stop and Clean Up
+
+```bash
+# Stop services
+docker compose -f docker/docker-compose/timescale/docker-compose.yaml down
+
+# Remove volumes (deletes all data)
+docker compose -f docker/docker-compose/timescale/docker-compose.yaml down -v
+```
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `HINDSIGHT_DB_PASSWORD` | PostgreSQL password | `hindsight_password` |
+| `HINDSIGHT_DB_USER` | PostgreSQL username | `hindsight_user` |
+| `HINDSIGHT_DB_NAME` | Database name | `hindsight_db` |
+| `HINDSIGHT_VERSION` | Hindsight Docker image version | `latest` |
+| `OPENAI_API_KEY` | OpenAI API key | (required) |
+| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider | `openai` |
+
+### Why Timescale Extensions?
+
+**pgvectorscale (DiskANN):**
+- 28x lower p95 latency vs dedicated vector databases
+- 16x higher query throughput at 99% recall
+- 60-75% cost reduction (disk is cheaper than RAM)
+- Best for large datasets (10M+ vectors)
+
+**pg_textsearch (BM25):**
+- High-performance keyword retrieval
+- Native BM25 ranking algorithm
+- Optimized for full-text search
+
+## Troubleshooting
+
+### Extensions not installed
+
+Check if extensions are available:
+
+```bash
+docker exec -it hindsight-db-timescale psql -U hindsight_user -d hindsight_db -c "\dx"
+```
+
+You should see:
+- `vector` (pgvector)
+- `vectorscale` (pgvectorscale/DiskANN)
+- `pg_textsearch` (BM25 search)
+
+### Build fails
+
+If the Docker build fails during pgvectorscale compilation:
+
+1. Ensure you have sufficient memory (recommended: 4GB+)
+2. Check Docker build logs for Rust compilation errors
+3. Try building with more resources: `docker compose build --no-cache --memory 4g`
+
+### Port conflicts
+
+If port 5438 is already in use, modify the `ports` section in docker-compose.yaml.
+
+## Learn More
+
+- [pgvectorscale GitHub](https://github.com/timescale/pgvectorscale)
+- [pg_textsearch GitHub](https://github.com/timescale/pg_textsearch)
+- [HNSW vs DiskANN](https://www.tigerdata.com/learn/hnsw-vs-diskann)
+- [Hindsight Documentation](https://hindsight.dev)

docker/docker-compose/timescale/docker-compose.yaml

@@ -0,0 +1,108 @@
+name: hindsight
+# Docker Compose file for Hindsight with Timescale extensions
+# - pgvectorscale: DiskANN vector search (disk-based, scalable)
+# - pg_textsearch: BM25 text search (high-performance keyword retrieval)
+#
+# Quick start:
+# docker compose -f docker/docker-compose/timescale/docker-compose.yaml up -d --build
+#
+# Required environment variables:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - OPENAI_API_KEY (or configure another LLM provider)
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+
+services:
+  db:
+    # Custom PostgreSQL image with Timescale extensions (pgvectorscale + pg_textsearch)
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: hindsight-db-timescale
+    restart: always
+    # Expose PostgreSQL port (using 5438 to avoid conflicts with other setups)
+    ports:
+      - "5438:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+    # Health check to ensure database is ready
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U hindsight_user"]
+      interval: 5s
+      timeout: 5s
+      retries: 5
+
+  timescale-init:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      db:
+        condition: service_healthy
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Installing Timescale extensions...';
+        echo '1/3: Installing pgvector (required by pgvectorscale)...';
+        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
+        echo '2/3: Installing pgvectorscale (DiskANN vector search)...';
+        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE;';
+        echo '3/3: Installing pg_textsearch (BM25 text search)...';
+        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_textsearch CASCADE;';
+        echo '';
+        echo '✅ Timescale extensions installed successfully';
+        echo '';
+        echo 'Installed extensions:';
+        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c \"\\dx\" | grep -E '(vector|vectorscale|pg_textsearch)';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app-timescale
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Timescale Extensions
+      # pgvectorscale: DiskANN algorithm for disk-based scalable vector search
+      HINDSIGHT_API_VECTOR_EXTENSION: pgvectorscale
+      # pg_textsearch: High-performance BM25 text search
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pg_textsearch
+
+    depends_on:
+      db:
+        condition: service_healthy
+      timescale-init:
+        condition: service_completed_successfully
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/docker-compose/vchord/docker-compose.yaml

@@ -0,0 +1,93 @@
+name: hindsight
+# Docker Compose file for Hindsight with PostgreSQL and vectorchord
+# docker compose -f docker/docker-compose/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/docker-compose.yaml up -d
+# Make sure to set the required environment variables before running:
+# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
+# - Configure LLM provider variables as needed (see below in the hindsight service)
+#
+# Usage:
+# docker compose up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
+
+services:
+  db:
+    # Use a PostgreSQL-Image with vectorchord extension pre-installed
+    image: tensorchord/vchord-suite:pg${HINDSIGHT_DB_VERSION:-18-latest}
+    container_name: hindsight-db
+    restart: always
+    # Expose PostgreSQL port
+    ports:
+      - "5436:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
+    networks:
+      - hindsight-net
+
+  vectorchord-init:
+    image: tensorchord/vchord-suite:pg18-latest
+    #container_name: vectorchord-init
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'Waiting for PostgreSQL to be ready...';
+        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
+          echo 'PostgreSQL is unavailable - sleeping';
+          sleep 2;
+        done;
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Creating extensions in hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;';
+        echo 'Creating llmlingua2 tokenizer';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c \"SELECT create_tokenizer('llmlingua2', \\$\\$ model = \\\"llmlingua2\\\" \\$\\$);\" 2>/dev/null || echo 'Tokenizer already exists or creation skipped';
+        echo 'Database and extensions created successfully';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration (uses OpenAI for testing vchord)
+      # LLM configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: vchord
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: vchord
+
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

docker/standalone/Dockerfile

@@ -8,6 +8,7 @@
 #                                     Set to false when using external providers (TEI, OpenAI, Cohere)
 #   PRELOAD_ML_MODELS=true/false   - Pre-download ML models during build (default: true)
 #                                     Only effective when INCLUDE_LOCAL_MODELS=true
+#                                     NOTE: tiktoken encodings are ALWAYS preloaded (required for air-gapped deployments)
 #
 # Examples:
 #   docker build -t hindsight .                                          # Both (standalone)
@@ -41,25 +42,24 @@ RUN apt-get update && apt-get install -y \
     && pip install --no-cache-dir uv
 
 # Copy dependency files and README (required by pyproject.toml)
-COPY hindsight-api/pyproject.toml ./api/
-COPY hindsight-api/README.md ./api/
+COPY hindsight-api-slim/pyproject.toml ./api/
+COPY hindsight-api-slim/README.md ./api/
 
 WORKDIR /app/api
 
-# Remove local ML model dependencies if INCLUDE_LOCAL_MODELS=false
-# This creates a smaller image when using external providers (TEI, OpenAI, Cohere)
-RUN if [ "$INCLUDE_LOCAL_MODELS" != "true" ]; then \
-        echo "Removing local-models dependencies (sentence-transformers, torch, transformers)..." && \
-        sed -i '/"sentence-transformers/d' pyproject.toml && \
-        sed -i '/"transformers/d' pyproject.toml && \
-        sed -i '/"torch/d' pyproject.toml; \
+# Sync dependencies using appropriate extras based on INCLUDE_LOCAL_MODELS
+# local-ml: torch, sentence-transformers, transformers, einops, flashrank, mlx (optional)
+# embedded-db: pg0-embedded (always included for embedded PostgreSQL support)
+# ONNX Runtime embeddings are intentionally not bundled into the official
+# standalone image; install the local-onnx extra in custom images when needed.
+RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
+        uv sync --extra local-ml --extra embedded-db; \
+    else \
+        uv sync --extra embedded-db; \
     fi
 
-# Sync dependencies (will create lock file if needed)
-RUN uv sync
-
 # Copy source code (alembic migrations are inside hindsight_api/)
-COPY hindsight-api/hindsight_api ./hindsight_api
+COPY hindsight-api-slim/hindsight_api ./hindsight_api
 
 # Install the local package (uv sync only installed dependencies, not the package itself)
 RUN uv pip install -e .
@@ -111,6 +111,10 @@ RUN rm -f package-lock.json && sed -i '/"@vectorize-io\/hindsight-client":/d' pa
 # Copy built SDK directly into node_modules (more reliable than npm link in Docker)
 COPY --from=sdk-builder /app/hindsight-clients/typescript ./node_modules/@vectorize-io/hindsight-client
 
+# Accept base path as build argument for reverse proxy deployments
+# Usage: docker build --build-arg NEXT_PUBLIC_BASE_PATH=/hindsight ...
+ARG NEXT_PUBLIC_BASE_PATH=""
+
 # Build Control Plane - run next build first, then custom standalone copy
 # (The build:standalone script expects a specific path structure that differs in Docker)
 RUN npm exec -- next build
@@ -165,20 +169,68 @@ RUN chown -R hindsight:hindsight /app
 
 USER hindsight
 
+# Create pg0 data directory as hindsight user so that Docker seeds new named
+# volumes with correct ownership (UID 1000) on first use, avoiding the
+# "Permission denied" error when mounting a fresh root-owned volume.
+RUN mkdir -p /home/hindsight/.pg0
+
+# Make /home/hindsight traversable when running with --user UID:GID overrides
+# (default 0700 blocks traversal by non-owner UIDs needed for bind-mount ownership matching)
+RUN chmod 755 /home/hindsight
+
 ENV PATH="/app/api/.venv/bin:${PATH}"
 
+# Pre-download tiktoken encoding (ALWAYS - required for token counting even in air-gapped envs)
+# Tiktoken is a core runtime dependency, not an optional ML model
+RUN MAX_RETRIES=3; \
+    RETRY_DELAY=5; \
+    for i in $(seq 1 $MAX_RETRIES); do \
+      echo "Attempt $i/$MAX_RETRIES: Downloading tiktoken encoding..."; \
+      /app/api/.venv/bin/python -c "\
+import tiktoken; \
+print('Downloading cl100k_base encoding...'); \
+tiktoken.get_encoding('cl100k_base'); \
+print('Tiktoken encoding cached successfully')" && break; \
+      if [ $i -lt $MAX_RETRIES ]; then \
+        echo "Attempt $i failed, retrying in ${RETRY_DELAY}s..."; \
+        sleep $RETRY_DELAY; \
+        RETRY_DELAY=$((RETRY_DELAY * 2)); \
+      fi; \
+    done; \
+    if [ $i -eq $MAX_RETRIES ]; then \
+      echo "ERROR: Failed to download tiktoken encoding after $MAX_RETRIES attempts"; \
+      exit 1; \
+    fi
+
 # Pre-download ML models to avoid runtime download (conditional)
 # Only runs if both PRELOAD_ML_MODELS=true AND INCLUDE_LOCAL_MODELS=true
+# Includes retry logic with exponential backoff for transient network failures
 ARG PRELOAD_ML_MODELS
 ARG INCLUDE_LOCAL_MODELS
+ENV HF_HUB_DOWNLOAD_TIMEOUT=600
 RUN if [ "$PRELOAD_ML_MODELS" = "true" ] && [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-    /app/api/.venv/bin/python -c "\
+    MAX_RETRIES=3; \
+    RETRY_DELAY=10; \
+    for i in $(seq 1 $MAX_RETRIES); do \
+      echo "Attempt $i/$MAX_RETRIES: Downloading ML models..."; \
+      /app/api/.venv/bin/python -c "\
+import os; os.environ['HF_HUB_DOWNLOAD_TIMEOUT'] = '600'; \
 from sentence_transformers import SentenceTransformer, CrossEncoder; \
 print('Downloading embedding model...'); \
 SentenceTransformer('BAAI/bge-small-en-v1.5'); \
 print('Downloading cross-encoder model...'); \
 CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2'); \
-print('Models cached successfully')"; \
+print('Models cached successfully')" && break; \
+      if [ $i -lt $MAX_RETRIES ]; then \
+        echo "Attempt $i failed, retrying in ${RETRY_DELAY}s..."; \
+        sleep $RETRY_DELAY; \
+        RETRY_DELAY=$((RETRY_DELAY * 2)); \
+      fi; \
+    done; \
+    if [ $i -eq $MAX_RETRIES ] && ! /app/api/.venv/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('BAAI/bge-small-en-v1.5')" 2>/dev/null; then \
+      echo "ERROR: Failed to download models after $MAX_RETRIES attempts"; \
+      exit 1; \
+    fi; \
     elif [ "$INCLUDE_LOCAL_MODELS" != "true" ]; then echo "Skipping ML model preload (local-models not included)"; \
     else echo "Skipping ML model preload"; fi
 
@@ -190,6 +242,10 @@ ENV HINDSIGHT_API_LOG_LEVEL=info
 ENV HINDSIGHT_ENABLE_API=true
 ENV HINDSIGHT_ENABLE_CP=false
 ENV PYTHONUNBUFFERED=1
+# Suppress verbose transformers/HuggingFace model loading warnings
+ENV TRANSFORMERS_VERBOSITY=error
+ENV HF_HUB_VERBOSITY=error
+ENV TOKENIZERS_PARALLELISM=false
 
 CMD ["/app/start-all.sh"]
 
@@ -273,20 +329,68 @@ RUN chown -R hindsight:hindsight /app
 
 USER hindsight
 
+# Create pg0 data directory as hindsight user so that Docker seeds new named
+# volumes with correct ownership (UID 1000) on first use, avoiding the
+# "Permission denied" error when mounting a fresh root-owned volume.
+RUN mkdir -p /home/hindsight/.pg0
+
+# Make /home/hindsight traversable when running with --user UID:GID overrides
+# (default 0700 blocks traversal by non-owner UIDs needed for bind-mount ownership matching)
+RUN chmod 755 /home/hindsight
+
 ENV PATH="/app/api/.venv/bin:${PATH}"
 
+# Pre-download tiktoken encoding (ALWAYS - required for token counting even in air-gapped envs)
+# Tiktoken is a core runtime dependency, not an optional ML model
+RUN MAX_RETRIES=3; \
+    RETRY_DELAY=5; \
+    for i in $(seq 1 $MAX_RETRIES); do \
+      echo "Attempt $i/$MAX_RETRIES: Downloading tiktoken encoding..."; \
+      /app/api/.venv/bin/python -c "\
+import tiktoken; \
+print('Downloading cl100k_base encoding...'); \
+tiktoken.get_encoding('cl100k_base'); \
+print('Tiktoken encoding cached successfully')" && break; \
+      if [ $i -lt $MAX_RETRIES ]; then \
+        echo "Attempt $i failed, retrying in ${RETRY_DELAY}s..."; \
+        sleep $RETRY_DELAY; \
+        RETRY_DELAY=$((RETRY_DELAY * 2)); \
+      fi; \
+    done; \
+    if [ $i -eq $MAX_RETRIES ]; then \
+      echo "ERROR: Failed to download tiktoken encoding after $MAX_RETRIES attempts"; \
+      exit 1; \
+    fi
+
 # Pre-download ML models to avoid runtime download (conditional)
 # Only runs if both PRELOAD_ML_MODELS=true AND INCLUDE_LOCAL_MODELS=true
+# Includes retry logic with exponential backoff for transient network failures
 ARG PRELOAD_ML_MODELS
 ARG INCLUDE_LOCAL_MODELS
+ENV HF_HUB_DOWNLOAD_TIMEOUT=600
 RUN if [ "$PRELOAD_ML_MODELS" = "true" ] && [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-    /app/api/.venv/bin/python -c "\
+    MAX_RETRIES=3; \
+    RETRY_DELAY=10; \
+    for i in $(seq 1 $MAX_RETRIES); do \
+      echo "Attempt $i/$MAX_RETRIES: Downloading ML models..."; \
+      /app/api/.venv/bin/python -c "\
+import os; os.environ['HF_HUB_DOWNLOAD_TIMEOUT'] = '600'; \
 from sentence_transformers import SentenceTransformer, CrossEncoder; \
 print('Downloading embedding model...'); \
 SentenceTransformer('BAAI/bge-small-en-v1.5'); \
 print('Downloading cross-encoder model...'); \
 CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2'); \
-print('Models cached successfully')"; \
+print('Models cached successfully')" && break; \
+      if [ $i -lt $MAX_RETRIES ]; then \
+        echo "Attempt $i failed, retrying in ${RETRY_DELAY}s..."; \
+        sleep $RETRY_DELAY; \
+        RETRY_DELAY=$((RETRY_DELAY * 2)); \
+      fi; \
+    done; \
+    if [ $i -eq $MAX_RETRIES ] && ! /app/api/.venv/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('BAAI/bge-small-en-v1.5')" 2>/dev/null; then \
+      echo "ERROR: Failed to download models after $MAX_RETRIES attempts"; \
+      exit 1; \
+    fi; \
     elif [ "$INCLUDE_LOCAL_MODELS" != "true" ]; then echo "Skipping ML model preload (local-models not included)"; \
     else echo "Skipping ML model preload"; fi
 
@@ -300,6 +404,10 @@ ENV HINDSIGHT_CP_DATAPLANE_API_URL=http://localhost:8888
 ENV HINDSIGHT_ENABLE_API=true
 ENV HINDSIGHT_ENABLE_CP=true
 ENV PYTHONUNBUFFERED=1
+# Suppress verbose transformers/HuggingFace model loading warnings
+ENV TRANSFORMERS_VERBOSITY=error
+ENV HF_HUB_VERBOSITY=error
+ENV TOKENIZERS_PARALLELISM=false
 
 CMD ["/app/start-all.sh"]
 

docker/standalone/start-all.sh

@@ -1,6 +1,99 @@
 #!/bin/bash
 set -e
 
+# =============================================================================
+# Embedded pg0 data integrity check (#675)
+#
+# When using embedded pg0, check if the data directory has existing PostgreSQL
+# data before starting. If the directory exists but appears empty/corrupt
+# (e.g., missing PG_VERSION file), log a warning. This helps diagnose data
+# loss scenarios where a container restart caused the data directory to be
+# wiped despite a volume mount being present.
+# =============================================================================
+pg0_has_pg_version() {
+    local pg0_data_dir="$1"
+
+    # pg0 has used more than one on-disk layout. Newer standalone images keep
+    # PostgreSQL data under instances/<name>/data, while older volumes may have
+    # placed PG_VERSION at or one level below the mount.
+    [ -f "$pg0_data_dir/PG_VERSION" ] && return 0
+    compgen -G "$pg0_data_dir"/*/PG_VERSION > /dev/null 2>&1 && return 0
+    compgen -G "$pg0_data_dir"/instances/*/data/PG_VERSION > /dev/null 2>&1 && return 0
+
+    return 1
+}
+
+check_pg0_data_integrity() {
+    local pg0_data_dir="$1"
+
+    if [ ! -d "$pg0_data_dir" ]; then
+        return 0
+    fi
+
+    # Look for actual PostgreSQL data directories (pg0 creates subdirs per instance)
+    if pg0_has_pg_version "$pg0_data_dir"; then
+        echo "✅ Existing pg0 data directory detected at $pg0_data_dir"
+    elif [ "$(ls -A "$pg0_data_dir" 2>/dev/null)" ]; then
+        echo "⚠️  WARNING: pg0 data directory exists at $pg0_data_dir but no PG_VERSION found."
+        echo "   This may indicate data corruption or an incomplete previous shutdown."
+        echo "   If you see all migrations running from scratch after this, your data may have been lost."
+        echo "   See: https://github.com/vectorize-io/hindsight/issues/675"
+    fi
+
+    return 0
+}
+
+# =============================================================================
+# Embedded pg0 writability pre-check (#1483)
+#
+# The container runs as the unprivileged `hindsight` user (UID 1000). When the
+# pg0 data directory is a host bind mount (e.g. `-v $HOME/dir:/home/hindsight/.pg0`)
+# that is not owned by UID 1000 — the default on macOS Docker Desktop and most
+# non-1000 Linux hosts — pg0 fails with the opaque "Permission denied (os error
+# 13)". We cannot chown it ourselves without root (and the image is deliberately
+# rootless), so we surface an actionable message up front instead.
+#
+# Docker *named* volumes are seeded with the image directory's ownership (UID
+# 1000) on first use, so they avoid this entirely — hence the named-volume
+# recommendation below and in the README.
+# =============================================================================
+check_pg0_writable() {
+    local pg0_data_dir="$1"
+
+    # Only relevant for embedded pg0; an external database doesn't use this dir.
+    if [ -n "${HINDSIGHT_API_DATABASE_URL:-}" ]; then
+        return 0
+    fi
+
+    mkdir -p "$pg0_data_dir" 2>/dev/null || true
+    if touch "$pg0_data_dir/.hindsight-write-test" 2>/dev/null; then
+        rm -f "$pg0_data_dir/.hindsight-write-test" 2>/dev/null || true
+        return 0
+    fi
+
+    echo "❌ The embedded database directory $pg0_data_dir is not writable by this container (UID $(id -u))."
+    echo ""
+    echo "   A host directory was bind-mounted but is not owned by the container user (UID 1000)."
+    echo "   Hindsight runs rootless and cannot fix this for you. Choose one:"
+    echo ""
+    echo "   • Recommended — use a Docker named volume (auto-owned by the container):"
+    echo "       -v hindsight-data:/home/hindsight/.pg0"
+    echo ""
+    echo "   • Or keep the host path and run as your host user, chowning it to match:"
+    echo "       sudo chown -R \$(id -u):\$(id -g) <host-directory>"
+    echo "       docker run --user \$(id -u):\$(id -g) -e HOME=/home/hindsight ..."
+    echo ""
+    echo "   See https://github.com/vectorize-io/hindsight/issues/1483"
+    return 1
+}
+
+if [ "${HINDSIGHT_START_ALL_SOURCE_ONLY:-false}" = "true" ]; then
+    return 0 2>/dev/null || exit 0
+fi
+
+check_pg0_data_integrity "${HOME}/.pg0"
+check_pg0_writable "${HOME}/.pg0" || exit 1
+
 # Service flags (default to true if not set)
 ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
 ENABLE_CP="${HINDSIGHT_ENABLE_CP:-true}"
@@ -71,24 +164,95 @@ if [ "${HINDSIGHT_WAIT_FOR_DEPS:-false}" = "true" ]; then
     done
 fi
 
+# =============================================================================
+# Graceful shutdown handler (#675)
+#
+# Docker sends SIGTERM on `docker stop`/`docker restart`. Without a trap, child
+# processes (hindsight-api + pg0, control-plane) are killed abruptly. For the
+# embedded pg0 database this can cause data loss when the data directory is on
+# a Docker volume that gets remounted after restart.
+#
+# The trap forwards SIGTERM to all tracked child PIDs so that:
+#   - hindsight-api receives the signal and can run its shutdown hooks
+#   - pg0 gets a clean PostgreSQL shutdown (checkpoint + WAL flush)
+#   - The control-plane Node.js process exits cleanly
+# =============================================================================
+# Guard against concurrent cleanup (e.g., child crash + SIGTERM arriving together)
+SHUTTING_DOWN=false
+
+cleanup() {
+    if $SHUTTING_DOWN; then return; fi
+    SHUTTING_DOWN=true
+
+    echo ""
+    echo "🛑 Received shutdown signal, stopping services gracefully..."
+    for pid in "${PIDS[@]}"; do
+        if kill -0 "$pid" 2>/dev/null; then
+            kill -TERM "$pid" 2>/dev/null
+        fi
+    done
+    # Give processes time to shut down cleanly (pg0 needs to flush WAL).
+    # NOTE: Docker's default stop_grace_period is 10s. If you use the default,
+    # either set stop_grace_period: 30s in your compose file / docker stop -t 30,
+    # or Docker will SIGKILL the container before this timeout expires.
+    local timeout=30
+    for ((i=1; i<=timeout; i++)); do
+        local all_stopped=true
+        for pid in "${PIDS[@]}"; do
+            if kill -0 "$pid" 2>/dev/null; then
+                all_stopped=false
+                break
+            fi
+        done
+        if $all_stopped; then
+            echo "✅ All services stopped cleanly"
+            exit 0
+        fi
+        sleep 1
+    done
+    # Force kill if still running after timeout
+    echo "⚠️  Timeout reached, forcing shutdown..."
+    for pid in "${PIDS[@]}"; do
+        if kill -0 "$pid" 2>/dev/null; then
+            kill -9 "$pid" 2>/dev/null
+        fi
+    done
+    exit 1
+}
+trap cleanup SIGTERM SIGINT
+
 # Track PIDs for wait
 PIDS=()
 
 # Start API if enabled
 if [ "$ENABLE_API" = "true" ]; then
     cd /app/api
+    API_HEALTH_URL="${HINDSIGHT_API_HEALTH_URL:-http://localhost:${HINDSIGHT_API_PORT:-8888}/health}"
+    API_STARTUP_WAIT_SECONDS="${HINDSIGHT_API_STARTUP_WAIT_SECONDS:-300}"
+
     # Run API directly - Python's PYTHONUNBUFFERED=1 handles output buffering
     hindsight-api &
     API_PID=$!
     PIDS+=($API_PID)
 
     # Wait for API to be ready
-    for i in {1..60}; do
-        if curl -sf http://localhost:8888/health &>/dev/null; then
+    api_ready=false
+    for ((i=1; i<=API_STARTUP_WAIT_SECONDS; i++)); do
+        if ! kill -0 "$API_PID" 2>/dev/null; then
+            wait "$API_PID"
+            exit $?
+        fi
+        if curl -sf "$API_HEALTH_URL" &>/dev/null; then
+            api_ready=true
             break
         fi
         sleep 1
     done
+
+    if [ "$api_ready" != "true" ]; then
+        echo "❌ API did not become healthy within ${API_STARTUP_WAIT_SECONDS}s"
+        exit 1
+    fi
 else
     echo "API disabled (HINDSIGHT_ENABLE_API=false)"
 fi
@@ -97,7 +261,8 @@ fi
 if [ "$ENABLE_CP" = "true" ]; then
     echo "🎛️  Starting Control Plane..."
     cd /app/control-plane
-    PORT=9999 node server.js &
+    export HOSTNAME="${HINDSIGHT_CP_HOSTNAME:-0.0.0.0}"
+    PORT="${HINDSIGHT_CP_PORT:-9999}" node server.js &
     CP_PID=$!
     PIDS+=($CP_PID)
 else
@@ -110,7 +275,7 @@ echo "✅ Hindsight is running!"
 echo ""
 echo "📍 Access:"
 if [ "$ENABLE_CP" = "true" ]; then
-    echo "   Control Plane: http://localhost:9999"
+    echo "   Control Plane: http://localhost:${HINDSIGHT_CP_PORT:-9999}"
 fi
 if [ "$ENABLE_API" = "true" ]; then
     echo "   API:           http://localhost:8888"
@@ -123,8 +288,21 @@ if [ ${#PIDS[@]} -eq 0 ]; then
     exit 1
 fi
 
-# Wait for any process to exit
-wait -n
-
-# Exit with status of first exited process
-exit $?
+# Wait for any process to exit (use wait -n with trap-safe loop)
+while true; do
+    # wait -n returns when any child exits; it also returns on signal delivery
+    # (the trap handler will run and exit, so this loop is just for robustness).
+    # `&& true` prevents `set -e` from killing the script when wait -n returns
+    # non-zero (child exited with error or no backgrounded children remain).
+    wait -n && true
+    # Check if any tracked PID has exited
+    for pid in "${PIDS[@]}"; do
+        if ! kill -0 "$pid" 2>/dev/null; then
+            wait "$pid" 2>/dev/null
+            exit_code=$?
+            echo "⚠️  Service (PID $pid) exited with code $exit_code"
+            # Trigger cleanup for remaining services
+            cleanup
+        fi
+    done
+done

docker/standalone/test-start-all.sh

@@ -0,0 +1,121 @@
+#!/bin/bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+HINDSIGHT_START_ALL_SOURCE_ONLY=true
+source "$SCRIPT_DIR/start-all.sh"
+unset HINDSIGHT_START_ALL_SOURCE_ONLY
+
+TMP_DIR="$(mktemp -d)"
+trap 'chmod -R u+rwx "$TMP_DIR" 2>/dev/null || true; rm -rf "$TMP_DIR"' EXIT
+
+assert_contains() {
+    local output="$1"
+    local expected="$2"
+
+    if [[ "$output" != *"$expected"* ]]; then
+        echo "Expected output to contain: $expected"
+        echo "Actual output:"
+        echo "$output"
+        exit 1
+    fi
+}
+
+assert_not_contains() {
+    local output="$1"
+    local unexpected="$2"
+
+    if [[ "$output" == *"$unexpected"* ]]; then
+        echo "Expected output not to contain: $unexpected"
+        echo "Actual output:"
+        echo "$output"
+        exit 1
+    fi
+}
+
+assert_empty() {
+    local output="$1"
+
+    if [ -n "$output" ]; then
+        echo "Expected no output, got:"
+        echo "$output"
+        exit 1
+    fi
+}
+
+mkdir -p "$TMP_DIR/empty"
+assert_empty "$(check_pg0_data_integrity "$TMP_DIR/empty")"
+
+mkdir -p "$TMP_DIR/direct"
+touch "$TMP_DIR/direct/PG_VERSION"
+direct_output="$(check_pg0_data_integrity "$TMP_DIR/direct")"
+assert_contains "$direct_output" "Existing pg0 data directory detected"
+assert_not_contains "$direct_output" "WARNING"
+
+mkdir -p "$TMP_DIR/legacy/instance"
+touch "$TMP_DIR/legacy/instance/PG_VERSION"
+legacy_output="$(check_pg0_data_integrity "$TMP_DIR/legacy")"
+assert_contains "$legacy_output" "Existing pg0 data directory detected"
+assert_not_contains "$legacy_output" "WARNING"
+
+mkdir -p "$TMP_DIR/nested/instances/hindsight/data"
+touch "$TMP_DIR/nested/instances/hindsight/data/PG_VERSION"
+nested_output="$(check_pg0_data_integrity "$TMP_DIR/nested")"
+assert_contains "$nested_output" "Existing pg0 data directory detected"
+assert_not_contains "$nested_output" "WARNING"
+
+mkdir -p "$TMP_DIR/nonempty/instances/hindsight"
+touch "$TMP_DIR/nonempty/instances/hindsight/instance.json"
+nonempty_output="$(check_pg0_data_integrity "$TMP_DIR/nonempty")"
+assert_contains "$nonempty_output" "WARNING: pg0 data directory exists"
+
+echo "start-all pg0 integrity checks passed"
+
+# =============================================================================
+# check_pg0_writable (#1483)
+# These rely on filesystem permissions, which root bypasses; skip under root.
+# =============================================================================
+if [ "$(id -u)" != "0" ]; then
+    # Writable directory: returns 0, prints nothing, leaves no artifact behind.
+    mkdir -p "$TMP_DIR/writable"
+    writable_output="$(check_pg0_writable "$TMP_DIR/writable")"
+    assert_empty "$writable_output"
+    if [ -e "$TMP_DIR/writable/.hindsight-write-test" ]; then
+        echo "check_pg0_writable left its write-test file behind"
+        exit 1
+    fi
+
+    # Non-writable directory: returns 1 with actionable guidance.
+    mkdir -p "$TMP_DIR/readonly"
+    chmod 000 "$TMP_DIR/readonly"
+    set +e
+    readonly_output="$(check_pg0_writable "$TMP_DIR/readonly" 2>&1)"
+    readonly_rc=$?
+    set -e
+    chmod 755 "$TMP_DIR/readonly"
+    if [ "$readonly_rc" -eq 0 ]; then
+        echo "check_pg0_writable should fail on a non-writable directory"
+        exit 1
+    fi
+    assert_contains "$readonly_output" "not writable"
+    assert_contains "$readonly_output" "hindsight-data:/home/hindsight/.pg0"
+    assert_contains "$readonly_output" "--user"
+
+    # External database configured: skip the check regardless of dir perms.
+    mkdir -p "$TMP_DIR/extdb"
+    chmod 000 "$TMP_DIR/extdb"
+    set +e
+    HINDSIGHT_API_DATABASE_URL="postgres://x" check_pg0_writable "$TMP_DIR/extdb" >/dev/null 2>&1
+    extdb_rc=$?
+    set -e
+    chmod 755 "$TMP_DIR/extdb"
+    if [ "$extdb_rc" -ne 0 ]; then
+        echo "check_pg0_writable should skip when an external database is configured"
+        exit 1
+    fi
+
+    echo "start-all pg0 writability checks passed"
+else
+    echo "⚠️  Running as root; skipping pg0 writability checks (permissions are bypassed)."
+fi

docker/test-image.sh

@@ -0,0 +1,235 @@
+#!/bin/bash
+#
+# Docker Smoke Test Script
+#
+# Tests that a Hindsight Docker image starts correctly and becomes healthy.
+# Can be run locally or in CI pipelines.
+#
+# Usage:
+#   ./docker/test-image.sh <image> [target]
+#
+# Arguments:
+#   image   - Docker image to test (e.g., hindsight-api:test, ghcr.io/vectorize-io/hindsight:latest)
+#   target  - Optional: 'cp-only' for control plane, otherwise assumes API image (default: api)
+#
+# Environment variables:
+#   HINDSIGHT_API_LLM_API_KEY                   - Required for API/standalone images (LLM verification)
+#   HINDSIGHT_API_LLM_PROVIDER                  - LLM provider (default: openai)
+#   HINDSIGHT_API_LLM_MODEL                     - LLM model (default: gpt-4o-mini)
+#   HINDSIGHT_API_EMBEDDINGS_PROVIDER           - Embeddings provider (optional, for slim images: openai, cohere, tei)
+#   HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY     - OpenAI API key for embeddings (optional)
+#   HINDSIGHT_API_RERANKER_PROVIDER             - Reranker provider (optional, for slim images: cohere, tei)
+#   HINDSIGHT_API_COHERE_API_KEY                - Cohere API key for reranking (optional)
+#   SMOKE_TEST_TIMEOUT                          - Timeout in seconds (default: 120)
+#   SMOKE_TEST_CONTAINER_NAME                   - Container name (default: hindsight-smoke-test)
+#
+# Examples:
+#   # Test a locally built full image
+#   ./docker/test-image.sh hindsight-api:test
+#
+#   # Test a released image
+#   ./docker/test-image.sh ghcr.io/vectorize-io/hindsight:latest
+#
+#   # Test control plane image
+#   ./docker/test-image.sh hindsight-control-plane:test cp-only
+#
+#   # Test slim image with external providers
+#   export HINDSIGHT_API_LLM_API_KEY=sk_xxx
+#   export HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
+#   export HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=sk-xxx
+#   export HINDSIGHT_API_RERANKER_PROVIDER=cohere
+#   export HINDSIGHT_API_COHERE_API_KEY=xxx
+#   ./docker/test-image.sh hindsight-slim:test
+#
+# Exit codes:
+#   0 - Success (container healthy)
+#   1 - Failure (container not healthy within timeout)
+#   2 - Invalid arguments
+#
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(dirname "$SCRIPT_DIR")"
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
+NC='\033[0m' # No Color
+
+# Configuration
+IMAGE="${1:-}"
+TARGET="${2:-api}"
+TIMEOUT="${SMOKE_TEST_TIMEOUT:-120}"
+CONTAINER_NAME="${SMOKE_TEST_CONTAINER_NAME:-hindsight-smoke-test}"
+LLM_PROVIDER="${HINDSIGHT_API_LLM_PROVIDER:-openai}"
+LLM_MODEL="${HINDSIGHT_API_LLM_MODEL:-gpt-4o-mini}"
+
+# Validate arguments
+if [ -z "$IMAGE" ]; then
+    echo -e "${RED}Error: Image argument is required${NC}"
+    echo ""
+    echo "Usage: $0 <image> [target]"
+    echo ""
+    echo "Examples:"
+    echo "  $0 hindsight-api:test"
+    echo "  $0 ghcr.io/vectorize-io/hindsight:latest"
+    echo "  $0 hindsight-control-plane:test cp-only"
+    exit 2
+fi
+
+# Determine health endpoint based on target
+if [ "$TARGET" = "cp-only" ]; then
+    HEALTH_PORT=9999
+    HEALTH_PATH="/api/health"
+    NEEDS_LLM=false
+else
+    HEALTH_PORT=8888
+    HEALTH_PATH="/health"
+    NEEDS_LLM=true
+fi
+
+# Check for required environment variables
+if [ "$NEEDS_LLM" = true ] && [ "$LLM_PROVIDER" != "vertexai" ] && [ -z "${HINDSIGHT_API_LLM_API_KEY:-}" ]; then
+    echo -e "${RED}Error: HINDSIGHT_API_LLM_API_KEY environment variable is required for API/standalone images${NC}"
+    echo "Set it with: export HINDSIGHT_API_LLM_API_KEY=your-api-key"
+    exit 2
+fi
+
+# Cleanup function
+cleanup() {
+    echo "Cleaning up..."
+    docker stop "$CONTAINER_NAME" 2>/dev/null || true
+    docker rm "$CONTAINER_NAME" 2>/dev/null || true
+}
+
+# Set trap to cleanup on exit
+trap cleanup EXIT
+
+echo -e "${YELLOW}Starting smoke test for: ${IMAGE}${NC}"
+echo "  Target: $TARGET"
+echo "  Health endpoint: http://localhost:${HEALTH_PORT}${HEALTH_PATH}"
+echo "  Timeout: ${TIMEOUT}s"
+echo ""
+
+# Remove any existing container with the same name
+docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
+
+# Start container based on target type
+echo "Starting container..."
+if [ "$TARGET" = "cp-only" ]; then
+    docker run -d --name "$CONTAINER_NAME" \
+        -p "${HEALTH_PORT}:${HEALTH_PORT}" \
+        "$IMAGE"
+else
+    # Build docker run command with required and optional env vars
+    DOCKER_CMD="docker run -d --name $CONTAINER_NAME"
+    DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_PROVIDER=$LLM_PROVIDER"
+    if [ -n "${HINDSIGHT_API_LLM_API_KEY:-}" ]; then
+        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_API_KEY=${HINDSIGHT_API_LLM_API_KEY}"
+    fi
+    DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_MODEL=$LLM_MODEL"
+
+    # Add Vertex AI config if provider is vertexai
+    if [ "$LLM_PROVIDER" = "vertexai" ]; then
+        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY:-}" ]; then
+            DOCKER_CMD="$DOCKER_CMD -v ${HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY}:/tmp/gcp-credentials.json:ro"
+            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/tmp/gcp-credentials.json"
+        fi
+        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID:-}" ]; then
+            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=${HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID}"
+        fi
+        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_REGION:-}" ]; then
+            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_REGION=${HINDSIGHT_API_LLM_VERTEXAI_REGION}"
+        fi
+    fi
+
+    # Add optional embeddings provider config
+    if [ -n "${HINDSIGHT_API_EMBEDDINGS_PROVIDER:-}" ]; then
+        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_EMBEDDINGS_PROVIDER=${HINDSIGHT_API_EMBEDDINGS_PROVIDER}"
+    fi
+    if [ -n "${HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY:-}" ]; then
+        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=${HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY}"
+    fi
+
+    # Add optional reranker provider config
+    if [ -n "${HINDSIGHT_API_RERANKER_PROVIDER:-}" ]; then
+        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_RERANKER_PROVIDER=${HINDSIGHT_API_RERANKER_PROVIDER}"
+    fi
+    if [ -n "${HINDSIGHT_API_COHERE_API_KEY:-}" ]; then
+        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_COHERE_API_KEY=${HINDSIGHT_API_COHERE_API_KEY}"
+    fi
+
+    DOCKER_CMD="$DOCKER_CMD -p ${HEALTH_PORT}:${HEALTH_PORT}"
+    DOCKER_CMD="$DOCKER_CMD $IMAGE"
+
+    eval $DOCKER_CMD
+fi
+
+# Wait for health endpoint
+echo "Waiting for health endpoint at http://localhost:${HEALTH_PORT}${HEALTH_PATH}..."
+start_time=$(date +%s)
+
+for i in $(seq 1 "$TIMEOUT"); do
+    if curl -sf "http://localhost:${HEALTH_PORT}${HEALTH_PATH}" > /dev/null 2>&1; then
+        end_time=$(date +%s)
+        duration=$((end_time - start_time))
+        echo ""
+        echo -e "${GREEN}Container is healthy after ${duration}s${NC}"
+        echo ""
+        echo "=== Health Response ==="
+        curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}" | python3 -m json.tool 2>/dev/null || curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}"
+        echo ""
+
+        # Run retain/recall smoke test for API targets
+        if [ "$TARGET" != "cp-only" ]; then
+            echo ""
+            echo "=== Retain/Recall Smoke Test ==="
+            if ! "$REPO_ROOT/scripts/smoke-test-slim.sh" "http://localhost:${HEALTH_PORT}"; then
+                echo ""
+                echo "=== Container Logs (last 50 lines) ==="
+                docker logs "$CONTAINER_NAME" 2>&1 | tail -50
+                echo ""
+                echo -e "${RED}Smoke test FAILED${NC}"
+                exit 1
+            fi
+        fi
+
+        echo ""
+        echo "=== Container Logs (last 50 lines) ==="
+        docker logs "$CONTAINER_NAME" 2>&1 | tail -50
+        echo ""
+        echo -e "${GREEN}Smoke test PASSED${NC}"
+        exit 0
+    fi
+
+    # Show progress every 10 seconds
+    if [ $((i % 10)) -eq 0 ]; then
+        echo "  Still waiting... (${i}s)"
+    fi
+
+    # Check if container is still running
+    if ! docker ps -q -f "name=$CONTAINER_NAME" | grep -q .; then
+        echo ""
+        echo -e "${RED}Container exited unexpectedly!${NC}"
+        echo ""
+        echo "=== Container Logs ==="
+        docker logs "$CONTAINER_NAME" 2>&1
+        echo ""
+        echo -e "${RED}Smoke test FAILED${NC}"
+        exit 1
+    fi
+
+    sleep 1
+done
+
+# Timeout reached
+echo ""
+echo -e "${RED}Container failed to become healthy after ${TIMEOUT}s${NC}"
+echo ""
+echo "=== Container Logs ==="
+docker logs "$CONTAINER_NAME" 2>&1
+echo ""
+echo -e "${RED}Smoke test FAILED${NC}"
+exit 1

docker/test-slim-local.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+#
+# Local Test Script for Slim Docker Images
+#
+# This script makes it easy to test slim images locally with external providers.
+# It expects API keys to be set in environment variables.
+#
+# Usage:
+#   export OPENAI_API_KEY=sk-xxx
+#   export COHERE_API_KEY=xxx
+#   ./docker/test-slim-local.sh
+#
+# Or inline:
+#   OPENAI_API_KEY=sk_xxx COHERE_API_KEY=xxx ./docker/test-slim-local.sh
+#
+
+set -euo pipefail
+
+# Check for required API keys
+if [ -z "${OPENAI_API_KEY:-}" ]; then
+    echo "❌ Error: OPENAI_API_KEY environment variable is required"
+    echo "Set it with: export OPENAI_API_KEY=sk-xxx"
+    exit 1
+fi
+
+if [ -z "${COHERE_API_KEY:-}" ]; then
+    echo "❌ Error: COHERE_API_KEY environment variable is required"
+    echo "Set it with: export COHERE_API_KEY=xxx"
+    exit 1
+fi
+
+# Configuration
+IMAGE="${1:-hindsight-slim:test}"
+echo "Testing image: $IMAGE"
+echo ""
+
+# Set up LLM and external providers
+export HINDSIGHT_API_LLM_PROVIDER=openai
+export HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY
+export HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
+export HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
+export HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=$OPENAI_API_KEY
+export HINDSIGHT_API_RERANKER_PROVIDER=cohere
+export HINDSIGHT_API_COHERE_API_KEY=$COHERE_API_KEY
+
+# Run the test
+exec "$(dirname "$0")/test-image.sh" "$IMAGE" standalone

helm/hindsight/Chart.lock

@@ -1,6 +0,0 @@
-dependencies:
-- name: postgresql
-  repository: https://charts.bitnami.com/bitnami
-  version: 15.5.38
-digest: sha256:f67c7612736803ece8a669f8ca6b0555f3b78557bc0ecb732aa2e43f0df7750d
-generated: "2025-12-10T17:20:57.058794+01:00"

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.3.0
-appVersion: "0.3.0"
+version: 0.7.2
+appVersion: "0.7.2"
 keywords:
   - ai
   - memory

helm/hindsight/templates/_helpers.tpl

@@ -127,6 +127,38 @@ API URL for control plane
 {{- printf "http://%s-api:%d" (include "hindsight.fullname" .) (.Values.api.service.port | int) }}
 {{- end }}
 
+{{/*
+TEI reranker labels
+*/}}
+{{- define "hindsight.tei.reranker.labels" -}}
+{{ include "hindsight.labels" . }}
+app.kubernetes.io/component: tei-reranker
+{{- end }}
+
+{{/*
+TEI reranker selector labels
+*/}}
+{{- define "hindsight.tei.reranker.selectorLabels" -}}
+{{ include "hindsight.selectorLabels" . }}
+app.kubernetes.io/component: tei-reranker
+{{- end }}
+
+{{/*
+TEI embedding labels
+*/}}
+{{- define "hindsight.tei.embedding.labels" -}}
+{{ include "hindsight.labels" . }}
+app.kubernetes.io/component: tei-embedding
+{{- end }}
+
+{{/*
+TEI embedding selector labels
+*/}}
+{{- define "hindsight.tei.embedding.selectorLabels" -}}
+{{ include "hindsight.selectorLabels" . }}
+app.kubernetes.io/component: tei-embedding
+{{- end }}
+
 {{/*
 Get the name of the secret to use
 */}}

helm/hindsight/templates/api-deployment.yaml

@@ -33,7 +33,7 @@ spec:
       - name: api
         securityContext:
           {{- toYaml .Values.securityContext | nindent 10 }}
-        image: "{{ .Values.api.image.repository }}:{{ .Values.api.image.tag | default .Values.version }}"
+        image: "{{ .Values.api.image.repository }}:{{ .Values.api.image.tag | default .Values.version | default .Chart.AppVersion }}"
         imagePullPolicy: {{ .Values.api.image.pullPolicy }}
         ports:
         - name: http
@@ -60,10 +60,25 @@ spec:
         - name: HINDSIGHT_API_WORKER_ENABLED
           value: "false"
         {{- end }}
+        {{- /* Explicitly set port to override K8s service discovery env var (HINDSIGHT_API_PORT) */}}
+        - name: HINDSIGHT_API_PORT
+          value: {{ .Values.api.service.targetPort | quote }}
         {{- range $key, $value := .Values.api.env }}
         - name: {{ $key }}
           value: {{ $value | quote }}
         {{- end }}
+        {{- if .Values.tei.reranker.enabled }}
+        - name: HINDSIGHT_API_RERANKER_PROVIDER
+          value: "tei"
+        - name: HINDSIGHT_API_RERANKER_TEI_URL
+          value: "http://{{ include "hindsight.fullname" . }}-tei-reranker:{{ .Values.tei.reranker.port }}"
+        {{- end }}
+        {{- if .Values.tei.embedding.enabled }}
+        - name: HINDSIGHT_API_EMBEDDINGS_PROVIDER
+          value: "tei"
+        - name: HINDSIGHT_API_EMBEDDINGS_TEI_URL
+          value: "http://{{ include "hindsight.fullname" . }}-tei-embedding:{{ .Values.tei.embedding.port }}"
+        {{- end }}
         {{- /* Only use api.secrets when not using existingSecret (for chart-managed secrets) */}}
         {{- if not .Values.existingSecret }}
         {{- range $key, $value := .Values.api.secrets }}
@@ -80,11 +95,32 @@ spec:
           {{- toYaml .Values.api.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.api.resources | nindent 10 }}
+        {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumeMounts }}
+        volumeMounts:
+        {{- if .Values.api.persistence.modelCache.enabled }}
+        - name: model-cache
+          mountPath: /home/hindsight/.cache
+        {{- end }}
+        {{- with .Values.api.extraVolumeMounts }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- end }}
+      {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumes }}
+      volumes:
+      {{- if .Values.api.persistence.modelCache.enabled }}
+      - name: model-cache
+        persistentVolumeClaim:
+          claimName: {{ include "hindsight.fullname" . }}-api-model-cache
+      {{- end }}
+      {{- with .Values.api.extraVolumes }}
+      {{- toYaml . | nindent 6 }}
+      {{- end }}
+      {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with (.Values.api.affinity | default .Values.affinity) }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}

helm/hindsight/templates/api-model-cache-pvc.yaml

@@ -0,0 +1,21 @@
+{{- if and .Values.api.enabled .Values.api.persistence.modelCache.enabled }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "hindsight.fullname" . }}-api-model-cache
+  labels:
+    {{- include "hindsight.api.labels" . | nindent 4 }}
+  {{- with .Values.api.persistence.modelCache.annotations }}
+  annotations:
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
+spec:
+  accessModes:
+    {{- toYaml .Values.api.persistence.modelCache.accessModes | nindent 4 }}
+  {{- if .Values.api.persistence.modelCache.storageClass }}
+  storageClassName: {{ .Values.api.persistence.modelCache.storageClass }}
+  {{- end }}
+  resources:
+    requests:
+      storage: {{ .Values.api.persistence.modelCache.size }}
+{{- end }}

helm/hindsight/templates/controlplane-deployment.yaml

@@ -33,7 +33,7 @@ spec:
       - name: control-plane
         securityContext:
           {{- toYaml .Values.securityContext | nindent 10 }}
-        image: "{{ .Values.controlPlane.image.repository }}:{{ .Values.controlPlane.image.tag | default .Values.version }}"
+        image: "{{ .Values.controlPlane.image.repository }}:{{ .Values.controlPlane.image.tag | default .Values.version | default .Chart.AppVersion }}"
         imagePullPolicy: {{ .Values.controlPlane.image.pullPolicy }}
         ports:
         - name: http
@@ -71,7 +71,7 @@ spec:
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with (.Values.controlPlane.affinity | default .Values.affinity) }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}

helm/hindsight/templates/pdb.yaml

@@ -0,0 +1,56 @@
+{{- if and .Values.api.enabled .Values.api.podDisruptionBudget.enabled }}
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ include "hindsight.fullname" . }}-api
+  labels:
+    {{- include "hindsight.api.labels" . | nindent 4 }}
+spec:
+  {{- if .Values.api.podDisruptionBudget.minAvailable }}
+  minAvailable: {{ .Values.api.podDisruptionBudget.minAvailable }}
+  {{- end }}
+  {{- if .Values.api.podDisruptionBudget.maxUnavailable }}
+  maxUnavailable: {{ .Values.api.podDisruptionBudget.maxUnavailable }}
+  {{- end }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.api.selectorLabels" . | nindent 6 }}
+{{- end }}
+---
+{{- if and .Values.controlPlane.enabled .Values.controlPlane.podDisruptionBudget.enabled }}
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ include "hindsight.fullname" . }}-control-plane
+  labels:
+    {{- include "hindsight.controlPlane.labels" . | nindent 4 }}
+spec:
+  {{- if .Values.controlPlane.podDisruptionBudget.minAvailable }}
+  minAvailable: {{ .Values.controlPlane.podDisruptionBudget.minAvailable }}
+  {{- end }}
+  {{- if .Values.controlPlane.podDisruptionBudget.maxUnavailable }}
+  maxUnavailable: {{ .Values.controlPlane.podDisruptionBudget.maxUnavailable }}
+  {{- end }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.controlPlane.selectorLabels" . | nindent 6 }}
+{{- end }}
+---
+{{- if and .Values.worker.enabled .Values.worker.podDisruptionBudget.enabled }}
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: {{ include "hindsight.fullname" . }}-worker
+  labels:
+    {{- include "hindsight.worker.labels" . | nindent 4 }}
+spec:
+  {{- if .Values.worker.podDisruptionBudget.minAvailable }}
+  minAvailable: {{ .Values.worker.podDisruptionBudget.minAvailable }}
+  {{- end }}
+  {{- if .Values.worker.podDisruptionBudget.maxUnavailable }}
+  maxUnavailable: {{ .Values.worker.podDisruptionBudget.maxUnavailable }}
+  {{- end }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.worker.selectorLabels" . | nindent 6 }}
+{{- end }}

helm/hindsight/templates/tei-embedding-deployment.yaml

@@ -0,0 +1,76 @@
+{{- if .Values.tei.embedding.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "hindsight.fullname" . }}-tei-embedding
+  labels:
+    {{- include "hindsight.tei.embedding.labels" . | nindent 4 }}
+spec:
+  replicas: {{ .Values.tei.embedding.replicaCount }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.tei.embedding.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      {{- with .Values.podAnnotations }}
+      annotations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      labels:
+        {{- include "hindsight.tei.embedding.selectorLabels" . | nindent 8 }}
+    spec:
+      {{- if .Values.serviceAccount.create }}
+      serviceAccountName: {{ include "hindsight.serviceAccountName" . }}
+      {{- end }}
+      securityContext:
+        {{- toYaml .Values.podSecurityContext | nindent 8 }}
+      containers:
+      - name: tei-embedding
+        securityContext:
+          {{- toYaml .Values.securityContext | nindent 10 }}
+        image: "{{ .Values.tei.embedding.image.repository }}:{{ .Values.tei.embedding.image.tag }}"
+        imagePullPolicy: {{ .Values.tei.embedding.image.pullPolicy }}
+        args:
+          - "--model-id"
+          - {{ .Values.tei.embedding.model | quote }}
+          - "--hostname"
+          - "0.0.0.0"
+          {{- range .Values.tei.embedding.args }}
+          - {{ . | quote }}
+          {{- end }}
+        ports:
+        - name: http
+          containerPort: {{ .Values.tei.embedding.port }}
+          protocol: TCP
+        env:
+        - name: PORT
+          value: {{ .Values.tei.embedding.port | quote }}
+        {{- range $key, $value := .Values.tei.embedding.env }}
+        - name: {{ $key }}
+          value: {{ $value | quote }}
+        {{- end }}
+        livenessProbe:
+          {{- toYaml .Values.tei.embedding.livenessProbe | nindent 10 }}
+        readinessProbe:
+          {{- toYaml .Values.tei.embedding.readinessProbe | nindent 10 }}
+        resources:
+          {{- toYaml .Values.tei.embedding.resources | nindent 10 }}
+        volumeMounts:
+        - name: model-cache
+          mountPath: /data
+      volumes:
+      - name: model-cache
+        emptyDir: {}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+{{- end }}

helm/hindsight/templates/tei-embedding-service.yaml

@@ -0,0 +1,17 @@
+{{- if .Values.tei.embedding.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "hindsight.fullname" . }}-tei-embedding
+  labels:
+    {{- include "hindsight.tei.embedding.labels" . | nindent 4 }}
+spec:
+  type: ClusterIP
+  ports:
+  - port: {{ .Values.tei.embedding.port }}
+    targetPort: http
+    protocol: TCP
+    name: http
+  selector:
+    {{- include "hindsight.tei.embedding.selectorLabels" . | nindent 4 }}
+{{- end }}

helm/hindsight/templates/tei-reranker-deployment.yaml

@@ -0,0 +1,76 @@
+{{- if .Values.tei.reranker.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "hindsight.fullname" . }}-tei-reranker
+  labels:
+    {{- include "hindsight.tei.reranker.labels" . | nindent 4 }}
+spec:
+  replicas: {{ .Values.tei.reranker.replicaCount }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.tei.reranker.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      {{- with .Values.podAnnotations }}
+      annotations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      labels:
+        {{- include "hindsight.tei.reranker.selectorLabels" . | nindent 8 }}
+    spec:
+      {{- if .Values.serviceAccount.create }}
+      serviceAccountName: {{ include "hindsight.serviceAccountName" . }}
+      {{- end }}
+      securityContext:
+        {{- toYaml .Values.podSecurityContext | nindent 8 }}
+      containers:
+      - name: tei-reranker
+        securityContext:
+          {{- toYaml .Values.securityContext | nindent 10 }}
+        image: "{{ .Values.tei.reranker.image.repository }}:{{ .Values.tei.reranker.image.tag }}"
+        imagePullPolicy: {{ .Values.tei.reranker.image.pullPolicy }}
+        args:
+          - "--model-id"
+          - {{ .Values.tei.reranker.model | quote }}
+          - "--hostname"
+          - "0.0.0.0"
+          {{- range .Values.tei.reranker.args }}
+          - {{ . | quote }}
+          {{- end }}
+        ports:
+        - name: http
+          containerPort: {{ .Values.tei.reranker.port }}
+          protocol: TCP
+        env:
+        - name: PORT
+          value: {{ .Values.tei.reranker.port | quote }}
+        {{- range $key, $value := .Values.tei.reranker.env }}
+        - name: {{ $key }}
+          value: {{ $value | quote }}
+        {{- end }}
+        livenessProbe:
+          {{- toYaml .Values.tei.reranker.livenessProbe | nindent 10 }}
+        readinessProbe:
+          {{- toYaml .Values.tei.reranker.readinessProbe | nindent 10 }}
+        resources:
+          {{- toYaml .Values.tei.reranker.resources | nindent 10 }}
+        volumeMounts:
+        - name: model-cache
+          mountPath: /data
+      volumes:
+      - name: model-cache
+        emptyDir: {}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+{{- end }}

helm/hindsight/templates/tei-reranker-service.yaml

@@ -0,0 +1,17 @@
+{{- if .Values.tei.reranker.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "hindsight.fullname" . }}-tei-reranker
+  labels:
+    {{- include "hindsight.tei.reranker.labels" . | nindent 4 }}
+spec:
+  type: ClusterIP
+  ports:
+  - port: {{ .Values.tei.reranker.port }}
+    targetPort: http
+    protocol: TCP
+    name: http
+  selector:
+    {{- include "hindsight.tei.reranker.selectorLabels" . | nindent 4 }}
+{{- end }}

helm/hindsight/templates/worker-statefulset.yaml

@@ -32,7 +32,7 @@ spec:
       - name: worker
         securityContext:
           {{- toYaml .Values.securityContext | nindent 10 }}
-        image: "{{ .Values.worker.image.repository }}:{{ .Values.worker.image.tag | default .Values.version }}"
+        image: "{{ .Values.worker.image.repository }}:{{ .Values.worker.image.tag | default .Values.version | default .Chart.AppVersion }}"
         imagePullPolicy: {{ .Values.worker.image.pullPolicy }}
         command: ["hindsight-worker"]
         ports:
@@ -95,11 +95,21 @@ spec:
           {{- toYaml .Values.worker.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.worker.resources | nindent 10 }}
+        {{- if or .Values.worker.persistence.modelCache.enabled .Values.worker.extraVolumeMounts }}
+        volumeMounts:
+        {{- if .Values.worker.persistence.modelCache.enabled }}
+        - name: model-cache
+          mountPath: /home/hindsight/.cache
+        {{- end }}
+        {{- with .Values.worker.extraVolumeMounts }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+        {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with (.Values.worker.affinity | default .Values.affinity) }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}
@@ -107,4 +117,26 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- with .Values.worker.extraVolumes }}
+      volumes:
+      {{- toYaml . | nindent 6 }}
+      {{- end }}
+  {{- if .Values.worker.persistence.modelCache.enabled }}
+  volumeClaimTemplates:
+  - metadata:
+      name: model-cache
+      {{- with .Values.worker.persistence.modelCache.annotations }}
+      annotations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+    spec:
+      accessModes:
+        {{- toYaml .Values.worker.persistence.modelCache.accessModes | nindent 8 }}
+      {{- if .Values.worker.persistence.modelCache.storageClass }}
+      storageClassName: {{ .Values.worker.persistence.modelCache.storageClass }}
+      {{- end }}
+      resources:
+        requests:
+          storage: {{ .Values.worker.persistence.modelCache.size }}
+  {{- end }}
 {{- end }}

helm/hindsight/values.yaml

@@ -1,7 +1,8 @@
 # Default values for hindsight
 
-# Chart version - use this to set a consistent image tag across all components
-version: "0.1.1"
+# Global version override - use this to set a consistent image tag across all components
+# If not set, defaults to Chart.appVersion from Chart.yaml
+# version: ""
 
 # Use an existing secret instead of creating one from values
 # When set, all keys from this secret are injected as environment variables via envFrom
@@ -57,6 +58,48 @@ api:
     timeoutSeconds: 3
     failureThreshold: 3
 
+  # Pod disruption budget
+  podDisruptionBudget:
+    enabled: false
+    minAvailable: 1
+    # maxUnavailable: 1
+
+  # Pod affinity/anti-affinity (overrides global affinity for this component)
+  # affinity: {}
+
+  # Persistent volume for local model cache (reranker, embeddings)
+  # Models are downloaded to /home/hindsight/.cache on first use.
+  # Without persistence, models are re-downloaded on every pod restart.
+  #
+  # For production, prefer baking models into a custom image instead of
+  # enabling this PVC: image layers are pulled once per node and cached
+  # for free, while a PVC adds storage cost, pins pods to a node
+  # (ReadWriteOnce), and needs lifecycle management on uninstall/upgrade.
+  # See docs: developer/installation#bundling-custom-models-in-a-custom-image
+  persistence:
+    modelCache:
+      enabled: false
+      size: 5Gi
+      storageClass: ""
+      accessModes:
+        - ReadWriteOnce
+      annotations: {}
+
+  # Extra volume mounts for the api container
+  # e.g.
+  # extraVolumeMounts:
+  #   - name: my-volume
+  #     mountPath: /mnt/my-volume
+  extraVolumeMounts: []
+
+  # Extra volumes for the api pod
+  # e.g.
+  # extraVolumes:
+  #   - name: my-volume
+  #     configMap:
+  #       name: my-configmap
+  extraVolumes: []
+
   # Environment variables
   env:
     #HINDSIGHT_API_LLM_PROVIDER: "groq"
@@ -75,7 +118,7 @@ worker:
   image:
     repository: ghcr.io/vectorize-io/hindsight-api
     pullPolicy: IfNotPresent
-    # tag defaults to .Values.version if not specified
+    # tag: ""  # defaults to .Values.version, then Chart.appVersion if not specified
 
   service:
     # Service for metrics scraping (headless for StatefulSet)
@@ -121,6 +164,44 @@ worker:
     # HTTP port for metrics/health (matches service.targetPort)
     HINDSIGHT_API_WORKER_HTTP_PORT: "8889"
 
+  # Pod disruption budget
+  podDisruptionBudget:
+    enabled: false
+    minAvailable: 1
+    # maxUnavailable: 1
+
+  # Pod affinity/anti-affinity (overrides global affinity for this component)
+  # affinity: {}
+
+  # Persistent volume for local model cache (reranker, embeddings)
+  # Uses volumeClaimTemplates since worker is a StatefulSet — one PVC per
+  # replica. For production, prefer baking models into a custom image; see
+  # api.persistence.modelCache above and docs:
+  # developer/installation#bundling-custom-models-in-a-custom-image
+  persistence:
+    modelCache:
+      enabled: false
+      size: 5Gi
+      storageClass: ""
+      accessModes:
+        - ReadWriteOnce
+      annotations: {}
+
+  # Extra volume mounts for the worker container
+  # e.g.
+  # extraVolumeMounts:
+  #   - name: my-volume
+  #     mountPath: /mnt/my-volume
+  extraVolumeMounts: []
+
+  # Extra volumes for the worker pod
+  # e.g.
+  # extraVolumes:
+  #   - name: my-volume
+  #     configMap:
+  #       name: my-configmap
+  extraVolumes: []
+
   # Secret environment variables (inherited from api.secrets if not specified)
   secrets: {}
 
@@ -164,6 +245,15 @@ controlPlane:
     timeoutSeconds: 3
     failureThreshold: 3
 
+  # Pod disruption budget
+  podDisruptionBudget:
+    enabled: false
+    minAvailable: 1
+    # maxUnavailable: 1
+
+  # Pod affinity/anti-affinity (overrides global affinity for this component)
+  # affinity: {}
+
   # Environment variables
   env:
     NODE_ENV: "production"
@@ -262,9 +352,87 @@ nodeSelector: {}
 # Tolerations
 tolerations: []
 
-# Affinity
+# Affinity (applied to all components unless overridden per-component)
 affinity: {}
 
+# TEI (Text Embeddings Inference) - optional standalone deployments
+# for reranking and/or embedding models
+tei:
+  reranker:
+    enabled: false
+    replicaCount: 1
+    image:
+      repository: ghcr.io/huggingface/text-embeddings-inference
+      tag: cpu-1.8.3
+      pullPolicy: IfNotPresent
+    model: "cross-encoder/ms-marco-MiniLM-L-6-v2"
+    port: 8090
+    args:
+      - "--auto-truncate"
+    env:
+      PAYLOAD_LIMIT: "10000000"
+      MAX_CLIENT_BATCH_SIZE: "256"
+    resources:
+      limits:
+        cpu: 2000m
+        memory: 2Gi
+      requests:
+        cpu: 500m
+        memory: 1Gi
+    livenessProbe:
+      httpGet:
+        path: /health
+        port: 8090
+      initialDelaySeconds: 30
+      periodSeconds: 10
+      timeoutSeconds: 5
+      failureThreshold: 6
+    readinessProbe:
+      httpGet:
+        path: /health
+        port: 8090
+      initialDelaySeconds: 15
+      periodSeconds: 5
+      timeoutSeconds: 3
+      failureThreshold: 3
+
+  embedding:
+    enabled: false
+    replicaCount: 1
+    image:
+      repository: ghcr.io/huggingface/text-embeddings-inference
+      tag: cpu-1.8.3
+      pullPolicy: IfNotPresent
+    model: "sentence-transformers/all-MiniLM-L6-v2"
+    port: 8091
+    args: []
+    env:
+      PAYLOAD_LIMIT: "10000000"
+      MAX_CLIENT_BATCH_SIZE: "256"
+    resources:
+      limits:
+        cpu: 2000m
+        memory: 2Gi
+      requests:
+        cpu: 500m
+        memory: 1Gi
+    livenessProbe:
+      httpGet:
+        path: /health
+        port: 8091
+      initialDelaySeconds: 30
+      periodSeconds: 10
+      timeoutSeconds: 5
+      failureThreshold: 6
+    readinessProbe:
+      httpGet:
+        path: /health
+        port: 8091
+      initialDelaySeconds: 15
+      periodSeconds: 5
+      timeoutSeconds: 3
+      failureThreshold: 3
+
 # Autoscaling
 autoscaling:
   enabled: false

hindsight-all-npm/.gitignore

@@ -0,0 +1,4 @@
+node_modules
+dist
+*.tgz
+.DS_Store

hindsight-all-npm/README.md

@@ -0,0 +1,80 @@
+# @vectorize-io/hindsight-all
+
+Node.js equivalent of the Python [`hindsight-all`](https://pypi.org/project/hindsight-all/) package — programmatic lifecycle manager for a local Hindsight daemon. Use this when you want to embed Hindsight in a Node application without hand-rolling subprocess management.
+
+This package deliberately does **not** ship an HTTP client. Once the daemon is running, talk to it with [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client) against `server.getBaseUrl()`. The two packages compose — one owns the daemon process, the other owns the HTTP API surface.
+
+## Requirements
+
+- **Node.js >= 22** — uses global `fetch` and `AbortSignal.timeout`.
+- **`uv` / `uvx`** on `PATH` — used to download and run the underlying `hindsight-embed` daemon on first use. Install via <https://docs.astral.sh/uv/>.
+
+## Install
+
+```bash
+npm install @vectorize-io/hindsight-all @vectorize-io/hindsight-client
+```
+
+## Example
+
+```ts
+import { HindsightServer, consoleLogger } from "@vectorize-io/hindsight-all";
+import { HindsightClient } from "@vectorize-io/hindsight-client";
+
+const server = new HindsightServer({
+  profile: "my-app",
+  port: 9077,
+  env: {
+    HINDSIGHT_API_LLM_PROVIDER: "anthropic",
+    HINDSIGHT_API_LLM_API_KEY: process.env.ANTHROPIC_API_KEY,
+    HINDSIGHT_API_LLM_MODEL: "claude-sonnet-4-20250514",
+    HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT: "0",
+  },
+  logger: consoleLogger,
+});
+
+await server.start();
+
+const client = new HindsightClient({ baseUrl: server.getBaseUrl() });
+
+await client.retain("user-123", "User prefers dark mode and concise answers.", {
+  documentId: "pref-2026-04-01",
+});
+
+const recall = await client.recall("user-123", "what are the user preferences?");
+console.log(recall.results);
+
+await server.stop();
+```
+
+For a remote Hindsight API, skip `HindsightServer` entirely and just point `HindsightClient` at the remote URL.
+
+## Open config — forward-compatible with new daemon flags
+
+`HindsightServerOptions` is designed so every new environment variable or CLI flag in the underlying Hindsight daemon can be used without waiting for a wrapper release:
+
+- **`env`** accepts an arbitrary `Record<string, string>`. Every entry is exported into the daemon process and written into the profile config via `--env KEY=VALUE`.
+- **`extraProfileCreateArgs`** / **`extraDaemonStartArgs`** append raw args to the respective commands.
+
+## Development against a local checkout
+
+If you're hacking on the Python `hindsight-embed` package in the same monorepo, point the server at the local path — it'll use `uv run --directory <path>` instead of `uvx`:
+
+```ts
+new HindsightServer({
+  embedPackagePath: "/path/to/hindsight-embed",
+  // ...
+});
+```
+
+## API surface
+
+- `HindsightServer` — daemon lifecycle (`start`, `stop`, `checkHealth`, `getBaseUrl`, `getProfile`).
+- `Logger` interface plus `silentLogger` (default) and `consoleLogger` helpers.
+- `getEmbedCommand(opts)` — low-level helper that returns the `[cmd, ...args]` tuple used to invoke the underlying Python CLI.
+
+For memory operations (retain, recall, reflect, bank management, stats) use [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client).
+
+## License
+
+MIT

hindsight-all-npm/package.json

@@ -0,0 +1,57 @@
+{
+  "name": "@vectorize-io/hindsight-all",
+  "version": "0.7.2",
+  "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "type": "module",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    }
+  },
+  "keywords": [
+    "hindsight",
+    "hindsight-all",
+    "memory",
+    "ai",
+    "agent",
+    "long-term-memory",
+    "llm",
+    "embedded-server"
+  ],
+  "author": "Vectorize <[email protected]>",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/vectorize-io/hindsight.git",
+    "directory": "hindsight-all-npm"
+  },
+  "files": [
+    "dist",
+    "README.md"
+  ],
+  "scripts": {
+    "build": "tsup",
+    "dev": "tsup --watch",
+    "clean": "rm -rf dist",
+    "test": "vitest run src",
+    "test:watch": "vitest src",
+    "prepublishOnly": "npm run clean && npm run build"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "tsup": "^8.5.1",
+    "typescript": "^5.7.0",
+    "vitest": "^4.1.2"
+  },
+  "engines": {
+    "node": ">=22"
+  },
+  "overrides": {
+    "rollup": "^4.59.0",
+    "picomatch": ">=2.3.2 <3.0.0 || >=4.0.4",
+    "vite": ">=8.0.5"
+  }
+}

hindsight-all-npm/src/command.test.ts

@@ -0,0 +1,36 @@
+import { describe, it, expect } from "vitest";
+import { getEmbedCommand } from "./command.js";
+
+describe("getEmbedCommand", () => {
+  it("defaults to uvx hindsight-embed@latest", () => {
+    expect(getEmbedCommand()).toEqual(["uvx", "hindsight-embed@latest"]);
+  });
+
+  it("honours an explicit version", () => {
+    expect(getEmbedCommand({ embedVersion: "0.5.0" })).toEqual(["uvx", "[email protected]"]);
+  });
+
+  it("treats an empty version as latest", () => {
+    expect(getEmbedCommand({ embedVersion: "" })).toEqual(["uvx", "hindsight-embed@latest"]);
+  });
+
+  it("uses uv run --directory when a local path is given", () => {
+    expect(getEmbedCommand({ embedPackagePath: "/abs/path" })).toEqual([
+      "uv",
+      "run",
+      "--directory",
+      "/abs/path",
+      "hindsight-embed",
+    ]);
+  });
+
+  it("local path takes precedence over version", () => {
+    expect(getEmbedCommand({ embedPackagePath: "/abs/path", embedVersion: "0.5.0" })).toEqual([
+      "uv",
+      "run",
+      "--directory",
+      "/abs/path",
+      "hindsight-embed",
+    ]);
+  });
+});

hindsight-all-npm/src/command.ts

@@ -0,0 +1,25 @@
+/**
+ * Resolve the command that invokes the `hindsight-embed` Python CLI.
+ *
+ * - If `embedPackagePath` is set, runs the package from a local checkout via
+ *   `uv run --directory <path> hindsight-embed`. Used for in-repo development.
+ * - Otherwise runs it via `uvx hindsight-embed@<version>` so no global install
+ *   is required.
+ *
+ * Returns the argv as `[command, ...baseArgs]` suitable for `spawn()` /
+ * `execFile()` (never shell-interpolated).
+ */
+export interface EmbedCommandOptions {
+  /** Version spec passed to uvx (e.g. "latest", "0.5.0"). Default: "latest". */
+  embedVersion?: string;
+  /** Local checkout path. When set, overrides `embedVersion` and uses `uv run`. */
+  embedPackagePath?: string;
+}
+
+export function getEmbedCommand(opts: EmbedCommandOptions = {}): string[] {
+  if (opts.embedPackagePath) {
+    return ["uv", "run", "--directory", opts.embedPackagePath, "hindsight-embed"];
+  }
+  const version = opts.embedVersion && opts.embedVersion.length > 0 ? opts.embedVersion : "latest";
+  return ["uvx", `hindsight-embed@${version}`];
+}

hindsight-all-npm/src/index.ts

@@ -0,0 +1,7 @@
+export { HindsightServer } from "./server.js";
+export { getEmbedCommand } from "./command.js";
+export { silentLogger, consoleLogger } from "./logger.js";
+
+export type { Logger } from "./logger.js";
+export type { EmbedCommandOptions } from "./command.js";
+export type { HindsightServerOptions } from "./types.js";

hindsight-all-npm/src/logger.ts

@@ -0,0 +1,29 @@
+/**
+ * Pluggable logger interface.
+ *
+ * This package does not own any logging infrastructure — consumers inject
+ * whatever they want (console, pino, openclaw's logger, a no-op). The default
+ * is silent so embedding this package never adds noise to an unrelated app.
+ */
+export interface Logger {
+  debug(msg: string): void;
+  info(msg: string): void;
+  warn(msg: string): void;
+  error(msg: string): void;
+}
+
+/** Logger that drops every call. Used when no logger is passed. */
+export const silentLogger: Logger = {
+  debug: () => {},
+  info: () => {},
+  warn: () => {},
+  error: () => {},
+};
+
+/** Logger that writes to the standard console. Handy for CLIs and tests. */
+export const consoleLogger: Logger = {
+  debug: (msg) => console.debug(msg),
+  info: (msg) => console.log(msg),
+  warn: (msg) => console.warn(msg),
+  error: (msg) => console.error(msg),
+};

hindsight-all-npm/src/server.test.ts

@@ -0,0 +1,35 @@
+import { describe, it, expect } from "vitest";
+import { HindsightServer } from "./server.js";
+
+describe("HindsightServer construction", () => {
+  it("defaults base URL to http://127.0.0.1:8888", () => {
+    const server = new HindsightServer();
+    expect(server.getBaseUrl()).toBe("http://127.0.0.1:8888");
+    expect(server.getProfile()).toBe("default");
+  });
+
+  it("honours custom profile, port, and host", () => {
+    const server = new HindsightServer({ profile: "app", port: 9077, host: "0.0.0.0" });
+    expect(server.getProfile()).toBe("app");
+    expect(server.getBaseUrl()).toBe("http://0.0.0.0:9077");
+  });
+
+  it("accepts open env pass-through without complaining about unknown keys", () => {
+    const server = new HindsightServer({
+      env: {
+        HINDSIGHT_API_LLM_PROVIDER: "openai",
+        HINDSIGHT_API_LLM_MODEL: "gpt-4o-mini",
+        // A field that does not exist today — should still be accepted
+        HINDSIGHT_FUTURE_FLAG: "enabled",
+      },
+    });
+    expect(server).toBeInstanceOf(HindsightServer);
+  });
+
+  it("exposes checkHealth that returns false when no daemon is running", async () => {
+    // Random high port that nothing is listening on.
+    const server = new HindsightServer({ port: 1, readyTimeoutMs: 100 });
+    const healthy = await server.checkHealth();
+    expect(healthy).toBe(false);
+  });
+});

hindsight-all-npm/src/server.ts

@@ -0,0 +1,322 @@
+import { spawn } from "child_process";
+import { getEmbedCommand } from "./command.js";
+import { silentLogger } from "./logger.js";
+import type { Logger } from "./logger.js";
+import type { HindsightServerOptions } from "./types.js";
+
+const DEFAULT_PORT = 8888;
+const DEFAULT_HOST = "127.0.0.1";
+const DEFAULT_PROFILE = "default";
+const DEFAULT_READY_TIMEOUT_MS = 30_000;
+const DEFAULT_READY_POLL_INTERVAL_MS = 1_000;
+
+/**
+ * Manages the lifecycle of a local Hindsight daemon from a Node.js process.
+ *
+ * On {@link start}, this class:
+ *   1. Resolves the `hindsight-embed` command (via `uvx` or a local `uv run`).
+ *   2. Runs `profile create <name> --merge --port <port> [--env K=V ...]`
+ *      with every entry in {@link HindsightServerOptions.env} forwarded as
+ *      an `--env` flag.
+ *   3. Runs `daemon --profile <name> start` and waits for the start command
+ *      to exit.
+ *   4. Polls `http://host:port/health` until it returns `200` or the
+ *      `readyTimeoutMs` budget is exhausted.
+ *
+ * On {@link stop}, it runs `daemon --profile <name> stop` and returns once
+ * the command exits (or after a short grace period).
+ *
+ * This is the Node.js equivalent of the Python `hindsight-all` package's
+ * `HindsightServer`: a thin programmatic lifecycle wrapper around the
+ * Hindsight daemon. It does NOT ship an HTTP client — once `start()`
+ * resolves, use `@vectorize-io/hindsight-client` against `getBaseUrl()` for
+ * retain / recall / reflect.
+ *
+ * The class is deliberately transparent about the daemon: new CLI flags or
+ * environment variables never require a code change here — callers can pass
+ * them via `env`, `extraProfileCreateArgs`, or `extraDaemonStartArgs`.
+ */
+export class HindsightServer {
+  private readonly profile: string;
+  private readonly port: number;
+  private readonly host: string;
+  private readonly baseUrl: string;
+  private readonly embedVersion: string | undefined;
+  private readonly embedPackagePath: string | undefined;
+  private readonly userEnv: Record<string, string | undefined>;
+  private readonly extraProfileCreateArgs: string[];
+  private readonly extraDaemonStartArgs: string[];
+  private readonly platformCpuWorkaround: boolean;
+  private readonly readyTimeoutMs: number;
+  private readonly readyPollIntervalMs: number;
+  private readonly logger: Logger;
+
+  constructor(opts: HindsightServerOptions = {}) {
+    this.profile = opts.profile ?? DEFAULT_PROFILE;
+    this.port = opts.port ?? DEFAULT_PORT;
+    this.host = opts.host ?? DEFAULT_HOST;
+    this.baseUrl = `http://${this.host}:${this.port}`;
+    this.embedVersion = opts.embedVersion;
+    this.embedPackagePath = opts.embedPackagePath;
+    this.userEnv = opts.env ?? {};
+    this.extraProfileCreateArgs = opts.extraProfileCreateArgs ?? [];
+    this.extraDaemonStartArgs = opts.extraDaemonStartArgs ?? [];
+    this.platformCpuWorkaround = opts.platformCpuWorkaround ?? process.platform === "darwin";
+    this.readyTimeoutMs = opts.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
+    this.readyPollIntervalMs = opts.readyPollIntervalMs ?? DEFAULT_READY_POLL_INTERVAL_MS;
+    this.logger = opts.logger ?? silentLogger;
+  }
+
+  /** The base URL the daemon listens on (`http://host:port`). */
+  getBaseUrl(): string {
+    return this.baseUrl;
+  }
+
+  /** The profile name this server operates on. */
+  getProfile(): string {
+    return this.profile;
+  }
+
+  /**
+   * Ensure the daemon is configured and running. Idempotent — the underlying
+   * `profile create --merge` and `daemon start` commands tolerate re-runs.
+   */
+  async start(): Promise<void> {
+    this.logger.info(`[hindsight] starting daemon for profile "${this.profile}"`);
+
+    const env = this.buildEnv();
+    await this.configureProfile(env);
+    await this.startDaemon(env);
+    await this.waitForReady();
+
+    this.logger.info(`[hindsight] daemon ready at ${this.baseUrl}`);
+  }
+
+  /** Stop the daemon. Never throws — logs and resolves even on failure. */
+  async stop(): Promise<void> {
+    this.logger.info(`[hindsight] stopping daemon for profile "${this.profile}"`);
+
+    const [cmd, ...baseArgs] = getEmbedCommand({
+      embedVersion: this.embedVersion,
+      embedPackagePath: this.embedPackagePath,
+    });
+    const args = [...baseArgs, "daemon", "--profile", this.profile, "stop"];
+
+    const child = spawn(cmd, args, { stdio: "pipe" });
+    this.pipeOutput(child, "daemon.stop");
+
+    await new Promise<void>((resolve) => {
+      const timeout = setTimeout(() => {
+        this.logger.warn(`[hindsight] daemon stop timed out after 5s`);
+        resolve();
+      }, 5_000);
+      child.on("exit", () => {
+        clearTimeout(timeout);
+        this.logger.info(`[hindsight] daemon stopped`);
+        resolve();
+      });
+      child.on("error", (err) => {
+        clearTimeout(timeout);
+        this.logger.warn(`[hindsight] error stopping daemon: ${err.message}`);
+        resolve();
+      });
+    });
+  }
+
+  /** Probe `/health` once with a short timeout. */
+  async checkHealth(): Promise<boolean> {
+    try {
+      const res = await fetch(`${this.baseUrl}/health`, {
+        signal: AbortSignal.timeout(2_000),
+      });
+      return res.ok;
+    } catch {
+      return false;
+    }
+  }
+
+  // -------------------------------------------------------------------------
+  // Internal
+  // -------------------------------------------------------------------------
+
+  /**
+   * Merge the process env, the caller-supplied `env`, and (on macOS) the
+   * embeddings CPU workaround. Caller-supplied values always win over the
+   * workaround; undefined values are dropped.
+   */
+  private buildEnv(): NodeJS.ProcessEnv {
+    const merged: NodeJS.ProcessEnv = { ...process.env };
+
+    if (this.platformCpuWorkaround && process.platform === "darwin") {
+      merged["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1";
+      merged["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1";
+    }
+
+    for (const [key, value] of Object.entries(this.userEnv)) {
+      if (value !== undefined) {
+        merged[key] = value;
+      }
+    }
+
+    return merged;
+  }
+
+  /**
+   * Run `profile create <name> --merge --port <port> [--env K=V ...]`.
+   * Every entry in the merged env that was passed via {@link userEnv} (or
+   * auto-applied by the CPU workaround) is forwarded as `--env`.
+   */
+  private async configureProfile(env: NodeJS.ProcessEnv): Promise<void> {
+    this.logger.info(`[hindsight] configuring profile "${this.profile}"`);
+
+    const [cmd, ...baseArgs] = getEmbedCommand({
+      embedVersion: this.embedVersion,
+      embedPackagePath: this.embedPackagePath,
+    });
+    const createArgs = [
+      ...baseArgs,
+      "profile",
+      "create",
+      this.profile,
+      "--merge",
+      "--port",
+      String(this.port),
+    ];
+
+    // Forward every env var that the caller intended for the daemon as --env.
+    // We only forward keys the caller explicitly set (userEnv) plus the CPU
+    // workaround values — not the entire process.env, to avoid leaking random
+    // host state into profile config.
+    const envForProfile = this.collectProfileEnv(env);
+    for (const [key, value] of Object.entries(envForProfile)) {
+      createArgs.push("--env", `${key}=${value}`);
+    }
+
+    createArgs.push(...this.extraProfileCreateArgs);
+
+    await this.runCommand(cmd, createArgs, env, "profile.create");
+  }
+
+  /** Collect only the env vars that should be written into the profile file. */
+  private collectProfileEnv(env: NodeJS.ProcessEnv): Record<string, string> {
+    const out: Record<string, string> = {};
+
+    // 1. User-supplied env — always forwarded.
+    for (const [key, value] of Object.entries(this.userEnv)) {
+      if (value !== undefined) {
+        out[key] = value;
+      }
+    }
+
+    // 2. CPU workaround — only if auto-applied and not already overridden.
+    if (this.platformCpuWorkaround && process.platform === "darwin") {
+      const cpuKeys = [
+        "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU",
+        "HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU",
+      ];
+      for (const key of cpuKeys) {
+        if (!(key in out) && env[key] !== undefined) {
+          out[key] = env[key] as string;
+        }
+      }
+    }
+
+    return out;
+  }
+
+  private async startDaemon(env: NodeJS.ProcessEnv): Promise<void> {
+    const [cmd, ...baseArgs] = getEmbedCommand({
+      embedVersion: this.embedVersion,
+      embedPackagePath: this.embedPackagePath,
+    });
+    const args = [
+      ...baseArgs,
+      "daemon",
+      "--profile",
+      this.profile,
+      "start",
+      ...this.extraDaemonStartArgs,
+    ];
+
+    await this.runCommand(cmd, args, env, "daemon.start");
+  }
+
+  /**
+   * Spawn `cmd` with `args`, pipe its output through the logger, and resolve
+   * once it exits with code 0. Rejects on non-zero exit or spawn error.
+   */
+  private async runCommand(
+    cmd: string,
+    args: string[],
+    env: NodeJS.ProcessEnv,
+    label: string
+  ): Promise<void> {
+    const child = spawn(cmd, args, { stdio: "pipe", env });
+    let output = "";
+    child.stdout?.on("data", (data: Buffer) => {
+      const text = data.toString();
+      output += text;
+      for (const line of text.trimEnd().split("\n")) {
+        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
+      }
+    });
+    child.stderr?.on("data", (data: Buffer) => {
+      const text = data.toString();
+      output += text;
+      for (const line of text.trimEnd().split("\n")) {
+        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
+      }
+    });
+
+    await new Promise<void>((resolve, reject) => {
+      child.on("exit", (code) => {
+        if (code === 0) {
+          resolve();
+        } else {
+          reject(new Error(`${label} failed with code ${code}: ${output.trim()}`));
+        }
+      });
+      child.on("error", (err) => {
+        reject(new Error(`${label} failed to spawn: ${err.message}`, { cause: err }));
+      });
+    });
+  }
+
+  /** Stream a spawned child's stdout/stderr through the logger without blocking. */
+  private pipeOutput(child: ReturnType<typeof spawn>, label: string): void {
+    child.stdout?.on("data", (data: Buffer) => {
+      for (const line of data.toString().trimEnd().split("\n")) {
+        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
+      }
+    });
+    child.stderr?.on("data", (data: Buffer) => {
+      for (const line of data.toString().trimEnd().split("\n")) {
+        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
+      }
+    });
+  }
+
+  /** Poll `/health` until it succeeds or `readyTimeoutMs` elapses. */
+  private async waitForReady(): Promise<void> {
+    const deadline = Date.now() + this.readyTimeoutMs;
+    let attempt = 0;
+    while (Date.now() < deadline) {
+      attempt++;
+      try {
+        const res = await fetch(`${this.baseUrl}/health`, {
+          signal: AbortSignal.timeout(this.readyPollIntervalMs),
+        });
+        if (res.ok) {
+          this.logger.debug(`[hindsight] health check passed (attempt ${attempt})`);
+          return;
+        }
+      } catch {
+        // expected while the daemon is still booting
+      }
+      await new Promise((resolve) => setTimeout(resolve, this.readyPollIntervalMs));
+    }
+    throw new Error(
+      `Hindsight daemon did not become ready within ${this.readyTimeoutMs}ms at ${this.baseUrl}`
+    );
+  }
+}

hindsight-all-npm/src/types.ts

@@ -0,0 +1,54 @@
+import type { Logger } from "./logger.js";
+
+/**
+ * Options for {@link HindsightServer}.
+ *
+ * The server is intentionally thin and pass-through: anything configurable
+ * on the daemon side (env vars or CLI flags) can be set here without needing
+ * a new dedicated option. Use {@link env} for `HINDSIGHT_*` / `OPENAI_API_KEY` /
+ * custom provider settings, and the two `extra*` arrays to append raw CLI
+ * args to `profile create` or `daemon start`.
+ *
+ * For talking to the daemon after `start()`, use `@vectorize-io/hindsight-client`
+ * against `server.getBaseUrl()`. This package does not ship its own HTTP
+ * client.
+ */
+export interface HindsightServerOptions {
+  /** Profile name used for `--profile <name>` on every sub-command. Default: `"default"`. */
+  profile?: string;
+  /** TCP port the daemon listens on. Default: `8888`. */
+  port?: number;
+  /** Hostname the daemon binds to (for health checks). Default: `127.0.0.1`. */
+  host?: string;
+  /** Version of the underlying `hindsight-embed` PyPI package to run via `uvx`. Default: `"latest"`. */
+  embedVersion?: string;
+  /** Local path to a `hindsight-embed` checkout — takes precedence over `embedVersion`. */
+  embedPackagePath?: string;
+  /**
+   * Environment variables passed to the daemon process AND written into the
+   * profile via repeated `--env KEY=VALUE` flags. This is the preferred way
+   * to surface any `HINDSIGHT_API_*` / `HINDSIGHT_EMBED_*` setting — adding a
+   * new daemon env var never requires a wrapper update.
+   *
+   * Values of `undefined` are dropped (so you can spread conditionally).
+   */
+  env?: Record<string, string | undefined>;
+  /** Extra args appended verbatim to `hindsight-embed profile create <name> --merge ...`. */
+  extraProfileCreateArgs?: string[];
+  /** Extra args appended verbatim to `hindsight-embed daemon --profile <name> start ...`. */
+  extraDaemonStartArgs?: string[];
+  /**
+   * On macOS, automatically set
+   * `HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1` and
+   * `HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1` to avoid Metal/MPS crashes in
+   * daemon mode. Default: `true` on `darwin`, ignored elsewhere. Any value set
+   * explicitly in {@link env} wins over the auto-applied value.
+   */
+  platformCpuWorkaround?: boolean;
+  /** Max time (ms) to wait for `/health` to return 200. Default: `30_000`. */
+  readyTimeoutMs?: number;
+  /** Polling interval (ms) while waiting for `/health`. Default: `1_000`. */
+  readyPollIntervalMs?: number;
+  /** Optional pluggable logger. Default: silent. */
+  logger?: Logger;
+}

hindsight-all-npm/tsconfig.json

@@ -0,0 +1,18 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "ES2022",
+    "lib": ["ES2022"],
+    "moduleResolution": "node",
+    "declaration": true,
+    "outDir": "./dist",
+    "rootDir": "./src",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "forceConsistentCasingInFileNames": true,
+    "resolveJsonModule": true
+  },
+  "include": ["src/**/*"],
+  "exclude": ["node_modules", "dist", "src/**/*.test.ts"]
+}

hindsight-all-npm/tsup.config.ts

@@ -0,0 +1,11 @@
+import { defineConfig } from "tsup";
+
+export default defineConfig({
+  entry: ["src/index.ts"],
+  format: ["esm"],
+  dts: true,
+  outDir: "dist",
+  clean: true,
+  sourcemap: true,
+  bundle: true,
+});

hindsight-all-npm/vitest.config.ts

@@ -0,0 +1,8 @@
+import { defineConfig } from "vitest/config";
+
+export default defineConfig({
+  test: {
+    include: ["src/**/*.test.ts"],
+    environment: "node",
+  },
+});

hindsight-all-slim/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hindsight-all-slim"
+version = "0.7.2"
+description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "hindsight-api-slim==0.7.2",
+    "hindsight-client>=0.0.7",
+    "hindsight-embed>=0.1.0",
+]
+
+[tool.uv.sources]
+hindsight-api-slim = { workspace = true }
+hindsight-client = { workspace = true }
+hindsight-embed = { workspace = true }
+
+[project.optional-dependencies]
+test = [
+    "pytest>=7.0.0",
+    "pytest-asyncio>=0.21.0",
+]
+
+[tool.setuptools]
+packages = []
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"

hindsight-all/README.md

@@ -0,0 +1,48 @@
+# hindsight-all
+
+All-in-one package for Hindsight - Agent Memory That Works Like Human Memory
+
+## Quick Start
+
+```python
+from hindsight import start_server, HindsightClient
+
+# Start server with embedded PostgreSQL
+server = start_server(
+    llm_provider="groq",
+    llm_api_key="your-api-key",
+    llm_model="openai/gpt-oss-120b"
+)
+
+# Create client
+client = HindsightClient(base_url=server.url)
+
+# Store memories
+client.put(agent_id="assistant", content="User prefers Python for data analysis")
+
+# Search memories
+results = client.search(agent_id="assistant", query="programming preferences")
+
+# Generate contextual response
+response = client.think(agent_id="assistant", query="What languages should I recommend?")
+
+# Stop server when done
+server.stop()
+```
+
+## Using Context Manager
+
+```python
+from hindsight import HindsightServer, HindsightClient
+
+with HindsightServer(llm_provider="groq", llm_api_key="...") as server:
+    client = HindsightClient(base_url=server.url)
+    # ... use client ...
+# Server automatically stops
+```
+
+## Installation
+
+```bash
+pip install hindsight-all
+```

hindsight-all/hindsight/__init__.py

@@ -3,7 +3,23 @@ Hindsight - All-in-one semantic memory system for AI agents.
 
 This package provides a simple way to run Hindsight locally with embedded PostgreSQL.
 
-Example:
+Easiest way - Embedded client (recommended):
+    ```python
+    from hindsight import HindsightEmbedded
+
+    # Server starts automatically on first use
+    client = HindsightEmbedded(
+        profile="myapp",
+        llm_provider="groq",
+        llm_api_key="your-api-key",
+    )
+
+    # Use immediately - no manual server management needed
+    client.retain(bank_id="alice", content="Alice loves AI")
+    results = client.recall(bank_id="alice", query="What does Alice like?")
+    ```
+
+Manual server management:
     ```python
     from hindsight import start_server, HindsightClient
 
@@ -18,13 +34,13 @@ Example:
     client = HindsightClient(base_url=server.url)
 
     # Store memories
-    client.put(agent_id="assistant", content="User prefers Python for data analysis")
+    client.retain(bank_id="assistant", content="User prefers Python for data analysis")
 
     # Search memories
-    results = client.search(agent_id="assistant", query="programming preferences")
+    results = client.recall(bank_id="assistant", query="programming preferences")
 
     # Generate contextual response
-    response = client.think(agent_id="assistant", query="What languages should I recommend?")
+    response = client.reflect(bank_id="assistant", query="What are my interests?")
 
     # Stop server when done
     server.stop()
@@ -41,13 +57,13 @@ Using context manager:
     ```
 """
 
+from .client_wrapper import HindsightClient
+from .embedded import HindsightEmbedded
 from .server import Server as HindsightServer, start_server
 
-# Re-export Client from hindsight-client
-from hindsight_client import Hindsight as HindsightClient
-
 __all__ = [
     "HindsightServer",
     "start_server",
     "HindsightClient",
+    "HindsightEmbedded",
 ]

hindsight-all/hindsight/api_namespaces.py

@@ -0,0 +1,193 @@
+"""
+API namespace classes for organizing client methods.
+
+These classes provide organized access to different parts of the Hindsight API
+while ensuring the daemon is running before each call.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    from .embedded import HindsightEmbedded
+
+
+class BanksAPI:
+    """Namespace for bank-related operations."""
+
+    def __init__(self, embedded: "HindsightEmbedded"):
+        self._embedded = embedded
+
+    def create(
+        self,
+        bank_id: str,
+        name: str | None = None,
+        mission: str | None = None,
+        disposition: dict[str, Any] | None = None,
+    ):
+        """Create a new bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.create_bank(
+            bank_id=bank_id,
+            name=name,
+            mission=mission,
+            disposition=disposition,
+        )
+
+    def delete(self, bank_id: str):
+        """Delete a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.delete_bank(bank_id=bank_id)
+
+    def set_mission(self, bank_id: str, mission: str):
+        """Set or update the mission for a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.set_mission(bank_id=bank_id, mission=mission)
+
+    def set_disposition(self, bank_id: str, disposition: dict[str, Any]):
+        """Set or update the disposition for a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.set_disposition(bank_id=bank_id, disposition=disposition)
+
+
+class MentalModelsAPI:
+    """Namespace for mental model operations."""
+
+    def __init__(self, embedded: "HindsightEmbedded"):
+        self._embedded = embedded
+
+    def create(
+        self,
+        bank_id: str,
+        name: str,
+        content: str,
+        tags: list[str] | None = None,
+    ):
+        """Create a new mental model."""
+        self._embedded._ensure_started()
+        return self._embedded._client.create_mental_model(
+            bank_id=bank_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def list(self, bank_id: str, tags: list[str] | None = None):
+        """List all mental models for a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.list_mental_models(bank_id=bank_id, tags=tags)
+
+    def get(self, bank_id: str, mental_model_id: str):
+        """Get a specific mental model."""
+        self._embedded._ensure_started()
+        return self._embedded._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+    def refresh(self, bank_id: str, mental_model_id: str):
+        """Refresh a mental model."""
+        self._embedded._ensure_started()
+        return self._embedded._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+    def update(
+        self,
+        bank_id: str,
+        mental_model_id: str,
+        name: str | None = None,
+        content: str | None = None,
+        tags: list[str] | None = None,
+    ):
+        """Update a mental model."""
+        self._embedded._ensure_started()
+        return self._embedded._client.update_mental_model(
+            bank_id=bank_id,
+            mental_model_id=mental_model_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def delete(self, bank_id: str, mental_model_id: str):
+        """Delete a mental model."""
+        self._embedded._ensure_started()
+        return self._embedded._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+
+class DirectivesAPI:
+    """Namespace for directive operations."""
+
+    def __init__(self, embedded: "HindsightEmbedded"):
+        self._embedded = embedded
+
+    def create(
+        self,
+        bank_id: str,
+        name: str,
+        content: str,
+        tags: list[str] | None = None,
+    ):
+        """Create a new directive."""
+        self._embedded._ensure_started()
+        return self._embedded._client.create_directive(
+            bank_id=bank_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def list(self, bank_id: str, tags: list[str] | None = None):
+        """List all directives for a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.list_directives(bank_id=bank_id, tags=tags)
+
+    def get(self, bank_id: str, directive_id: str):
+        """Get a specific directive."""
+        self._embedded._ensure_started()
+        return self._embedded._client.get_directive(bank_id=bank_id, directive_id=directive_id)
+
+    def update(
+        self,
+        bank_id: str,
+        directive_id: str,
+        name: str | None = None,
+        content: str | None = None,
+        tags: list[str] | None = None,
+    ):
+        """Update a directive."""
+        self._embedded._ensure_started()
+        return self._embedded._client.update_directive(
+            bank_id=bank_id,
+            directive_id=directive_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def delete(self, bank_id: str, directive_id: str):
+        """Delete a directive."""
+        self._embedded._ensure_started()
+        return self._embedded._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
+
+
+class MemoriesAPI:
+    """Namespace for memory operations."""
+
+    def __init__(self, embedded: "HindsightEmbedded"):
+        self._embedded = embedded
+
+    def list(
+        self,
+        bank_id: str,
+        type: str | None = None,
+        search_query: str | None = None,
+        limit: int = 100,
+        offset: int = 0,
+    ):
+        """List memories in a bank."""
+        self._embedded._ensure_started()
+        return self._embedded._client.list_memories(
+            bank_id=bank_id,
+            type=type,
+            search_query=search_query,
+            limit=limit,
+            offset=offset,
+        )

hindsight-all/hindsight/client_wrapper.py

@@ -0,0 +1,423 @@
+"""
+Wrapper for Hindsight client that adds API namespaces.
+
+Provides organized access to different parts of the Hindsight API through
+namespaces like .banks, .mental_models, etc.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from hindsight_client import Hindsight
+
+
+class BanksAPI:
+    """Namespace for bank-related operations.
+    
+    Provides methods to create, delete, and manage memory banks.
+    """
+
+    def __init__(self, client: Hindsight):
+        self._client = client
+
+    def create(
+        self,
+        bank_id: str,
+        name: str | None = None,
+        mission: str | None = None,
+        disposition: dict[str, Any] | None = None,
+    ) -> Any:
+        """Create a new bank.
+        
+        Args:
+            bank_id: Unique identifier for the bank.
+            name: Optional display name for the bank.
+            mission: Optional mission statement for the bank.
+            disposition: Optional disposition configuration dict.
+            
+        Returns:
+            Bank creation response from the API.
+        """
+        return self._client.create_bank(
+            bank_id=bank_id,
+            name=name,
+            mission=mission,
+            disposition=disposition,
+        )
+
+    def delete(self, bank_id: str) -> Any:
+        """Delete a bank.
+        
+        Args:
+            bank_id: The ID of the bank to delete.
+            
+        Returns:
+            Deletion response from the API.
+        """
+        return self._client.delete_bank(bank_id=bank_id)
+
+    def set_mission(self, bank_id: str, mission: str) -> Any:
+        """Set or update the mission for a bank.
+        
+        Args:
+            bank_id: The ID of the bank.
+            mission: The mission statement to set.
+            
+        Returns:
+            API response confirming the update.
+        """
+        return self._client.set_mission(bank_id=bank_id, mission=mission)
+
+    def set_disposition(self, bank_id: str, disposition: dict[str, Any]) -> Any:
+        """Set or update the disposition for a bank.
+        
+        Args:
+            bank_id: The ID of the bank.
+            disposition: The disposition configuration dict.
+            
+        Returns:
+            API response confirming the update.
+        """
+        return self._client.set_disposition(bank_id=bank_id, disposition=disposition)
+
+    def list(self) -> Any:
+        """List all banks.
+        
+        Returns:
+            List of banks from the API.
+        """
+        from hindsight_client.hindsight_client import _run_async
+
+        return _run_async(self._client._banks_api.list_banks())
+
+
+class MentalModelsAPI:
+    """Namespace for mental model operations.
+    
+    Mental models are reusable knowledge structures that guide agent behavior.
+    """
+
+    def __init__(self, client: Hindsight):
+        self._client = client
+
+    def create(
+        self,
+        bank_id: str,
+        name: str,
+        content: str,
+        tags: list[str] | None = None,
+    ) -> Any:
+        """Create a new mental model.
+        
+        Args:
+            bank_id: The ID of the bank to add the model to.
+            name: Name for the mental model.
+            content: The content/instructions for the mental model.
+            tags: Optional list of tags for categorization.
+            
+        Returns:
+            Creation response from the API.
+        """
+        return self._client.create_mental_model(
+            bank_id=bank_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
+        """List all mental models for a bank.
+        
+        Args:
+            bank_id: The ID of the bank.
+            tags: Optional filter by tags.
+            
+        Returns:
+            List of mental models.
+        """
+        return self._client.list_mental_models(bank_id=bank_id, tags=tags)
+
+    def get(self, bank_id: str, mental_model_id: str) -> Any:
+        """Get a specific mental model.
+        
+        Args:
+            bank_id: The ID of the bank.
+            mental_model_id: The ID of the mental model.
+            
+        Returns:
+            The mental model details.
+        """
+        return self._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+    def refresh(self, bank_id: str, mental_model_id: str) -> Any:
+        """Refresh a mental model.
+        
+        Args:
+            bank_id: The ID of the bank.
+            mental_model_id: The ID of the mental model to refresh.
+            
+        Returns:
+            Refresh response from the API.
+        """
+        return self._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+    def update(
+        self,
+        bank_id: str,
+        mental_model_id: str,
+        name: str | None = None,
+        content: str | None = None,
+        tags: list[str] | None = None,
+    ) -> Any:
+        """Update a mental model.
+        
+        Args:
+            bank_id: The ID of the bank.
+            mental_model_id: The ID of the mental model to update.
+            name: Optional new name.
+            content: Optional new content.
+            tags: Optional new tags list.
+            
+        Returns:
+            Update response from the API.
+        """
+        return self._client.update_mental_model(
+            bank_id=bank_id,
+            mental_model_id=mental_model_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def delete(self, bank_id: str, mental_model_id: str) -> Any:
+        """Delete a mental model.
+        
+        Args:
+            bank_id: The ID of the bank.
+            mental_model_id: The ID of the mental model to delete.
+            
+        Returns:
+            Deletion response from the API.
+        """
+        return self._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
+
+
+class DirectivesAPI:
+    """Namespace for directive operations.
+    
+    Directives are explicit instructions that guide agent behavior.
+    """
+
+    def __init__(self, client: Hindsight):
+        self._client = client
+
+    def create(
+        self,
+        bank_id: str,
+        name: str,
+        content: str,
+        tags: list[str] | None = None,
+    ) -> Any:
+        """Create a new directive.
+        
+        Args:
+            bank_id: The ID of the bank to add the directive to.
+            name: Name for the directive.
+            content: The directive content/instructions.
+            tags: Optional list of tags for categorization.
+            
+        Returns:
+            Creation response from the API.
+        """
+        return self._client.create_directive(
+            bank_id=bank_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
+        """List all directives for a bank.
+        
+        Args:
+            bank_id: The ID of the bank.
+            tags: Optional filter by tags.
+            
+        Returns:
+            List of directives.
+        """
+        return self._client.list_directives(bank_id=bank_id, tags=tags)
+
+    def get(self, bank_id: str, directive_id: str) -> Any:
+        """Get a specific directive.
+        
+        Args:
+            bank_id: The ID of the bank.
+            directive_id: The ID of the directive.
+            
+        Returns:
+            The directive details.
+        """
+        return self._client.get_directive(bank_id=bank_id, directive_id=directive_id)
+
+    def update(
+        self,
+        bank_id: str,
+        directive_id: str,
+        name: str | None = None,
+        content: str | None = None,
+        tags: list[str] | None = None,
+    ) -> Any:
+        """Update a directive.
+        
+        Args:
+            bank_id: The ID of the bank.
+            directive_id: The ID of the directive to update.
+            name: Optional new name.
+            content: Optional new content.
+            tags: Optional new tags list.
+            
+        Returns:
+            Update response from the API.
+        """
+        return self._client.update_directive(
+            bank_id=bank_id,
+            directive_id=directive_id,
+            name=name,
+            content=content,
+            tags=tags,
+        )
+
+    def delete(self, bank_id: str, directive_id: str) -> Any:
+        """Delete a directive.
+        
+        Args:
+            bank_id: The ID of the bank.
+            directive_id: The ID of the directive to delete.
+            
+        Returns:
+            Deletion response from the API.
+        """
+        return self._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
+
+
+class MemoriesAPI:
+    """Namespace for memory operations.
+    
+    Provides methods to query and retrieve stored memories.
+    """
+
+    def __init__(self, client: Hindsight):
+        self._client = client
+
+    def list(
+        self,
+        bank_id: str,
+        type: str | None = None,
+        search_query: str | None = None,
+        limit: int = 100,
+        offset: int = 0,
+    ) -> Any:
+        """List memories in a bank.
+        
+        Args:
+            bank_id: The ID of the bank to query.
+            type: Optional filter by memory type.
+            search_query: Optional search query for filtering.
+            limit: Maximum number of results to return (default: 100).
+            offset: Number of results to skip for pagination (default: 0).
+            
+        Returns:
+            List of memories matching the criteria.
+        """
+        return self._client.list_memories(
+            bank_id=bank_id,
+            type=type,
+            search_query=search_query,
+            limit=limit,
+            offset=offset,
+        )
+
+
+class HindsightClient(Hindsight):
+    """
+    Enhanced Hindsight client with organized API namespaces.
+
+    This wrapper extends the auto-generated Hindsight client with organized
+    access to different parts of the API through namespaces.
+
+    Example:
+        ```python
+        from hindsight import HindsightClient
+
+        client = HindsightClient(base_url="http://localhost:8888")
+
+        # Core operations (inherited from Hindsight)
+        client.retain(bank_id="test", content="Hello")
+        results = client.recall(bank_id="test", query="Hello")
+
+        # Organized API access through namespaces
+        client.banks.create(bank_id="test", name="Test Bank")
+        models = client.mental_models.list(bank_id="test")
+        directives = client.directives.list(bank_id="test")
+        memories = client.memories.list(bank_id="test")
+        ```
+        
+    Attributes:
+        banks: Namespace for bank management operations.
+        mental_models: Namespace for mental model operations.
+        directives: Namespace for directive operations.
+        memories: Namespace for memory listing operations.
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        super().__init__(*args, **kwargs)
+        self._banks_namespace: BanksAPI | None = None
+        self._mental_models_namespace: MentalModelsAPI | None = None
+        self._directives_namespace: DirectivesAPI | None = None
+        self._memories_namespace: MemoriesAPI | None = None
+
+    @property
+    def banks(self) -> BanksAPI:
+        """Access bank management operations.
+        
+        Returns:
+            BanksAPI instance for bank operations.
+        """
+        if self._banks_namespace is None:
+            self._banks_namespace = BanksAPI(self)
+        return self._banks_namespace
+
+    @property
+    def mental_models(self) -> MentalModelsAPI:
+        """Access mental model operations.
+        
+        Returns:
+            MentalModelsAPI instance for mental model operations.
+        """
+        if self._mental_models_namespace is None:
+            self._mental_models_namespace = MentalModelsAPI(self)
+        return self._mental_models_namespace
+
+    @property
+    def directives(self) -> DirectivesAPI:
+        """Access directive operations.
+        
+        Returns:
+            DirectivesAPI instance for directive operations.
+        """
+        if self._directives_namespace is None:
+            self._directives_namespace = DirectivesAPI(self)
+        return self._directives_namespace
+
+    @property
+    def memories(self) -> MemoriesAPI:
+        """Access memory listing operations.
+        
+        Returns:
+            MemoriesAPI instance for memory operations.
+        """
+        if self._memories_namespace is None:
+            self._memories_namespace = MemoriesAPI(self)
+        return self._memories_namespace

hindsight-all/hindsight/embedded.py

@@ -0,0 +1,445 @@
+"""
+Embedded Hindsight client with automatic daemon lifecycle management.
+
+This module provides HindsightEmbedded, a client that uses the same daemon
+management interface as hindsight-embed CLI, ensuring full compatibility.
+
+Example:
+    ```python
+    from hindsight import HindsightEmbedded
+
+    # Daemon starts automatically on first use
+    client = HindsightEmbedded(
+        profile="myapp",
+        llm_provider="groq",
+        llm_api_key="your-api-key",
+    )
+
+    # Use just like HindsightClient
+    client.retain(bank_id="alice", content="Alice loves AI")
+    results = client.recall(bank_id="alice", query="What does Alice like?")
+
+    # Optional cleanup
+    client.close()
+    ```
+
+Using context manager:
+    ```python
+    from hindsight import HindsightEmbedded
+
+    with HindsightEmbedded(profile="myapp") as client:
+        client.retain(bank_id="alice", content="Alice loves AI")
+        # Daemon managed automatically
+    ```
+"""
+
+import logging
+import threading
+from typing import Optional
+
+from hindsight_client import Hindsight
+from hindsight_embed import get_embed_manager
+
+from .api_namespaces import BanksAPI, DirectivesAPI, MemoriesAPI, MentalModelsAPI
+
+logger = logging.getLogger(__name__)
+
+
+class HindsightEmbedded:
+    """
+    Hindsight client with automatic daemon lifecycle management.
+
+    This client uses the same daemon management interface as hindsight-embed CLI,
+    ensuring full compatibility and shared profiles. The daemon is started automatically
+    on first use and manages profile-specific databases.
+
+    Profile data is stored in: ~/.pg0/instances/hindsight-embed-{profile}/
+
+    All methods from HindsightClient are available:
+    - retain(), retain_batch()
+    - recall()
+    - reflect()
+    - create_bank(), set_mission(), delete_bank()
+    - create_mental_model(), list_mental_models(), etc.
+    - create_directive(), list_directives(), etc.
+    - And all async variants (aretain, arecall, areflect, etc.)
+
+    Args:
+        profile: Profile name for data isolation (default: "default")
+        llm_provider: LLM provider ("groq", "openai", "ollama", "gemini", "anthropic", "lmstudio")
+        llm_api_key: API key for the LLM provider
+        llm_model: Model name to use
+        llm_base_url: Optional custom base URL for LLM API
+        database_url: Optional database URL override (default: profile-specific pg0)
+        idle_timeout: Seconds before daemon auto-exits when idle (default: 0, disabled)
+        log_level: Daemon log level (default: "info")
+        ui: Whether to start the control plane web UI alongside the daemon (default: False)
+        ui_port: Port for the UI. Defaults to daemon_port + 10000.
+        ui_hostname: Hostname to bind the UI to. Defaults to "0.0.0.0".
+    """
+
+    def __init__(
+        self,
+        profile: str = "default",
+        llm_provider: str = "groq",
+        llm_api_key: str = "",
+        llm_model: str = "openai/gpt-oss-120b",
+        llm_base_url: Optional[str] = None,
+        database_url: Optional[str] = None,
+        idle_timeout: int = 0,
+        log_level: str = "info",
+        ui: bool = False,
+        ui_port: Optional[int] = None,
+        ui_hostname: str = "0.0.0.0",
+    ):
+        """
+        Initialize the embedded client (daemon starts on first use).
+
+        Args:
+            profile: Profile name for data isolation
+            llm_provider: LLM provider
+            llm_api_key: API key for the LLM provider
+            llm_model: Model name to use
+            llm_base_url: Optional custom base URL for LLM API
+            database_url: Optional database URL override
+            idle_timeout: Seconds before daemon auto-exits when idle (0 = disabled)
+            log_level: Daemon log level
+            ui: Whether to start the control plane web UI alongside the daemon
+            ui_port: Port for the UI (defaults to daemon_port + 10000)
+            ui_hostname: Hostname to bind the UI to (defaults to "0.0.0.0")
+        """
+        self.profile = profile
+
+        # Build config dict for daemon (matches CLI format)
+        self.config = {
+            "HINDSIGHT_API_LLM_PROVIDER": llm_provider,
+            "HINDSIGHT_API_LLM_API_KEY": llm_api_key,
+            "HINDSIGHT_API_LLM_MODEL": llm_model,
+            "HINDSIGHT_API_LOG_LEVEL": log_level,
+            "HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT": str(idle_timeout),
+        }
+
+        if llm_base_url:
+            self.config["HINDSIGHT_API_LLM_BASE_URL"] = llm_base_url
+
+        if database_url:
+            self.config["HINDSIGHT_EMBED_API_DATABASE_URL"] = database_url
+
+        self._ui = ui
+        self._ui_port = ui_port
+        self._ui_hostname = ui_hostname
+
+        self._client: Optional[Hindsight] = None
+        self._lock = threading.Lock()
+        self._started = False
+        self._closed = False
+        self._manager = get_embed_manager()
+
+        # API namespaces (initialized once, lazily)
+        self._banks_api: Optional[BanksAPI] = None
+        self._mental_models_api: Optional[MentalModelsAPI] = None
+        self._directives_api: Optional[DirectivesAPI] = None
+        self._memories_api: Optional[MemoriesAPI] = None
+
+    def _ensure_started(self):
+        """Ensure daemon is running (thread-safe), restarting if crashed."""
+        if self._started and self._client is not None:
+            if self._manager.is_running(self.profile):
+                return
+            # Daemon crashed — reset state and fall through to restart
+            logger.warning(
+                "Daemon for profile '%s' is no longer responsive, restarting...",
+                self.profile,
+            )
+            try:
+                self._client.close()
+            except Exception:
+                logger.debug("Error closing stale client", exc_info=True)
+            self._client = None
+            self._started = False
+
+        with self._lock:
+            # Double-check after acquiring lock
+            if self._started and self._client is not None:
+                if self._manager.is_running(self.profile):
+                    return
+                logger.warning(
+                    "Daemon for profile '%s' is no longer responsive (lock path), restarting...",
+                    self.profile,
+                )
+                try:
+                    self._client.close()
+                except Exception:
+                    logger.debug("Error closing stale client", exc_info=True)
+                self._client = None
+                self._started = False
+
+            if self._closed:
+                raise RuntimeError(
+                    "Cannot use HindsightEmbedded after it has been closed"
+                )
+
+            # Use embed manager interface for daemon management
+            logger.info(f"Ensuring daemon is running for profile '{self.profile}'...")
+            success = self._manager.ensure_running(self.config, self.profile)
+            if not success:
+                raise RuntimeError(
+                    f"Failed to start daemon for profile '{self.profile}'"
+                )
+
+            # Get daemon URL and create client
+            daemon_url = self._manager.get_url(self.profile)
+            self._client = Hindsight(base_url=daemon_url)
+            self._started = True
+            logger.info(f"Connected to daemon at {daemon_url}")
+
+            # Start UI if requested
+            if self._ui:
+                logger.info(f"Starting UI for profile '{self.profile}'...")
+                ui_started = self._manager.start_ui(
+                    self.profile, self._ui_port, self._ui_hostname
+                )
+                if not ui_started:
+                    logger.warning(f"Failed to start UI for profile '{self.profile}'")
+
+    def _cleanup(self, stop_daemon_on_close: bool = False):
+        """
+        Cleanup client resources (idempotent).
+
+        Args:
+            stop_daemon_on_close: If True, stops the daemon. Otherwise, daemon continues
+                running (it will auto-stop after idle timeout).
+        """
+        if self._closed:
+            return
+
+        acquired = self._lock.acquire(timeout=5.0)
+        if not acquired:
+            # Lock is held by another thread (e.g. _ensure_started).
+            # Mark closed to prevent new operations but skip shared-state
+            # teardown — the daemon's idle timeout handles the rest.
+            logger.warning(
+                "Cleanup lock acquisition timed out for profile '%s'; "
+                "marking closed, daemon will idle-stop on its own",
+                self.profile,
+            )
+            self._closed = True
+            return
+
+        try:
+            if self._closed:
+                return
+
+            if self._client is not None:
+                try:
+                    self._client.close()
+                except Exception:
+                    logger.debug(
+                        "Error closing client for profile '%s'",
+                        self.profile,
+                        exc_info=True,
+                    )
+                self._client = None
+
+            # Stop UI if it was started
+            if self._ui and self._started:
+                logger.info(f"Stopping UI for profile '{self.profile}'...")
+                self._manager.stop_ui(self.profile, self._ui_port)
+
+            # Optionally stop daemon (daemon has idle timeout, so not required)
+            if stop_daemon_on_close and self._started:
+                logger.info(f"Stopping daemon for profile '{self.profile}'...")
+                self._manager.stop(self.profile)
+
+            self._closed = True
+        finally:
+            self._lock.release()
+
+    def close(self, stop_daemon: bool = False):
+        """
+        Explicitly close the client.
+
+        Args:
+            stop_daemon: If True, stops the daemon. Otherwise, daemon continues running
+                and will auto-stop after idle timeout (default: False).
+
+        Note:
+            The daemon may be shared with other clients or the CLI, so stopping it
+            might affect other users. By default, we rely on the daemon's idle timeout.
+        """
+        self._cleanup(stop_daemon_on_close=stop_daemon)
+
+    def __getattr__(self, name: str):
+        """
+        Proxy all method calls to the underlying Hindsight client.
+
+        This allows HindsightEmbedded to expose all HindsightClient methods
+        without manually wrapping each one.
+        """
+        # Ensure server is started (and restart if crashed) before proxying
+        self._ensure_started()
+
+        return getattr(self._client, name)
+
+    def __enter__(self):
+        """Context manager entry - ensures server is started."""
+        self._ensure_started()
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """Context manager exit - stops the server."""
+        self.close()
+
+    def __del__(self):
+        """Cleanup on garbage collection."""
+        self._cleanup()
+
+    @property
+    def banks(self) -> BanksAPI:
+        """
+        Access bank management operations.
+
+        Each method call ensures the daemon is running before executing.
+
+        Example:
+            ```python
+            from hindsight import HindsightEmbedded
+
+            embedded = HindsightEmbedded(profile="myapp", ...)
+
+            # Create a bank
+            embedded.banks.create(bank_id="test", name="Test Bank")
+
+            # Set mission
+            embedded.banks.set_mission(bank_id="test", mission="Help users")
+            ```
+        """
+        if self._banks_api is None:
+            self._banks_api = BanksAPI(self)
+        return self._banks_api
+
+    @property
+    def mental_models(self) -> MentalModelsAPI:
+        """
+        Access mental model operations.
+
+        Each method call ensures the daemon is running before executing.
+
+        Example:
+            ```python
+            from hindsight import HindsightEmbedded
+
+            embedded = HindsightEmbedded(profile="myapp", ...)
+
+            # Create a mental model
+            embedded.mental_models.create(
+                bank_id="test",
+                name="User Preferences",
+                content="User prefers dark mode"
+            )
+
+            # List mental models
+            models = embedded.mental_models.list(bank_id="test")
+            ```
+        """
+        if self._mental_models_api is None:
+            self._mental_models_api = MentalModelsAPI(self)
+        return self._mental_models_api
+
+    @property
+    def directives(self) -> DirectivesAPI:
+        """
+        Access directive operations.
+
+        Each method call ensures the daemon is running before executing.
+
+        Example:
+            ```python
+            from hindsight import HindsightEmbedded
+
+            embedded = HindsightEmbedded(profile="myapp", ...)
+
+            # Create a directive
+            embedded.directives.create(
+                bank_id="test",
+                name="Response Style",
+                content="Always be concise and friendly"
+            )
+
+            # List directives
+            directives = embedded.directives.list(bank_id="test")
+            ```
+        """
+        if self._directives_api is None:
+            self._directives_api = DirectivesAPI(self)
+        return self._directives_api
+
+    @property
+    def memories(self) -> MemoriesAPI:
+        """
+        Access memory listing operations.
+
+        Each method call ensures the daemon is running before executing.
+
+        Example:
+            ```python
+            from hindsight import HindsightEmbedded
+
+            embedded = HindsightEmbedded(profile="myapp", ...)
+
+            # List memories
+            memories = embedded.memories.list(
+                bank_id="test",
+                type="world",
+                limit=50
+            )
+            ```
+        """
+        if self._memories_api is None:
+            self._memories_api = MemoriesAPI(self)
+        return self._memories_api
+
+    @property
+    def client(self) -> Hindsight:
+        """
+        Get the underlying Hindsight client for direct access.
+
+        Ensures daemon is started (and restarts it if it has crashed) before
+        returning the client.
+
+        Returns:
+            Hindsight: The underlying client instance
+
+        Example:
+            ```python
+            from hindsight import HindsightEmbedded
+
+            embedded = HindsightEmbedded(profile="myapp", ...)
+
+            client = embedded.client
+            banks = client.list_banks()
+            ```
+        """
+        self._ensure_started()
+        return self._client
+
+    @property
+    def url(self) -> str:
+        """Get the daemon URL (starts daemon if needed)."""
+        self._ensure_started()
+        return self._manager.get_url(self.profile)
+
+    @property
+    def is_running(self) -> bool:
+        """Check if the client is initialized and the daemon is responsive."""
+        return (
+            self._started
+            and not self._closed
+            and self._client is not None
+            and self._manager.is_running(self.profile)
+        )
+
+    @property
+    def ui_url(self) -> str:
+        """Get the UI URL for this profile."""
+        return self._manager.get_ui_url(self.profile)

hindsight-all/pyproject.toml

@@ -4,20 +4,25 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-all"
-version = "0.3.0"
+version = "0.7.2"
 description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
 readme = "README.md"
 requires-python = ">=3.11"
 dependencies = [
-    "hindsight-api>=0.0.7",
+    "hindsight-api-slim[all]==0.7.2",
     "hindsight-client>=0.0.7",
+    "hindsight-embed>=0.1.0",
 ]
 
 [tool.uv.sources]
-hindsight-api = { workspace = true }
+hindsight-api-slim = { workspace = true }
 hindsight-client = { workspace = true }
+hindsight-embed = { workspace = true }
 
 [project.optional-dependencies]
+local-llm = [
+    "hindsight-api-slim[local-llm]==0.7.2",
+]
 test = [
     "pytest>=7.0.0",
     "pytest-asyncio>=0.21.0",

hindsight-all/tests/test_cleanup_timeout.py

@@ -0,0 +1,56 @@
+"""
+Unit test for _cleanup lock timeout behavior.
+
+Verifies that _cleanup completes even when the lock is held by another thread,
+instead of hanging indefinitely (fixes #952).
+"""
+
+import threading
+import time
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+
+def test_cleanup_completes_when_lock_held():
+    """
+    _cleanup should complete (best-effort) even when self._lock is held
+    by another thread, e.g. during a long _ensure_started call.
+    """
+    with patch.dict("sys.modules", {
+        "hindsight_client": MagicMock(),
+        "hindsight_embed": MagicMock(),
+        "hindsight.api_namespaces": MagicMock(),
+    }):
+        from hindsight.embedded import HindsightEmbedded
+
+        client = HindsightEmbedded.__new__(HindsightEmbedded)
+        client.profile = "test"
+        client._lock = threading.Lock()
+        client._closed = False
+        client._client = None
+        client._started = False
+        client._ui = False
+
+        # Simulate another thread holding the lock
+        client._lock.acquire()
+
+        cleanup_done = threading.Event()
+
+        def run_cleanup():
+            client._cleanup()
+            cleanup_done.set()
+
+        t = threading.Thread(target=run_cleanup)
+        t.start()
+
+        # Cleanup should complete within the timeout (5s) + margin
+        assert cleanup_done.wait(timeout=8.0), (
+            "_cleanup hung instead of timing out on lock acquisition"
+        )
+
+        # Release the lock from the simulating thread
+        client._lock.release()
+        t.join(timeout=1.0)
+
+        assert client._closed, "Client should be marked as closed after cleanup"

hindsight-all/tests/test_embedded.py

@@ -0,0 +1,442 @@
+"""
+Integration tests for HindsightEmbedded client.
+
+Tests the embedded client with automatic server lifecycle management:
+1. Lazy server startup on first use
+2. Server reuse across multiple operations
+3. Context manager support
+4. Method proxying to underlying HindsightClient
+5. Proper cleanup
+
+Note: Each test uses random bank_ids to avoid conflicts and allow safe parallel execution.
+"""
+
+import os
+import uuid
+
+import pytest
+import urllib.request
+import json
+
+from hindsight import HindsightEmbedded
+
+
+@pytest.fixture(scope="session")
+def llm_config():
+    """Get LLM configuration from environment (session-scoped)."""
+    # Try both naming conventions
+    provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv(
+        "HINDSIGHT_LLM_PROVIDER", "groq"
+    )
+    api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv(
+        "HINDSIGHT_LLM_API_KEY", ""
+    )
+    model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv(
+        "HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b"
+    )
+
+    if not api_key:
+        pytest.skip(
+            "LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY."
+        )
+
+    return {
+        "llm_provider": provider,
+        "llm_api_key": api_key,
+        "llm_model": model,
+    }
+
+
+def test_embedded_lazy_start(llm_config):
+    """
+    Test that HindsightEmbedded starts server lazily on first use.
+    """
+    profile = f"test_lazy_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    # Create client - should NOT start server yet
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+    assert not client.is_running, "Server should not be running after initialization"
+
+    # First call should start server
+    result = client.retain(bank_id=bank_id, content="Test content for lazy start")
+
+    # Verify server is now running
+    assert client.is_running, "Server should be running after first call"
+    assert result.success, "Retain should succeed"
+    assert result.items_count >= 1, "Should have stored at least 1 item"
+
+    # Cleanup
+    client.close()
+    assert not client.is_running, "Server should stop after close()"
+
+
+def test_embedded_context_manager(llm_config):
+    """
+    Test HindsightEmbedded with context manager.
+    """
+    profile = f"test_ctx_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    # Use context manager
+    with HindsightEmbedded(profile=profile, log_level="info", **llm_config) as client:
+        assert client.is_running, "Server should be running inside context"
+
+        # Store memory
+        result = client.retain(bank_id=bank_id, content="Testing context manager")
+        assert result.success, "Retain should succeed"
+
+        # Recall memory
+        recall_results = client.recall(bank_id=bank_id, query="context")
+        assert isinstance(recall_results.results, list), (
+            "Recall should return results list"
+        )
+
+    # Server should be stopped after context exit
+    # Note: We can't check client.is_running here as client is out of scope
+
+
+def test_embedded_complete_workflow(llm_config):
+    """
+    Test complete workflow with HindsightEmbedded.
+
+    This test:
+    1. Creates a client with lazy start
+    2. Creates a memory bank
+    3. Stores multiple memories
+    4. Recalls memories
+    5. Reflects on memories
+    6. Tests cleanup
+    """
+    profile = f"test_workflow_{uuid.uuid4().hex[:8]}"
+    bank_id = f"assistant_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    try:
+        # Step 1: Create a memory bank
+        print(f"\n1. Creating memory bank: {bank_id}")
+        bank_response = client.create_bank(
+            bank_id=bank_id,
+            name="Test Assistant",
+            mission="Help with programming tasks",
+        )
+        assert bank_response.bank_id == bank_id
+
+        # Step 2: Store memories (single)
+        print("\n2. Storing single memory...")
+        retain_response = client.retain(
+            bank_id=bank_id,
+            content="User prefers Python for data analysis.",
+            context="Programming preferences",
+        )
+        assert retain_response.success
+        assert retain_response.items_count >= 1
+
+        # Step 3: Store batch memories
+        print("\n3. Storing batch memories...")
+        batch_response = client.retain_batch(
+            bank_id=bank_id,
+            items=[
+                {"content": "User works with pandas and numpy."},
+                {"content": "User likes matplotlib for visualization."},
+                {
+                    "content": "User is interested in machine learning with scikit-learn."
+                },
+            ],
+        )
+        assert batch_response.success
+        assert batch_response.items_count >= 3
+
+        # Step 4: Recall memories
+        print("\n4. Recalling memories...")
+        recall_response = client.recall(
+            bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000
+        )
+        assert isinstance(recall_response.results, list)
+        assert len(recall_response.results) > 0
+        print(f"   Found {len(recall_response.results)} relevant memories")
+
+        # Step 5: Reflect on memories
+        print("\n5. Reflecting on memories...")
+        reflect_response = client.reflect(
+            bank_id=bank_id,
+            query="What programming tools should I recommend?",
+            budget="low",
+        )
+        assert reflect_response.text
+        assert len(reflect_response.text) > 0
+        print(f"   Answer: {reflect_response.text[:150]}...")
+
+        # Verify answer mentions relevant tools
+        answer_lower = reflect_response.text.lower()
+        assert any(
+            term in answer_lower for term in ["python", "pandas", "numpy", "data"]
+        )
+
+        # Step 6: List memories
+        print("\n6. Listing memories...")
+        list_response = client.list_memories(bank_id=bank_id, limit=10)
+        assert len(list_response.items) > 0
+        print(f"   Listed {len(list_response.items)} memories")
+
+    finally:
+        # Cleanup
+        client.close()
+
+
+def test_embedded_server_reuse(llm_config):
+    """
+    Test that the same server is reused across multiple calls.
+    """
+    profile = f"test_reuse_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    try:
+        # First call starts server
+        result1 = client.retain(bank_id=bank_id, content="First message")
+        url1 = client.url
+        assert client.is_running
+
+        # Second call should reuse the same server
+        result2 = client.retain(bank_id=bank_id, content="Second message")
+        url2 = client.url
+
+        # URLs should be identical (same server)
+        assert url1 == url2, "Server URL should remain the same across calls"
+        assert result1.success and result2.success
+
+        # Third call should also reuse
+        recall_result = client.recall(bank_id=bank_id, query="message")
+        url3 = client.url
+        assert url3 == url1, "Server URL should remain the same for recall"
+        assert isinstance(recall_result.results, list)
+
+    finally:
+        client.close()
+
+
+def test_embedded_method_proxying(llm_config):
+    """
+    Test that all HindsightClient methods are properly proxied.
+
+    This ensures __getattr__ proxying works for various method types.
+    """
+    profile = f"test_proxy_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    try:
+        # Test bank operations
+        bank = client.create_bank(bank_id=bank_id, name="Proxy Test")
+        assert bank.bank_id == bank_id
+
+        # Test mission setting
+        mission_response = client.set_mission(
+            bank_id=bank_id, mission="Test mission for proxying"
+        )
+        assert mission_response.bank_id == bank_id
+
+        # Test retain
+        retain_result = client.retain(bank_id=bank_id, content="Test content")
+        assert retain_result.success
+
+        # Test retain_batch
+        batch_result = client.retain_batch(
+            bank_id=bank_id, items=[{"content": "Item 1"}, {"content": "Item 2"}]
+        )
+        assert batch_result.success
+        assert batch_result.items_count >= 2
+
+        # Test recall
+        recall_result = client.recall(bank_id=bank_id, query="test")
+        assert hasattr(recall_result, "results")
+
+        # Test reflect
+        reflect_result = client.reflect(bank_id=bank_id, query="What is stored?")
+        assert hasattr(reflect_result, "text")
+
+        # Test list_memories
+        list_result = client.list_memories(bank_id=bank_id, limit=5)
+        assert hasattr(list_result, "items")
+
+        print("✓ All methods successfully proxied")
+
+    finally:
+        client.close()
+
+
+def test_embedded_multiple_banks(llm_config):
+    """
+    Test that HindsightEmbedded can work with multiple banks.
+    """
+    profile = f"test_multibank_{uuid.uuid4().hex[:8]}"
+    bank1_id = f"bank1_{uuid.uuid4().hex[:8]}"
+    bank2_id = f"bank2_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    try:
+        # Create first bank and store data
+        client.create_bank(bank_id=bank1_id, name="Bank 1")
+        client.retain(bank_id=bank1_id, content="Alice prefers Python for data science")
+
+        # Create second bank and store data
+        client.create_bank(bank_id=bank2_id, name="Bank 2")
+        client.retain(
+            bank_id=bank2_id, content="Bob uses JavaScript for web development"
+        )
+
+        # Recall from both banks
+        results1 = client.recall(bank_id=bank1_id, query="programming language")
+        results2 = client.recall(bank_id=bank2_id, query="programming language")
+
+        assert len(results1.results) > 0
+        assert len(results2.results) > 0
+
+        # Verify banks are isolated (each should only see their own content)
+        # This is a basic check - content isolation is tested more thoroughly in other tests
+        assert results1.results[0].text != results2.results[0].text or len(
+            results1.results
+        ) != len(results2.results)
+
+    finally:
+        client.close()
+
+
+def test_embedded_profile_isolation(llm_config):
+    """
+    Test that different profiles create isolated data stores.
+    """
+    profile1 = f"test_iso1_{uuid.uuid4().hex[:8]}"
+    profile2 = f"test_iso2_{uuid.uuid4().hex[:8]}"
+    bank_id = "shared_bank_name"  # Same bank_id in both profiles
+
+    client1 = HindsightEmbedded(profile=profile1, log_level="info", **llm_config)
+    client2 = HindsightEmbedded(profile=profile2, log_level="info", **llm_config)
+
+    try:
+        # Store data in profile1
+        client1.retain(
+            bank_id=bank_id, content="User likes TypeScript for frontend development"
+        )
+
+        # Store different data in profile2
+        client2.retain(
+            bank_id=bank_id, content="User prefers Rust for systems programming"
+        )
+
+        # Each profile should only see its own data
+        results1 = client1.recall(bank_id=bank_id, query="programming preference")
+        results2 = client2.recall(bank_id=bank_id, query="programming preference")
+
+        # Both should have results
+        assert len(results1.results) > 0
+        assert len(results2.results) > 0
+
+        # Results should be different (basic isolation check)
+        # Note: This is a basic sanity check. Full isolation is ensured by pg0's data directory separation
+
+    finally:
+        client1.close()
+        client2.close()
+
+
+def test_embedded_error_after_close(llm_config):
+    """
+    Test that using HindsightEmbedded after close() raises an error.
+    """
+    profile = f"test_error_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    # Use it once to start server
+    client.retain(bank_id=bank_id, content="Test")
+
+    # Close the client
+    client.close()
+    assert not client.is_running
+
+    # Trying to use it after close should raise an error
+    with pytest.raises(
+        RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"
+    ):
+        client.retain(bank_id=bank_id, content="This should fail")
+
+
+def test_embedded_ui_flag(llm_config):
+    """
+    Test that ui=True starts the control plane UI alongside the daemon,
+    and that the UI's health endpoint reports a connected dataplane.
+    """
+    profile = f"test_ui_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", ui=True, **llm_config)
+
+    try:
+        # First use triggers daemon + UI startup
+        result = client.retain(bank_id=bank_id, content="UI integration test content")
+        assert result.success, "Retain should succeed"
+        assert client.is_running, "Daemon should be running"
+
+        # Verify UI is reachable and reports connected dataplane
+        ui_url = client.ui_url
+        assert isinstance(ui_url, str) and ui_url, "ui_url should be a non-empty string"
+
+        health_url = f"{ui_url}/api/health"
+        with urllib.request.urlopen(health_url, timeout=10) as resp:
+            health = json.loads(resp.read().decode())
+
+        assert health["status"] == "ok", (
+            f"UI health status should be 'ok', got: {health['status']}"
+        )
+        assert health["dataplane"]["status"] == "connected", (
+            f"Dataplane should be connected, got: {health['dataplane']}"
+        )
+
+    finally:
+        client.close()
+
+
+def test_embedded_daemon_crash_recovery(llm_config):
+    """
+    Test that HindsightEmbedded recovers when the daemon crashes.
+
+    Simulates a crash by stopping the daemon, then verifies
+    that the next operation transparently restarts it.
+    """
+    profile = f"test_crash_{uuid.uuid4().hex[:8]}"
+    bank_id = f"bank_{uuid.uuid4().hex[:8]}"
+
+    client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
+
+    try:
+        # Start daemon and store a memory
+        result = client.retain(bank_id=bank_id, content="Before crash")
+        assert result.success, "Initial retain should succeed"
+        assert client.is_running, "Daemon should be running"
+
+        original_url = client.url
+
+        # Simulate daemon crash by stopping it
+        client._manager.stop(client.profile)
+        assert not client._manager.is_running(client.profile), (
+            "Daemon should be stopped after simulated crash"
+        )
+
+        # Next operation should transparently restart the daemon
+        result2 = client.retain(bank_id=bank_id, content="After crash recovery")
+        assert result2.success, "Retain after crash recovery should succeed"
+        assert client.is_running, "Daemon should be running again after recovery"
+
+        # Verify recall still works
+        recall_result = client.recall(bank_id=bank_id, query="crash")
+        assert isinstance(recall_result.results, list), "Recall should return results"
+
+    finally:
+        client.close()

hindsight-all/tests/test_embedded_namespaces.py

@@ -0,0 +1,170 @@
+"""Test that API namespaces ensure daemon is started before each call."""
+
+from unittest.mock import Mock, patch
+
+import pytest
+
+from hindsight import HindsightEmbedded
+
+
+@pytest.fixture
+def embedded_client():
+    """Create an embedded client for testing."""
+    return HindsightEmbedded(
+        profile="test",
+        llm_provider="openai",
+        llm_api_key="test-key",
+    )
+
+
+def test_banks_create_ensures_daemon_started(embedded_client):
+    """Test that banks.create() calls _ensure_started()."""
+    # Mock _ensure_started to track calls
+    with patch.object(embedded_client, "_ensure_started") as mock_ensure:
+        # Mock the underlying client to avoid actual API call
+        mock_client = Mock()
+        embedded_client._client = mock_client
+
+        # Call namespace method
+        try:
+            embedded_client.banks.create(bank_id="test", name="Test Bank")
+        except Exception:
+            pass  # We don't care if the actual call fails
+
+        # Verify _ensure_started was called
+        mock_ensure.assert_called_once()
+
+
+def test_mental_models_list_ensures_daemon_started(embedded_client):
+    """Test that mental_models.list() calls _ensure_started()."""
+    with patch.object(embedded_client, "_ensure_started") as mock_ensure:
+        mock_client = Mock()
+        embedded_client._client = mock_client
+
+        try:
+            embedded_client.mental_models.list(bank_id="test")
+        except Exception:
+            pass
+
+        mock_ensure.assert_called_once()
+
+
+def test_directives_list_ensures_daemon_started(embedded_client):
+    """Test that directives.list() calls _ensure_started()."""
+    with patch.object(embedded_client, "_ensure_started") as mock_ensure:
+        mock_client = Mock()
+        embedded_client._client = mock_client
+
+        try:
+            embedded_client.directives.list(bank_id="test")
+        except Exception:
+            pass
+
+        mock_ensure.assert_called_once()
+
+
+def test_memories_list_ensures_daemon_started(embedded_client):
+    """Test that memories.list() calls _ensure_started()."""
+    with patch.object(embedded_client, "_ensure_started") as mock_ensure:
+        mock_client = Mock()
+        embedded_client._client = mock_client
+
+        try:
+            embedded_client.memories.list(bank_id="test")
+        except Exception:
+            pass
+
+        mock_ensure.assert_called_once()
+
+
+def test_multiple_calls_ensure_daemon_each_time(embedded_client):
+    """Test that each namespace call ensures daemon is started."""
+    with patch.object(embedded_client, "_ensure_started") as mock_ensure:
+        mock_client = Mock()
+        embedded_client._client = mock_client
+
+        # Make multiple calls
+        try:
+            embedded_client.banks.create(bank_id="test", name="Test")
+        except Exception:
+            pass
+
+        try:
+            embedded_client.mental_models.list(bank_id="test")
+        except Exception:
+            pass
+
+        try:
+            embedded_client.directives.list(bank_id="test")
+        except Exception:
+            pass
+
+        # Should be called 3 times (once per namespace method call)
+        assert mock_ensure.call_count == 3
+
+
+def test_daemon_restart_handling(embedded_client):
+    """Test that namespace methods can recover from daemon crash."""
+    call_count = 0
+
+    def mock_ensure_started():
+        """Mock that simulates daemon restart."""
+        nonlocal call_count
+        call_count += 1
+        # Create a new mock client each time (simulating daemon restart)
+        embedded_client._client = Mock()
+        embedded_client._started = True
+
+    with patch.object(embedded_client, "_ensure_started", side_effect=mock_ensure_started):
+        # First call - daemon starts
+        embedded_client.banks.create(bank_id="test", name="Test")
+        assert call_count == 1
+
+        # Simulate daemon crash by clearing client
+        embedded_client._client = None
+        embedded_client._started = False
+
+        # Second call - daemon restarts
+        embedded_client.banks.create(bank_id="test", name="Test")
+        assert call_count == 2
+
+
+def test_ensure_started_calls_manager(embedded_client):
+    """Test that _ensure_started actually starts the daemon via manager."""
+    # Mock the manager
+    mock_manager = Mock()
+    mock_manager.ensure_running.return_value = True
+    mock_manager.get_url.return_value = "http://localhost:54321"
+
+    embedded_client._manager = mock_manager
+
+    # Mock Hindsight client constructor
+    with patch("hindsight.embedded.Hindsight") as mock_hindsight_class:
+        mock_client = Mock()
+        mock_hindsight_class.return_value = mock_client
+
+        # Call _ensure_started
+        embedded_client._ensure_started()
+
+        # Verify manager was called
+        mock_manager.ensure_running.assert_called_once_with(
+            embedded_client.config, embedded_client.profile
+        )
+        mock_manager.get_url.assert_called_once_with(embedded_client.profile)
+
+        # Verify Hindsight client was created
+        mock_hindsight_class.assert_called_once_with(base_url="http://localhost:54321")
+
+
+def test_namespace_singleton_behavior(embedded_client):
+    """Test that namespace properties return the same instance."""
+    banks1 = embedded_client.banks
+    banks2 = embedded_client.banks
+
+    # Should be the same instance
+    assert banks1 is banks2
+
+    # Same for other namespaces
+    assert embedded_client.mental_models is embedded_client.mental_models
+    assert embedded_client.directives is embedded_client.directives
+    assert embedded_client.memories is embedded_client.memories

hindsight-all/tests/test_server_integration.py

@@ -26,7 +26,10 @@ def llm_config():
     api_key = os.getenv("HINDSIGHT_LLM_API_KEY", "")
     model = os.getenv("HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b")
 
-    if not api_key:
+    # vertexai uses GCP service account credentials (HINDSIGHT_API_LLM_VERTEXAI_*),
+    # not a traditional API key
+    providers_without_api_key = ("vertexai", "ollama")
+    if not api_key and provider not in providers_without_api_key:
         raise Exception("LLM API key not configured. Set HINDSIGHT_LLM_API_KEY environment variable.")
 
     return {
@@ -75,10 +78,9 @@ def test_server_context_manager_basic_workflow(client):
     bank_response = client.create_bank(
         bank_id=bank_id,
         name="Test Assistant",
-        background="An AI assistant that helps with programming and data analysis tasks."
+        mission="An AI assistant that helps with programming and data analysis tasks."
     )
-    assert "bank_id" in bank_response
-    assert bank_response["bank_id"] == bank_id
+    assert bank_response.bank_id == bank_id
 
     # Step 2: Store some memories about user preferences
     print("\n2. Storing memories...")
@@ -89,7 +91,7 @@ def test_server_context_manager_basic_workflow(client):
         content="User prefers Python over JavaScript for data analysis projects.",
         context="User conversation about programming languages"
     )
-    assert retain_response1.get("success") is True
+    assert retain_response1.success is True
 
     # Store second memory
     retain_response2 = client.retain(
@@ -97,7 +99,7 @@ def test_server_context_manager_basic_workflow(client):
         content="User is working on a machine learning project using scikit-learn.",
         context="Discussion about ML frameworks"
     )
-    assert retain_response2.get("success") is True
+    assert retain_response2.success is True
 
     # Store third memory
     retain_response3 = client.retain(
@@ -105,7 +107,7 @@ def test_server_context_manager_basic_workflow(client):
         content="User likes visualizing data with matplotlib and seaborn.",
         context="Conversation about data visualization"
     )
-    assert retain_response3.get("success") is True
+    assert retain_response3.success is True
 
     # Store batch memories
     batch_response = client.retain_batch(
@@ -116,7 +118,7 @@ def test_server_context_manager_basic_workflow(client):
         ]
     )
     # Check if the batch was submitted successfully (items_count shows how many were submitted)
-    assert batch_response.get("items_count", 0) >= 2
+    assert batch_response.items_count >= 2
 
     # Step 3: Recall memories based on a query
     print("\n3. Recalling memories about programming preferences...")
@@ -127,14 +129,13 @@ def test_server_context_manager_basic_workflow(client):
     )
 
     # Verify recall results
-    assert isinstance(recall_results, list)
-    assert len(recall_results) > 0
-    print(f"   Found {len(recall_results)} relevant memories")
+    assert isinstance(recall_results.results, list)
+    assert len(recall_results.results) > 0
+    print(f"   Found {len(recall_results.results)} relevant memories")
 
     # Check that results have expected structure
-    for result in recall_results:
-        assert "content" in result or "text" in result
-        print(f"   - {result.get('content') or result.get('text', '')[:100]}")
+    for result in recall_results.results:
+        print(f"   - {result.text[:100]}")
 
     # Step 4: Recall memories about machine learning
     print("\n4. Recalling memories about machine learning...")
@@ -145,11 +146,11 @@ def test_server_context_manager_basic_workflow(client):
     )
 
     # Verify recall results
-    assert isinstance(ml_recall_results, list)
-    assert len(ml_recall_results) > 0
-    print(f"   Found {len(ml_recall_results)} ML-related memories")
-    for result in ml_recall_results[:3]:  # Show first 3
-        print(f"   - {result.get('content') or result.get('text', '')[:100]}")
+    assert isinstance(ml_recall_results.results, list)
+    assert len(ml_recall_results.results) > 0
+    print(f"   Found {len(ml_recall_results.results)} ML-related memories")
+    for result in ml_recall_results.results[:3]:  # Show first 3
+        print(f"   - {result.text[:100]}")
 
     # Step 5: Reflect (generate contextual answer based on memories)
     print("\n5. Reflecting on query about recommendations...")
@@ -160,10 +161,7 @@ def test_server_context_manager_basic_workflow(client):
     )
 
     # Verify reflection response
-    assert isinstance(reflect_response, dict)
-    assert "answer" in reflect_response or "text" in reflect_response
-
-    answer = reflect_response.get("answer") or reflect_response.get("text", "")
+    answer = reflect_response.text
     assert len(answer) > 0
     print(f"   Answer: {answer[:200]}...")
 
@@ -180,8 +178,7 @@ def test_server_context_manager_basic_workflow(client):
         context="The user is starting a new deep learning project"
     )
 
-    assert isinstance(reflect_with_context, dict)
-    context_answer = reflect_with_context.get("answer") or reflect_with_context.get("text", "")
+    context_answer = reflect_with_context.text
     assert len(context_answer) > 0
     print(f"   Context-aware answer: {context_answer[:150]}...")
 
@@ -200,21 +197,21 @@ def test_server_manual_start_stop(client):
         bank_id=bank_id,
         name="Manual Test"
     )
-    assert bank_response["bank_id"] == bank_id
+    assert bank_response.bank_id == bank_id
 
     # Store a memory
     retain_response = client.retain(
         bank_id=bank_id,
         content="Testing manual server lifecycle."
     )
-    assert retain_response.get("success") is True
+    assert retain_response.success is True
 
     # Recall the memory
     recall_results = client.recall(
         bank_id=bank_id,
         query="server testing"
     )
-    assert len(recall_results) >= 0  # May or may not find results immediately
+    assert len(recall_results.results) >= 0  # May or may not find results immediately
 
 
 def test_server_with_client_context_manager(client):
@@ -233,11 +230,11 @@ def test_server_with_client_context_manager(client):
         bank_id=bank_id,
         content="Testing nested context managers."
     )
-    assert response.get("success") is True
+    assert response.success is True
 
     # Verify we can recall
     results = client.recall(bank_id=bank_id, query="context")
-    assert isinstance(results, list)
+    assert isinstance(results.results, list)
 
 
 def test_list_banks(client, shared_server):
@@ -252,21 +249,11 @@ def test_list_banks(client, shared_server):
     bank1_id = f"test_bank_1_{test_suffix}"
     bank2_id = f"test_bank_2_{test_suffix}"
 
-    client.create_bank(bank_id=bank1_id, name="Test Bank 1", background="First test bank")
-    client.create_bank(bank_id=bank2_id, name="Test Bank 2", background="Second test bank")
+    client.create_bank(bank_id=bank1_id, name="Test Bank 1", mission="First test bank")
+    client.create_bank(bank_id=bank2_id, name="Test Bank 2", mission="Second test bank")
 
-    # List all banks using the generated client
-    import hindsight_client_api
-    from hindsight_client_api.api import default_api
-
-    config = hindsight_client_api.Configuration(host=shared_server.url)
-    api_client = hindsight_client_api.ApiClient(config)
-    api = default_api.DefaultApi(api_client)
-
-    # Call list_banks endpoint
-    import asyncio
-    loop = asyncio.get_event_loop()
-    response = loop.run_until_complete(api.list_banks())
+    # List all banks using the namespace API
+    response = client.banks.list()
 
     # Verify response structure
     assert hasattr(response, 'banks'), "Response should have 'banks' attribute"
@@ -274,9 +261,8 @@ def test_list_banks(client, shared_server):
 
     # Verify each bank has bank_id (not agent_id)
     for bank in response.banks:
-        bank_dict = bank.to_dict() if hasattr(bank, 'to_dict') else bank
-        assert 'bank_id' in bank_dict, f"Bank should have 'bank_id' field, got: {bank_dict.keys()}"
-        assert 'agent_id' not in bank_dict, f"Bank should NOT have 'agent_id' field, got: {bank_dict.keys()}"
+        assert hasattr(bank, 'bank_id'), f"Bank should have 'bank_id' attribute"
+        assert bank.bank_id is not None, "Bank ID should not be None"
 
     # Find our test banks
     bank_ids = [b.bank_id if hasattr(b, 'bank_id') else b['bank_id'] for b in response.banks]
@@ -284,6 +270,3 @@ def test_list_banks(client, shared_server):
     assert bank2_id in bank_ids, f"Should find {bank2_id} in bank list"
 
     print(f"✓ Successfully listed {len(response.banks)} banks with correct bank_id field")
-
-    # Cleanup
-    loop.run_until_complete(api_client.close())

hindsight-api-slim/README.md

@@ -0,0 +1,137 @@
+# Hindsight API
+
+**Memory System for AI Agents** — Temporal + Semantic + Entity Memory Architecture using PostgreSQL with pgvector.
+
+Hindsight gives AI agents persistent memory that works like human memory: it stores facts, tracks entities and relationships, handles temporal reasoning ("what happened last spring?"), and forms opinions based on configurable disposition traits.
+
+## Installation
+
+```bash
+pip install hindsight-api
+```
+
+## Quick Start
+
+### Run the Server
+
+```bash
+# Set your LLM provider
+export HINDSIGHT_API_LLM_PROVIDER=openai
+export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
+
+# Start the server (uses embedded PostgreSQL by default)
+hindsight-api
+```
+
+The server starts at http://localhost:8888 with:
+- REST API for memory operations
+- MCP server at `/mcp` for tool-use integration
+
+### Use the Python API
+
+```python
+from hindsight_api import MemoryEngine
+
+# Create and initialize the memory engine
+memory = MemoryEngine()
+await memory.initialize()
+
+# Create a memory bank for your agent
+bank = await memory.create_memory_bank(
+    name="my-assistant",
+    background="A helpful coding assistant"
+)
+
+# Store a memory
+await memory.retain(
+    memory_bank_id=bank.id,
+    content="The user prefers Python for data science projects"
+)
+
+# Recall memories
+results = await memory.recall(
+    memory_bank_id=bank.id,
+    query="What programming language does the user prefer?"
+)
+
+# Reflect with reasoning
+response = await memory.reflect(
+    memory_bank_id=bank.id,
+    query="Should I recommend Python or R for this ML project?"
+)
+```
+
+## CLI Options
+
+```bash
+hindsight-api --help
+
+# Common options
+hindsight-api --port 9000          # Custom port (default: 8888)
+hindsight-api --host 127.0.0.1     # Bind to localhost only
+hindsight-api --workers 4          # Multiple worker processes
+hindsight-api --log-level debug    # Verbose logging
+```
+
+## Configuration
+
+Configure via environment variables:
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `HINDSIGHT_API_DATABASE_URL` | PostgreSQL connection string | `pg0` (embedded) |
+| `HINDSIGHT_API_LLM_PROVIDER` | `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio` | `openai` |
+| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
+| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-4o-mini` |
+| `HINDSIGHT_API_HOST` | Server bind address | `0.0.0.0` |
+| `HINDSIGHT_API_PORT` | Server port | `8888` |
+
+### Example with External PostgreSQL
+
+```bash
+export HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@localhost:5432/hindsight
+export HINDSIGHT_API_LLM_PROVIDER=groq
+export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx
+
+hindsight-api
+```
+
+## Docker
+
+```bash
+docker run -it --name hindsight --restart unless-stopped -p 8888:8888 \
+  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
+  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
+  ghcr.io/vectorize-io/hindsight:latest
+```
+
+## MCP Server
+
+For local MCP integration without running the full API server:
+
+```bash
+hindsight-local-mcp
+```
+
+This runs a stdio-based MCP server that can be used directly with MCP-compatible clients.
+
+## Key Features
+
+- **Multi-Strategy Retrieval (TEMPR)** — Semantic, keyword, graph, and temporal search combined with RRF fusion
+- **Entity Graph** — Automatic entity extraction and relationship tracking
+- **Temporal Reasoning** — Native support for time-based queries
+- **Disposition Traits** — Configurable skepticism, literalism, and empathy influence opinion formation
+- **Three Memory Types** — World facts, bank actions, and formed opinions with confidence scores
+
+## Documentation
+
+Full documentation: [https://hindsight.vectorize.io](https://hindsight.vectorize.io)
+
+- [Installation Guide](https://hindsight.vectorize.io/developer/installation)
+- [Configuration Reference](https://hindsight.vectorize.io/developer/configuration)
+- [API Reference](https://hindsight.vectorize.io/api-reference)
+- [Python SDK](https://hindsight.vectorize.io/sdks/python)
+
+## License
+
+Apache 2.0

hindsight-api-slim/hindsight_api/__init__.py

@@ -4,6 +4,13 @@ Memory System for AI Agents.
 Temporal + Semantic Memory Architecture using PostgreSQL with pgvector.
 """
 
+# Cap native ML thread pools (OpenBLAS/OpenMP/MKL) before any import pulls in
+# numpy/torch/onnxruntime — they read these env vars only at load time. See
+# hindsight_api/_thread_limits.py for the rationale.
+from ._thread_limits import apply_default_thread_limits
+
+apply_default_thread_limits()
+
 from .config import HindsightConfig, get_config
 from .engine.cross_encoder import CrossEncoderModel, LocalSTCrossEncoder, RemoteTEICrossEncoder
 from .engine.embeddings import Embeddings, LocalSTEmbeddings, RemoteTEIEmbeddings
@@ -46,4 +53,4 @@ __all__ = [
     "RemoteTEICrossEncoder",
     "LLMConfig",
 ]
-__version__ = "0.1.0"
+__version__ = "0.7.2"

hindsight-api-slim/hindsight_api/_pg_search.py

@@ -0,0 +1,85 @@
+"""Helpers for ParadeDB pg_search index configuration."""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Sequence
+
+PG_SEARCH_TOKENIZER_ENV = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION_PG_SEARCH_TOKENIZER"
+
+_SIMPLE_TOKENIZERS = {
+    "unicode_words",
+    "simple",
+    "whitespace",
+    "literal",
+    "literal_normalized",
+    "chinese_compatible",
+    "icu",
+    "jieba",
+    "source_code",
+}
+
+_TOKENIZER_ALIASES = {
+    "chinese_lindera": "lindera(chinese)",
+    "japanese_lindera": "lindera(japanese)",
+    "korean_lindera": "lindera(korean)",
+    "lindera_chinese": "lindera(chinese)",
+    "lindera_japanese": "lindera(japanese)",
+    "lindera_korean": "lindera(korean)",
+}
+
+
+def normalize_pg_search_tokenizer(value: str | None) -> str:
+    """Validate and normalize a ParadeDB pg_search tokenizer setting.
+
+    Returns an empty string when unset. The returned value is safe to embed after
+    ``pdb.`` in a CREATE INDEX expression.
+    """
+
+    tokenizer = (value or "").strip().lower()
+    if not tokenizer:
+        return ""
+
+    if tokenizer in _TOKENIZER_ALIASES:
+        return _TOKENIZER_ALIASES[tokenizer]
+
+    if tokenizer in _SIMPLE_TOKENIZERS:
+        return tokenizer
+
+    lindera_match = re.fullmatch(r"lindera\((chinese|japanese|korean)\)", tokenizer)
+    if lindera_match:
+        return tokenizer
+
+    ngram_match = re.fullmatch(r"(ngram|edge_ngram)\((\d{1,3}),\s*(\d{1,3})\)", tokenizer)
+    if ngram_match:
+        kind, min_gram, max_gram = ngram_match.groups()
+        min_value = int(min_gram)
+        max_value = int(max_gram)
+        if min_value <= 0 or min_value > max_value:
+            raise ValueError(
+                f"Invalid {PG_SEARCH_TOKENIZER_ENV}: {value!r}. "
+                "ngram and edge_ngram require positive min/max gram sizes with min <= max."
+            )
+        return f"{kind}({min_value},{max_value})"
+
+    raise ValueError(
+        f"Invalid {PG_SEARCH_TOKENIZER_ENV}: {value!r}. "
+        "Supported values are: unicode_words, simple, whitespace, literal, "
+        "literal_normalized, chinese_compatible, icu, jieba, source_code, "
+        "chinese_lindera, japanese_lindera, korean_lindera, or "
+        "lindera(chinese|japanese|korean), ngram(min,max), or edge_ngram(min,max)."
+    )
+
+
+def pg_search_bm25_columns(
+    key_field: str,
+    text_fields: Sequence[str],
+    tokenizer: str | None,
+) -> str:
+    """Build a ParadeDB BM25 column list for CREATE INDEX."""
+
+    normalized = normalize_pg_search_tokenizer(tokenizer)
+    if not normalized:
+        return ", ".join([key_field, *text_fields])
+
+    return ", ".join([key_field, *(f"({field}::pdb.{normalized})" for field in text_fields)])

hindsight-api-slim/hindsight_api/_thread_limits.py

@@ -0,0 +1,107 @@
+"""Process-level caps for native ML thread pools.
+
+OpenBLAS, OpenMP, and MKL each spawn a worker pool sized to the host CPU count
+the first time they are loaded (numpy pulls in OpenBLAS eagerly; torch and
+onnxruntime load their pools lazily on first inference). Hindsight already
+parallelizes at the request level via thread-pool executors (embeddings on the
+default executor, the reranker on its own pool), so these native intra-op pools
+oversubscribe the CPU: on a many-core host the process accumulates 100+ native
+threads, which inflates memory and, under contention, can degrade throughput.
+
+We bound each pool to ``_MAX_NATIVE_THREADS`` (or the available CPU count, if
+smaller). "Available" is the CPU budget actually granted to the process, not
+``os.cpu_count()``: in a CPU-limited container ``os.cpu_count()`` still reports
+the host's cores, so sizing pools by it oversubscribes the container's real
+quota — the exact failure mode this guards against. We therefore take the
+smallest of the CPU-affinity set, the cgroup CPU quota, and ``os.cpu_count()``.
+
+Every cap is applied with ``setdefault`` so an operator who has deliberately
+tuned one of these variables keeps their value. This must run *before* numpy,
+torch, or onnxruntime are imported — those libraries read the variables only at
+load time — which is why it is invoked at the very top of
+``hindsight_api/__init__.py``, ahead of the package's other imports.
+"""
+
+from __future__ import annotations
+
+import os
+
+# Native threading env vars, each read by the respective library at load time.
+_NATIVE_THREAD_VARS = (
+    "OMP_NUM_THREADS",  # OpenMP — torch, onnxruntime, some BLAS builds
+    "OPENBLAS_NUM_THREADS",  # OpenBLAS — numpy's default BLAS
+    "MKL_NUM_THREADS",  # Intel MKL — numpy/torch when MKL-backed
+    "NUMEXPR_NUM_THREADS",  # numexpr expression engine
+)
+
+# Upper bound on intra-op threads per native pool. Bounds runaway growth on
+# many-core hosts without serialising single-request inference.
+_MAX_NATIVE_THREADS = 16
+
+
+def _quota_to_cpus(quota: int, period: int) -> int | None:
+    """Whole CPUs from a CFS quota/period pair, or None if unlimited."""
+    if quota > 0 and period > 0:
+        # Floor (never round up) so we never exceed the granted budget.
+        return max(1, quota // period)
+    return None
+
+
+def _parse_cgroup_v2_cpu_max(text: str) -> int | None:
+    """Parse cgroup v2 ``cpu.max`` ("<quota> <period>", or "max <period>")."""
+    parts = text.split()
+    if len(parts) >= 2 and parts[0] != "max":
+        try:
+            return _quota_to_cpus(int(parts[0]), int(parts[1]))
+        except ValueError:
+            return None
+    return None
+
+
+def _cgroup_cpu_quota() -> int | None:
+    """Effective CPUs from the cgroup CPU quota, or None if unlimited/unknown."""
+    try:  # cgroup v2
+        with open("/sys/fs/cgroup/cpu.max") as fh:
+            return _parse_cgroup_v2_cpu_max(fh.read())
+    except OSError:
+        pass
+    try:  # cgroup v1
+        with open("/sys/fs/cgroup/cpu/cpu.cfs_quota_us") as fh:
+            quota = int(fh.read())
+        with open("/sys/fs/cgroup/cpu/cpu.cfs_period_us") as fh:
+            period = int(fh.read())
+        return _quota_to_cpus(quota, period)
+    except (OSError, ValueError):
+        return None
+
+
+def _available_cpu_count() -> int:
+    """CPUs actually available to this process.
+
+    The smallest of the CPU-affinity set (cpuset / ``--cpuset-cpus``), the
+    cgroup CPU quota (``--cpus``), and ``os.cpu_count()`` — each captures a
+    different way the budget can be constrained, and the last alone overcounts
+    inside a limited container.
+    """
+    candidates = [os.cpu_count() or 1]
+    if hasattr(os, "sched_getaffinity"):
+        try:
+            candidates.append(len(os.sched_getaffinity(0)))
+        except OSError:
+            pass
+    quota = _cgroup_cpu_quota()
+    if quota is not None:
+        candidates.append(quota)
+    return max(1, min(candidates))
+
+
+def default_native_thread_count() -> int:
+    """Per-pool cap: ``_MAX_NATIVE_THREADS``, or available CPUs if fewer."""
+    return min(_MAX_NATIVE_THREADS, _available_cpu_count())
+
+
+def apply_default_thread_limits() -> None:
+    """Cap native ML thread pools unless the operator has set the var already."""
+    value = str(default_native_thread_count())
+    for var in _NATIVE_THREAD_VARS:
+        os.environ.setdefault(var, value)

hindsight-api-slim/hindsight_api/_vector_index.py

@@ -0,0 +1,228 @@
+"""Shared PostgreSQL vector-extension dispatch helpers."""
+
+from __future__ import annotations
+
+import logging
+import os
+
+from sqlalchemy import text
+from sqlalchemy.engine import Connection
+
+logger = logging.getLogger(__name__)
+
+# Extensions a user can set via HINDSIGHT_API_VECTOR_EXTENSION.
+CONFIGURABLE_EXTENSIONS = ("pgvector", "pgvectorscale", "vchord", "scann")
+
+# Extensions detect_vector_extension() can return. pg_diskann is a runtime-only
+# resolution from a configured "pgvectorscale" backend on Azure (uses a different
+# WITH clause), never a value the user sets directly.
+RESOLVED_EXTENSIONS = (*CONFIGURABLE_EXTENSIONS, "pg_diskann")
+
+# Backwards-compatible alias for older imports.
+VALID_EXTENSIONS = CONFIGURABLE_EXTENSIONS
+
+SCANN_MIN_ROWS_FOR_AUTO_INDEX = 10_000
+
+
+_EXTENSION_NAMES = {
+    "pgvector": "vector",
+    "pgvectorscale": "vectorscale",
+    "vchord": "vchord",
+    "scann": "alloydb_scann",
+}
+
+_INDEX_USING_CLAUSES = {
+    "pgvector": "USING hnsw (embedding vector_cosine_ops)",
+    "pgvectorscale": "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)",
+    "pg_diskann": "USING diskann (embedding vector_cosine_ops) WITH (max_neighbors = 50)",
+    "vchord": "USING vchordrq (embedding vector_cosine_ops)",
+    "scann": "USING scann (embedding cosine) WITH (mode = 'AUTO')",
+}
+
+_INDEX_TYPE_KEYWORDS = {
+    "pgvector": "hnsw",
+    "pgvectorscale": "diskann",
+    "pg_diskann": "diskann",
+    "vchord": "vchordrq",
+    "scann": "scann",
+}
+
+# Per-backend ANN search-time tuning GUCs. Each entry is a tuple of
+# (guc_name, value) pairs the caller can apply with SET or SET LOCAL.
+#
+# - pgvector exposes hnsw.ef_search. The 60 / 200 pair is unchanged from the
+#   pre-dispatcher code (internal benchmarks tuned around our embedding count
+#   and recall floor; see the link_utils / pool init call sites for the
+#   latency-vs-recall framing).
+# - vchord exposes vchordrq.probes (no default; see VectorChord issue #392)
+#   and vchordrq.epsilon (default 1.9). probes = 10 / 30 are starting
+#   defaults pending a workload-specific sweep — vchordrq's recall curve
+#   shape differs from HNSW's, so the pgvector numbers don't translate
+#   directly. Revisit with a per-cluster benchmark once we have production
+#   recall data; until then these are deliberately conservative on the
+#   high-recall path. We leave epsilon at its default; tightening it is a
+#   separate trade-off.
+# - pgvectorscale / pg_diskann / scann do not expose an equivalent per-statement
+#   knob in the engine today, so the dispatcher returns no statements for them.
+_ANN_TUNING_LOW_LATENCY: dict[str, tuple[tuple[str, str], ...]] = {
+    "pgvector": (("hnsw.ef_search", "60"),),
+    "vchord": (("vchordrq.probes", "10"),),
+}
+_ANN_TUNING_HIGH_RECALL: dict[str, tuple[tuple[str, str], ...]] = {
+    "pgvector": (("hnsw.ef_search", "200"),),
+    "vchord": (("vchordrq.probes", "30"),),
+}
+
+_EXTENSION_INSTALL_SQL = {
+    "pgvector": ("CREATE EXTENSION IF NOT EXISTS vector",),
+    "pgvectorscale": (
+        "CREATE EXTENSION IF NOT EXISTS vector",
+        "CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE",
+    ),
+    "vchord": ("CREATE EXTENSION IF NOT EXISTS vchord CASCADE",),
+    "scann": (
+        "CREATE EXTENSION IF NOT EXISTS vector",
+        "CREATE EXTENSION IF NOT EXISTS alloydb_scann CASCADE",
+    ),
+}
+
+_INSTALL_HINTS = {
+    "pgvector": "CREATE EXTENSION vector;",
+    "pgvectorscale": "CREATE EXTENSION vector; then CREATE EXTENSION vectorscale CASCADE; (or pg_diskann on Azure)",
+    "vchord": "CREATE EXTENSION vchord CASCADE;",
+    "scann": "CREATE EXTENSION vector; then CREATE EXTENSION alloydb_scann CASCADE;",
+}
+
+
+def configured_vector_extension() -> str:
+    """Return the user-configured vector backend extension.
+
+    Reads ``HINDSIGHT_API_VECTOR_EXTENSION`` (default ``"pgvector"``) and
+    validates it via :func:`validate_extension`. This is the single source of
+    truth for runtime code that needs to dispatch behaviour by vector backend;
+    callers should prefer this over reading the env var directly, so the
+    default value and the lookup mechanism live in one place.
+    """
+    return validate_extension(os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector"))
+
+
+def validate_extension(name: str) -> str:
+    """Return a normalized configurable vector extension name or raise.
+
+    Used at the user-facing config boundary; pg_diskann is rejected here because
+    it is a detection-time alias, never a value the user sets directly.
+    """
+    ext = name.lower()
+    if ext not in CONFIGURABLE_EXTENSIONS:
+        valid = ", ".join(CONFIGURABLE_EXTENSIONS)
+        raise ValueError(f"Invalid vector_extension: {name}. Must be one of: {valid}")
+    return ext
+
+
+def _normalize_resolved(name: str) -> str:
+    """Normalize either a user-configurable or detect-time extension name."""
+    ext = name.lower()
+    if ext not in RESOLVED_EXTENSIONS:
+        valid = ", ".join(RESOLVED_EXTENSIONS)
+        raise ValueError(f"Unknown vector extension: {name}. Must be one of: {valid}")
+    return ext
+
+
+def pg_extension_name(ext: str) -> str:
+    """Return the PostgreSQL extension name for a configured vector backend."""
+    return _EXTENSION_NAMES[validate_extension(ext)]
+
+
+def index_using_clause(ext: str) -> str:
+    """Return the CREATE INDEX USING clause for the vector backend."""
+    return _INDEX_USING_CLAUSES[_normalize_resolved(ext)]
+
+
+def index_type_keyword(ext: str) -> str:
+    """Return the keyword that identifies this index type in pg_indexes.indexdef."""
+    return _INDEX_TYPE_KEYWORDS[_normalize_resolved(ext)]
+
+
+def minimum_rows_for_index(ext: str) -> int:
+    """Return the minimum populated embedding rows before creating this index type."""
+    return SCANN_MIN_ROWS_FOR_AUTO_INDEX if _normalize_resolved(ext) == "scann" else 0
+
+
+def should_defer_index_creation(ext: str, row_count: int) -> bool:
+    """Return True when index creation should wait for more embeddings."""
+    minimum_rows = minimum_rows_for_index(ext)
+    return minimum_rows > 0 and row_count < minimum_rows
+
+
+def ann_search_tuning_settings(ext: str, *, kind: str) -> tuple[tuple[str, str], ...]:
+    """Return per-backend (guc_name, value) pairs for ANN search-time tuning.
+
+    ``kind`` is ``"low_latency"`` for retain-side link probing (smaller probe
+    count, lower recall, lower latency) and ``"high_recall"`` for connection
+    init in the pool (larger probe count, higher recall). Callers wrap each
+    pair with ``SET LOCAL`` or ``SET`` themselves so the same dispatcher works
+    for both transaction-scoped and session-scoped use. Returns an empty tuple
+    for backends without an equivalent knob.
+    """
+    if kind == "low_latency":
+        table = _ANN_TUNING_LOW_LATENCY
+    elif kind == "high_recall":
+        table = _ANN_TUNING_HIGH_RECALL
+    else:
+        raise ValueError(f"Unknown ANN tuning kind: {kind!r}")
+    return table.get(_normalize_resolved(ext), ())
+
+
+def uses_per_bank_vector_indexes(ext: str) -> bool:
+    """Return whether the backend should create per-bank partial vector indexes."""
+    return _normalize_resolved(ext) != "scann"
+
+
+def bootstrap_extension(conn: Connection, ext: str) -> None:
+    """Install the configured vector extension and any prerequisites if possible."""
+    normalized = validate_extension(ext)
+    for statement in _EXTENSION_INSTALL_SQL[normalized]:
+        conn.execute(text(statement))
+
+
+def detect_vector_extension(conn: Connection, vector_extension: str = "pgvector") -> str:
+    """Validate the configured vector extension exists and return the index backend."""
+    configured_ext = validate_extension(vector_extension)
+
+    if configured_ext == "pgvectorscale":
+        pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
+        if not pgvector_check:
+            raise RuntimeError(
+                "DiskANN (pgvectorscale/pg_diskann) requires pgvector to be installed. "
+                f"Install it with: {_INSTALL_HINTS['pgvectorscale']}"
+            )
+
+        vectorscale_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vectorscale'")).scalar()
+        pg_diskann_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'pg_diskann'")).scalar()
+
+        if vectorscale_check:
+            logger.debug("Using vector extension: pgvectorscale (DiskANN)")
+            return "pgvectorscale"
+        if pg_diskann_check:
+            logger.debug("Using vector extension: pg_diskann (Azure DiskANN)")
+            return "pg_diskann"
+
+        raise RuntimeError(
+            "Configured vector extension 'pgvectorscale' not found. Install either:\n"
+            "  - pgvectorscale (open source): CREATE EXTENSION vectorscale CASCADE;\n"
+            "  - pg_diskann (Azure): CREATE EXTENSION pg_diskann CASCADE;"
+        )
+
+    extension_name = pg_extension_name(configured_ext)
+    extension_check = conn.execute(
+        text("SELECT 1 FROM pg_extension WHERE extname = :extension_name"),
+        {"extension_name": extension_name},
+    ).scalar()
+    if not extension_check:
+        raise RuntimeError(
+            f"Configured vector extension '{configured_ext}' not found. "
+            f"Install it with: {_INSTALL_HINTS[configured_ext]}"
+        )
+
+    logger.debug("Using configured vector extension: %s", configured_ext)
+    return configured_ext

hindsight-api-slim/hindsight_api/admin/cli.py

@@ -0,0 +1,669 @@
+"""PostgreSQL-only admin utilities (backup, restore, migration, worker management).
+
+Not supported on Oracle backends. Uses asyncpg.connect() directly, binary COPY,
+TRUNCATE CASCADE, and REFRESH MATERIALIZED VIEW — all inherently PG-specific.
+"""
+
+import asyncio
+import io
+import json
+import logging
+import zipfile
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import asyncpg
+import typer
+
+from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
+from ..engine.memory_engine import _current_schema
+from ..engine.schema import fq_table_explicit as _fq_table
+from ..engine.transfer import export_bank
+from ..extensions import TenantExtension, load_extension
+from ..pg0 import parse_pg0_url, resolve_database_url
+
+# Setup logging
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(message)s",
+)
+logger = logging.getLogger(__name__)
+
+app = typer.Typer(name="hindsight-admin", help="Hindsight administrative commands")
+
+# Tables to backup/restore in foreign-key dependency order (parents first).
+# Restore COPYs in this order and TRUNCATEs in reverse, so every child must
+# appear after the tables it references.
+#
+# This must cover EVERY persistent PostgreSQL table in the schema — a missing
+# entry silently drops that table's data on restore (and, worse, restore's
+# `TRUNCATE banks CASCADE` wipes any FK-to-banks child like mental_models even
+# when it was never backed up). test_admin_backup_restore.py asserts this list
+# equals the live schema's tables, so adding a migration that creates a table
+# without adding it here fails CI. Oracle-only tables (e.g. observation_sources)
+# are intentionally absent — admin backup/restore is PostgreSQL-only.
+BACKUP_TABLES = [
+    "banks",
+    "documents",
+    "entities",
+    "chunks",
+    "memory_units",
+    "unit_entities",
+    "entity_cooccurrences",
+    "memory_links",
+    "observation_history",
+    "mental_models",
+    "mental_model_history",
+    "directives",
+    "async_operations",
+    "webhooks",
+    "file_storage",
+    "audit_log",
+    "llm_requests",
+    "graph_maintenance_queue",
+]
+
+MANIFEST_VERSION = "1"
+
+
+async def _admin_connect(db_url: str) -> asyncpg.Connection:
+    """Open a raw asyncpg connection to an admin DB URL.
+
+    ``resolve_database_url`` handles both plain ``postgres://`` (passthrough) and
+    ``pg0://`` (boots the embedded server and returns its real libpq URL), so this
+    is the only step needed to connect. JSON codecs are registered so ``jsonb``
+    columns decode to Python objects (used by the export row dumps).
+    """
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    conn = await asyncpg.connect(await resolve_database_url(db_url))
+    for type_name in ("json", "jsonb"):
+        await conn.set_type_codec(type_name, encoder=json.dumps, decoder=json.loads, schema="pg_catalog")
+    return conn
+
+
+async def _backup(database_url: str, output_path: Path, schema: str = "public") -> dict[str, Any]:
+    """Backup all tables to a zip file using binary COPY protocol."""
+    conn = await asyncpg.connect(database_url)
+    try:
+        tables: dict[str, Any] = {}
+        manifest: dict[str, Any] = {
+            "version": MANIFEST_VERSION,
+            "created_at": datetime.now(timezone.utc).isoformat(),
+            "schema": schema,
+            "tables": tables,
+        }
+
+        # Use a transaction with REPEATABLE READ isolation to get a consistent
+        # snapshot across all tables. This prevents race conditions where
+        # entity_cooccurrences could reference entities created after the
+        # entities table was backed up.
+        async with conn.transaction(isolation="repeatable_read"):
+            with zipfile.ZipFile(output_path, "w", zipfile.ZIP_DEFLATED) as zf:
+                for i, table in enumerate(BACKUP_TABLES, 1):
+                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Backing up {table}...", nl=False)
+
+                    buffer = io.BytesIO()
+
+                    # Use binary COPY for exact type preservation
+                    # asyncpg requires schema_name as separate parameter
+                    await conn.copy_from_table(table, schema_name=schema, output=buffer, format="binary")
+
+                    data = buffer.getvalue()
+                    zf.writestr(f"{table}.bin", data)
+
+                    # Get row count for manifest
+                    qualified_table = _fq_table(table, schema)
+                    row_count = await conn.fetchval(f"SELECT COUNT(*) FROM {qualified_table}")
+                    tables[table] = {
+                        "rows": row_count,
+                        "size_bytes": len(data),
+                    }
+
+                    typer.echo(f" {row_count} rows")
+
+                zf.writestr("manifest.json", json.dumps(manifest, indent=2))
+
+        return manifest
+    finally:
+        await conn.close()
+
+
+async def _restore(database_url: str, input_path: Path, schema: str = "public") -> dict[str, Any]:
+    """Restore all tables from a zip file using binary COPY protocol."""
+    conn = await asyncpg.connect(database_url)
+    try:
+        with zipfile.ZipFile(input_path, "r") as zf:
+            # Read and validate manifest
+            manifest: dict[str, Any] = json.loads(zf.read("manifest.json"))
+            if manifest.get("version") != MANIFEST_VERSION:
+                raise ValueError(f"Unsupported backup version: {manifest.get('version')}")
+
+            # Use a transaction for atomic restore - either all tables are
+            # restored or none are, preventing partial/inconsistent state.
+            async with conn.transaction():
+                typer.echo("  Clearing existing data...")
+                # Truncate tables in reverse order (respects FK constraints)
+                for table in reversed(BACKUP_TABLES):
+                    qualified_table = _fq_table(table, schema)
+                    await conn.execute(f"TRUNCATE TABLE {qualified_table} CASCADE")
+
+                # Restore tables in forward order
+                for i, table in enumerate(BACKUP_TABLES, 1):
+                    filename = f"{table}.bin"
+                    if filename not in zf.namelist():
+                        typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] {table}: skipped (not in backup)")
+                        continue
+
+                    expected_rows = manifest["tables"].get(table, {}).get("rows", "?")
+                    typer.echo(f"  [{i}/{len(BACKUP_TABLES)}] Restoring {table}... {expected_rows} rows")
+
+                    data = zf.read(filename)
+                    buffer = io.BytesIO(data)
+                    # asyncpg requires schema_name as separate parameter
+                    await conn.copy_to_table(table, schema_name=schema, source=buffer, format="binary")
+
+                # Refresh materialized view
+                typer.echo("  Refreshing materialized views...")
+                await conn.execute(f"REFRESH MATERIALIZED VIEW {_fq_table('memory_units_bm25', schema)}")
+
+        return manifest
+    finally:
+        await conn.close()
+
+
+async def _run_backup(db_url: str, output: Path, schema: str = "public") -> dict[str, Any]:
+    """Resolve database URL and run backup."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+    return await _backup(resolved_url, output, schema)
+
+
+async def _run_restore(db_url: str, input_file: Path, schema: str = "public") -> dict[str, Any]:
+    """Resolve database URL and run restore."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+    return await _restore(resolved_url, input_file, schema)
+
+
+@app.command()
+def backup(
+    output: Path = typer.Argument(..., help="Output file path (.zip)"),
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema to backup"),
+):
+    """Backup the Hindsight database to a zip file."""
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if output.suffix != ".zip":
+        output = output.with_suffix(".zip")
+
+    typer.echo(f"Backing up database (schema: {schema}) to {output}...")
+
+    manifest = asyncio.run(_run_backup(config.database_url, output, schema))
+
+    total_rows = sum(t["rows"] for t in manifest["tables"].values())
+    typer.echo(f"Backed up {total_rows} rows across {len(BACKUP_TABLES)} tables")
+    typer.echo(f"Backup saved to {output}")
+
+
+@app.command()
+def restore(
+    input_file: Path = typer.Argument(..., help="Input backup file (.zip)"),
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema to restore to"),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
+):
+    """Restore the database from a backup file. WARNING: This deletes all existing data."""
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if not input_file.exists():
+        typer.echo(f"Error: File not found: {input_file}", err=True)
+        raise typer.Exit(1)
+
+    if not yes:
+        typer.confirm(
+            "This will DELETE all existing data and replace it with the backup. Continue?",
+            abort=True,
+        )
+
+    typer.echo(f"Restoring database (schema: {schema}) from {input_file}...")
+
+    manifest = asyncio.run(_run_restore(config.database_url, input_file, schema))
+
+    total_rows = sum(t["rows"] for t in manifest["tables"].values())
+    typer.echo(f"Restored {total_rows} rows across {len(BACKUP_TABLES)} tables")
+    typer.echo("Restore complete")
+
+
+async def _run_migration(
+    db_url: str,
+    schema: str | None = None,
+    base_schema: str = DEFAULT_DATABASE_SCHEMA,
+    embedding_dimension: int | None = None,
+) -> list[str]:
+    """Resolve database URL and run migrations for one schema or all discovered schemas."""
+    from ..migrations import (
+        ensure_embedding_dimension,
+        ensure_text_search_extension,
+        ensure_vector_extension,
+        run_migrations,
+    )
+
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+
+    config = HindsightConfig.from_env()
+    if schema:
+        schemas = [schema]
+    else:
+        tenant_extension = load_extension("TENANT", TenantExtension)
+
+        schemas = [base_schema or DEFAULT_DATABASE_SCHEMA]
+        if tenant_extension:
+            tenants = await tenant_extension.list_tenants()
+            schemas.extend(tenant.schema for tenant in tenants if tenant.schema)
+
+        # Preserve order while removing duplicates.
+        schemas = list(dict.fromkeys(schemas))
+
+    for schema in schemas:
+        run_migrations(resolved_url, schema=schema, migration_database_url=config.migration_database_url)
+
+    if embedding_dimension is not None:
+        for schema in schemas:
+            ensure_embedding_dimension(
+                resolved_url,
+                embedding_dimension,
+                schema=schema,
+                vector_extension=config.vector_extension,
+            )
+
+    for schema in schemas:
+        ensure_vector_extension(
+            resolved_url,
+            vector_extension=config.vector_extension,
+            schema=schema,
+        )
+
+    for schema in schemas:
+        ensure_text_search_extension(
+            resolved_url,
+            text_search_extension=config.text_search_extension,
+            pg_search_tokenizer=config.text_search_extension_pg_search_tokenizer,
+            schema=schema,
+        )
+
+    return schemas
+
+
+@app.command(name="run-db-migration")
+def run_db_migration(
+    schema: str | None = typer.Option(
+        None,
+        "--schema",
+        "-s",
+        help="Database schema to run migrations on. If omitted, migrate the base schema and all discovered tenant schemas.",
+    ),
+    embedding_dimension: int | None = typer.Option(
+        None,
+        "--embedding-dimension",
+        help="Expected embedding dimension to enforce after migrations. Omit to skip dimension sync.",
+    ),
+):
+    """Run database migrations to the latest version."""
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if schema:
+        typer.echo(f"Running database migrations for schema: {schema}...")
+    else:
+        typer.echo("Running database migrations for base schema and all discovered tenant schemas...")
+
+    schemas = asyncio.run(
+        _run_migration(
+            config.database_url,
+            schema=schema,
+            base_schema=config.database_schema,
+            embedding_dimension=embedding_dimension,
+        )
+    )
+
+    typer.echo(f"Database migrations completed successfully for {len(schemas)} schema(s)")
+
+
+async def _run_export_bank(db_url: str, bank_id: str, output: Path, schema: str, include_history: bool) -> int:
+    """Export a whole bank to a ZIP archive."""
+    conn = await _admin_connect(db_url)
+    try:
+        # export_bank resolves table names via fq_table (the _current_schema
+        # contextvar); set it so the raw connection targets the right schema.
+        _current_schema.set(schema)
+        data = await export_bank(conn, bank_id, include_history=include_history)
+    finally:
+        await conn.close()
+
+    output.write_bytes(data)
+    return len(data)
+
+
+@app.command(name="export-bank")
+def export_bank_command(
+    bank_id: str = typer.Option(..., "--bank", "-b", help="Bank id to export."),
+    output: Path = typer.Option(..., "--output", "-o", help="Path to write the .zip archive."),
+    schema: str | None = typer.Option(
+        None,
+        "--schema",
+        "-s",
+        help="Database schema the bank lives in. Defaults to the configured base schema.",
+    ),
+    include_history: bool = typer.Option(
+        False,
+        "--include-history",
+        help="Also export operational history (audit_log, llm_requests). Off by default.",
+    ),
+):
+    """Export an entire bank to a portable ZIP (no embeddings — regenerated on import).
+
+    Carries documents, facts, observations, bank config, mental models, directives
+    and webhooks so the bank can be imported into a new instance configured with a
+    different embedding model / vector / text-search backend.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
+    typer.echo(f"Exporting bank '{bank_id}' from schema '{target_schema}'...")
+
+    size = asyncio.run(_run_export_bank(config.database_url, bank_id, output, target_schema, include_history))
+
+    typer.echo(f"Exported bank '{bank_id}' to {output} ({size} bytes)")
+
+
+async def _run_import_bank(archive_path: Path, schema: str, target_bank_id: str | None, include_history: bool):
+    """Boot a MemoryEngine (for the target's embedding model) and restore a bank archive."""
+    # MemoryEngine is heavy (loads embeddings); import it lazily so other admin
+    # commands don't pay for it. _current_schema is imported at module top.
+    from ..engine.memory_engine import MemoryEngine
+    from ..models import RequestContext
+
+    archive_bytes = archive_path.read_bytes()
+    # run_migrations=True so a fresh target instance is provisioned at this
+    # instance's embedding dimension / vector / text-search backend before restore.
+    engine = MemoryEngine(run_migrations=True)
+    await engine.initialize()
+    try:
+        _current_schema.set(schema)
+        context = RequestContext(internal=True, user_initiated=True)
+        return await engine.import_bank_async(
+            archive_bytes,
+            context,
+            target_bank_id=target_bank_id,
+            include_history=include_history,
+        )
+    finally:
+        await engine.close()
+
+
+@app.command(name="import-bank")
+def import_bank_command(
+    archive: Path = typer.Option(..., "--archive", "-a", help="Path to the .zip produced by export-bank."),
+    schema: str | None = typer.Option(
+        None, "--schema", "-s", help="Target schema. Defaults to the configured base schema."
+    ),
+    target_bank: str | None = typer.Option(
+        None, "--target-bank", help="Override the bank id (defaults to the archive's source bank)."
+    ),
+    include_history: bool = typer.Option(
+        False, "--include-history", help="Also restore operational history if present in the archive."
+    ),
+):
+    """Restore a whole bank from an export-bank archive into THIS instance.
+
+    Re-embeds facts with this instance's configured embedding model and rebuilds
+    links and indexes — the import half of a cross-instance migration. Run against
+    an instance configured with the desired embedding / vector / text-search backend.
+    The target bank must not already exist (import restores a whole bank, not a merge).
+    """
+    config = HindsightConfig.from_env()
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    target_schema = schema or config.database_schema or DEFAULT_DATABASE_SCHEMA
+    typer.echo(f"Importing bank archive '{archive}' into schema '{target_schema}'...")
+
+    result = asyncio.run(_run_import_bank(archive, target_schema, target_bank, include_history))
+
+    typer.echo(
+        f"Imported bank '{result.bank_id}': {result.documents_imported} doc(s), "
+        f"{result.facts_imported} fact(s), {result.observations_imported} observation(s), "
+        f"{result.mental_models_imported} mental model(s), {result.directives_imported} directive(s), "
+        f"{result.webhooks_imported} webhook(s), {result.history_rows_imported} history row(s)"
+    )
+
+
+async def _decommission_worker(db_url: str, worker_id: str, schema: str = "public") -> int:
+    """Release all tasks owned by a worker, setting them back to pending status."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+
+    conn = await asyncpg.connect(resolved_url)
+    try:
+        table = _fq_table("async_operations", schema)
+        result = await conn.fetch(
+            f"""
+            UPDATE {table}
+            SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
+            WHERE worker_id = $1 AND status = 'processing'
+            RETURNING operation_id
+            """,
+            worker_id,
+        )
+        return len(result)
+    finally:
+        await conn.close()
+
+
+@app.command(name="decommission-worker")
+def decommission_worker(
+    worker_id: str = typer.Argument(..., help="Worker ID to decommission"),
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
+):
+    """Release all tasks owned by a worker (sets status back to pending).
+
+    Use this command when a worker has crashed or been removed without graceful shutdown.
+    All tasks that were being processed by the worker will be released back to the queue
+    so other workers can pick them up.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if not yes:
+        typer.confirm(
+            f"This will release all tasks owned by worker '{worker_id}' back to pending. Continue?",
+            abort=True,
+        )
+
+    typer.echo(f"Decommissioning worker '{worker_id}' (schema: {schema})...")
+
+    count = asyncio.run(_decommission_worker(config.database_url, worker_id, schema))
+
+    if count > 0:
+        typer.echo(f"Released {count} task(s) from worker '{worker_id}'")
+    else:
+        typer.echo(f"No tasks found for worker '{worker_id}'")
+
+
+async def _decommission_all_workers(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
+    """Release all processing tasks from all workers, setting them back to pending status."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+
+    conn = await asyncpg.connect(resolved_url)
+    try:
+        table = _fq_table("async_operations", schema)
+        rows = await conn.fetch(
+            f"""
+            UPDATE {table}
+            SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
+            WHERE status = 'processing'
+            RETURNING operation_id, worker_id, operation_type
+            """,
+        )
+        return [dict(r) for r in rows]
+    finally:
+        await conn.close()
+
+
+@app.command(name="decommission-workers")
+def decommission_workers(
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
+):
+    """Release all processing tasks from all workers (sets status back to pending).
+
+    Use this command to recover from situations where one or more workers have crashed
+    or been removed without graceful shutdown. All tasks currently in 'processing' status
+    will be released back to the queue regardless of which worker owns them.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if not yes:
+        typer.confirm(
+            "This will release ALL processing tasks from ALL workers back to pending. Continue?",
+            abort=True,
+        )
+
+    typer.echo(f"Decommissioning all workers (schema: {schema})...")
+
+    released = asyncio.run(_decommission_all_workers(config.database_url, schema))
+
+    if released:
+        # Group by worker_id for summary
+        by_worker: dict[str, int] = {}
+        for row in released:
+            wid = row["worker_id"] or "unknown"
+            by_worker[wid] = by_worker.get(wid, 0) + 1
+
+        typer.echo(f"Released {len(released)} task(s):")
+        for wid, count in by_worker.items():
+            typer.echo(f"  {wid}: {count} task(s)")
+    else:
+        typer.echo("No processing tasks found")
+
+
+async def _worker_status(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
+    """Get all processing tasks grouped by worker with their last update time."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+
+    conn = await asyncpg.connect(resolved_url)
+    try:
+        table = _fq_table("async_operations", schema)
+        rows = await conn.fetch(
+            f"""
+            SELECT worker_id, operation_id, operation_type, bank_id,
+                   claimed_at, updated_at,
+                   now() - claimed_at AS running_for,
+                   now() - updated_at AS last_update_ago
+            FROM {table}
+            WHERE status = 'processing'
+            ORDER BY worker_id, claimed_at
+            """,
+        )
+        return [dict(r) for r in rows]
+    finally:
+        await conn.close()
+
+
+@app.command(name="worker-status")
+def worker_status(
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
+):
+    """Show all currently processing tasks grouped by worker.
+
+    Displays each worker's active tasks with operation type, bank, how long
+    the task has been running, and when it was last updated. Useful for
+    identifying dead workers with orphaned tasks.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    rows = asyncio.run(_worker_status(config.database_url, schema))
+
+    if not rows:
+        typer.echo("No processing tasks found")
+        return
+
+    # Group by worker_id
+    by_worker: dict[str, list[dict[str, Any]]] = {}
+    for row in rows:
+        wid = row["worker_id"] or "unknown"
+        by_worker.setdefault(wid, []).append(row)
+
+    typer.echo(f"Processing tasks across {len(by_worker)} worker(s):\n")
+    for wid, tasks in by_worker.items():
+        typer.echo(f"Worker: {wid} ({len(tasks)} task(s))")
+        for task in tasks:
+            op_id = str(task["operation_id"])[:8]
+            running_for = task["running_for"]
+            last_update = task["last_update_ago"]
+            typer.echo(
+                f"  {op_id}  {task['operation_type']:<20s} bank={task['bank_id']}"
+                f"  running={running_for}  last_update={last_update} ago"
+            )
+        typer.echo("")
+
+
+def main():
+    app()
+
+
+if __name__ == "__main__":
+    main()

hindsight-api-slim/hindsight_api/alembic/_dialect.py

@@ -0,0 +1,42 @@
+"""Dialect dispatcher for Alembic migrations.
+
+Each migration file declares a ``_pg_upgrade``/``_oracle_upgrade`` (and matching
+downgrades) function and routes ``upgrade()``/``downgrade()`` through
+``run_for_dialect``. The helper inspects the live connection's dialect name and
+runs the matching function — or no-ops if the migration doesn't apply to the
+current backend.
+
+Use ``None`` (or omit the kwarg) when a migration intentionally has no effect
+on a dialect; the helper treats it as a no-op.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+from alembic import op
+
+DialectFn = Callable[[], None]
+_SUPPORTED = ("postgresql", "oracle")
+
+
+def run_for_dialect(
+    *,
+    pg: DialectFn | None = None,
+    oracle: DialectFn | None = None,
+) -> None:
+    """Dispatch to the function matching the current bind's dialect.
+
+    Args:
+        pg: Function to run when the active bind is PostgreSQL.
+        oracle: Function to run when the active bind is Oracle.
+
+    Unrecognized dialects raise; an explicit ``None`` for the active dialect
+    is a no-op (the migration deliberately does nothing here).
+    """
+    name = op.get_bind().dialect.name
+    if name not in _SUPPORTED:
+        raise RuntimeError(f"Unsupported dialect for migration dispatch: {name!r}. Expected one of {_SUPPORTED}.")
+    fn = {"postgresql": pg, "oracle": oracle}[name]
+    if fn is not None:
+        fn()