feat: support litellm-sdk for reranker endpoint

.claude-plugin/marketplace.json

@@ -1,15 +0,0 @@
-{
-  "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
-  "name": "hindsight",
-  "description": "Official Hindsight integrations for Claude Code",
-  "owner": {
-    "name": "vectorize-io"
-  },
-  "plugins": [
-    {
-      "name": "hindsight-memory",
-      "description": "Automatic long-term memory for Claude Code via Hindsight",
-      "source": "./hindsight-integrations/claude-code"
-    }
-  ]
-}

.claude/skills/code-review/SKILL.md

@@ -1,205 +0,0 @@
----
-name: code-review
-description: Review changed code against project standards. Checks for missing tests, dead code, type safety, lint issues, and coding conventions. Run after completing any implementation work.
-user_invocable: true
----
-
-# Code Review
-
-Review all changed code against the project's quality standards and coding conventions.
-
-## Code Standards
-
-Read and internalize these standards before writing code. The review steps below verify compliance.
-
-### Python Style
-- Python 3.11+, type hints required
-- Async throughout (asyncpg, async FastAPI)
-- Pydantic models for request/response
-- Ruff for linting (line-length 120)
-- No Python files at project root - maintain clean directory structure
-- **Never use multi-item tuple return values** — not even for internal/private functions. Always use a dataclass or Pydantic model. No exceptions, no "it's just two values" shortcuts. If a function returns more than one value, define a named type for it.
-
-### Type Safety with Pydantic Models
-**NEVER use raw `dict` types for structured data** — this applies to all code, including internal helpers and private functions. If the dict has known keys, it must be a dataclass or Pydantic model:
-- Use Pydantic `BaseModel` for all data structures passed between functions
-- Use `@dataclass` for lightweight internal data containers when Pydantic validation isn't needed
-- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
-- Avoid `dict.get()` patterns - use typed model attributes instead
-- Parse external data (JSON, API responses) into Pydantic models at the boundary
-- This catches type errors at parse time, not deep in business logic
-- The only acceptable `dict` usage is for truly dynamic/unknown keys (e.g., arbitrary metadata, JSON blobs with no fixed schema)
-
-```python
-# BAD - error-prone dict access
-def process(data: dict) -> str:
-    return data.get("name", "")  # No validation, silent failures
-
-# GOOD - typed and validated
-class UserData(BaseModel):
-    name: str
-    created_at: datetime
-
-    @field_validator("created_at", mode="before")
-    @classmethod
-    def ensure_tz_aware(cls, v):
-        if isinstance(v, str):
-            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
-        if v.tzinfo is None:
-            return v.replace(tzinfo=timezone.utc)
-        return v
-
-def process(data: UserData) -> str:
-    return data.name  # Type-safe, validated at construction
-```
-
-### TypeScript Style
-- Next.js App Router for control plane
-- Tailwind CSS with shadcn/ui components
-
-### Code Comments
-- **Always comment non-trivial technical decisions** with the reasoning behind the choice. If someone would ask "why is it done this way?", there should be a comment.
-- **Keep comments up to date with history** — when changing an approach, update the comment to explain what was tried before and why it was changed. Comments serve as a tracker of previous implementations that likely had problems.
-- Don't comment obvious code — only where the "why" isn't self-evident from the code itself.
-
-```python
-# BAD - no context for future readers
-results = await asyncio.gather(*tasks, return_exceptions=True)
-
-# GOOD - explains the non-obvious choice
-# Use return_exceptions=True to avoid cancelling sibling tasks on failure.
-# Previously we used TaskGroup but it cancelled all tasks when one failed,
-# causing partial writes that left orphaned entity links (see #412).
-results = await asyncio.gather(*tasks, return_exceptions=True)
-```
-
-### Branch Hygiene
-- **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
-- **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
-
-### General Principles
-- Don't add features, refactor code, or make "improvements" beyond what was asked
-- Don't add unnecessary error handling for impossible scenarios
-- Don't create helpers or abstractions for one-time operations
-- No backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
-- Three similar lines of code is better than a premature abstraction
-
-## Review Steps
-
-### 1. Check branch hygiene
-
-- Run `git log --oneline main..HEAD` to list all commits on the branch.
-- Verify every commit is relevant to the feature/PR. Flag any unrelated commits.
-- Check the branch is based on a recent `origin/main` (no stale base).
-
-### 2. Identify changed files
-
-Run `git diff --name-only HEAD` (unstaged) and `git diff --cached --name-only` (staged) to get all changed files. If there are no local changes, diff against the base branch using `git diff main...HEAD --name-only` and `git diff main...HEAD` to review all commits on the current branch.
-
-### 3. Run linters
-
-```bash
-./scripts/hooks/lint.sh
-```
-
-Report any failures. Do NOT fix them yourself — just report.
-
-### 4. Check for dead code
-
-For each changed Python file, check for:
-- Unused imports (Ruff should catch these, but verify)
-- Functions/methods/classes that were added but are never called from anywhere
-- Variables assigned but never read
-- Commented-out code blocks that should be removed
-
-For each changed TypeScript file, check for:
-- Unused imports
-- Unused variables or functions
-- Commented-out code
-
-### 5. Check type safety (Python)
-
-For each changed Python file, check for violations:
-- **No raw `dict` for structured data** — must use Pydantic model or dataclass, even for internal/private functions (only exception: truly dynamic/unknown keys)
-- **No multi-item tuple returns** — must use dataclass or Pydantic model, even for internal/private functions (no exceptions)
-- **Missing type hints** on function parameters and return types
-- **Missing `@field_validator`** for datetime fields that should be timezone-aware
-
-### 6. Check for missing tests
-
-For each new or significantly changed function/endpoint/class:
-- Check if there is a corresponding test addition or update
-- New API endpoints MUST have integration tests
-- New utility functions MUST have unit tests
-- Bug fixes SHOULD have a regression test
-
-Flag any new logic that lacks test coverage.
-
-### 7. Check API consistency
-
-If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
-- Were the OpenAPI specs regenerated? (`./scripts/generate-openapi.sh`)
-- Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
-- Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
-
-### 8. Check code comments
-
-For each non-trivial change:
-- **New non-obvious logic** — is there a comment explaining the reasoning?
-- **Changed approach** — does the comment include what was done before and why it changed?
-- **Stale comments** — do existing comments near the changed code still accurately describe the behavior?
-
-### 9. Check integration completeness
-
-If any files in `hindsight-integrations/` were added or changed, verify:
-- **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
-- **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
-- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
-- **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
-
-### 10. Check MCP tool registration completeness
-
-If any new MCP tools were added or existing tools renamed in `hindsight-api-slim/hindsight_api/mcp_tools.py`:
-- **`_ALL_TOOLS` set** in `mcp_tools.py` — must include the new tool name
-- **`tools_to_register` default set** in `register_mcp_tools()` in `mcp_tools.py` — must include the new tool name
-- **`_SINGLE_BANK_TOOLS` set** in `hindsight-api-slim/hindsight_api/api/mcp.py` — must include the new tool if it is bank-scoped (not a bank-management tool like `list_banks`/`create_bank`)
-- **`MCP_TOOL_GROUPS`** in `hindsight-control-plane/src/components/bank-config-view.tsx` — must include the new tool in the appropriate group for the UI tool selector
-- **Tool count assertions** in tests (e.g., `test_mcp_tools.py`) — must be updated to reflect the new count
-
-### 11. Review against other coding standards
-
-Check the diff for violations of the standards listed above:
-- Python files at project root (not allowed)
-- Missing async patterns (should be async throughout)
-- Pydantic models for request/response
-- Line length > 120 chars
-- New features/code beyond what was asked (over-engineering)
-- Unnecessary error handling for impossible scenarios
-- Premature abstractions or speculative helpers
-- Backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
-
-### 12. Report findings
-
-Present a clear summary organized by severity:
-
-**Must fix** — issues that will break CI or violate hard project rules:
-- Unrelated commits on the branch
-- Lint failures
-- Missing type hints on public functions
-- Raw dict usage for structured data (including internal code)
-- Multi-item tuple returns (including internal code)
-- Missing tests for new endpoints
-- New integration missing tests, CI job, or release-integration.sh entry
-
-**Should fix** — issues that hurt code quality:
-- Dead code / unused imports missed by linter
-- Missing tests for non-trivial utility functions
-- Over-engineering beyond the task scope
-
-**Note** — observations that may or may not need action:
-- API changes that might need client regeneration
-- Patterns that deviate from nearby code style
-
-For each finding, include the file path, line number, and a brief explanation.
-
-Do NOT auto-fix any issues. Report all findings and let the user decide what to address. If there are no findings, confirm the code looks good.

.env.example

@@ -2,10 +2,10 @@
 # Copy this file to .env and fill in your values
 
 # LLM Configuration (Required)
-# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, volcano
+# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai
 HINDSIGHT_API_LLM_PROVIDER=openai
 HINDSIGHT_API_LLM_API_KEY=your-api-key-here
-HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
+HINDSIGHT_API_LLM_MODEL=o3-mini
 HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 
 # Example: Anthropic Claude configuration
@@ -20,11 +20,6 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
 # HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/service-account-key.json  # Optional, uses ADC if not set
 
-# Example: MiniMax configuration (1M context window)
-# HINDSIGHT_API_LLM_PROVIDER=minimax
-# HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
-# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
-
 # Example: LM Studio local configuration (Qwen 2.5 32B recommended)
 # HINDSIGHT_API_LLM_PROVIDER=lmstudio
 # HINDSIGHT_API_LLM_API_KEY=lmstudio
@@ -44,15 +39,8 @@ HINDSIGHT_API_LOG_LEVEL=info
 
 # Database (Optional - uses embedded pg0 by default)
 # HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
-# HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
 # HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
 
-# Vector Extension (Optional - uses pgvector by default)
-# Options: "pgvector" (default), "vchord", "pgvectorscale" (DiskANN)
-# HINDSIGHT_API_VECTOR_EXTENSION=pgvector
-# For Azure PostgreSQL with DiskANN:
-# HINDSIGHT_API_VECTOR_EXTENSION=pgvectorscale  # Auto-detects pg_diskann on Azure
-
 # Embeddings Configuration (Optional - uses local by default)
 # Provider: "local" (default) or "tei" (HuggingFace Text Embeddings Inference)
 # HINDSIGHT_API_EMBEDDINGS_PROVIDER=local

.github/dependabot.yml

@@ -1,6 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "weekly"

.github/workflows/deploy-docs.yml

@@ -21,20 +21,17 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
-      - uses: actions/setup-node@v6
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
         with:
           node-version: 20
           cache: npm
           cache-dependency-path: package-lock.json
-      - uses: astral-sh/setup-uv@v7
+      - uses: astral-sh/setup-uv@v4
       - run: npm ci --workspace=hindsight-docs
       - run: uv run generate-llms-full
       - run: npm run build --workspace=hindsight-docs
-        env:
-          UMAMI_URL: https://analytics.hindsight.vectorize.io
-          UMAMI_WEBSITE_ID: ${{ secrets.UMAMI_WEBSITE_ID }}
-      - uses: actions/upload-pages-artifact@v4
+      - uses: actions/upload-pages-artifact@v3
         with:
           path: hindsight-docs/build
   deploy:
@@ -44,5 +41,5 @@ jobs:
     runs-on: ubuntu-latest
     needs: build
     steps:
-      - uses: actions/deploy-pages@v5
+      - uses: actions/deploy-pages@v4
         id: deployment

.github/workflows/release-integration.yml

@@ -1,120 +0,0 @@
-name: Release Integration
-
-on:
-  push:
-    tags:
-      - 'integrations/**'
-
-jobs:
-  publish:
-    runs-on: ubuntu-latest
-    permissions:
-      id-token: write  # for PyPI trusted publishing
-
-    steps:
-    - uses: actions/checkout@v6
-
-    - name: Extract integration info
-      id: info
-      run: |
-        # refs/tags/integrations/litellm/v0.1.0 → integration=litellm, version=0.1.0
-        TAG="${GITHUB_REF#refs/tags/}"
-        INTEGRATION=$(echo "$TAG" | cut -d'/' -f2)
-        VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
-        echo "integration=$INTEGRATION" >> $GITHUB_OUTPUT
-        echo "version=$VERSION" >> $GITHUB_OUTPUT
-        echo "tag=$TAG" >> $GITHUB_OUTPUT
-        echo "Integration: $INTEGRATION, Version: $VERSION"
-
-    - name: Detect integration type
-      id: type
-      run: |
-        if [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/pyproject.toml" ]; then
-          echo "type=python" >> $GITHUB_OUTPUT
-        elif [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/package.json" ]; then
-          echo "type=typescript" >> $GITHUB_OUTPUT
-        else
-          echo "type=plugin" >> $GITHUB_OUTPUT
-        fi
-
-    # ── Python integrations (litellm, pydantic-ai, crewai) ──────────────────
-
-    - name: Install uv
-      if: steps.type.outputs.type == 'python'
-      uses: astral-sh/setup-uv@v7
-      with:
-        enable-cache: true
-
-    - name: Set up Python
-      if: steps.type.outputs.type == 'python'
-      uses: actions/setup-python@v6
-      with:
-        python-version-file: ".python-version"
-
-    - name: Build Python package
-      if: steps.type.outputs.type == 'python'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: uv build --out-dir dist
-
-    - name: Publish Python package to PyPI
-      if: steps.type.outputs.type == 'python'
-      uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        packages-dir: ./hindsight-integrations/${{ steps.info.outputs.integration }}/dist
-        skip-existing: true
-
-    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
-
-    # ── Plugin integrations (claude-code) — no package to publish ───────────
-
-    - name: Plugin release
-      if: steps.type.outputs.type == 'plugin'
-      run: |
-        echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
-        echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
-
-    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
-
-    - name: Set up Node.js
-      if: steps.type.outputs.type == 'typescript'
-      uses: actions/setup-node@v6
-      with:
-        node-version: '22'
-        registry-url: 'https://registry.npmjs.org'
-
-    # Guard: fail fast if the integration's lockfile resolves any dep from a
-    # monorepo workspace (link=true) or a relative file path. The release
-    # runner has no pre-built workspace `dist/` so `npm run build` would
-    # later fail at tsc with "Cannot find module". See:
-    # https://github.com/vectorize-io/hindsight/issues/… (0.6.0 openclaw retry)
-    - name: Check integration lockfile
-      if: steps.type.outputs.type == 'typescript'
-      run: ./scripts/check-integration-lockfiles.sh
-
-    - name: Install dependencies
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: npm ci
-
-    - name: Build TypeScript package
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: npm run build
-
-    - name: Publish TypeScript package to npm
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: |
-        set +e
-        OUTPUT=$(npm publish --access public 2>&1)
-        EXIT_CODE=$?
-        echo "$OUTPUT"
-        if [ $EXIT_CODE -ne 0 ]; then
-          if echo "$OUTPUT" | grep -q "cannot publish over"; then
-            echo "Package version already published, skipping..."
-            exit 0
-          fi
-          exit $EXIT_CODE
-        fi
-      env:
-        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/release.yml

@@ -13,15 +13,15 @@ jobs:
       id-token: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install uv
-      uses: astral-sh/setup-uv@v7
+      uses: astral-sh/setup-uv@v5
       with:
         enable-cache: true
 
     - name: Set up Python
-      uses: actions/setup-python@v6
+      uses: actions/setup-python@v5
       with:
         python-version-file: ".python-version"
 
@@ -30,39 +30,29 @@ jobs:
       working-directory: ./hindsight-clients/python
       run: uv build --out-dir dist
 
-    - name: Build hindsight-api-slim
-      working-directory: ./hindsight-api-slim
-      run: uv build --out-dir dist
-
     - name: Build hindsight-api
       working-directory: ./hindsight-api
       run: uv build --out-dir dist
 
     - name: Build hindsight-all
-      working-directory: ./hindsight-all
+      working-directory: ./hindsight
       run: uv build --out-dir dist
 
-    - name: Build hindsight-all-slim
-      working-directory: ./hindsight-all-slim
+    - name: Build hindsight-litellm
+      working-directory: ./hindsight-integrations/litellm
       run: uv build --out-dir dist
 
     - name: Build hindsight-embed
       working-directory: ./hindsight-embed
       run: uv build --out-dir dist
 
-    # Publish in order (client and api-slim first, then api/all wrappers which depend on them)
+    # Publish in order (client and api first, then hindsight-all which depends on them)
     - name: Publish hindsight-client to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         packages-dir: ./hindsight-clients/python/dist
         skip-existing: true
 
-    - name: Publish hindsight-api-slim to PyPI
-      uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        packages-dir: ./hindsight-api-slim/dist
-        skip-existing: true
-
     - name: Publish hindsight-api to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
@@ -72,13 +62,13 @@ jobs:
     - name: Publish hindsight-all to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight-all/dist
+        packages-dir: ./hindsight/dist
         skip-existing: true
 
-    - name: Publish hindsight-all-slim to PyPI
+    - name: Publish hindsight-litellm to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight-all-slim/dist
+        packages-dir: ./hindsight-integrations/litellm/dist
         skip-existing: true
 
     - name: Publish hindsight-embed to PyPI
@@ -89,15 +79,14 @@ jobs:
 
     # Upload artifacts for GitHub release
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: python-packages
         path: |
           hindsight-clients/python/dist/*
-          hindsight-api-slim/dist/*
           hindsight-api/dist/*
-          hindsight-all/dist/*
-          hindsight-all-slim/dist/*
+          hindsight/dist/*
+          hindsight-integrations/litellm/dist/*
           hindsight-embed/dist/*
         retention-days: 1
 
@@ -106,10 +95,10 @@ jobs:
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -144,35 +133,35 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: typescript-client
         path: hindsight-clients/typescript/*.tgz
         retention-days: 1
 
-  release-hindsight-all-npm:
+  release-openclaw-integration:
     runs-on: ubuntu-latest
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '22'
         registry-url: 'https://registry.npmjs.org'
-        cache: 'npm'
-        cache-dependency-path: package-lock.json
 
     - name: Install dependencies
-      run: npm ci --workspace=hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
+      run: npm ci
 
     - name: Build
-      run: npm run build --workspace=hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
+      run: npm run build
 
     - name: Publish to npm
-      working-directory: ./hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
       run: |
         set +e
         OUTPUT=$(npm publish --access public 2>&1)
@@ -189,14 +178,63 @@ jobs:
         NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
 
     - name: Pack for GitHub release
-      working-directory: ./hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
-        name: hindsight-all-npm
-        path: hindsight-all-npm/*.tgz
+        name: openclaw-integration
+        path: hindsight-integrations/openclaw/*.tgz
+        retention-days: 1
+
+  release-ai-sdk-integration:
+    runs-on: ubuntu-latest
+    environment: npm
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm ci
+
+    - name: Build
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm run build
+
+    - name: Publish to npm
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+    - name: Pack for GitHub release
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm pack
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: ai-sdk-integration
+        path: hindsight-integrations/ai-sdk/*.tgz
         retention-days: 1
 
   release-control-plane:
@@ -204,10 +242,10 @@ jobs:
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -230,14 +268,11 @@ jobs:
     - name: Build
       run: npm run build --workspace=hindsight-control-plane
 
-    - name: Verify standalone build
-      run: test -f hindsight-control-plane/standalone/server.js || (echo 'standalone/server.js missing - build failed' && exit 1)
-
     - name: Publish to npm
       working-directory: ./hindsight-control-plane
       run: |
         set +e
-        OUTPUT=$(npm publish --access public --ignore-scripts 2>&1)
+        OUTPUT=$(npm publish --access public 2>&1)
         EXIT_CODE=$?
         echo "$OUTPUT"
         if [ $EXIT_CODE -ne 0 ]; then
@@ -255,7 +290,7 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: control-plane
         path: hindsight-control-plane/*.tgz
@@ -278,13 +313,9 @@ jobs:
             target: aarch64-apple-darwin
             artifact_name: hindsight
             asset_name: hindsight-darwin-arm64
-          - os: ubuntu-24.04-arm
-            target: aarch64-unknown-linux-gnu
-            artifact_name: hindsight
-            asset_name: hindsight-linux-arm64
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install Rust
       uses: dtolnay/rust-toolchain@stable
@@ -302,7 +333,7 @@ jobs:
         chmod +x artifacts/${{ matrix.asset_name }}
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: rust-cli-${{ matrix.asset_name }}
         path: artifacts/${{ matrix.asset_name }}
@@ -343,7 +374,7 @@ jobs:
               PRELOAD_ML_MODELS=false
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Free Disk Space
       uses: jlumbroso/free-disk-space@main
@@ -357,13 +388,13 @@ jobs:
         swap-storage: true
 
     - name: Set up QEMU
-      uses: docker/setup-qemu-action@v4
+      uses: docker/setup-qemu-action@v3
 
     - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@v4
+      uses: docker/setup-buildx-action@v3
 
     - name: Log in to GitHub Container Registry
-      uses: docker/login-action@v4
+      uses: docker/login-action@v3
       with:
         registry: ghcr.io
         username: ${{ github.actor }}
@@ -375,7 +406,7 @@ jobs:
 
     - name: Extract metadata for release tags
       id: meta
-      uses: docker/metadata-action@v6
+      uses: docker/metadata-action@v5
       with:
         images: ghcr.io/${{ github.repository_owner }}/${{ matrix.image_name }}
         flavor: |
@@ -391,7 +422,7 @@ jobs:
     # # Step 1: Build for local testing (single platform, no push)
     # # This creates an identical image to what will be released, just for one platform
     # - name: Build image for testing
-    #   uses: docker/build-push-action@v7
+    #   uses: docker/build-push-action@v6
     #   with:
     #     context: .
     #     file: docker/standalone/Dockerfile
@@ -410,7 +441,7 @@ jobs:
 
     # Build multi-platform and push to release tags
     - name: Build and push release images
-      uses: docker/build-push-action@v7
+      uses: docker/build-push-action@v6
       with:
         context: .
         file: docker/standalone/Dockerfile
@@ -428,10 +459,10 @@ jobs:
       packages: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install Helm
-      uses: azure/setup-helm@v5
+      uses: azure/setup-helm@v4
       with:
         version: 'latest'
 
@@ -448,7 +479,7 @@ jobs:
       run: helm push helm-packages/*.tgz oci://ghcr.io/${{ github.repository_owner }}/charts
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: helm-chart
         path: helm-packages/*.tgz
@@ -456,61 +487,67 @@ jobs:
 
   create-github-release:
     runs-on: ubuntu-latest
-    needs: [release-python-packages, release-typescript-client, release-hindsight-all-npm, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
+    needs: [release-python-packages, release-typescript-client, release-openclaw-integration, release-ai-sdk-integration, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
     permissions:
       contents: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Extract version from tag
       id: get_version
       run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
 
     - name: Download Python packages
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: python-packages
         path: ./artifacts/python-packages
 
     - name: Download TypeScript client
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: typescript-client
         path: ./artifacts/typescript-client
 
+    - name: Download OpenClaw Integration
+      uses: actions/download-artifact@v4
+      with:
+        name: openclaw-integration
+        path: ./artifacts/openclaw-integration
+
+    - name: Download AI SDK Integration
+      uses: actions/download-artifact@v4
+      with:
+        name: ai-sdk-integration
+        path: ./artifacts/ai-sdk-integration
+
     - name: Download Control Plane
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: control-plane
         path: ./artifacts/control-plane
 
-    - name: Download hindsight-embed npm wrapper
-      uses: actions/download-artifact@v8
-      with:
-        name: hindsight-all-npm
-        path: ./artifacts/hindsight-all-npm
-
     - name: Download Rust CLI (Linux)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-linux-amd64
         path: ./artifacts/rust-cli-linux
 
     - name: Download Rust CLI (macOS Intel)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-darwin-amd64
         path: ./artifacts/rust-cli-darwin-amd64
 
     - name: Download Rust CLI (macOS ARM)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-darwin-arm64
         path: ./artifacts/rust-cli-darwin-arm64
 
     - name: Download Helm chart
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: helm-chart
         path: ./artifacts/helm-chart
@@ -520,15 +557,16 @@ jobs:
         mkdir -p release-assets
         # Python packages
         cp artifacts/python-packages/hindsight-clients/python/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-api-slim/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-api/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-all/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-all-slim/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-integrations/litellm/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-embed/dist/* release-assets/ || true
         # TypeScript client
         cp artifacts/typescript-client/*.tgz release-assets/ || true
-        # hindsight-embed npm wrapper
-        cp artifacts/hindsight-all-npm/*.tgz release-assets/ || true
+        # OpenClaw Integration
+        cp artifacts/openclaw-integration/*.tgz release-assets/ || true
+        # AI SDK Integration
+        cp artifacts/ai-sdk-integration/*.tgz release-assets/ || true
         # Control Plane
         cp artifacts/control-plane/*.tgz release-assets/ || true
         # Rust CLI binaries
@@ -540,7 +578,7 @@ jobs:
         ls -la release-assets/
 
     - name: Create GitHub Release
-      uses: softprops/action-gh-release@v3
+      uses: softprops/action-gh-release@v2
       with:
         files: release-assets/*
         generate_release_notes: true

.gitignore

@@ -46,12 +46,10 @@ hindsight-docs/static/llms-full.txt
 hindsight-dev/benchmarks/locomo/results/
 hindsight-dev/benchmarks/longmemeval/results/
 hindsight-dev/benchmarks/consolidation/results/
-hindsight-dev/benchmarks/perf/results/
 benchmarks/results/
 hindsight-cli/target
 hindsight-clients/rust/target
-.claude/*
-!.claude/skills/
+.claude
 whats-next.md
 TASK.md
 # Changelog is now tracked in hindsight-docs/src/pages/changelog.md

.prettierrc.json

@@ -1,7 +0,0 @@
-{
-  "semi": true,
-  "singleQuote": false,
-  "tabWidth": 2,
-  "trailingComma": "es5",
-  "printWidth": 100
-}

CLAUDE.md

@@ -11,32 +11,26 @@ Hindsight is an agent memory system that provides long-term memory for AI agents
 
 ## Development Commands
 
-### Local Development (API + UI)
-```bash
-# Start both API server and control plane UI
-./scripts/dev/start.sh
-```
-
 ### API Server (Python/FastAPI)
 ```bash
-# Start API server only (loads .env automatically)
+# Start API server (loads .env automatically)
 ./scripts/dev/start-api.sh
 
 # Run all tests (parallelized with pytest-xdist)
-cd hindsight-api-slim && uv run pytest tests/
+cd hindsight-api && uv run pytest tests/
 
 # Run specific test file
-cd hindsight-api-slim && uv run pytest tests/test_http_api_integration.py -v
+cd hindsight-api && uv run pytest tests/test_http_api_integration.py -v
 
 # Run single test function
-cd hindsight-api-slim && uv run pytest tests/test_retain.py::test_retain_simple -v
+cd hindsight-api && uv run pytest tests/test_retain.py::test_retain_simple -v
 
 # Lint and format
-cd hindsight-api-slim && uv run ruff check .
-cd hindsight-api-slim && uv run ruff format .
+cd hindsight-api && uv run ruff check .
+cd hindsight-api && uv run ruff format .
 
 # Type checking (uses ty - extremely fast type checker from Astral)
-cd hindsight-api-slim && uv run ty check hindsight_api/
+cd hindsight-api && uv run ty check hindsight_api/
 ```
 
 ### Control Plane (Next.js)
@@ -63,32 +57,26 @@ cd hindsight-control-plane && npm run dev
 
 ### Benchmarks
 ```bash
-# Accuracy benchmarks
 ./scripts/benchmarks/run-longmemeval.sh
 ./scripts/benchmarks/run-locomo.sh
-
-# Performance benchmarks
-./scripts/benchmarks/run-consolidation.sh
-./scripts/benchmarks/run-retain-perf.sh --document <path>  # Requires API server running
-
-# Results viewer
 ./scripts/benchmarks/start-visualizer.sh  # View results at localhost:8001
 ```
 
 ## Architecture
 
 ### Monorepo Structure
-- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)
+- **hindsight-api/**: Core FastAPI server with memory engine (Python, uv)
+- **hindsight/**: Embedded Python bundle (hindsight-all package)
 - **hindsight-control-plane/**: Admin UI (Next.js, npm)
 - **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)
 - **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)
 - **hindsight-docs/**: Docusaurus documentation site
-- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)
+- **hindsight-integrations/**: Framework integrations (LiteLLM, OpenAI)
 - **hindsight-dev/**: Development tools and benchmarks
 
-### Core Engine (hindsight-api-slim/hindsight_api/engine/)
-- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
-- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
+### Core Engine (hindsight-api/hindsight_api/engine/)
+- `memory_engine.py`: Main orchestrator (~170KB) for retain/recall/reflect operations
+- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, Groq, Ollama, LM Studio
 - `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
 - `cross_encoder.py`: Reranking (local or TEI)
 - `entity_resolver.py`: Entity extraction and normalization
@@ -101,13 +89,13 @@ cd hindsight-control-plane && npm run dev
 
 **search/**: Multi-strategy retrieval
 - `retrieval.py`: Main retrieval orchestrator
-- `graph_retrieval.py`: Graph retrieval abstract base class
-- `link_expansion_retrieval.py`: Link expansion graph retrieval
+- `graph_retrieval.py`: Entity/relationship graph traversal
+- `mpfp_retrieval.py`: Multi-Path Fact Propagation retrieval
 - `fusion.py`: Reciprocal rank fusion for combining results
 - `reranking.py`: Cross-encoder reranking
 
-### API Layer (hindsight-api-slim/hindsight_api/api/)
-- `http.py`: FastAPI HTTP routers for all REST endpoints
+### API Layer (hindsight-api/hindsight_api/api/)
+- `http.py`: FastAPI HTTP routers (~80KB) for all REST endpoints
 - `mcp.py`: Model Context Protocol server implementation
 
 Main operations:
@@ -116,13 +104,13 @@ Main operations:
 - **Reflect**: Disposition-aware reasoning using memories and mental models.
 
 ### Database
-PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api-slim/hindsight_api/alembic/`. Migrations run automatically on API startup.
+PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api/hindsight_api/alembic/`. Migrations run automatically on API startup.
 
 Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 
 ### Adding Database Migrations
 
-1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:
+1. **Create a new migration file** in `hindsight-api/hindsight_api/alembic/versions/`:
    - File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)
    - Use a unique hex revision ID (12 chars)
    - Set `down_revision` to the previous migration's revision ID
@@ -159,7 +147,7 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 
 3. **Run migrations locally**:
    ```bash
-   # Set database URL and run migrations for the base schema plus all tenants
+   # Set database URL and run migrations
    uv run hindsight-admin run-db-migration
 
    # Run on a specific tenant schema
@@ -169,17 +157,11 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 ## Key Conventions
 
 ### Code Quality
-
-**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).
-
 **Always run the lint script after making Python or TypeScript/Node changes:**
 ```bash
 ./scripts/hooks/lint.sh
 ```
-
-**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
-
-**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
+This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Prettier for TypeScript).
 
 ### Memory Banks
 - Each bank is an isolated memory store (like a "brain" for one user/agent)
@@ -211,20 +193,48 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
    - Update the client type definition in `lib/api.ts`
    - Update any UI components that need to use the new parameter
 
-### Adding New Integrations
+### Python Style
+- Python 3.11+, type hints required
+- Async throughout (asyncpg, async FastAPI)
+- Pydantic models for request/response
+- Ruff for linting (line-length 120)
+- No Python files at project root - maintain clean directory structure
+- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
 
-Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
+### Type Safety with Pydantic Models
+**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
+- Use Pydantic `BaseModel` for all data structures passed between functions
+- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
+- Avoid `dict.get()` patterns - use typed model attributes instead
+- Parse external data (JSON, API responses) into Pydantic models at the boundary
+- This catches type errors at parse time, not deep in business logic
 
-1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.
-2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.
-3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.
-4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).
+```python
+# BAD - error-prone dict access
+def process(data: dict) -> str:
+    return data.get("name", "")  # No validation, silent failures
 
-If any of these are missing, the integration is incomplete and must not be pushed or merged.
+# GOOD - typed and validated
+class UserData(BaseModel):
+    name: str
+    created_at: datetime
 
-### Changelogs
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def ensure_tz_aware(cls, v):
+        if isinstance(v, str):
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if v.tzinfo is None:
+            return v.replace(tzinfo=timezone.utc)
+        return v
 
-Never add "Unreleased" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.
+def process(data: UserData) -> str:
+    return data.name  # Type-safe, validated at construction
+```
+
+### TypeScript Style
+- Next.js App Router for control plane
+- Tailwind CSS with shadcn/ui components
 
 ### Adding New API Configuration Flags
 
@@ -234,24 +244,24 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
 
 #### Adding a New Configuration Field
 
-1. **config.py** (`hindsight-api-slim/hindsight_api/config.py`):
+1. **config.py** (`hindsight-api/hindsight_api/config.py`):
    - Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
    - Add `DEFAULT_*` constant for the default value
    - Add field to `HindsightConfig` dataclass with type annotation
-   - **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API
+   - **Mark as hierarchical or static** by adding to `_HIERARCHICAL_FIELDS` set (hierarchical) or leaving it out (static)
    - Add initialization in `from_env()` method
 
    ```python
-   # Configurable field (can be overridden per-tenant/bank via API)
-   _CONFIGURABLE_FIELDS = {
+   # Hierarchical field (can be overridden per-bank)
+   _HIERARCHICAL_FIELDS = {
        ...,
-       "my_setting",  # Add here for configurable
+       "my_setting",  # Add here for hierarchical
    }
 
-   # Static field - just don't add to _CONFIGURABLE_FIELDS
+   # Static field - just don't add to _HIERARCHICAL_FIELDS
    ```
 
-2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
+2. **main.py** (`hindsight-api/hindsight_api/main.py`):
    - Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
 
 3. **Use hierarchical config in MemoryEngine**:
@@ -291,19 +301,19 @@ cp .env.example .env
 # Edit .env with LLM API key
 
 # Python deps
-uv sync --directory hindsight-api-slim/
+uv sync --directory hindsight-api/
 
 # Node deps (uses npm workspaces)
 npm install
 ```
 
 Required env vars:
-- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio
+- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, ollama, lmstudio
 - `HINDSIGHT_API_LLM_API_KEY`: Your API key
-- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., gpt-4o-mini, claude-sonnet-4-20250514)
+- `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., o3-mini, claude-sonnet-4-20250514)
 
 Optional (uses local models by default):
 - `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei
 - `HINDSIGHT_API_RERANKER_PROVIDER`: local (default) or tei
 - `HINDSIGHT_API_DATABASE_URL`: External PostgreSQL (uses embedded pg0 by default)
-- `HINDSIGHT_API_ENABLE_BANK_CONFIG_API`: Enable per-bank config API (default: true)
+- `HINDSIGHT_API_ENABLE_BANK_CONFIG_API`: Enable per-bank config API (default: false, disabled for security)

README.md

@@ -7,12 +7,10 @@
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
 [![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
-<br/>
 
-<a href="https://trendshift.io/repositories/15603" target="_blank"><img src="https://trendshift.io/api/badge/repositories/15603" alt="vectorize-io%2Fhindsight | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
+
 </div>
 
 ---
@@ -38,7 +36,7 @@ Hindsight is being used in production at Fortune 500 enterprises and by a growin
 
 ## Adding Hindsight to Your AI Agents
 
-The easiest way to use Hindsight with an existing agent is with the LLM Wrapper. You can add memory to your agent with 2 lines of code. That will swap your current LLM client out with the Hindsight wrapper. After that, memories will be stored and retrieved automatically as you make LLM calls.
+The easiest way use Hindsight with an existing agent is with the LLM Wrapper. You can add memory to your agent with 2 lines of code. That will swap your current LLM client out with the Hindsight wrapper. After that, memories will be stored and retrieved automatically as you make LLM calls.
 
 If you need more control over how and when your agent stores and recalls memories, there's also a simple API you can integrate with using the SDKs or directly via HTTP.
 
@@ -71,7 +69,7 @@ docker run --rm -it --pull always -p 8888:8888 -p 9999:9999 \
 >API: http://localhost:8888
 >UI: http://localhost:9999
 
-You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `minimax`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
+You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, and `lmstudio`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
 
 
 
@@ -183,7 +181,7 @@ Satisfying these requirements in Hindsight is straightforward. When new user inp
 
 ![Overview](./hindsight-docs/static/img/hindsight-overview.webp)
 
-Most agent memory implementations rely on basic vector search or sometimes use a knowledge graph. Hindsight uses biomimetic data structures to organize agent memories in a way that is more like how human memory works:
+Most agent memory implementation rely on basic vector search or sometimes use a knowledge graph. Hindsight uses biomimetic data structures to organize agent memories in a way that is more like how human memory works:
 
 - **World:** Facts about the world ("The stove gets hot")
 - **Experiences:** Agent's own experiences ("I touched the stove and it really hurt")
@@ -309,5 +307,3 @@ MIT — see [LICENSE](./LICENSE)
 ---
 
 Built by [Vectorize.io](https://vectorize.io)
-
-<img src="https://umami-pixel.chris-latimer.workers.dev/?id=a8b043e6-6964-454d-80df-69b69d3f0d50&host=github.com&url=/vectorize-io/hindsight" width="1" height="1" alt="" />

deno.lock

@@ -1,139 +0,0 @@
-{
-  "version": "5",
-  "specifiers": {
-    "jsr:@std/assert@^1.0.17": "1.0.19",
-    "jsr:@std/assert@^1.0.19": "1.0.19",
-    "jsr:@std/expect@*": "1.0.18",
-    "jsr:@std/internal@^1.0.12": "1.0.12",
-    "jsr:@std/path@^1.1.4": "1.1.4",
-    "jsr:@std/testing@*": "1.0.17"
-  },
-  "jsr": {
-    "@std/[email protected]": {
-      "integrity": "eaada96ee120cb980bc47e040f82814d786fe8162ecc53c91d8df60b8755991e",
-      "dependencies": [
-        "jsr:@std/internal"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "8566eab35200466f8609eb7e7aed062ed0db314e9a258d5d201b1b8997ce801a",
-      "dependencies": [
-        "jsr:@std/assert@^1.0.19",
-        "jsr:@std/internal",
-        "jsr:@std/path"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "972a634fd5bc34b242024402972cd5143eac68d8dffaca5eaa4dba30ce17b027"
-    },
-    "@std/[email protected]": {
-      "integrity": "1d2d43f39efb1b42f0b1882a25486647cb851481862dc7313390b2bb044314b5",
-      "dependencies": [
-        "jsr:@std/internal"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "87bdc2700fa98249d48a17cd72413352d3d3680dcfbdb64947fd0982d6bbf681",
-      "dependencies": [
-        "jsr:@std/assert@^1.0.17",
-        "jsr:@std/internal"
-      ]
-    }
-  },
-  "workspace": {
-    "members": {
-      "hindsight-clients/typescript": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@hey-api/[email protected]",
-            "npm:@types/jest@29",
-            "npm:@types/node@20",
-            "npm:jest@29",
-            "npm:ts-jest@29",
-            "npm:tsup@^8.5.1",
-            "npm:typescript@5"
-          ]
-        }
-      },
-      "hindsight-control-plane": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@eslint/eslintrc@^3.3.3",
-            "npm:@eslint/js@^9.39.2",
-            "npm:@radix-ui/react-alert-dialog@^1.1.15",
-            "npm:@radix-ui/react-checkbox@^1.3.3",
-            "npm:@radix-ui/react-dialog@^1.1.15",
-            "npm:@radix-ui/react-dropdown-menu@^2.1.16",
-            "npm:@radix-ui/react-label@^2.1.8",
-            "npm:@radix-ui/react-popover@^1.1.15",
-            "npm:@radix-ui/react-radio-group@^1.3.8",
-            "npm:@radix-ui/react-select@^2.2.6",
-            "npm:@radix-ui/react-slider@^1.3.6",
-            "npm:@radix-ui/react-slot@^1.2.4",
-            "npm:@radix-ui/react-switch@^1.2.6",
-            "npm:@radix-ui/react-tabs@^1.1.13",
-            "npm:@radix-ui/react-tooltip@^1.2.8",
-            "npm:@tailwindcss/postcss@^4.1.17",
-            "npm:@tailwindcss/typography@~0.5.19",
-            "npm:@types/cytoscape@^3.21.9",
-            "npm:@types/node@^24.10.0",
-            "npm:@types/react-dom@^19.2.2",
-            "npm:@types/react@^19.2.2",
-            "npm:autoprefixer@^10.4.21",
-            "npm:class-variance-authority@~0.7.1",
-            "npm:clsx@^2.1.1",
-            "npm:cmdk@^1.1.1",
-            "npm:cytoscape-fcose@^2.2.0",
-            "npm:cytoscape@^3.33.1",
-            "npm:eslint-config-next@^16.0.1",
-            "npm:eslint-plugin-react-hooks@^7.0.1",
-            "npm:eslint-plugin-react@^7.37.5",
-            "npm:eslint@^9.39.1",
-            "npm:[email protected]",
-            "npm:next-themes@~0.4.6",
-            "npm:next@^16.1.6",
-            "npm:postcss@^8.5.6",
-            "npm:prettier@^3.7.4",
-            "npm:react-chrono@^2.9.1",
-            "npm:react-dom@^19.2.0",
-            "npm:react-markdown@^10.1.0",
-            "npm:react18-json-view@~0.2.9",
-            "npm:react@^19.2.0",
-            "npm:recharts@^3.5.1",
-            "npm:remark-gfm@^4.0.1",
-            "npm:sonner@^2.0.7",
-            "npm:tailwind-merge@^3.4.0",
-            "npm:tailwindcss-animate@^1.0.7",
-            "npm:tailwindcss@^4.1.17",
-            "npm:[email protected]",
-            "npm:typescript-eslint@^8.50.0",
-            "npm:typescript@^5.9.3"
-          ]
-        }
-      },
-      "hindsight-docs": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/theme-common@^3.9.2",
-            "npm:@docusaurus/theme-mermaid@^3.9.2",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@easyops-cn/docusaurus-search-local@~0.52.2",
-            "npm:@mdx-js/react@3",
-            "npm:clsx@2",
-            "npm:prism-react-renderer@^2.3.0",
-            "npm:raw-loader@^4.0.2",
-            "npm:react-dom@19",
-            "npm:react-icons@^5.6.0",
-            "npm:react@19",
-            "npm:redocusaurus@^2.5.0",
-            "npm:typescript@~5.6.2"
-          ]
-        }
-      }
-    }
-  }
-}

docker/docker-compose/pg_textsearch/Dockerfile

@@ -1,32 +0,0 @@
-# PostgreSQL with pgvector and pg_textsearch extensions
-# Note: pg_textsearch requires PostgreSQL 17+
-FROM postgres:17
-
-# Install build dependencies
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    git \
-    postgresql-server-dev-17 \
-    libpq-dev \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install pgvector
-RUN cd /tmp && \
-    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
-    cd pgvector && \
-    make && \
-    make install
-
-# Install pg_textsearch
-RUN cd /tmp && \
-    git clone https://github.com/timescale/pg_textsearch.git && \
-    cd pg_textsearch && \
-    make && \
-    make install
-
-# Clean up source files and build dependencies
-RUN rm -rf /tmp/pgvector /tmp/pg_textsearch && \
-    apt-get purge -y --auto-remove build-essential git postgresql-server-dev-17
-
-# Ensure extensions are preloaded
-RUN echo "shared_preload_libraries = 'pg_textsearch'" >> /usr/share/postgresql/postgresql.conf.sample

docker/docker-compose/pg_textsearch/docker-compose.yaml

@@ -1,91 +0,0 @@
-name: hindsight
-# Docker Compose file for Hindsight with PostgreSQL and Timescale pg_textsearch
-# docker compose -f docker/docker-compose/pg_textsearch/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/pg_textsearch/docker-compose.yaml up -d
-# Make sure to set the required environment variables before running:
-# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
-# - Configure LLM provider variables as needed (see below in the hindsight service)
-#
-# Usage:
-# docker compose up -d
-#
-# Optional environment variables with defaults:
-# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
-# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
-# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
-
-services:
-  db:
-    # Use custom PostgreSQL image with pgvector and pg_textsearch extensions
-    build:
-      context: .
-      dockerfile: Dockerfile
-    container_name: hindsight-db
-    restart: always
-    # Expose PostgreSQL port
-    ports:
-      - "5437:5432"
-    environment:
-      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
-      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
-      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
-    volumes:
-      - pg_data:/var/lib/postgresql/data
-    networks:
-      - hindsight-net
-
-  pg-textsearch-init:
-    build:
-      context: .
-      dockerfile: Dockerfile
-    depends_on:
-      - db
-    environment:
-      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
-    command: >
-      bash -c "
-        echo 'Waiting for PostgreSQL to be ready...';
-        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
-          echo 'PostgreSQL is unavailable - sleeping';
-          sleep 2;
-        done;
-        echo 'PostgreSQL is ready - creating hindsight_db database';
-        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
-        echo 'Creating extensions in hindsight_db database';
-        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
-        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_textsearch CASCADE;';
-        echo 'Database and extensions created successfully';
-      "
-    restart: "no"
-    networks:
-      - hindsight-net
-
-  hindsight:
-    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
-    container_name: hindsight-app
-    ports:
-      - "8888:8888"
-      - "9999:9999"
-    environment:
-      # LLM Configuration
-      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
-      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
-
-      # Database Configuration
-      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
-
-      # Vector and Text Search Extensions
-      HINDSIGHT_API_VECTOR_EXTENSION: pgvector
-      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pg_textsearch
-
-    depends_on:
-      - db
-    networks:
-      - hindsight-net
-
-
-networks:
-  hindsight-net:
-    driver: bridge
-
-volumes:
-  pg_data:

docker/docker-compose/s3-file-storage/docker-compose.yaml

@@ -1,83 +0,0 @@
-# Docker Compose file for Hindsight with S3 file storage (SeaweedFS)
-#
-# SeaweedFS (Apache 2.0) provides an S3-compatible object storage backend
-# for storing uploaded files instead of PostgreSQL BYTEA storage.
-#
-# Make sure to set the required environment variables before running:
-# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
-# - Configure LLM provider variables as needed (see below in the hindsight service)
-#
-# Usage:
-# docker compose up -d
-#
-# Optional environment variables with defaults:
-# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
-# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
-# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
-# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
-# - SEAWEEDFS_S3_ACCESS_KEY: S3 access key (default: hindsight_s3_key)
-# - SEAWEEDFS_S3_SECRET_KEY: S3 secret key (default: hindsight_s3_secret)
-
-services:
-  db:
-    image: pgvector/pgvector:pg${HINDSIGHT_DB_VERSION:-18}
-    container_name: hindsight-db
-    restart: always
-    environment:
-      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
-      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}
-      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
-    volumes:
-      - pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
-    networks:
-      - hindsight-net
-
-  seaweedfs:
-    image: chrislusf/seaweedfs:latest
-    container_name: hindsight-seaweedfs
-    restart: always
-    # Single-node mode: master + volume + filer + S3 gateway all in one process
-    command: >
-      server
-      -s3
-      -s3.port=8333
-      -s3.config=/etc/seaweedfs/s3.json
-      -ip.bind=0.0.0.0
-    volumes:
-      - seaweedfs_data:/data
-      - ./s3.json:/etc/seaweedfs/s3.json:ro
-    # Expose S3 API port (uncomment to access from host)
-    # ports:
-    #   - "8333:8333"
-    networks:
-      - hindsight-net
-
-  hindsight:
-    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
-    container_name: hindsight-app
-    ports:
-      - "8888:8888"
-      - "9999:9999"
-    environment:
-      - HINDSIGHT_API_LLM_API_KEY=${OPENAI_API_KEY?Please set the OPENAI_API_KEY env variable}
-      - HINDSIGHT_API_DATABASE_URL=postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:?Please set the HINDSIGHT_DB_PASSWORD env variable}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
-      # S3 file storage configuration (SeaweedFS)
-      - HINDSIGHT_API_FILE_STORAGE_TYPE=s3
-      - HINDSIGHT_API_FILE_STORAGE_S3_BUCKET=hindsight
-      - HINDSIGHT_API_FILE_STORAGE_S3_ENDPOINT=http://seaweedfs:8333
-      - HINDSIGHT_API_FILE_STORAGE_S3_REGION=us-east-1
-      - HINDSIGHT_API_FILE_STORAGE_S3_ACCESS_KEY_ID=${SEAWEEDFS_S3_ACCESS_KEY:-hindsight_s3_key}
-      - HINDSIGHT_API_FILE_STORAGE_S3_SECRET_ACCESS_KEY=${SEAWEEDFS_S3_SECRET_KEY:-hindsight_s3_secret}
-    depends_on:
-      - db
-      - seaweedfs
-    networks:
-      - hindsight-net
-
-networks:
-  hindsight-net:
-    driver: bridge
-
-volumes:
-  pg_data:
-  seaweedfs_data:

docker/docker-compose/s3-file-storage/s3.json

@@ -1,19 +0,0 @@
-{
-  "identities": [
-    {
-      "name": "hindsight",
-      "credentials": [
-        {
-          "accessKey": "hindsight_s3_key",
-          "secretKey": "hindsight_s3_secret"
-        }
-      ],
-      "actions": [
-        "Admin",
-        "Read",
-        "Write",
-        "List"
-      ]
-    }
-  ]
-}

docker/docker-compose/timescale/.dockerignore

@@ -1,16 +0,0 @@
-# Git
-.git
-.gitignore
-.gitattributes
-
-# Docker
-docker-compose.yaml
-.dockerignore
-
-# Documentation
-README.md
-*.md
-
-# Environment
-.env
-.env.example

docker/docker-compose/timescale/.env.example

@@ -1,25 +0,0 @@
-# PostgreSQL Configuration
-HINDSIGHT_DB_USER=hindsight_user
-HINDSIGHT_DB_PASSWORD=change-me-to-secure-password
-HINDSIGHT_DB_NAME=hindsight_db
-
-# Hindsight Version
-HINDSIGHT_VERSION=latest
-
-# LLM Configuration
-HINDSIGHT_API_LLM_PROVIDER=openai
-OPENAI_API_KEY=your-openai-api-key-here
-
-# Alternative LLM providers (uncomment and configure as needed):
-# HINDSIGHT_API_LLM_PROVIDER=anthropic
-# ANTHROPIC_API_KEY=your-anthropic-api-key
-
-# HINDSIGHT_API_LLM_PROVIDER=gemini
-# GEMINI_API_KEY=your-gemini-api-key
-
-# HINDSIGHT_API_LLM_PROVIDER=groq
-# GROQ_API_KEY=your-groq-api-key
-
-# Vector and Text Search (already configured in docker-compose.yaml)
-# HINDSIGHT_API_VECTOR_EXTENSION=pgvectorscale
-# HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pg_textsearch

docker/docker-compose/timescale/Dockerfile

@@ -1,55 +0,0 @@
-# PostgreSQL with pgvector, pgvectorscale, and pg_textsearch extensions
-# All three extensions from Timescale/pgvector for high-performance vector and text search
-# Note: Requires PostgreSQL 16+
-FROM postgres:17
-
-# Install build dependencies and Rust toolchain
-RUN apt-get update && apt-get install -y \
-    build-essential \
-    git \
-    postgresql-server-dev-17 \
-    libpq-dev \
-    cmake \
-    curl \
-    pkg-config \
-    libssl-dev \
-    && rm -rf /var/lib/apt/lists/*
-
-# Install Rust toolchain (required for pgvectorscale)
-RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
-ENV PATH="/root/.cargo/bin:${PATH}"
-
-# Install pgvector (required by pgvectorscale)
-RUN cd /tmp && \
-    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
-    cd pgvector && \
-    make && \
-    make install && \
-    rm -rf /tmp/pgvector
-
-# Install cargo-pgrx (PostgreSQL extension framework for Rust)
-RUN cargo install cargo-pgrx --version 0.12.5 --locked && \
-    cargo pgrx init --pg17 /usr/bin/pg_config
-
-# Install pgvectorscale (DiskANN index support)
-RUN cd /tmp && \
-    git clone --branch 0.5.1 https://github.com/timescale/pgvectorscale.git && \
-    cd pgvectorscale/pgvectorscale && \
-    cargo pgrx install --release && \
-    rm -rf /tmp/pgvectorscale
-
-# Install pg_textsearch (BM25 text search)
-RUN cd /tmp && \
-    git clone https://github.com/timescale/pg_textsearch.git && \
-    cd pg_textsearch && \
-    make && \
-    make install && \
-    rm -rf /tmp/pg_textsearch
-
-# Clean up build dependencies (keep runtime dependencies)
-RUN apt-get purge -y --auto-remove git cmake curl && \
-    rm -rf /root/.cargo/registry /root/.cargo/git
-
-# Ensure extensions are preloaded (pg_textsearch requires preloading)
-RUN echo "shared_preload_libraries = 'pg_textsearch'" >> /usr/share/postgresql/postgresql.conf.sample
-

docker/docker-compose/timescale/README.md

@@ -1,101 +0,0 @@
-# Hindsight with Timescale Extensions
-
-This Docker Compose setup provides a complete Hindsight deployment with **Timescale extensions**:
-- **pgvectorscale** - DiskANN algorithm for disk-based scalable vector search  
-- **pg_textsearch** - High-performance BM25 text search
-
-Both extensions are from [Timescale](https://github.com/timescale) and provide production-grade performance.
-
-## Prerequisites
-
-- Docker and Docker Compose installed
-- OpenAI API key (or another LLM provider)
-
-## Quick Start
-
-```bash
-# Set environment variables
-export HINDSIGHT_DB_PASSWORD="your-secure-password"
-export OPENAI_API_KEY="your-openai-api-key"
-
-# Build and start
-docker compose -f docker/docker-compose/timescale/docker-compose.yaml up -d --build
-
-# Check logs
-
-docker compose -f docker/docker-compose/timescale/docker-compose.yaml logs -f
-```
-
-**Access:**
-- API: http://localhost:8888
-- Control Plane: http://localhost:9999
-
-## Stop and Clean Up
-
-```bash
-# Stop services
-docker compose -f docker/docker-compose/timescale/docker-compose.yaml down
-
-# Remove volumes (deletes all data)
-docker compose -f docker/docker-compose/timescale/docker-compose.yaml down -v
-```
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `HINDSIGHT_DB_PASSWORD` | PostgreSQL password | `hindsight_password` |
-| `HINDSIGHT_DB_USER` | PostgreSQL username | `hindsight_user` |
-| `HINDSIGHT_DB_NAME` | Database name | `hindsight_db` |
-| `HINDSIGHT_VERSION` | Hindsight Docker image version | `latest` |
-| `OPENAI_API_KEY` | OpenAI API key | (required) |
-| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider | `openai` |
-
-### Why Timescale Extensions?
-
-**pgvectorscale (DiskANN):**
-- 28x lower p95 latency vs dedicated vector databases
-- 16x higher query throughput at 99% recall
-- 60-75% cost reduction (disk is cheaper than RAM)
-- Best for large datasets (10M+ vectors)
-
-**pg_textsearch (BM25):**
-- High-performance keyword retrieval
-- Native BM25 ranking algorithm
-- Optimized for full-text search
-
-## Troubleshooting
-
-### Extensions not installed
-
-Check if extensions are available:
-
-```bash
-docker exec -it hindsight-db-timescale psql -U hindsight_user -d hindsight_db -c "\dx"
-```
-
-You should see:
-- `vector` (pgvector)
-- `vectorscale` (pgvectorscale/DiskANN)
-- `pg_textsearch` (BM25 search)
-
-### Build fails
-
-If the Docker build fails during pgvectorscale compilation:
-
-1. Ensure you have sufficient memory (recommended: 4GB+)
-2. Check Docker build logs for Rust compilation errors
-3. Try building with more resources: `docker compose build --no-cache --memory 4g`
-
-### Port conflicts
-
-If port 5438 is already in use, modify the `ports` section in docker-compose.yaml.
-
-## Learn More
-
-- [pgvectorscale GitHub](https://github.com/timescale/pgvectorscale)
-- [pg_textsearch GitHub](https://github.com/timescale/pg_textsearch)
-- [HNSW vs DiskANN](https://www.tigerdata.com/learn/hnsw-vs-diskann)
-- [Hindsight Documentation](https://hindsight.dev)

docker/docker-compose/timescale/docker-compose.yaml

@@ -1,108 +0,0 @@
-name: hindsight
-# Docker Compose file for Hindsight with Timescale extensions
-# - pgvectorscale: DiskANN vector search (disk-based, scalable)
-# - pg_textsearch: BM25 text search (high-performance keyword retrieval)
-#
-# Quick start:
-# docker compose -f docker/docker-compose/timescale/docker-compose.yaml up -d --build
-#
-# Required environment variables:
-# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
-# - OPENAI_API_KEY (or configure another LLM provider)
-#
-# Optional environment variables with defaults:
-# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
-# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
-# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
-
-services:
-  db:
-    # Custom PostgreSQL image with Timescale extensions (pgvectorscale + pg_textsearch)
-    build:
-      context: .
-      dockerfile: Dockerfile
-    container_name: hindsight-db-timescale
-    restart: always
-    # Expose PostgreSQL port (using 5438 to avoid conflicts with other setups)
-    ports:
-      - "5438:5432"
-    environment:
-      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
-      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
-      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
-    volumes:
-      - pg_data:/var/lib/postgresql/data
-    networks:
-      - hindsight-net
-    # Health check to ensure database is ready
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -U hindsight_user"]
-      interval: 5s
-      timeout: 5s
-      retries: 5
-
-  timescale-init:
-    build:
-      context: .
-      dockerfile: Dockerfile
-    depends_on:
-      db:
-        condition: service_healthy
-    environment:
-      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
-    command: >
-      bash -c "
-        echo 'PostgreSQL is ready - creating hindsight_db database';
-        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
-        echo 'Installing Timescale extensions...';
-        echo '1/3: Installing pgvector (required by pgvectorscale)...';
-        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
-        echo '2/3: Installing pgvectorscale (DiskANN vector search)...';
-        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE;';
-        echo '3/3: Installing pg_textsearch (BM25 text search)...';
-        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_textsearch CASCADE;';
-        echo '';
-        echo '✅ Timescale extensions installed successfully';
-        echo '';
-        echo 'Installed extensions:';
-        psql -h hindsight-db-timescale -p 5432 -U hindsight_user -d hindsight_db -c \"\\dx\" | grep -E '(vector|vectorscale|pg_textsearch)';
-      "
-    restart: "no"
-    networks:
-      - hindsight-net
-
-  hindsight:
-    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
-    container_name: hindsight-app-timescale
-    ports:
-      - "8888:8888"
-      - "9999:9999"
-    environment:
-      # LLM Configuration
-      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
-      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
-
-      # Database Configuration
-      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
-
-      # Timescale Extensions
-      # pgvectorscale: DiskANN algorithm for disk-based scalable vector search
-      HINDSIGHT_API_VECTOR_EXTENSION: pgvectorscale
-      # pg_textsearch: High-performance BM25 text search
-      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pg_textsearch
-
-    depends_on:
-      db:
-        condition: service_healthy
-      timescale-init:
-        condition: service_completed_successfully
-    networks:
-      - hindsight-net
-
-
-networks:
-  hindsight-net:
-    driver: bridge
-
-volumes:
-  pg_data:

docker/standalone/Dockerfile

@@ -42,22 +42,25 @@ RUN apt-get update && apt-get install -y \
     && pip install --no-cache-dir uv
 
 # Copy dependency files and README (required by pyproject.toml)
-COPY hindsight-api-slim/pyproject.toml ./api/
-COPY hindsight-api-slim/README.md ./api/
+COPY hindsight-api/pyproject.toml ./api/
+COPY hindsight-api/README.md ./api/
 
 WORKDIR /app/api
 
-# Sync dependencies using appropriate extras based on INCLUDE_LOCAL_MODELS
-# local-ml: torch, sentence-transformers, transformers, einops, flashrank, mlx (optional)
-# embedded-db: pg0-embedded (always included for embedded PostgreSQL support)
-RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-        uv sync --extra local-ml --extra embedded-db; \
-    else \
-        uv sync --extra embedded-db; \
+# Remove local ML model dependencies if INCLUDE_LOCAL_MODELS=false
+# This creates a smaller image when using external providers (TEI, OpenAI, Cohere)
+RUN if [ "$INCLUDE_LOCAL_MODELS" != "true" ]; then \
+        echo "Removing local-models dependencies (sentence-transformers, torch, transformers)..." && \
+        sed -i '/"sentence-transformers/d' pyproject.toml && \
+        sed -i '/"transformers/d' pyproject.toml && \
+        sed -i '/"torch/d' pyproject.toml; \
     fi
 
+# Sync dependencies (will create lock file if needed)
+RUN uv sync
+
 # Copy source code (alembic migrations are inside hindsight_api/)
-COPY hindsight-api-slim/hindsight_api ./hindsight_api
+COPY hindsight-api/hindsight_api ./hindsight_api
 
 # Install the local package (uv sync only installed dependencies, not the package itself)
 RUN uv pip install -e .
@@ -167,11 +170,6 @@ RUN chown -R hindsight:hindsight /app
 
 USER hindsight
 
-# Create pg0 data directory as hindsight user so that Docker seeds new named
-# volumes with correct ownership (UID 1000) on first use, avoiding the
-# "Permission denied" error when mounting a fresh root-owned volume.
-RUN mkdir -p /home/hindsight/.pg0
-
 ENV PATH="/app/api/.venv/bin:${PATH}"
 
 # Pre-download tiktoken encoding (ALWAYS - required for token counting even in air-gapped envs)
@@ -323,11 +321,6 @@ RUN chown -R hindsight:hindsight /app
 
 USER hindsight
 
-# Create pg0 data directory as hindsight user so that Docker seeds new named
-# volumes with correct ownership (UID 1000) on first use, avoiding the
-# "Permission denied" error when mounting a fresh root-owned volume.
-RUN mkdir -p /home/hindsight/.pg0
-
 ENV PATH="/app/api/.venv/bin:${PATH}"
 
 # Pre-download tiktoken encoding (ALWAYS - required for token counting even in air-gapped envs)

docker/standalone/start-all.sh

@@ -1,28 +1,6 @@
 #!/bin/bash
 set -e
 
-# =============================================================================
-# Embedded pg0 data integrity check (#675)
-#
-# When using embedded pg0, check if the data directory has existing PostgreSQL
-# data before starting. If the directory exists but appears empty/corrupt
-# (e.g., missing PG_VERSION file), log a warning. This helps diagnose data
-# loss scenarios where a container restart caused the data directory to be
-# wiped despite a volume mount being present.
-# =============================================================================
-PG0_DATA_DIR="${HOME}/.pg0"
-if [ -d "$PG0_DATA_DIR" ]; then
-    # Look for actual PostgreSQL data directories (pg0 creates subdirs per instance)
-    if compgen -G "$PG0_DATA_DIR"/*/PG_VERSION > /dev/null 2>&1; then
-        echo "✅ Existing pg0 data directory detected at $PG0_DATA_DIR"
-    elif [ "$(ls -A "$PG0_DATA_DIR" 2>/dev/null)" ]; then
-        echo "⚠️  WARNING: pg0 data directory exists at $PG0_DATA_DIR but no PG_VERSION found."
-        echo "   This may indicate data corruption or an incomplete previous shutdown."
-        echo "   If you see all migrations running from scratch after this, your data may have been lost."
-        echo "   See: https://github.com/vectorize-io/hindsight/issues/675"
-    fi
-fi
-
 # Service flags (default to true if not set)
 ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
 ENABLE_CP="${HINDSIGHT_ENABLE_CP:-true}"
@@ -93,95 +71,24 @@ if [ "${HINDSIGHT_WAIT_FOR_DEPS:-false}" = "true" ]; then
     done
 fi
 
-# =============================================================================
-# Graceful shutdown handler (#675)
-#
-# Docker sends SIGTERM on `docker stop`/`docker restart`. Without a trap, child
-# processes (hindsight-api + pg0, control-plane) are killed abruptly. For the
-# embedded pg0 database this can cause data loss when the data directory is on
-# a Docker volume that gets remounted after restart.
-#
-# The trap forwards SIGTERM to all tracked child PIDs so that:
-#   - hindsight-api receives the signal and can run its shutdown hooks
-#   - pg0 gets a clean PostgreSQL shutdown (checkpoint + WAL flush)
-#   - The control-plane Node.js process exits cleanly
-# =============================================================================
-# Guard against concurrent cleanup (e.g., child crash + SIGTERM arriving together)
-SHUTTING_DOWN=false
-
-cleanup() {
-    if $SHUTTING_DOWN; then return; fi
-    SHUTTING_DOWN=true
-
-    echo ""
-    echo "🛑 Received shutdown signal, stopping services gracefully..."
-    for pid in "${PIDS[@]}"; do
-        if kill -0 "$pid" 2>/dev/null; then
-            kill -TERM "$pid" 2>/dev/null
-        fi
-    done
-    # Give processes time to shut down cleanly (pg0 needs to flush WAL).
-    # NOTE: Docker's default stop_grace_period is 10s. If you use the default,
-    # either set stop_grace_period: 30s in your compose file / docker stop -t 30,
-    # or Docker will SIGKILL the container before this timeout expires.
-    local timeout=30
-    for ((i=1; i<=timeout; i++)); do
-        local all_stopped=true
-        for pid in "${PIDS[@]}"; do
-            if kill -0 "$pid" 2>/dev/null; then
-                all_stopped=false
-                break
-            fi
-        done
-        if $all_stopped; then
-            echo "✅ All services stopped cleanly"
-            exit 0
-        fi
-        sleep 1
-    done
-    # Force kill if still running after timeout
-    echo "⚠️  Timeout reached, forcing shutdown..."
-    for pid in "${PIDS[@]}"; do
-        if kill -0 "$pid" 2>/dev/null; then
-            kill -9 "$pid" 2>/dev/null
-        fi
-    done
-    exit 1
-}
-trap cleanup SIGTERM SIGINT
-
 # Track PIDs for wait
 PIDS=()
 
 # Start API if enabled
 if [ "$ENABLE_API" = "true" ]; then
     cd /app/api
-    API_HEALTH_URL="${HINDSIGHT_API_HEALTH_URL:-http://localhost:8888/health}"
-    API_STARTUP_WAIT_SECONDS="${HINDSIGHT_API_STARTUP_WAIT_SECONDS:-300}"
-
     # Run API directly - Python's PYTHONUNBUFFERED=1 handles output buffering
     hindsight-api &
     API_PID=$!
     PIDS+=($API_PID)
 
     # Wait for API to be ready
-    api_ready=false
-    for ((i=1; i<=API_STARTUP_WAIT_SECONDS; i++)); do
-        if ! kill -0 "$API_PID" 2>/dev/null; then
-            wait "$API_PID"
-            exit $?
-        fi
-        if curl -sf "$API_HEALTH_URL" &>/dev/null; then
-            api_ready=true
+    for i in {1..60}; do
+        if curl -sf http://localhost:8888/health &>/dev/null; then
             break
         fi
         sleep 1
     done
-
-    if [ "$api_ready" != "true" ]; then
-        echo "❌ API did not become healthy within ${API_STARTUP_WAIT_SECONDS}s"
-        exit 1
-    fi
 else
     echo "API disabled (HINDSIGHT_ENABLE_API=false)"
 fi
@@ -190,8 +97,7 @@ fi
 if [ "$ENABLE_CP" = "true" ]; then
     echo "🎛️  Starting Control Plane..."
     cd /app/control-plane
-    export HOSTNAME="${HINDSIGHT_CP_HOSTNAME:-0.0.0.0}"
-    PORT="${HINDSIGHT_CP_PORT:-9999}" node server.js &
+    PORT=9999 node server.js &
     CP_PID=$!
     PIDS+=($CP_PID)
 else
@@ -204,7 +110,7 @@ echo "✅ Hindsight is running!"
 echo ""
 echo "📍 Access:"
 if [ "$ENABLE_CP" = "true" ]; then
-    echo "   Control Plane: http://localhost:${HINDSIGHT_CP_PORT:-9999}"
+    echo "   Control Plane: http://localhost:9999"
 fi
 if [ "$ENABLE_API" = "true" ]; then
     echo "   API:           http://localhost:8888"
@@ -217,21 +123,8 @@ if [ ${#PIDS[@]} -eq 0 ]; then
     exit 1
 fi
 
-# Wait for any process to exit (use wait -n with trap-safe loop)
-while true; do
-    # wait -n returns when any child exits; it also returns on signal delivery
-    # (the trap handler will run and exit, so this loop is just for robustness).
-    # `&& true` prevents `set -e` from killing the script when wait -n returns
-    # non-zero (child exited with error or no backgrounded children remain).
-    wait -n && true
-    # Check if any tracked PID has exited
-    for pid in "${PIDS[@]}"; do
-        if ! kill -0 "$pid" 2>/dev/null; then
-            wait "$pid" 2>/dev/null
-            exit_code=$?
-            echo "⚠️  Service (PID $pid) exited with code $exit_code"
-            # Trigger cleanup for remaining services
-            cleanup
-        fi
-    done
-done
+# Wait for any process to exit
+wait -n
+
+# Exit with status of first exited process
+exit $?

docker/test-image.sh

@@ -13,9 +13,9 @@
 #   target  - Optional: 'cp-only' for control plane, otherwise assumes API image (default: api)
 #
 # Environment variables:
-#   HINDSIGHT_API_LLM_API_KEY                   - Required for API/standalone images (LLM verification)
-#   HINDSIGHT_API_LLM_PROVIDER                  - LLM provider (default: openai)
-#   HINDSIGHT_API_LLM_MODEL                     - LLM model (default: gpt-4o-mini)
+#   GROQ_API_KEY                                - Required for API/standalone images (LLM verification)
+#   HINDSIGHT_API_LLM_PROVIDER                  - LLM provider (default: groq)
+#   HINDSIGHT_API_LLM_MODEL                     - LLM model (default: llama-3.3-70b-versatile)
 #   HINDSIGHT_API_EMBEDDINGS_PROVIDER           - Embeddings provider (optional, for slim images: openai, cohere, tei)
 #   HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY     - OpenAI API key for embeddings (optional)
 #   HINDSIGHT_API_RERANKER_PROVIDER             - Reranker provider (optional, for slim images: cohere, tei)
@@ -34,7 +34,7 @@
 #   ./docker/test-image.sh hindsight-control-plane:test cp-only
 #
 #   # Test slim image with external providers
-#   export HINDSIGHT_API_LLM_API_KEY=sk_xxx
+#   export GROQ_API_KEY=gsk_xxx
 #   export HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
 #   export HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=sk-xxx
 #   export HINDSIGHT_API_RERANKER_PROVIDER=cohere
@@ -49,9 +49,6 @@
 
 set -euo pipefail
 
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-REPO_ROOT="$(dirname "$SCRIPT_DIR")"
-
 # Colors for output
 RED='\033[0;31m'
 GREEN='\033[0;32m'
@@ -63,8 +60,8 @@ IMAGE="${1:-}"
 TARGET="${2:-api}"
 TIMEOUT="${SMOKE_TEST_TIMEOUT:-120}"
 CONTAINER_NAME="${SMOKE_TEST_CONTAINER_NAME:-hindsight-smoke-test}"
-LLM_PROVIDER="${HINDSIGHT_API_LLM_PROVIDER:-openai}"
-LLM_MODEL="${HINDSIGHT_API_LLM_MODEL:-gpt-4o-mini}"
+LLM_PROVIDER="${HINDSIGHT_API_LLM_PROVIDER:-groq}"
+LLM_MODEL="${HINDSIGHT_API_LLM_MODEL:-llama-3.3-70b-versatile}"
 
 # Validate arguments
 if [ -z "$IMAGE" ]; then
@@ -91,9 +88,9 @@ else
 fi
 
 # Check for required environment variables
-if [ "$NEEDS_LLM" = true ] && [ "$LLM_PROVIDER" != "vertexai" ] && [ -z "${HINDSIGHT_API_LLM_API_KEY:-}" ]; then
-    echo -e "${RED}Error: HINDSIGHT_API_LLM_API_KEY environment variable is required for API/standalone images${NC}"
-    echo "Set it with: export HINDSIGHT_API_LLM_API_KEY=your-api-key"
+if [ "$NEEDS_LLM" = true ] && [ -z "${GROQ_API_KEY:-}" ]; then
+    echo -e "${RED}Error: GROQ_API_KEY environment variable is required for API/standalone images${NC}"
+    echo "Set it with: export GROQ_API_KEY=your-api-key"
     exit 2
 fi
 
@@ -126,25 +123,9 @@ else
     # Build docker run command with required and optional env vars
     DOCKER_CMD="docker run -d --name $CONTAINER_NAME"
     DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_PROVIDER=$LLM_PROVIDER"
-    if [ -n "${HINDSIGHT_API_LLM_API_KEY:-}" ]; then
-        DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_API_KEY=${HINDSIGHT_API_LLM_API_KEY}"
-    fi
+    DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_API_KEY=${GROQ_API_KEY}"
     DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_MODEL=$LLM_MODEL"
 
-    # Add Vertex AI config if provider is vertexai
-    if [ "$LLM_PROVIDER" = "vertexai" ]; then
-        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY:-}" ]; then
-            DOCKER_CMD="$DOCKER_CMD -v ${HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY}:/tmp/gcp-credentials.json:ro"
-            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/tmp/gcp-credentials.json"
-        fi
-        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID:-}" ]; then
-            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=${HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID}"
-        fi
-        if [ -n "${HINDSIGHT_API_LLM_VERTEXAI_REGION:-}" ]; then
-            DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_LLM_VERTEXAI_REGION=${HINDSIGHT_API_LLM_VERTEXAI_REGION}"
-        fi
-    fi
-
     # Add optional embeddings provider config
     if [ -n "${HINDSIGHT_API_EMBEDDINGS_PROVIDER:-}" ]; then
         DOCKER_CMD="$DOCKER_CMD -e HINDSIGHT_API_EMBEDDINGS_PROVIDER=${HINDSIGHT_API_EMBEDDINGS_PROVIDER}"
@@ -181,21 +162,6 @@ for i in $(seq 1 "$TIMEOUT"); do
         echo "=== Health Response ==="
         curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}" | python3 -m json.tool 2>/dev/null || curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}"
         echo ""
-
-        # Run retain/recall smoke test for API targets
-        if [ "$TARGET" != "cp-only" ]; then
-            echo ""
-            echo "=== Retain/Recall Smoke Test ==="
-            if ! "$REPO_ROOT/scripts/smoke-test-slim.sh" "http://localhost:${HEALTH_PORT}"; then
-                echo ""
-                echo "=== Container Logs (last 50 lines) ==="
-                docker logs "$CONTAINER_NAME" 2>&1 | tail -50
-                echo ""
-                echo -e "${RED}Smoke test FAILED${NC}"
-                exit 1
-            fi
-        fi
-
         echo ""
         echo "=== Container Logs (last 50 lines) ==="
         docker logs "$CONTAINER_NAME" 2>&1 | tail -50

docker/test-slim-local.sh

@@ -6,17 +6,24 @@
 # It expects API keys to be set in environment variables.
 #
 # Usage:
+#   export GROQ_API_KEY=gsk_xxx
 #   export OPENAI_API_KEY=sk-xxx
 #   export COHERE_API_KEY=xxx
 #   ./docker/test-slim-local.sh
 #
 # Or inline:
-#   OPENAI_API_KEY=sk_xxx COHERE_API_KEY=xxx ./docker/test-slim-local.sh
+#   GROQ_API_KEY=gsk_xxx OPENAI_API_KEY=sk_xxx COHERE_API_KEY=xxx ./docker/test-slim-local.sh
 #
 
 set -euo pipefail
 
 # Check for required API keys
+if [ -z "${GROQ_API_KEY:-}" ]; then
+    echo "❌ Error: GROQ_API_KEY environment variable is required"
+    echo "Set it with: export GROQ_API_KEY=gsk_xxx"
+    exit 1
+fi
+
 if [ -z "${OPENAI_API_KEY:-}" ]; then
     echo "❌ Error: OPENAI_API_KEY environment variable is required"
     echo "Set it with: export OPENAI_API_KEY=sk-xxx"
@@ -34,10 +41,7 @@ IMAGE="${1:-hindsight-slim:test}"
 echo "Testing image: $IMAGE"
 echo ""
 
-# Set up LLM and external providers
-export HINDSIGHT_API_LLM_PROVIDER=openai
-export HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY
-export HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
+# Set up external providers
 export HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
 export HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=$OPENAI_API_KEY
 export HINDSIGHT_API_RERANKER_PROVIDER=cohere

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.5.3
-appVersion: "0.5.3"
+version: 0.4.10
+appVersion: "0.4.10"
 keywords:
   - ai
   - memory

helm/hindsight/templates/api-deployment.yaml

@@ -95,27 +95,6 @@ spec:
           {{- toYaml .Values.api.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.api.resources | nindent 10 }}
-        {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumeMounts }}
-        volumeMounts:
-        {{- if .Values.api.persistence.modelCache.enabled }}
-        - name: model-cache
-          mountPath: /home/hindsight/.cache
-        {{- end }}
-        {{- with .Values.api.extraVolumeMounts }}
-        {{- toYaml . | nindent 8 }}
-        {{- end }}
-        {{- end }}
-      {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumes }}
-      volumes:
-      {{- if .Values.api.persistence.modelCache.enabled }}
-      - name: model-cache
-        persistentVolumeClaim:
-          claimName: {{ include "hindsight.fullname" . }}-api-model-cache
-      {{- end }}
-      {{- with .Values.api.extraVolumes }}
-      {{- toYaml . | nindent 6 }}
-      {{- end }}
-      {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}

helm/hindsight/templates/api-model-cache-pvc.yaml

@@ -1,21 +0,0 @@
-{{- if and .Values.api.enabled .Values.api.persistence.modelCache.enabled }}
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: {{ include "hindsight.fullname" . }}-api-model-cache
-  labels:
-    {{- include "hindsight.api.labels" . | nindent 4 }}
-  {{- with .Values.api.persistence.modelCache.annotations }}
-  annotations:
-    {{- toYaml . | nindent 4 }}
-  {{- end }}
-spec:
-  accessModes:
-    {{- toYaml .Values.api.persistence.modelCache.accessModes | nindent 4 }}
-  {{- if .Values.api.persistence.modelCache.storageClass }}
-  storageClassName: {{ .Values.api.persistence.modelCache.storageClass }}
-  {{- end }}
-  resources:
-    requests:
-      storage: {{ .Values.api.persistence.modelCache.size }}
-{{- end }}

helm/hindsight/templates/worker-statefulset.yaml

@@ -95,16 +95,6 @@ spec:
           {{- toYaml .Values.worker.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.worker.resources | nindent 10 }}
-        {{- if or .Values.worker.persistence.modelCache.enabled .Values.worker.extraVolumeMounts }}
-        volumeMounts:
-        {{- if .Values.worker.persistence.modelCache.enabled }}
-        - name: model-cache
-          mountPath: /home/hindsight/.cache
-        {{- end }}
-        {{- with .Values.worker.extraVolumeMounts }}
-        {{- toYaml . | nindent 8 }}
-        {{- end }}
-        {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -117,26 +107,4 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.worker.extraVolumes }}
-      volumes:
-      {{- toYaml . | nindent 6 }}
-      {{- end }}
-  {{- if .Values.worker.persistence.modelCache.enabled }}
-  volumeClaimTemplates:
-  - metadata:
-      name: model-cache
-      {{- with .Values.worker.persistence.modelCache.annotations }}
-      annotations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-    spec:
-      accessModes:
-        {{- toYaml .Values.worker.persistence.modelCache.accessModes | nindent 8 }}
-      {{- if .Values.worker.persistence.modelCache.storageClass }}
-      storageClassName: {{ .Values.worker.persistence.modelCache.storageClass }}
-      {{- end }}
-      resources:
-        requests:
-          storage: {{ .Values.worker.persistence.modelCache.size }}
-  {{- end }}
 {{- end }}

helm/hindsight/values.yaml

@@ -67,33 +67,6 @@ api:
   # Pod affinity/anti-affinity (overrides global affinity for this component)
   # affinity: {}
 
-  # Persistent volume for local model cache (reranker, embeddings)
-  # Models are downloaded to /home/hindsight/.cache on first use.
-  # Without persistence, models are re-downloaded on every pod restart.
-  persistence:
-    modelCache:
-      enabled: false
-      size: 5Gi
-      storageClass: ""
-      accessModes:
-        - ReadWriteOnce
-      annotations: {}
-
-  # Extra volume mounts for the api container
-  # e.g.
-  # extraVolumeMounts:
-  #   - name: my-volume
-  #     mountPath: /mnt/my-volume
-  extraVolumeMounts: []
-
-  # Extra volumes for the api pod
-  # e.g.
-  # extraVolumes:
-  #   - name: my-volume
-  #     configMap:
-  #       name: my-configmap
-  extraVolumes: []
-
   # Environment variables
   env:
     #HINDSIGHT_API_LLM_PROVIDER: "groq"
@@ -167,32 +140,6 @@ worker:
   # Pod affinity/anti-affinity (overrides global affinity for this component)
   # affinity: {}
 
-  # Persistent volume for local model cache (reranker, embeddings)
-  # Uses volumeClaimTemplates since worker is a StatefulSet.
-  persistence:
-    modelCache:
-      enabled: false
-      size: 5Gi
-      storageClass: ""
-      accessModes:
-        - ReadWriteOnce
-      annotations: {}
-
-  # Extra volume mounts for the worker container
-  # e.g.
-  # extraVolumeMounts:
-  #   - name: my-volume
-  #     mountPath: /mnt/my-volume
-  extraVolumeMounts: []
-
-  # Extra volumes for the worker pod
-  # e.g.
-  # extraVolumes:
-  #   - name: my-volume
-  #     configMap:
-  #       name: my-configmap
-  extraVolumes: []
-
   # Secret environment variables (inherited from api.secrets if not specified)
   secrets: {}
 

hindsight-all-npm/.gitignore

@@ -1,4 +0,0 @@
-node_modules
-dist
-*.tgz
-.DS_Store

hindsight-all-npm/README.md

@@ -1,80 +0,0 @@
-# @vectorize-io/hindsight-all
-
-Node.js equivalent of the Python [`hindsight-all`](https://pypi.org/project/hindsight-all/) package — programmatic lifecycle manager for a local Hindsight daemon. Use this when you want to embed Hindsight in a Node application without hand-rolling subprocess management.
-
-This package deliberately does **not** ship an HTTP client. Once the daemon is running, talk to it with [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client) against `server.getBaseUrl()`. The two packages compose — one owns the daemon process, the other owns the HTTP API surface.
-
-## Requirements
-
-- **Node.js >= 22** — uses global `fetch` and `AbortSignal.timeout`.
-- **`uv` / `uvx`** on `PATH` — used to download and run the underlying `hindsight-embed` daemon on first use. Install via <https://docs.astral.sh/uv/>.
-
-## Install
-
-```bash
-npm install @vectorize-io/hindsight-all @vectorize-io/hindsight-client
-```
-
-## Example
-
-```ts
-import { HindsightServer, consoleLogger } from '@vectorize-io/hindsight-all';
-import { HindsightClient } from '@vectorize-io/hindsight-client';
-
-const server = new HindsightServer({
-  profile: 'my-app',
-  port: 9077,
-  env: {
-    HINDSIGHT_API_LLM_PROVIDER: 'anthropic',
-    HINDSIGHT_API_LLM_API_KEY: process.env.ANTHROPIC_API_KEY,
-    HINDSIGHT_API_LLM_MODEL: 'claude-sonnet-4-20250514',
-    HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT: '0',
-  },
-  logger: consoleLogger,
-});
-
-await server.start();
-
-const client = new HindsightClient({ baseUrl: server.getBaseUrl() });
-
-await client.retain('user-123', 'User prefers dark mode and concise answers.', {
-  documentId: 'pref-2026-04-01',
-});
-
-const recall = await client.recall('user-123', 'what are the user preferences?');
-console.log(recall.results);
-
-await server.stop();
-```
-
-For a remote Hindsight API, skip `HindsightServer` entirely and just point `HindsightClient` at the remote URL.
-
-## Open config — forward-compatible with new daemon flags
-
-`HindsightServerOptions` is designed so every new environment variable or CLI flag in the underlying Hindsight daemon can be used without waiting for a wrapper release:
-
-- **`env`** accepts an arbitrary `Record<string, string>`. Every entry is exported into the daemon process and written into the profile config via `--env KEY=VALUE`.
-- **`extraProfileCreateArgs`** / **`extraDaemonStartArgs`** append raw args to the respective commands.
-
-## Development against a local checkout
-
-If you're hacking on the Python `hindsight-embed` package in the same monorepo, point the server at the local path — it'll use `uv run --directory <path>` instead of `uvx`:
-
-```ts
-new HindsightServer({
-  embedPackagePath: '/path/to/hindsight-embed',
-  // ...
-});
-```
-
-## API surface
-
-- `HindsightServer` — daemon lifecycle (`start`, `stop`, `checkHealth`, `getBaseUrl`, `getProfile`).
-- `Logger` interface plus `silentLogger` (default) and `consoleLogger` helpers.
-- `getEmbedCommand(opts)` — low-level helper that returns the `[cmd, ...args]` tuple used to invoke the underlying Python CLI.
-
-For memory operations (retain, recall, reflect, bank management, stats) use [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client).
-
-## License
-
-MIT

hindsight-all-npm/package.json

@@ -1,57 +0,0 @@
-{
-  "name": "@vectorize-io/hindsight-all",
-  "version": "0.5.3",
-  "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
-  "main": "dist/index.js",
-  "types": "dist/index.d.ts",
-  "type": "module",
-  "exports": {
-    ".": {
-      "types": "./dist/index.d.ts",
-      "import": "./dist/index.js"
-    }
-  },
-  "keywords": [
-    "hindsight",
-    "hindsight-all",
-    "memory",
-    "ai",
-    "agent",
-    "long-term-memory",
-    "llm",
-    "embedded-server"
-  ],
-  "author": "Vectorize <[email protected]>",
-  "license": "MIT",
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/vectorize-io/hindsight.git",
-    "directory": "hindsight-all-npm"
-  },
-  "files": [
-    "dist",
-    "README.md"
-  ],
-  "scripts": {
-    "build": "tsup",
-    "dev": "tsup --watch",
-    "clean": "rm -rf dist",
-    "test": "vitest run src",
-    "test:watch": "vitest src",
-    "prepublishOnly": "npm run clean && npm run build"
-  },
-  "devDependencies": {
-    "@types/node": "^22.0.0",
-    "tsup": "^8.5.1",
-    "typescript": "^5.7.0",
-    "vitest": "^4.1.2"
-  },
-  "engines": {
-    "node": ">=22"
-  },
-  "overrides": {
-    "rollup": "^4.59.0",
-    "picomatch": ">=2.3.2 <3.0.0 || >=4.0.4",
-    "vite": ">=8.0.5"
-  }
-}

hindsight-all-npm/src/command.test.ts

@@ -1,32 +0,0 @@
-import { describe, it, expect } from 'vitest';
-import { getEmbedCommand } from './command.js';
-
-describe('getEmbedCommand', () => {
-  it('defaults to uvx hindsight-embed@latest', () => {
-    expect(getEmbedCommand()).toEqual(['uvx', 'hindsight-embed@latest']);
-  });
-
-  it('honours an explicit version', () => {
-    expect(getEmbedCommand({ embedVersion: '0.5.0' })).toEqual(['uvx', '[email protected]']);
-  });
-
-  it('treats an empty version as latest', () => {
-    expect(getEmbedCommand({ embedVersion: '' })).toEqual(['uvx', 'hindsight-embed@latest']);
-  });
-
-  it('uses uv run --directory when a local path is given', () => {
-    expect(getEmbedCommand({ embedPackagePath: '/abs/path' })).toEqual([
-      'uv',
-      'run',
-      '--directory',
-      '/abs/path',
-      'hindsight-embed',
-    ]);
-  });
-
-  it('local path takes precedence over version', () => {
-    expect(
-      getEmbedCommand({ embedPackagePath: '/abs/path', embedVersion: '0.5.0' }),
-    ).toEqual(['uv', 'run', '--directory', '/abs/path', 'hindsight-embed']);
-  });
-});

hindsight-all-npm/src/command.ts

@@ -1,25 +0,0 @@
-/**
- * Resolve the command that invokes the `hindsight-embed` Python CLI.
- *
- * - If `embedPackagePath` is set, runs the package from a local checkout via
- *   `uv run --directory <path> hindsight-embed`. Used for in-repo development.
- * - Otherwise runs it via `uvx hindsight-embed@<version>` so no global install
- *   is required.
- *
- * Returns the argv as `[command, ...baseArgs]` suitable for `spawn()` /
- * `execFile()` (never shell-interpolated).
- */
-export interface EmbedCommandOptions {
-  /** Version spec passed to uvx (e.g. "latest", "0.5.0"). Default: "latest". */
-  embedVersion?: string;
-  /** Local checkout path. When set, overrides `embedVersion` and uses `uv run`. */
-  embedPackagePath?: string;
-}
-
-export function getEmbedCommand(opts: EmbedCommandOptions = {}): string[] {
-  if (opts.embedPackagePath) {
-    return ['uv', 'run', '--directory', opts.embedPackagePath, 'hindsight-embed'];
-  }
-  const version = opts.embedVersion && opts.embedVersion.length > 0 ? opts.embedVersion : 'latest';
-  return ['uvx', `hindsight-embed@${version}`];
-}

hindsight-all-npm/src/index.ts

@@ -1,7 +0,0 @@
-export { HindsightServer } from './server.js';
-export { getEmbedCommand } from './command.js';
-export { silentLogger, consoleLogger } from './logger.js';
-
-export type { Logger } from './logger.js';
-export type { EmbedCommandOptions } from './command.js';
-export type { HindsightServerOptions } from './types.js';

hindsight-all-npm/src/logger.ts

@@ -1,29 +0,0 @@
-/**
- * Pluggable logger interface.
- *
- * This package does not own any logging infrastructure — consumers inject
- * whatever they want (console, pino, openclaw's logger, a no-op). The default
- * is silent so embedding this package never adds noise to an unrelated app.
- */
-export interface Logger {
-  debug(msg: string): void;
-  info(msg: string): void;
-  warn(msg: string): void;
-  error(msg: string): void;
-}
-
-/** Logger that drops every call. Used when no logger is passed. */
-export const silentLogger: Logger = {
-  debug: () => {},
-  info: () => {},
-  warn: () => {},
-  error: () => {},
-};
-
-/** Logger that writes to the standard console. Handy for CLIs and tests. */
-export const consoleLogger: Logger = {
-  debug: (msg) => console.debug(msg),
-  info: (msg) => console.log(msg),
-  warn: (msg) => console.warn(msg),
-  error: (msg) => console.error(msg),
-};

hindsight-all-npm/src/server.test.ts

@@ -1,35 +0,0 @@
-import { describe, it, expect } from 'vitest';
-import { HindsightServer } from './server.js';
-
-describe('HindsightServer construction', () => {
-  it('defaults base URL to http://127.0.0.1:8888', () => {
-    const server = new HindsightServer();
-    expect(server.getBaseUrl()).toBe('http://127.0.0.1:8888');
-    expect(server.getProfile()).toBe('default');
-  });
-
-  it('honours custom profile, port, and host', () => {
-    const server = new HindsightServer({ profile: 'app', port: 9077, host: '0.0.0.0' });
-    expect(server.getProfile()).toBe('app');
-    expect(server.getBaseUrl()).toBe('http://0.0.0.0:9077');
-  });
-
-  it('accepts open env pass-through without complaining about unknown keys', () => {
-    const server = new HindsightServer({
-      env: {
-        HINDSIGHT_API_LLM_PROVIDER: 'openai',
-        HINDSIGHT_API_LLM_MODEL: 'gpt-4o-mini',
-        // A field that does not exist today — should still be accepted
-        HINDSIGHT_FUTURE_FLAG: 'enabled',
-      },
-    });
-    expect(server).toBeInstanceOf(HindsightServer);
-  });
-
-  it('exposes checkHealth that returns false when no daemon is running', async () => {
-    // Random high port that nothing is listening on.
-    const server = new HindsightServer({ port: 1, readyTimeoutMs: 100 });
-    const healthy = await server.checkHealth();
-    expect(healthy).toBe(false);
-  });
-});

hindsight-all-npm/src/server.ts

@@ -1,322 +0,0 @@
-import { spawn } from 'child_process';
-import { getEmbedCommand } from './command.js';
-import { silentLogger } from './logger.js';
-import type { Logger } from './logger.js';
-import type { HindsightServerOptions } from './types.js';
-
-const DEFAULT_PORT = 8888;
-const DEFAULT_HOST = '127.0.0.1';
-const DEFAULT_PROFILE = 'default';
-const DEFAULT_READY_TIMEOUT_MS = 30_000;
-const DEFAULT_READY_POLL_INTERVAL_MS = 1_000;
-
-/**
- * Manages the lifecycle of a local Hindsight daemon from a Node.js process.
- *
- * On {@link start}, this class:
- *   1. Resolves the `hindsight-embed` command (via `uvx` or a local `uv run`).
- *   2. Runs `profile create <name> --merge --port <port> [--env K=V ...]`
- *      with every entry in {@link HindsightServerOptions.env} forwarded as
- *      an `--env` flag.
- *   3. Runs `daemon --profile <name> start` and waits for the start command
- *      to exit.
- *   4. Polls `http://host:port/health` until it returns `200` or the
- *      `readyTimeoutMs` budget is exhausted.
- *
- * On {@link stop}, it runs `daemon --profile <name> stop` and returns once
- * the command exits (or after a short grace period).
- *
- * This is the Node.js equivalent of the Python `hindsight-all` package's
- * `HindsightServer`: a thin programmatic lifecycle wrapper around the
- * Hindsight daemon. It does NOT ship an HTTP client — once `start()`
- * resolves, use `@vectorize-io/hindsight-client` against `getBaseUrl()` for
- * retain / recall / reflect.
- *
- * The class is deliberately transparent about the daemon: new CLI flags or
- * environment variables never require a code change here — callers can pass
- * them via `env`, `extraProfileCreateArgs`, or `extraDaemonStartArgs`.
- */
-export class HindsightServer {
-  private readonly profile: string;
-  private readonly port: number;
-  private readonly host: string;
-  private readonly baseUrl: string;
-  private readonly embedVersion: string | undefined;
-  private readonly embedPackagePath: string | undefined;
-  private readonly userEnv: Record<string, string | undefined>;
-  private readonly extraProfileCreateArgs: string[];
-  private readonly extraDaemonStartArgs: string[];
-  private readonly platformCpuWorkaround: boolean;
-  private readonly readyTimeoutMs: number;
-  private readonly readyPollIntervalMs: number;
-  private readonly logger: Logger;
-
-  constructor(opts: HindsightServerOptions = {}) {
-    this.profile = opts.profile ?? DEFAULT_PROFILE;
-    this.port = opts.port ?? DEFAULT_PORT;
-    this.host = opts.host ?? DEFAULT_HOST;
-    this.baseUrl = `http://${this.host}:${this.port}`;
-    this.embedVersion = opts.embedVersion;
-    this.embedPackagePath = opts.embedPackagePath;
-    this.userEnv = opts.env ?? {};
-    this.extraProfileCreateArgs = opts.extraProfileCreateArgs ?? [];
-    this.extraDaemonStartArgs = opts.extraDaemonStartArgs ?? [];
-    this.platformCpuWorkaround = opts.platformCpuWorkaround ?? (process.platform === 'darwin');
-    this.readyTimeoutMs = opts.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
-    this.readyPollIntervalMs = opts.readyPollIntervalMs ?? DEFAULT_READY_POLL_INTERVAL_MS;
-    this.logger = opts.logger ?? silentLogger;
-  }
-
-  /** The base URL the daemon listens on (`http://host:port`). */
-  getBaseUrl(): string {
-    return this.baseUrl;
-  }
-
-  /** The profile name this server operates on. */
-  getProfile(): string {
-    return this.profile;
-  }
-
-  /**
-   * Ensure the daemon is configured and running. Idempotent — the underlying
-   * `profile create --merge` and `daemon start` commands tolerate re-runs.
-   */
-  async start(): Promise<void> {
-    this.logger.info(`[hindsight] starting daemon for profile "${this.profile}"`);
-
-    const env = this.buildEnv();
-    await this.configureProfile(env);
-    await this.startDaemon(env);
-    await this.waitForReady();
-
-    this.logger.info(`[hindsight] daemon ready at ${this.baseUrl}`);
-  }
-
-  /** Stop the daemon. Never throws — logs and resolves even on failure. */
-  async stop(): Promise<void> {
-    this.logger.info(`[hindsight] stopping daemon for profile "${this.profile}"`);
-
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const args = [...baseArgs, 'daemon', '--profile', this.profile, 'stop'];
-
-    const child = spawn(cmd, args, { stdio: 'pipe' });
-    this.pipeOutput(child, 'daemon.stop');
-
-    await new Promise<void>((resolve) => {
-      const timeout = setTimeout(() => {
-        this.logger.warn(`[hindsight] daemon stop timed out after 5s`);
-        resolve();
-      }, 5_000);
-      child.on('exit', () => {
-        clearTimeout(timeout);
-        this.logger.info(`[hindsight] daemon stopped`);
-        resolve();
-      });
-      child.on('error', (err) => {
-        clearTimeout(timeout);
-        this.logger.warn(`[hindsight] error stopping daemon: ${err.message}`);
-        resolve();
-      });
-    });
-  }
-
-  /** Probe `/health` once with a short timeout. */
-  async checkHealth(): Promise<boolean> {
-    try {
-      const res = await fetch(`${this.baseUrl}/health`, {
-        signal: AbortSignal.timeout(2_000),
-      });
-      return res.ok;
-    } catch {
-      return false;
-    }
-  }
-
-  // -------------------------------------------------------------------------
-  // Internal
-  // -------------------------------------------------------------------------
-
-  /**
-   * Merge the process env, the caller-supplied `env`, and (on macOS) the
-   * embeddings CPU workaround. Caller-supplied values always win over the
-   * workaround; undefined values are dropped.
-   */
-  private buildEnv(): NodeJS.ProcessEnv {
-    const merged: NodeJS.ProcessEnv = { ...process.env };
-
-    if (this.platformCpuWorkaround && process.platform === 'darwin') {
-      merged['HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU'] = '1';
-      merged['HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU'] = '1';
-    }
-
-    for (const [key, value] of Object.entries(this.userEnv)) {
-      if (value !== undefined) {
-        merged[key] = value;
-      }
-    }
-
-    return merged;
-  }
-
-  /**
-   * Run `profile create <name> --merge --port <port> [--env K=V ...]`.
-   * Every entry in the merged env that was passed via {@link userEnv} (or
-   * auto-applied by the CPU workaround) is forwarded as `--env`.
-   */
-  private async configureProfile(env: NodeJS.ProcessEnv): Promise<void> {
-    this.logger.info(`[hindsight] configuring profile "${this.profile}"`);
-
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const createArgs = [
-      ...baseArgs,
-      'profile',
-      'create',
-      this.profile,
-      '--merge',
-      '--port',
-      String(this.port),
-    ];
-
-    // Forward every env var that the caller intended for the daemon as --env.
-    // We only forward keys the caller explicitly set (userEnv) plus the CPU
-    // workaround values — not the entire process.env, to avoid leaking random
-    // host state into profile config.
-    const envForProfile = this.collectProfileEnv(env);
-    for (const [key, value] of Object.entries(envForProfile)) {
-      createArgs.push('--env', `${key}=${value}`);
-    }
-
-    createArgs.push(...this.extraProfileCreateArgs);
-
-    await this.runCommand(cmd, createArgs, env, 'profile.create');
-  }
-
-  /** Collect only the env vars that should be written into the profile file. */
-  private collectProfileEnv(env: NodeJS.ProcessEnv): Record<string, string> {
-    const out: Record<string, string> = {};
-
-    // 1. User-supplied env — always forwarded.
-    for (const [key, value] of Object.entries(this.userEnv)) {
-      if (value !== undefined) {
-        out[key] = value;
-      }
-    }
-
-    // 2. CPU workaround — only if auto-applied and not already overridden.
-    if (this.platformCpuWorkaround && process.platform === 'darwin') {
-      const cpuKeys = [
-        'HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU',
-        'HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU',
-      ];
-      for (const key of cpuKeys) {
-        if (!(key in out) && env[key] !== undefined) {
-          out[key] = env[key] as string;
-        }
-      }
-    }
-
-    return out;
-  }
-
-  private async startDaemon(env: NodeJS.ProcessEnv): Promise<void> {
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const args = [
-      ...baseArgs,
-      'daemon',
-      '--profile',
-      this.profile,
-      'start',
-      ...this.extraDaemonStartArgs,
-    ];
-
-    await this.runCommand(cmd, args, env, 'daemon.start');
-  }
-
-  /**
-   * Spawn `cmd` with `args`, pipe its output through the logger, and resolve
-   * once it exits with code 0. Rejects on non-zero exit or spawn error.
-   */
-  private async runCommand(
-    cmd: string,
-    args: string[],
-    env: NodeJS.ProcessEnv,
-    label: string,
-  ): Promise<void> {
-    const child = spawn(cmd, args, { stdio: 'pipe', env });
-    let output = '';
-    child.stdout?.on('data', (data: Buffer) => {
-      const text = data.toString();
-      output += text;
-      for (const line of text.trimEnd().split('\n')) {
-        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
-      }
-    });
-    child.stderr?.on('data', (data: Buffer) => {
-      const text = data.toString();
-      output += text;
-      for (const line of text.trimEnd().split('\n')) {
-        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
-      }
-    });
-
-    await new Promise<void>((resolve, reject) => {
-      child.on('exit', (code) => {
-        if (code === 0) {
-          resolve();
-        } else {
-          reject(new Error(`${label} failed with code ${code}: ${output.trim()}`));
-        }
-      });
-      child.on('error', (err) => {
-        reject(new Error(`${label} failed to spawn: ${err.message}`, { cause: err }));
-      });
-    });
-  }
-
-  /** Stream a spawned child's stdout/stderr through the logger without blocking. */
-  private pipeOutput(child: ReturnType<typeof spawn>, label: string): void {
-    child.stdout?.on('data', (data: Buffer) => {
-      for (const line of data.toString().trimEnd().split('\n')) {
-        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
-      }
-    });
-    child.stderr?.on('data', (data: Buffer) => {
-      for (const line of data.toString().trimEnd().split('\n')) {
-        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
-      }
-    });
-  }
-
-  /** Poll `/health` until it succeeds or `readyTimeoutMs` elapses. */
-  private async waitForReady(): Promise<void> {
-    const deadline = Date.now() + this.readyTimeoutMs;
-    let attempt = 0;
-    while (Date.now() < deadline) {
-      attempt++;
-      try {
-        const res = await fetch(`${this.baseUrl}/health`, {
-          signal: AbortSignal.timeout(this.readyPollIntervalMs),
-        });
-        if (res.ok) {
-          this.logger.debug(`[hindsight] health check passed (attempt ${attempt})`);
-          return;
-        }
-      } catch {
-        // expected while the daemon is still booting
-      }
-      await new Promise((resolve) => setTimeout(resolve, this.readyPollIntervalMs));
-    }
-    throw new Error(
-      `Hindsight daemon did not become ready within ${this.readyTimeoutMs}ms at ${this.baseUrl}`,
-    );
-  }
-}

hindsight-all-npm/src/types.ts

@@ -1,54 +0,0 @@
-import type { Logger } from './logger.js';
-
-/**
- * Options for {@link HindsightServer}.
- *
- * The server is intentionally thin and pass-through: anything configurable
- * on the daemon side (env vars or CLI flags) can be set here without needing
- * a new dedicated option. Use {@link env} for `HINDSIGHT_*` / `OPENAI_API_KEY` /
- * custom provider settings, and the two `extra*` arrays to append raw CLI
- * args to `profile create` or `daemon start`.
- *
- * For talking to the daemon after `start()`, use `@vectorize-io/hindsight-client`
- * against `server.getBaseUrl()`. This package does not ship its own HTTP
- * client.
- */
-export interface HindsightServerOptions {
-  /** Profile name used for `--profile <name>` on every sub-command. Default: `"default"`. */
-  profile?: string;
-  /** TCP port the daemon listens on. Default: `8888`. */
-  port?: number;
-  /** Hostname the daemon binds to (for health checks). Default: `127.0.0.1`. */
-  host?: string;
-  /** Version of the underlying `hindsight-embed` PyPI package to run via `uvx`. Default: `"latest"`. */
-  embedVersion?: string;
-  /** Local path to a `hindsight-embed` checkout — takes precedence over `embedVersion`. */
-  embedPackagePath?: string;
-  /**
-   * Environment variables passed to the daemon process AND written into the
-   * profile via repeated `--env KEY=VALUE` flags. This is the preferred way
-   * to surface any `HINDSIGHT_API_*` / `HINDSIGHT_EMBED_*` setting — adding a
-   * new daemon env var never requires a wrapper update.
-   *
-   * Values of `undefined` are dropped (so you can spread conditionally).
-   */
-  env?: Record<string, string | undefined>;
-  /** Extra args appended verbatim to `hindsight-embed profile create <name> --merge ...`. */
-  extraProfileCreateArgs?: string[];
-  /** Extra args appended verbatim to `hindsight-embed daemon --profile <name> start ...`. */
-  extraDaemonStartArgs?: string[];
-  /**
-   * On macOS, automatically set
-   * `HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1` and
-   * `HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1` to avoid Metal/MPS crashes in
-   * daemon mode. Default: `true` on `darwin`, ignored elsewhere. Any value set
-   * explicitly in {@link env} wins over the auto-applied value.
-   */
-  platformCpuWorkaround?: boolean;
-  /** Max time (ms) to wait for `/health` to return 200. Default: `30_000`. */
-  readyTimeoutMs?: number;
-  /** Polling interval (ms) while waiting for `/health`. Default: `1_000`. */
-  readyPollIntervalMs?: number;
-  /** Optional pluggable logger. Default: silent. */
-  logger?: Logger;
-}

hindsight-all-npm/tsconfig.json

@@ -1,18 +0,0 @@
-{
-  "compilerOptions": {
-    "target": "ES2022",
-    "module": "ES2022",
-    "lib": ["ES2022"],
-    "moduleResolution": "node",
-    "declaration": true,
-    "outDir": "./dist",
-    "rootDir": "./src",
-    "strict": true,
-    "esModuleInterop": true,
-    "skipLibCheck": true,
-    "forceConsistentCasingInFileNames": true,
-    "resolveJsonModule": true
-  },
-  "include": ["src/**/*"],
-  "exclude": ["node_modules", "dist", "src/**/*.test.ts"]
-}

hindsight-all-npm/tsup.config.ts

@@ -1,11 +0,0 @@
-import { defineConfig } from 'tsup';
-
-export default defineConfig({
-  entry: ['src/index.ts'],
-  format: ['esm'],
-  dts: true,
-  outDir: 'dist',
-  clean: true,
-  sourcemap: true,
-  bundle: true,
-});

hindsight-all-npm/vitest.config.ts

@@ -1,8 +0,0 @@
-import { defineConfig } from 'vitest/config';
-
-export default defineConfig({
-  test: {
-    include: ['src/**/*.test.ts'],
-    environment: 'node',
-  },
-});

hindsight-all-slim/pyproject.toml

@@ -1,33 +0,0 @@
-[build-system]
-requires = ["setuptools>=61"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "hindsight-all-slim"
-version = "0.5.3"
-description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
-readme = "README.md"
-requires-python = ">=3.11"
-dependencies = [
-    "hindsight-api-slim>=0.4.17",
-    "hindsight-client>=0.0.7",
-    "hindsight-embed>=0.1.0",
-]
-
-[tool.uv.sources]
-hindsight-api-slim = { workspace = true }
-hindsight-client = { workspace = true }
-hindsight-embed = { workspace = true }
-
-[project.optional-dependencies]
-test = [
-    "pytest>=7.0.0",
-    "pytest-asyncio>=0.21.0",
-]
-
-[tool.setuptools]
-packages = []
-
-[tool.pytest.ini_options]
-asyncio_mode = "auto"
-asyncio_default_fixture_loop_scope = "function"

hindsight-all/README.md

@@ -1,48 +0,0 @@
-# hindsight-all
-
-All-in-one package for Hindsight - Agent Memory That Works Like Human Memory
-
-## Quick Start
-
-```python
-from hindsight import start_server, HindsightClient
-
-# Start server with embedded PostgreSQL
-server = start_server(
-    llm_provider="groq",
-    llm_api_key="your-api-key",
-    llm_model="openai/gpt-oss-120b"
-)
-
-# Create client
-client = HindsightClient(base_url=server.url)
-
-# Store memories
-client.put(agent_id="assistant", content="User prefers Python for data analysis")
-
-# Search memories
-results = client.search(agent_id="assistant", query="programming preferences")
-
-# Generate contextual response
-response = client.think(agent_id="assistant", query="What languages should I recommend?")
-
-# Stop server when done
-server.stop()
-```
-
-## Using Context Manager
-
-```python
-from hindsight import HindsightServer, HindsightClient
-
-with HindsightServer(llm_provider="groq", llm_api_key="...") as server:
-    client = HindsightClient(base_url=server.url)
-    # ... use client ...
-# Server automatically stops
-```
-
-## Installation
-
-```bash
-pip install hindsight-all
-```

hindsight-all/hindsight/client_wrapper.py

@@ -1,423 +0,0 @@
-"""
-Wrapper for Hindsight client that adds API namespaces.
-
-Provides organized access to different parts of the Hindsight API through
-namespaces like .banks, .mental_models, etc.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-
-from hindsight_client import Hindsight
-
-
-class BanksAPI:
-    """Namespace for bank-related operations.
-    
-    Provides methods to create, delete, and manage memory banks.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str | None = None,
-        mission: str | None = None,
-        disposition: dict[str, Any] | None = None,
-    ) -> Any:
-        """Create a new bank.
-        
-        Args:
-            bank_id: Unique identifier for the bank.
-            name: Optional display name for the bank.
-            mission: Optional mission statement for the bank.
-            disposition: Optional disposition configuration dict.
-            
-        Returns:
-            Bank creation response from the API.
-        """
-        return self._client.create_bank(
-            bank_id=bank_id,
-            name=name,
-            mission=mission,
-            disposition=disposition,
-        )
-
-    def delete(self, bank_id: str) -> Any:
-        """Delete a bank.
-        
-        Args:
-            bank_id: The ID of the bank to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_bank(bank_id=bank_id)
-
-    def set_mission(self, bank_id: str, mission: str) -> Any:
-        """Set or update the mission for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mission: The mission statement to set.
-            
-        Returns:
-            API response confirming the update.
-        """
-        return self._client.set_mission(bank_id=bank_id, mission=mission)
-
-    def set_disposition(self, bank_id: str, disposition: dict[str, Any]) -> Any:
-        """Set or update the disposition for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            disposition: The disposition configuration dict.
-            
-        Returns:
-            API response confirming the update.
-        """
-        return self._client.set_disposition(bank_id=bank_id, disposition=disposition)
-
-    def list(self) -> Any:
-        """List all banks.
-        
-        Returns:
-            List of banks from the API.
-        """
-        from hindsight_client.hindsight_client import _run_async
-
-        return _run_async(self._client._banks_api.list_banks())
-
-
-class MentalModelsAPI:
-    """Namespace for mental model operations.
-    
-    Mental models are reusable knowledge structures that guide agent behavior.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str,
-        content: str,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Create a new mental model.
-        
-        Args:
-            bank_id: The ID of the bank to add the model to.
-            name: Name for the mental model.
-            content: The content/instructions for the mental model.
-            tags: Optional list of tags for categorization.
-            
-        Returns:
-            Creation response from the API.
-        """
-        return self._client.create_mental_model(
-            bank_id=bank_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
-        """List all mental models for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            tags: Optional filter by tags.
-            
-        Returns:
-            List of mental models.
-        """
-        return self._client.list_mental_models(bank_id=bank_id, tags=tags)
-
-    def get(self, bank_id: str, mental_model_id: str) -> Any:
-        """Get a specific mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model.
-            
-        Returns:
-            The mental model details.
-        """
-        return self._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-    def refresh(self, bank_id: str, mental_model_id: str) -> Any:
-        """Refresh a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to refresh.
-            
-        Returns:
-            Refresh response from the API.
-        """
-        return self._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-    def update(
-        self,
-        bank_id: str,
-        mental_model_id: str,
-        name: str | None = None,
-        content: str | None = None,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Update a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to update.
-            name: Optional new name.
-            content: Optional new content.
-            tags: Optional new tags list.
-            
-        Returns:
-            Update response from the API.
-        """
-        return self._client.update_mental_model(
-            bank_id=bank_id,
-            mental_model_id=mental_model_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def delete(self, bank_id: str, mental_model_id: str) -> Any:
-        """Delete a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-
-class DirectivesAPI:
-    """Namespace for directive operations.
-    
-    Directives are explicit instructions that guide agent behavior.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str,
-        content: str,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Create a new directive.
-        
-        Args:
-            bank_id: The ID of the bank to add the directive to.
-            name: Name for the directive.
-            content: The directive content/instructions.
-            tags: Optional list of tags for categorization.
-            
-        Returns:
-            Creation response from the API.
-        """
-        return self._client.create_directive(
-            bank_id=bank_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
-        """List all directives for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            tags: Optional filter by tags.
-            
-        Returns:
-            List of directives.
-        """
-        return self._client.list_directives(bank_id=bank_id, tags=tags)
-
-    def get(self, bank_id: str, directive_id: str) -> Any:
-        """Get a specific directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive.
-            
-        Returns:
-            The directive details.
-        """
-        return self._client.get_directive(bank_id=bank_id, directive_id=directive_id)
-
-    def update(
-        self,
-        bank_id: str,
-        directive_id: str,
-        name: str | None = None,
-        content: str | None = None,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Update a directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive to update.
-            name: Optional new name.
-            content: Optional new content.
-            tags: Optional new tags list.
-            
-        Returns:
-            Update response from the API.
-        """
-        return self._client.update_directive(
-            bank_id=bank_id,
-            directive_id=directive_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def delete(self, bank_id: str, directive_id: str) -> Any:
-        """Delete a directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
-
-
-class MemoriesAPI:
-    """Namespace for memory operations.
-    
-    Provides methods to query and retrieve stored memories.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def list(
-        self,
-        bank_id: str,
-        type: str | None = None,
-        search_query: str | None = None,
-        limit: int = 100,
-        offset: int = 0,
-    ) -> Any:
-        """List memories in a bank.
-        
-        Args:
-            bank_id: The ID of the bank to query.
-            type: Optional filter by memory type.
-            search_query: Optional search query for filtering.
-            limit: Maximum number of results to return (default: 100).
-            offset: Number of results to skip for pagination (default: 0).
-            
-        Returns:
-            List of memories matching the criteria.
-        """
-        return self._client.list_memories(
-            bank_id=bank_id,
-            type=type,
-            search_query=search_query,
-            limit=limit,
-            offset=offset,
-        )
-
-
-class HindsightClient(Hindsight):
-    """
-    Enhanced Hindsight client with organized API namespaces.
-
-    This wrapper extends the auto-generated Hindsight client with organized
-    access to different parts of the API through namespaces.
-
-    Example:
-        ```python
-        from hindsight import HindsightClient
-
-        client = HindsightClient(base_url="http://localhost:8888")
-
-        # Core operations (inherited from Hindsight)
-        client.retain(bank_id="test", content="Hello")
-        results = client.recall(bank_id="test", query="Hello")
-
-        # Organized API access through namespaces
-        client.banks.create(bank_id="test", name="Test Bank")
-        models = client.mental_models.list(bank_id="test")
-        directives = client.directives.list(bank_id="test")
-        memories = client.memories.list(bank_id="test")
-        ```
-        
-    Attributes:
-        banks: Namespace for bank management operations.
-        mental_models: Namespace for mental model operations.
-        directives: Namespace for directive operations.
-        memories: Namespace for memory listing operations.
-    """
-
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        super().__init__(*args, **kwargs)
-        self._banks_namespace: BanksAPI | None = None
-        self._mental_models_namespace: MentalModelsAPI | None = None
-        self._directives_namespace: DirectivesAPI | None = None
-        self._memories_namespace: MemoriesAPI | None = None
-
-    @property
-    def banks(self) -> BanksAPI:
-        """Access bank management operations.
-        
-        Returns:
-            BanksAPI instance for bank operations.
-        """
-        if self._banks_namespace is None:
-            self._banks_namespace = BanksAPI(self)
-        return self._banks_namespace
-
-    @property
-    def mental_models(self) -> MentalModelsAPI:
-        """Access mental model operations.
-        
-        Returns:
-            MentalModelsAPI instance for mental model operations.
-        """
-        if self._mental_models_namespace is None:
-            self._mental_models_namespace = MentalModelsAPI(self)
-        return self._mental_models_namespace
-
-    @property
-    def directives(self) -> DirectivesAPI:
-        """Access directive operations.
-        
-        Returns:
-            DirectivesAPI instance for directive operations.
-        """
-        if self._directives_namespace is None:
-            self._directives_namespace = DirectivesAPI(self)
-        return self._directives_namespace
-
-    @property
-    def memories(self) -> MemoriesAPI:
-        """Access memory listing operations.
-        
-        Returns:
-            MemoriesAPI instance for memory operations.
-        """
-        if self._memories_namespace is None:
-            self._memories_namespace = MemoriesAPI(self)
-        return self._memories_namespace

hindsight-all/tests/test_cleanup_timeout.py

@@ -1,56 +0,0 @@
-"""
-Unit test for _cleanup lock timeout behavior.
-
-Verifies that _cleanup completes even when the lock is held by another thread,
-instead of hanging indefinitely (fixes #952).
-"""
-
-import threading
-import time
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-
-def test_cleanup_completes_when_lock_held():
-    """
-    _cleanup should complete (best-effort) even when self._lock is held
-    by another thread, e.g. during a long _ensure_started call.
-    """
-    with patch.dict("sys.modules", {
-        "hindsight_client": MagicMock(),
-        "hindsight_embed": MagicMock(),
-        "hindsight.api_namespaces": MagicMock(),
-    }):
-        from hindsight.embedded import HindsightEmbedded
-
-        client = HindsightEmbedded.__new__(HindsightEmbedded)
-        client.profile = "test"
-        client._lock = threading.Lock()
-        client._closed = False
-        client._client = None
-        client._started = False
-        client._ui = False
-
-        # Simulate another thread holding the lock
-        client._lock.acquire()
-
-        cleanup_done = threading.Event()
-
-        def run_cleanup():
-            client._cleanup()
-            cleanup_done.set()
-
-        t = threading.Thread(target=run_cleanup)
-        t.start()
-
-        # Cleanup should complete within the timeout (5s) + margin
-        assert cleanup_done.wait(timeout=8.0), (
-            "_cleanup hung instead of timing out on lock acquisition"
-        )
-
-        # Release the lock from the simulating thread
-        client._lock.release()
-        t.join(timeout=1.0)
-
-        assert client._closed, "Client should be marked as closed after cleanup"

hindsight-api-slim/README.md

@@ -1,137 +0,0 @@
-# Hindsight API
-
-**Memory System for AI Agents** — Temporal + Semantic + Entity Memory Architecture using PostgreSQL with pgvector.
-
-Hindsight gives AI agents persistent memory that works like human memory: it stores facts, tracks entities and relationships, handles temporal reasoning ("what happened last spring?"), and forms opinions based on configurable disposition traits.
-
-## Installation
-
-```bash
-pip install hindsight-api
-```
-
-## Quick Start
-
-### Run the Server
-
-```bash
-# Set your LLM provider
-export HINDSIGHT_API_LLM_PROVIDER=openai
-export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
-
-# Start the server (uses embedded PostgreSQL by default)
-hindsight-api
-```
-
-The server starts at http://localhost:8888 with:
-- REST API for memory operations
-- MCP server at `/mcp` for tool-use integration
-
-### Use the Python API
-
-```python
-from hindsight_api import MemoryEngine
-
-# Create and initialize the memory engine
-memory = MemoryEngine()
-await memory.initialize()
-
-# Create a memory bank for your agent
-bank = await memory.create_memory_bank(
-    name="my-assistant",
-    background="A helpful coding assistant"
-)
-
-# Store a memory
-await memory.retain(
-    memory_bank_id=bank.id,
-    content="The user prefers Python for data science projects"
-)
-
-# Recall memories
-results = await memory.recall(
-    memory_bank_id=bank.id,
-    query="What programming language does the user prefer?"
-)
-
-# Reflect with reasoning
-response = await memory.reflect(
-    memory_bank_id=bank.id,
-    query="Should I recommend Python or R for this ML project?"
-)
-```
-
-## CLI Options
-
-```bash
-hindsight-api --help
-
-# Common options
-hindsight-api --port 9000          # Custom port (default: 8888)
-hindsight-api --host 127.0.0.1     # Bind to localhost only
-hindsight-api --workers 4          # Multiple worker processes
-hindsight-api --log-level debug    # Verbose logging
-```
-
-## Configuration
-
-Configure via environment variables:
-
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `HINDSIGHT_API_DATABASE_URL` | PostgreSQL connection string | `pg0` (embedded) |
-| `HINDSIGHT_API_LLM_PROVIDER` | `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio` | `openai` |
-| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
-| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-4o-mini` |
-| `HINDSIGHT_API_HOST` | Server bind address | `0.0.0.0` |
-| `HINDSIGHT_API_PORT` | Server port | `8888` |
-
-### Example with External PostgreSQL
-
-```bash
-export HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@localhost:5432/hindsight
-export HINDSIGHT_API_LLM_PROVIDER=groq
-export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx
-
-hindsight-api
-```
-
-## Docker
-
-```bash
-docker run --rm -it -p 8888:8888 \
-  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
-  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
-  ghcr.io/vectorize-io/hindsight:latest
-```
-
-## MCP Server
-
-For local MCP integration without running the full API server:
-
-```bash
-hindsight-local-mcp
-```
-
-This runs a stdio-based MCP server that can be used directly with MCP-compatible clients.
-
-## Key Features
-
-- **Multi-Strategy Retrieval (TEMPR)** — Semantic, keyword, graph, and temporal search combined with RRF fusion
-- **Entity Graph** — Automatic entity extraction and relationship tracking
-- **Temporal Reasoning** — Native support for time-based queries
-- **Disposition Traits** — Configurable skepticism, literalism, and empathy influence opinion formation
-- **Three Memory Types** — World facts, bank actions, and formed opinions with confidence scores
-
-## Documentation
-
-Full documentation: [https://hindsight.vectorize.io](https://hindsight.vectorize.io)
-
-- [Installation Guide](https://hindsight.vectorize.io/developer/installation)
-- [Configuration Reference](https://hindsight.vectorize.io/developer/configuration)
-- [API Reference](https://hindsight.vectorize.io/api-reference)
-- [Python SDK](https://hindsight.vectorize.io/sdks/python)
-
-## License
-
-Apache 2.0

hindsight-api-slim/hindsight_api/alembic/versions/2eee35aa3cfc_case_insensitive_entities_trgm_index.py

@@ -1,45 +0,0 @@
-"""Recreate entities trigram index on LOWER(canonical_name) for case-insensitive matching
-
-The previous GIN trigram index on canonical_name was case-sensitive, causing
-"Alice" and "alice" to have different trigram sets. This recreates it on
-LOWER(canonical_name) so the % operator matches case-insensitively.
-
-Revision ID: 2eee35aa3cfc
-Revises: d6e7f8a9b0c1
-Create Date: 2026-03-31
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "2eee35aa3cfc"
-down_revision: str | Sequence[str] | None = "d6e7f8a9b0c1"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Drop the old case-sensitive trigram index
-    op.execute("DROP INDEX IF EXISTS entities_canonical_name_trgm_idx")
-    # Create case-insensitive trigram index on LOWER(canonical_name)
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS entities_canonical_name_lower_trgm_idx "
-        f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops)"
-    )
-
-
-def downgrade() -> None:
-    op.execute("DROP INDEX IF EXISTS entities_canonical_name_lower_trgm_idx")
-    schema = _get_schema_prefix()
-    # Restore original case-sensitive index
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS entities_canonical_name_trgm_idx "
-        f"ON {schema}entities USING GIN (canonical_name gin_trgm_ops)"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/a1b2c3d4e5f6_add_file_storage_table.py

@@ -1,70 +0,0 @@
-"""Add file_storage table for BYTEA-based file storage
-
-Revision ID: a1b2c3d4e5f6
-Revises: y0t1u2v3w4x5
-Create Date: 2026-02-16
-
-Creates a dedicated table for storing uploaded files using BYTEA.
-This provides zero-config file storage that "just works" for development
-and small deployments. For production/scale, use S3-compatible storage.
-
-Files are stored in a separate table to avoid bloating the documents table.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a1b2c3d4e5f6"
-down_revision: str | Sequence[str] | None = "y0t1u2v3w4x5"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    """Create file_storage table for BYTEA storage."""
-    schema = _get_schema_prefix()
-
-    # Create file_storage table (minimal: just key + data)
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}file_storage (
-            storage_key TEXT PRIMARY KEY,
-            data BYTEA NOT NULL
-        )
-    """
-    )
-
-    # Add file tracking columns to documents table
-    op.execute(
-        f"""
-        ALTER TABLE {schema}documents
-        ADD COLUMN IF NOT EXISTS file_storage_key TEXT,
-        ADD COLUMN IF NOT EXISTS file_original_name TEXT,
-        ADD COLUMN IF NOT EXISTS file_content_type TEXT
-    """
-    )
-
-
-def downgrade() -> None:
-    """Remove file_storage table and related columns."""
-    schema = _get_schema_prefix()
-
-    # Drop columns from documents table
-    op.execute(
-        f"""
-        ALTER TABLE {schema}documents
-        DROP COLUMN IF EXISTS file_storage_key,
-        DROP COLUMN IF EXISTS file_original_name,
-        DROP COLUMN IF EXISTS file_content_type
-    """
-    )
-
-    # Drop file_storage table
-    op.execute(f"DROP TABLE IF EXISTS {schema}file_storage")

hindsight-api-slim/hindsight_api/alembic/versions/a2b3c4d5e6f7_add_text_signals_column.py

@@ -1,88 +0,0 @@
-"""Add text_signals column to memory_units for enriched BM25 indexing.
-
-text_signals stores a denormalized space-separated string of entity names
-(and future signals) to improve full-text search recall without polluting
-the stored fact text.
-
-- vchord: text_signals included in tokenize() at insert time
-- native: search_vector GENERATED column regenerated to include text_signals
-- pg_textsearch: no change (index only supports a single base column)
-
-Revision ID: a2b3c4d5e6f7
-Revises: z1u2v3w4x5y6
-Create Date: 2026-02-28
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a2b3c4d5e6f7"
-down_revision: str | Sequence[str] | None = "aa2b3c4d5e6f"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _detect_text_search_extension() -> str:
-    return os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    table = f"{schema}memory_units"
-    text_search_ext = _detect_text_search_extension()
-
-    # Add text_signals column (nullable TEXT, populated at retain time)
-    op.execute(f"ALTER TABLE {table} ADD COLUMN IF NOT EXISTS text_signals TEXT")
-
-    if text_search_ext == "native":
-        # Native PostgreSQL: drop and recreate the GENERATED tsvector column to include text_signals
-        op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS search_vector")
-        op.execute(f"""
-            ALTER TABLE {table}
-            ADD COLUMN search_vector tsvector
-            GENERATED ALWAYS AS (
-                to_tsvector('english',
-                    COALESCE(text, '') || ' ' ||
-                    COALESCE(context, '') || ' ' ||
-                    COALESCE(text_signals, '')
-                )
-            ) STORED
-        """)
-        # Recreate GIN index (was dropped with the column)
-        op.execute(f"""
-            CREATE INDEX IF NOT EXISTS idx_memory_units_text_search
-            ON {table} USING gin(search_vector)
-        """)
-
-    # vchord: tokenize() call in fact_storage.py is updated to include text_signals at insert time
-    # pg_textsearch: no change — index operates on the base `text` column only
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    table = f"{schema}memory_units"
-    text_search_ext = _detect_text_search_extension()
-
-    if text_search_ext == "native":
-        op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_text_search")
-        op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS search_vector")
-        op.execute(f"""
-            ALTER TABLE {table}
-            ADD COLUMN search_vector tsvector
-            GENERATED ALWAYS AS (
-                to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))
-            ) STORED
-        """)
-        op.execute(f"""
-            CREATE INDEX idx_memory_units_text_search
-            ON {table} USING gin(search_vector)
-        """)
-
-    op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS text_signals")

hindsight-api-slim/hindsight_api/alembic/versions/a2b3c4d5e6f8_add_gin_index_source_memory_ids.py

@@ -1,54 +0,0 @@
-"""Add GIN index on source_memory_ids for observation lookup performance
-
-Without this index, queries using the array overlap operator (&&) or array
-containment (@>) on source_memory_ids require a full sequential scan over all
-observation memory_units. At ~77k observations this was measured at 45ms per
-query, becoming a bottleneck during consolidation recall (57-64s timeouts) and
-user recall (18-27s average).
-
-The GIN index reduces these queries to index scans: 45ms → 0.049ms (927x
-speedup). Recall dropped from 18-27s to ~6s, and consolidation recall
-stabilised from timeout to ~15s.
-
-Created with CONCURRENTLY so the migration does not block reads or writes.
-CONCURRENTLY requires running outside a transaction block, so the migration
-emits an explicit COMMIT before the statement and uses IF NOT EXISTS for
-idempotency.
-
-Revision ID: a2b3c4d5e6f8
-Revises: f7g8h9i0j1k2
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a2b3c4d5e6f8"
-down_revision: str | Sequence[str] | None = "f7g8h9i0j1k2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
-    # Commit the current Alembic transaction first.
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")

hindsight-api-slim/hindsight_api/alembic/versions/a2v3w4x5y6z7_add_last_refreshed_source_query.py

@@ -1,38 +0,0 @@
-"""Add last_refreshed_source_query column to mental_models
-
-Revision ID: a2v3w4x5y6z7
-Revises: z1u2v3w4x5y6
-Create Date: 2026-04-15
-
-Tracks the source_query that was used during the most recent refresh.
-Used by delta-mode refresh to detect when the query has changed: if it has,
-delta mode falls back to a full regeneration because the surgical-edit
-assumption (same topic, new facts) no longer holds.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a2v3w4x5y6z7"
-down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"""
-        ALTER TABLE {schema}mental_models
-        ADD COLUMN IF NOT EXISTS last_refreshed_source_query TEXT
-    """)
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS last_refreshed_source_query")

hindsight-api-slim/hindsight_api/alembic/versions/a3b4c5d6e7f8_add_consolidation_failed_at_to_memory_units.py

@@ -1,52 +0,0 @@
-"""Add consolidation_failed_at column to memory_units for tracking persistent LLM failures.
-
-When all LLM retries are exhausted on a single-memory batch, the memory is marked
-with consolidation_failed_at instead of consolidated_at, so it is not silently lost
-and can be retried later via the API.
-
-Revision ID: a3b4c5d6e7f8
-Revises: g7h8i9j0k1l2
-Create Date: 2026-03-17
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a3b4c5d6e7f8"
-down_revision: str | Sequence[str] | None = "g7h8i9j0k1l2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        ALTER TABLE {schema}memory_units
-        ADD COLUMN IF NOT EXISTS consolidation_failed_at TIMESTAMPTZ DEFAULT NULL
-        """
-    )
-
-    # Index to efficiently query memories that failed consolidation for a given bank
-    op.execute(
-        f"""
-        CREATE INDEX IF NOT EXISTS idx_memory_units_consolidation_failed
-        ON {schema}memory_units (bank_id, consolidation_failed_at)
-        WHERE consolidation_failed_at IS NOT NULL AND fact_type IN ('experience', 'world')
-        """
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_consolidation_failed")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidation_failed_at")

hindsight-api-slim/hindsight_api/alembic/versions/a4b5c6d7e8f9_fix_per_bank_vector_index_type.py

@@ -1,142 +0,0 @@
-"""Fix per-bank vector indexes to match configured extension
-
-Revision ID: a4b5c6d7e8f9
-Revises: 2eee35aa3cfc
-Create Date: 2026-04-01
-
-Migration d5e6f7a8b9c0 hardcoded HNSW when creating per-bank partial vector
-indexes, ignoring HINDSIGHT_API_VECTOR_EXTENSION. Banks that existed when that
-migration ran got HNSW indexes even when pgvectorscale (DiskANN) or vchord
-was configured.
-
-This migration detects the mismatch and recreates the affected indexes with
-the correct type. Skipped entirely when the configured extension is pgvector
-(the default), since those indexes are already correct.
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import text
-
-revision: str = "a4b5c6d7e8f9"
-down_revision: str | Sequence[str] | None = "2eee35aa3cfc"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_FACT_TYPES: dict[str, str] = {
-    "world": "worl",
-    "experience": "expr",
-    "observation": "obsv",
-}
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _target_index_type() -> str | None:
-    """Return the target index type, or None if pgvector (no fix needed)."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "diskann"
-    elif ext == "vchord":
-        return "vchordrq"
-    return None
-
-
-def _vector_index_using_clause() -> str:
-    """Return the USING clause based on the configured vector extension."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
-    elif ext == "vchord":
-        return "USING vchordrq (embedding vector_l2_ops)"
-    else:
-        return "USING hnsw (embedding vector_cosine_ops)"
-
-
-def upgrade() -> None:
-    target = _target_index_type()
-    if target is None:
-        # pgvector — indexes are already HNSW, nothing to fix
-        return
-
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    schema = _get_schema_prefix()
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-    using_clause = _vector_index_using_clause()
-    pg_schema = schema_name or "public"
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-
-            # Check if this index exists and what type it is
-            idx_info = bind.execute(
-                text("SELECT indexdef FROM pg_indexes WHERE schemaname = :schema AND indexname = :idx"),
-                {"schema": pg_schema, "idx": idx_name},
-            ).fetchone()
-
-            if idx_info is None:
-                # Index doesn't exist — create it with the correct type
-                bind.execute(
-                    text(
-                        f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                        f"ON {table_ref} {using_clause} "
-                        f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                    )
-                )
-                continue
-
-            indexdef = idx_info[0].lower()
-            if target in indexdef:
-                # Already the correct type
-                continue
-
-            # Wrong type — drop and recreate
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} {using_clause} "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )
-
-
-def downgrade() -> None:
-    # Downgrade recreates indexes as HNSW (the original hardcoded behavior)
-    target = _target_index_type()
-    if target is None:
-        return
-
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    schema = _get_schema_prefix()
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )

hindsight-api-slim/hindsight_api/alembic/versions/aa2b3c4d5e6f_nullable_event_date.py

@@ -1,36 +0,0 @@
-"""Make event_date nullable in memory_units to support timestamp-free content
-
-Revision ID: aa2b3c4d5e6f
-Revises: z1u2v3w4x5y6
-Create Date: 2026-03-02
-
-When callers retain content without a timestamp (e.g. fictional documents, static text),
-the event_date column should be allowed to be NULL rather than defaulting to utcnow().
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "aa2b3c4d5e6f"
-down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date DROP NOT NULL")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    # Backfill NULLs with now() before restoring the NOT NULL constraint
-    op.execute(f"UPDATE {schema}memory_units SET event_date = now() WHERE event_date IS NULL")
-    op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date SET NOT NULL")

hindsight-api-slim/hindsight_api/alembic/versions/b3c4d5e6f7a8_add_content_hash_to_chunks.py

@@ -1,32 +0,0 @@
-"""add content_hash to chunks table for delta retain
-
-Revision ID: b3c4d5e6f7a8
-Revises: a3b4c5d6e7f8
-Create Date: 2026-03-25
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b3c4d5e6f7a8"
-down_revision: str | Sequence[str] | None = "a3b4c5d6e7f8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Add content_hash column to chunks table for delta comparison
-    op.execute(f"ALTER TABLE {schema}chunks ADD COLUMN IF NOT EXISTS content_hash TEXT")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}chunks DROP COLUMN IF EXISTS content_hash")

hindsight-api-slim/hindsight_api/alembic/versions/b3c4d5e6f7g8_add_temporal_date_indexes.py

@@ -1,68 +0,0 @@
-"""Add partial indexes on memory_units temporal date fields for fast temporal retrieval
-
-Revision ID: b3c4d5e6f7g8
-Revises: c1a2b3d4e5f6
-Create Date: 2026-03-02
-
-The temporal retrieval entry-point query filters memory_units by occurred_start,
-occurred_end, and mentioned_at using OR conditions. Without dedicated indexes the
-planner falls back to a sequential scan of all bank rows after applying the
-(bank_id, fact_type) index, then re-checks each date field.
-
-These three partial indexes give the planner bitmap-index scan options for the
-three most common date predicates, dramatically reducing the row set before any
-embedding computation is required.
-
-All indexes are created CONCURRENTLY so the migration does not block writes on
-memory_units during production deployments. CONCURRENTLY requires running outside
-a transaction block; see migrations.py for how this is handled safely.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b3c4d5e6f7g8"
-down_revision: str | Sequence[str] | None = "c1a2b3d4e5f6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Partial index on occurred_start (covers "occurred_start BETWEEN $4 AND $5")
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_bank_occurred_start "
-        f"ON {schema}memory_units(bank_id, fact_type, occurred_start) "
-        f"WHERE occurred_start IS NOT NULL"
-    )
-    # Partial index on occurred_end (covers "occurred_end BETWEEN $4 AND $5")
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_bank_occurred_end "
-        f"ON {schema}memory_units(bank_id, fact_type, occurred_end) "
-        f"WHERE occurred_end IS NOT NULL"
-    )
-    # Partial index on mentioned_at (covers "mentioned_at BETWEEN $4 AND $5")
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_bank_mentioned_at "
-        f"ON {schema}memory_units(bank_id, fact_type, mentioned_at) "
-        f"WHERE mentioned_at IS NOT NULL"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_mentioned_at")
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_end")
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_start")

hindsight-api-slim/hindsight_api/alembic/versions/b3w4x5y6z7a8_add_structured_content_to_mental_models.py

@@ -1,44 +0,0 @@
-"""Add structured_content JSONB column to mental_models
-
-Revision ID: b3w4x5y6z7a8
-Revises: a2v3w4x5y6z7
-Create Date: 2026-04-16
-
-Stores the structured representation of a mental model document (sections,
-blocks). The plain ``content`` column remains the rendered markdown shown to
-users. ``structured_content`` is the source of truth for delta-mode refreshes:
-each refresh applies a list of typed operations to the structured doc, then
-re-renders to markdown — so unchanged sections come through byte-identical
-without an LLM round-trip.
-
-Nullable: existing markdown-only mental models continue to work in full mode;
-the column is populated lazily the first time a model is refreshed in delta
-mode.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b3w4x5y6z7a8"
-down_revision: str | Sequence[str] | None = "a2v3w4x5y6z7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"""
-        ALTER TABLE {schema}mental_models
-        ADD COLUMN IF NOT EXISTS structured_content JSONB
-    """)
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS structured_content")

hindsight-api-slim/hindsight_api/alembic/versions/b4c5d6e7f8a9_backfill_observation_scopes.py

@@ -1,34 +0,0 @@
-"""Backfill observation_scopes column if missing.
-
-This migration ensures observation_scopes exists even on databases that had
-revision z1u2v3w4x5y6 applied when it referred to the old text_signals migration
-(before it was renamed to a2b3c4d5e6f7). The ADD COLUMN IF NOT EXISTS makes this
-a no-op on databases that already have the column.
-
-Revision ID: b4c5d6e7f8a9
-Revises: a2b3c4d5e6f7
-Create Date: 2026-03-02
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b4c5d6e7f8a9"
-down_revision: str | Sequence[str] | None = "a2b3c4d5e6f7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS observation_scopes JSONB")
-
-
-def downgrade() -> None:
-    pass  # intentionally no-op — safe to leave the column in place

hindsight-api-slim/hindsight_api/alembic/versions/c1a2b3d4e5f6_enable_pg_trgm_and_entities_trgm_index.py

@@ -1,59 +0,0 @@
-"""Enable pg_trgm extension and add GIN trigram index on entities.canonical_name
-
-Revision ID: c1a2b3d4e5f6
-Revises: b4c5d6e7f8a9
-Create Date: 2026-03-02
-
-Index is created CONCURRENTLY so the migration does not block writes on entities
-during production deployments. CONCURRENTLY requires running outside a transaction
-block; see migrations.py for how this is handled safely.
-"""
-
-from collections.abc import Sequence
-
-import sqlalchemy as sa
-from alembic import context, op
-
-revision: str = "c1a2b3d4e5f6"
-down_revision: str | Sequence[str] | None = "b4c5d6e7f8a9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    # pg_trgm ships with most PostgreSQL installations as a contrib module.
-    # It enables fast similarity lookups via GIN indexes, used for entity name matching.
-    # On managed services (e.g. Azure Flexible Server), the extension may not be
-    # available or may require manual enablement.  We gracefully skip the index
-    # creation if the extension cannot be loaded — the entity resolver will
-    # auto-detect and fall back to the "full" lookup strategy at runtime.  See #626.
-    conn = op.get_bind()
-    try:
-        conn.execute(sa.text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
-    except Exception:
-        # Extension not available (managed Postgres, insufficient privileges, etc.)
-        # Roll back the failed statement and skip index creation.
-        conn.execute(sa.text("ROLLBACK"))
-        conn.execute(sa.text("BEGIN"))
-        return
-
-    schema = _get_schema_prefix()
-    # GIN index on canonical_name enables sub-millisecond trigram similarity queries
-    # (% operator, similarity()) instead of full-table scans across all bank entities.
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS entities_canonical_name_trgm_idx "
-        f"ON {schema}entities USING GIN (canonical_name gin_trgm_ops)"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}entities_canonical_name_trgm_idx")
-    # Note: not dropping pg_trgm extension as other indexes may depend on it

hindsight-api-slim/hindsight_api/alembic/versions/c2d3e4f5g6h7_add_audit_log_table.py

@@ -1,61 +0,0 @@
-"""Add audit_log table for feature usage tracking.
-
-Merge migration that combines the two existing heads (a3b4c5d6e7f8 + c8e5f2a3b4d1).
-
-Stores raw request/response as JSONB for expandability without future migrations.
-The metadata JSONB column allows adding arbitrary fields in the future.
-
-Revision ID: c2d3e4f5g6h7
-Revises: a3b4c5d6e7f8, c8e5f2a3b4d1
-Create Date: 2026-03-26
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c2d3e4f5g6h7"
-down_revision: str | Sequence[str] | None = ("a3b4c5d6e7f8", "c8e5f2a3b4d1")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}audit_log (
-            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-            action TEXT NOT NULL,
-            transport TEXT NOT NULL,
-            bank_id TEXT,
-            started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-            ended_at TIMESTAMPTZ,
-            request JSONB,
-            response JSONB,
-            metadata JSONB DEFAULT '{{}}'::jsonb
-        )
-        """
-    )
-
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_audit_log_action_started ON {schema}audit_log (action, started_at DESC)"
-    )
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_bank_started ON {schema}audit_log (bank_id, started_at DESC)")
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_started ON {schema}audit_log (started_at DESC)")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_bank_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_action_started")
-    op.execute(f"DROP TABLE IF EXISTS {schema}audit_log")

hindsight-api-slim/hindsight_api/alembic/versions/c3d4e5f6g7h8_add_history_to_mental_models.py

@@ -1,30 +0,0 @@
-"""Add history column to mental_models
-
-Revision ID: c3d4e5f6g7h8
-Revises: a2b3c4d5e6f7, a2b3c4d5e6f8
-Create Date: 2026-03-06
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c3d4e5f6g7h8"
-down_revision: str | Sequence[str] | None = ("a2b3c4d5e6f7", "a2b3c4d5e6f8")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")

hindsight-api-slim/hindsight_api/alembic/versions/c4x5y6z7a8b9_backsweep_orphan_observations_v2.py

@@ -1,66 +0,0 @@
-"""backsweep_orphan_observations_v2
-
-Re-run of Pass 2 from migration ``g7h8i9j0k1l2_backsweep_orphan_observations``
-to sweep observations that became orphaned between then and now.
-
-Why we need it again:
-  ``fact_storage.handle_document_tracking`` (the retain/upsert path) deleted
-  the existing document via the FK cascade — which removes the source
-  ``memory_units`` — but never invalidated the observations derived from
-  them.  Only the explicit ``MemoryEngine.delete_document`` API called
-  ``_delete_stale_observations_for_memories``.  Every document re-ingest
-  therefore left orphan observations whose ``source_memory_ids`` arrays
-  pointed at IDs that no longer existed in ``memory_units``.
-
-  ``handle_document_tracking`` now calls the same cleanup helper before the
-  cascade, so no new orphans will accumulate going forward.  This migration
-  cleans up the historical residue.
-
-Identical to Pass 2 of g7h8i9j0k1l2.  Pass 1 (memory_units whose bank is
-gone) is intentionally not re-run; that scenario has no fresh source.
-
-Revision ID: c4x5y6z7a8b9
-Revises: b3w4x5y6z7a8
-Create Date: 2026-04-16
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c4x5y6z7a8b9"
-down_revision: str | Sequence[str] | None = "b3w4x5y6z7a8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    mu = f"{schema}memory_units"
-
-    # Delete observations whose every source_memory_id refers to a now-deleted
-    # memory_unit (or the array is empty).  Observations with at least one
-    # surviving source are left alone — the consolidation engine will refresh
-    # their text on the next pass.
-    op.execute(
-        f"""
-        DELETE FROM {mu} orphan
-        WHERE orphan.fact_type = 'observation'
-          AND NOT EXISTS (
-              SELECT 1
-              FROM {mu} src
-              WHERE src.id = ANY(orphan.source_memory_ids)
-                AND src.bank_id = orphan.bank_id
-          )
-        """
-    )
-
-
-def downgrade() -> None:
-    # Deleted rows cannot be restored.
-    pass

hindsight-api-slim/hindsight_api/alembic/versions/c5d6e7f8a9b0_add_bank_id_to_memory_links.py

@@ -1,48 +0,0 @@
-"""Add bank_id column to memory_links for direct filtering
-
-The stats endpoint JOINs memory_links to memory_units just to filter by
-bank_id.  With millions of links this takes 18+ seconds.  Adding bank_id
-directly to memory_links lets Postgres push the filter down before the JOIN.
-
-Revision ID: c5d6e7f8a9b0
-Revises: b3c4d5e6f7a8
-Create Date: 2026-03-26
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c5d6e7f8a9b0"
-down_revision: str | Sequence[str] | None = "b3c4d5e6f7a8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Add nullable column
-    op.execute(f"ALTER TABLE {schema}memory_links ADD COLUMN IF NOT EXISTS bank_id TEXT")
-
-    # 2. Backfill from memory_units
-    op.execute(f"""
-        UPDATE {schema}memory_links ml
-        SET bank_id = mu.bank_id
-        FROM {schema}memory_units mu
-        WHERE ml.from_unit_id = mu.id
-          AND ml.bank_id IS NULL
-    """)
-
-    # 3. Set NOT NULL
-    op.execute(f"ALTER TABLE {schema}memory_links ALTER COLUMN bank_id SET NOT NULL")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_links DROP COLUMN IF EXISTS bank_id")

hindsight-api-slim/hindsight_api/alembic/versions/d2e3f4a5b6c7_add_memory_links_expansion_indexes.py

@@ -1,83 +0,0 @@
-"""Add covering and composite indexes to speed up link expansion graph retrieval.
-
-Two indexes target the two bottlenecks identified by EXPLAIN ANALYZE on a 17M-row
-memory_links table:
-
-1. idx_memory_links_to_type_weight  (to_unit_id, link_type, weight DESC)
-   The semantic incoming direction — finding facts that consider seeds as their
-   nearest neighbour — currently hits an expensive BitmapAnd of two separate
-   bitmap scans (to_unit_id bitmap ∩ link_type bitmap).  A composite index
-   on (to_unit_id, link_type) turns this into a single index scan and reduces
-   latency from ~36 ms to < 5 ms per query.
-
-2. idx_memory_links_entity_covering  (from_unit_id) INCLUDE (to_unit_id, entity_id)
-   WHERE link_type = 'entity'
-   The entity co-occurrence expansion uses COUNT(DISTINCT ml.entity_id) and
-   joins on ml.to_unit_id.  Without a covering index the planner must read
-   ~2 500 heap pages to fetch entity_id and to_unit_id after the bitmap index
-   scan, adding ~230 ms of random I/O.  INCLUDE adds those two columns to the
-   index leaf pages so the entire query can be served from the index (index-only
-   scan), eliminating the heap reads entirely.
-   Partial index (WHERE link_type = 'entity') keeps index size ~40 % smaller.
-
-Both indexes are created with CONCURRENTLY so the migration does not block
-concurrent reads or writes on memory_links.  CONCURRENTLY requires running
-outside a transaction block, so the migration emits an explicit COMMIT before
-each statement and uses IF NOT EXISTS for idempotency.
-
-Revision ID: d2e3f4a5b6c7
-Revises: b3c4d5e6f7g8
-Create Date: 2026-03-02
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "d2e3f4a5b6c7"
-down_revision: str | Sequence[str] | None = "b3c4d5e6f7g8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
-    # Commit the current Alembic transaction, then issue each CONCURRENTLY
-    # statement in its own implicit autocommit transaction.
-    # IF NOT EXISTS makes each statement idempotent if the migration is retried.
-
-    # Index for the semantic *incoming* direction in link_expansion_retrieval.py.
-    # Replaces the BitmapAnd of idx_memory_links_to_unit ∩ idx_memory_links_link_type
-    # with a single composite index scan.
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_links_to_type_weight "
-        f"ON {schema}memory_links(to_unit_id, link_type, weight DESC)"
-    )
-
-    # Covering index for entity co-occurrence expansion.
-    # Enables an index-only scan: entity_id and to_unit_id are read from the
-    # index leaf pages instead of the heap, eliminating ~2 500 random heap-page
-    # reads per expansion query.
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_links_entity_covering "
-        f"ON {schema}memory_links(from_unit_id) "
-        f"INCLUDE (to_unit_id, entity_id) "
-        f"WHERE link_type = 'entity'"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_entity_covering")
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_to_type_weight")

hindsight-api-slim/hindsight_api/alembic/versions/d4e5f6g7h8i9_gin_source_memory_ids_fastupdate_off.py

@@ -1,53 +0,0 @@
-"""Recreate idx_memory_units_source_memory_ids GIN index with fastupdate=off
-
-GIN indexes use a "fastupdate" pending list by default: small writes are
-buffered there and flushed to the main GIN tree in bulk. Flushing requires
-AccessExclusiveLock on the index. Under high insert concurrency (e.g. 8
-parallel pytest-xdist workers all calling retain_async) two transactions can
-each trigger a flush simultaneously and deadlock.
-
-Disabling fastupdate makes every insert write directly to the GIN tree
-(slightly slower per insert, but no pending-list lock cycles).
-
-Revision ID: d4e5f6g7h8i9
-Revises: d5e6f7a8b9c0
-Create Date: 2026-03-11
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "d4e5f6g7h8i9"
-down_revision: str | Sequence[str] | None = "d5e6f7a8b9c0"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # DROP + CREATE CONCURRENTLY must run outside a transaction block.
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WITH (fastupdate=off) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/d5e6f7a8b9c0_add_bank_internal_id_and_per_bank_hnsw.py

@@ -1,139 +0,0 @@
-"""Add internal_id to banks and per-(bank, fact_type) partial vector indexes
-
-Revision ID: d5e6f7a8b9c0
-Revises: a3b4c5d6e7f8
-Create Date: 2026-03-11
-
-This migration:
-1. Adds internal_id UUID column to banks (stable identifier for index naming)
-2. Drops the global vector index (competes with per-bank partial indexes)
-3. Creates per-(bank_id, fact_type) partial vector indexes for all existing banks
-   using the configured vector extension (HNSW for pgvector, DiskANN for
-   pgvectorscale, vchordrq for vchord).
-   (new banks get indexes created at bank-creation time via bank_utils.create_bank_vector_indexes)
-
-Why per-(bank, fact_type) indexes:
-- fact_type-only partial indexes are never chosen by the planner when bank_id is in the WHERE
-  clause, because the idx_memory_units_bank_id B-tree index always wins at planning time.
-- Per-(bank, fact_type) partial indexes have both predicates matching → planner selects them.
-- The global vector index competes for larger partitions (world, observation) and must be dropped.
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import text
-
-revision: str = "d5e6f7a8b9c0"
-down_revision: str | Sequence[str] | None = "c3d4e5f6g7h8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_FACT_TYPES: dict[str, str] = {
-    "world": "worl",
-    "experience": "expr",
-    "observation": "obsv",
-}
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _vector_index_using_clause() -> str:
-    """Return the USING clause based on the configured vector extension."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
-    elif ext == "vchord":
-        return "USING vchordrq (embedding vector_l2_ops)"
-    else:
-        return "USING hnsw (embedding vector_cosine_ops)"
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Add internal_id column to banks
-    op.execute(
-        f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS internal_id UUID DEFAULT gen_random_uuid() NOT NULL"
-    )
-    op.execute(f"ALTER TABLE {schema}banks ADD CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)")
-
-    # 2. Drop any fact_type-only partial indexes that may exist from prior migrations
-    #    (bank_id B-tree always wins over them when bank_id is in the WHERE clause)
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_world")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_observation")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_experience")
-
-    # 4. Drop global vector index (competes with per-bank partial indexes)
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_embedding")
-
-    # 5. Create per-(bank, fact_type) partial vector indexes for all existing banks
-    #    using the configured extension (HNSW / DiskANN / vchordrq)
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-    using_clause = _vector_index_using_clause()
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            # Index name is schema-unqualified (indexes live in the schema of their table)
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} {using_clause} "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Drop per-bank HNSW indexes (iterate existing banks)
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-
-    rows = bind.execute(text(f"SELECT internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        internal_id = str(row[0]).replace("-", "")[:16]
-        for ft_short in _HNSW_FACT_TYPES.values():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-
-    # Restore the global HNSW index
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_memory_units_embedding ON {table_ref} USING hnsw (embedding vector_cosine_ops)"
-    )
-
-    # Restore old fact_type-only partial indexes
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_world "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'world'"
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_observation "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'observation'"
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_experience "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'experience'"
-    )
-
-    # Drop internal_id column
-    op.execute(f"ALTER TABLE {schema}banks DROP CONSTRAINT IF EXISTS banks_internal_id_unique")
-    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS internal_id")

hindsight-api-slim/hindsight_api/alembic/versions/d6e7f8a9b0c1_drop_documents_metadata_column.py

@@ -1,39 +0,0 @@
-"""Drop unused metadata column from documents table
-
-Revision ID: d6e7f8a9b0c1
-Revises: c2d3e4f5g6h7, c5d6e7f8a9b0
-Create Date: 2026-03-30
-
-The metadata column on documents was always stored as an empty dict {}.
-Actual document metadata is stored inside retain_params.metadata.
-
-This migration was originally shipped in v0.4.22, then its file was deleted
-in v0.5.0 (and its revision ID accidentally reused by 2eee35aa3cfc).
-Restoring the file so that databases stamped at this revision can upgrade
-cleanly to v0.5.x+.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "d6e7f8a9b0c1"
-down_revision: str | Sequence[str] | None = ("c2d3e4f5g6h7", "c5d6e7f8a9b0")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS metadata")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS metadata jsonb DEFAULT '{{}}'")

hindsight-api-slim/hindsight_api/alembic/versions/e4f5a6b7c8d9_add_webhooks_tables.py

@@ -1,62 +0,0 @@
-"""Add webhooks table and next_retry_at to async_operations.
-
-Webhook deliveries are handled as async_operations tasks (operation_type='webhook_delivery')
-rather than a dedicated webhook_deliveries table.
-
-Revision ID: e4f5a6b7c8d9
-Revises: d2e3f4a5b6c7
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "e4f5a6b7c8d9"
-down_revision: str | Sequence[str] | None = "d2e3f4a5b6c7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}webhooks (
-            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-            bank_id TEXT,
-            url TEXT NOT NULL,
-            secret TEXT,
-            event_types TEXT[] NOT NULL DEFAULT '{{}}',
-            enabled BOOLEAN NOT NULL DEFAULT TRUE,
-            created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-            updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
-        )
-        """
-    )
-
-    # Index for bank-scoped webhook lookup
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_webhooks_bank_id ON {schema}webhooks(bank_id)")
-
-    # Add next_retry_at to async_operations for task-owned retry scheduling
-    op.execute(f"ALTER TABLE {schema}async_operations ADD COLUMN IF NOT EXISTS next_retry_at TIMESTAMPTZ NULL")
-
-    # Index for polling: status + next_retry_at
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_async_operations_status_retry "
-        f"ON {schema}async_operations(status, next_retry_at)"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_status_retry")
-    op.execute(f"ALTER TABLE {schema}async_operations DROP COLUMN IF EXISTS next_retry_at")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_webhooks_bank_id")
-    op.execute(f"DROP TABLE IF EXISTS {schema}webhooks")

hindsight-api-slim/hindsight_api/alembic/versions/e5f6g7h8i9j0_cascade_delete_ops_on_bank_delete.py

@@ -1,73 +0,0 @@
-"""Add CASCADE DELETE FK from async_operations and webhooks to banks.
-
-When a bank is deleted, all its async_operations and webhooks rows are
-automatically deleted by the database. This ensures that any in-flight
-worker tasks detect the deletion via _check_op_alive() and abort early.
-
-Revision ID: e5f6g7h8i9j0
-Revises: d4e5f6g7h8i9
-Create Date: 2026-03-11
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "e5f6g7h8i9j0"
-down_revision: str | Sequence[str] | None = "d4e5f6g7h8i9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Remove orphaned async_operations rows whose bank no longer exists
-    # (can happen because there was no FK before this migration).
-    op.execute(
-        f"""
-        DELETE FROM {schema}async_operations
-        WHERE bank_id IS NOT NULL
-          AND bank_id NOT IN (SELECT bank_id FROM {schema}banks)
-        """
-    )
-
-    # Remove orphaned webhooks rows whose bank no longer exists.
-    op.execute(
-        f"""
-        DELETE FROM {schema}webhooks
-        WHERE bank_id IS NOT NULL
-          AND bank_id NOT IN (SELECT bank_id FROM {schema}banks)
-        """
-    )
-
-    # Add FK with ON DELETE CASCADE so that deleting a bank automatically
-    # cleans up all its pending/processing operations and webhook configs.
-    op.execute(
-        f"""
-        ALTER TABLE {schema}async_operations
-            ADD CONSTRAINT fk_async_operations_bank_id
-            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id)
-            ON DELETE CASCADE
-        """
-    )
-
-    op.execute(
-        f"""
-        ALTER TABLE {schema}webhooks
-            ADD CONSTRAINT fk_webhooks_bank_id
-            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id)
-            ON DELETE CASCADE
-        """
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS fk_async_operations_bank_id")
-    op.execute(f"ALTER TABLE {schema}webhooks DROP CONSTRAINT IF EXISTS fk_webhooks_bank_id")

hindsight-api-slim/hindsight_api/alembic/versions/f6g7h8i9j0k1_chunk_fk_cascade_delete.py

@@ -1,57 +0,0 @@
-"""chunk_fk_cascade_delete
-
-Revision ID: f6g7h8i9j0k1
-Revises: e5f6g7h8i9j0
-Create Date: 2026-03-16 00:00:00.000000
-
-"""
-
-from collections.abc import Sequence
-
-from alembic import op
-
-# revision identifiers, used by Alembic.
-revision: str = "f6g7h8i9j0k1"
-down_revision: str | Sequence[str] | None = "e5f6g7h8i9j0"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def upgrade() -> None:
-    """Change memory_units.chunk_id FK from SET NULL to CASCADE.
-
-    When a document is deleted the CASCADE reaches chunks first; with SET NULL
-    the memory_units rows survived with chunk_id = NULL, leaving ghost records.
-    Switching to CASCADE ensures they are removed together with their chunk.
-    """
-    from alembic import context
-
-    schema = context.config.get_main_option("target_schema")
-    schema_prefix = f'"{schema}".' if schema else ""
-    # Use raw SQL with IF EXISTS so this is safe on schemas where the FK was
-    # already dropped or never existed under this name.
-    op.execute(f"ALTER TABLE {schema_prefix}memory_units DROP CONSTRAINT IF EXISTS memory_units_chunk_fkey")
-    # Use a DO block so the ADD is also idempotent: if the FK already exists (e.g.
-    # the schema was provisioned after the base migration already added it) the
-    # duplicate_object exception is swallowed rather than failing the migration.
-    op.execute(
-        f"""
-        DO $$ BEGIN
-            ALTER TABLE {schema_prefix}memory_units
-                ADD CONSTRAINT memory_units_chunk_fkey
-                FOREIGN KEY (chunk_id)
-                REFERENCES {schema_prefix}chunks (chunk_id)
-                ON DELETE CASCADE;
-        EXCEPTION
-            WHEN duplicate_object THEN NULL;
-        END $$;
-        """
-    )
-
-
-def downgrade() -> None:
-    """Revert to SET NULL behaviour."""
-    op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
-    op.create_foreign_key(
-        "memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="SET NULL"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/f7g8h9i0j1k2_add_webhook_http_config.py

@@ -1,33 +0,0 @@
-"""Add http_config JSONB column to webhooks table.
-
-Stores HTTP delivery configuration (method, timeout, headers, params) as a
-single JSONB column rather than separate columns.
-
-Revision ID: f7g8h9i0j1k2
-Revises: e4f5a6b7c8d9
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "f7g8h9i0j1k2"
-down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}webhooks ADD COLUMN IF NOT EXISTS http_config JSONB NOT NULL DEFAULT '{{}}'")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}webhooks DROP COLUMN IF EXISTS http_config")

hindsight-api-slim/hindsight_api/alembic/versions/g2h3i4j5k6l7_remove_opinion_fact_type.py

@@ -1,83 +0,0 @@
-"""remove_opinion_fact_type
-
-Revision ID: g2h3i4j5k6l7
-Revises: f1a2b3c4d5e6
-Create Date: 2026-04-02
-
-Remove the deprecated 'opinion' fact type: drop opinion-specific indexes,
-update CHECK constraints, delete any remaining opinion rows, and drop the
-confidence_score column (was only used for opinions, always NULL otherwise).
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-# revision identifiers, used by Alembic.
-revision: str = "g2h3i4j5k6l7"
-down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Delete any remaining opinion rows
-    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
-
-    # 2. Drop opinion-specific indexes
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_confidence")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_date")
-
-    # 3. Drop confidence_score constraints and column (only used for opinions, always NULL otherwise)
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS confidence_score_fact_type_check")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_confidence_score_check")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS confidence_score")
-
-    # 4. Replace fact_type CHECK constraint
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
-        f"CHECK (fact_type IN ('world', 'experience', 'observation'))"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Restore confidence_score column
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS confidence_score float")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_confidence_score_check "
-        f"CHECK (confidence_score IS NULL OR (confidence_score >= 0.0 AND confidence_score <= 1.0))"
-    )
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT confidence_score_fact_type_check "
-        f"CHECK ((fact_type = 'opinion' AND confidence_score IS NOT NULL) OR "
-        f"(fact_type = 'observation') OR "
-        f"(fact_type NOT IN ('opinion', 'observation') AND confidence_score IS NULL))"
-    )
-
-    # Restore original fact_type CHECK constraint (with opinion)
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
-        f"CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))"
-    )
-
-    # Recreate opinion indexes
-    op.execute(
-        f"CREATE INDEX idx_memory_units_opinion_confidence ON {schema}memory_units "
-        f"(bank_id, confidence_score DESC) WHERE fact_type = 'opinion'"
-    )
-    op.execute(
-        f"CREATE INDEX idx_memory_units_opinion_date ON {schema}memory_units "
-        f"(bank_id, event_date DESC) WHERE fact_type = 'opinion'"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/g7h8i9j0k1l2_backsweep_orphan_observations.py

@@ -1,71 +0,0 @@
-"""backsweep_orphan_memory_units
-
-Two-pass cleanup of memory_units rows that were never removed by earlier bugs:
-
-Pass 1 — any fact_type, bank gone:
-  memory_units whose bank_id no longer exists in banks.  These accumulate when
-  a bank is deleted without a proper cascade (no FK from memory_units to banks
-  exists in the schema).
-
-Pass 2 — observations only, all sources gone:
-  observation rows whose bank still exists but every source_memory_id points
-  to a deleted memory unit.  These were left behind before PR #580 fixed the
-  chunk FK cascade and before delete_document() called
-  _delete_stale_observations_for_memories.
-
-Revision ID: g7h8i9j0k1l2
-Revises: f6g7h8i9j0k1
-Create Date: 2026-03-16
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "g7h8i9j0k1l2"
-down_revision: str | Sequence[str] | None = "f6g7h8i9j0k1"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    mu = f"{schema}memory_units"
-    banks = f"{schema}banks"
-
-    # Pass 1: delete all memory_units (any fact_type) whose bank no longer exists.
-    # There is no FK from memory_units to banks, so these never cascade away.
-    op.execute(
-        f"""
-        DELETE FROM {mu}
-        WHERE NOT EXISTS (
-            SELECT 1 FROM {banks} b WHERE b.bank_id = {mu}.bank_id
-        )
-        """
-    )
-
-    # Pass 2: delete orphaned observations whose bank still exists but every
-    # source_memory_id refers to a now-deleted memory unit (or the array is
-    # empty).  Observations with at least one surviving source are left alone.
-    op.execute(
-        f"""
-        DELETE FROM {mu} orphan
-        WHERE orphan.fact_type = 'observation'
-          AND NOT EXISTS (
-              SELECT 1
-              FROM {mu} src
-              WHERE src.id = ANY(orphan.source_memory_ids)
-                AND src.bank_id = orphan.bank_id
-          )
-        """
-    )
-
-
-def downgrade() -> None:
-    # Deleted rows cannot be restored.
-    pass

hindsight-api-slim/hindsight_api/alembic/versions/h3i4j5k6l7m8_merge_heads_and_add_unit_entities_index.py

@@ -1,42 +0,0 @@
-"""Merge 3 migration heads and add unit_entities composite index
-
-Revision ID: h3i4j5k6l7m8
-Revises: a4b5c6d7e8f9, g2h3i4j5k6l7
-Create Date: 2026-04-07
-
-Merges three unmerged migration heads into one, and adds a composite index
-(entity_id, unit_id) on unit_entities for index-only scans in the LATERAL
-entity expansion query.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "h3i4j5k6l7m8"
-down_revision: str | Sequence[str] | None = ("a4b5c6d7e8f9", "g2h3i4j5k6l7")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Composite index enables index-only scans for entity_id -> unit_id lookups
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity_unit ON {schema}unit_entities (entity_id, unit_id)"
-    )
-    # Drop the now-redundant single-column index
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity_unit")
-    # Restore the single-column index
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity ON {schema}unit_entities (entity_id)")

hindsight-api-slim/hindsight_api/alembic/versions/y0t1u2v3w4x5_add_result_metadata_gin_index.py

@@ -1,49 +0,0 @@
-"""Add GIN index on async_operations.result_metadata for parent_operation_id queries
-
-Revision ID: y0t1u2v3w4x5
-Revises: x9s0t1u2v3w4
-Create Date: 2026-02-13
-
-This migration adds a GIN index on the result_metadata JSONB column in the
-async_operations table to support efficient queries for child operations by
-parent_operation_id.
-
-The index enables fast lookups when querying for child operations:
-  SELECT * FROM async_operations
-  WHERE result_metadata::jsonb @> '{"parent_operation_id": "uuid"}'::jsonb
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "y0t1u2v3w4x5"
-down_revision: str | Sequence[str] | None = "x9s0t1u2v3w4"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    """Add GIN index on result_metadata for efficient parent_operation_id queries."""
-    schema = _get_schema_prefix()
-
-    # Add GIN index for JSONB containment queries (@> operator)
-    op.execute(f"""
-        CREATE INDEX IF NOT EXISTS idx_async_operations_result_metadata
-        ON {schema}async_operations
-        USING gin(result_metadata)
-    """)
-
-
-def downgrade() -> None:
-    """Remove GIN index on result_metadata."""
-    schema = _get_schema_prefix()
-
-    # Drop index
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_result_metadata")

hindsight-api-slim/hindsight_api/alembic/versions/z1u2v3w4x5y6_add_observation_tags_to_memory_units.py

@@ -1,35 +0,0 @@
-"""Add observation_scopes column to memory_units table
-
-Revision ID: z1u2v3w4x5y6
-Revises: a1b2c3d4e5f6
-Create Date: 2026-02-25
-
-Adds observation_scopes JSONB column to memory_units to control how observations
-are scoped during consolidation. Accepts "per_tag", "combined", or an explicit
-list of tag-set lists for custom multi-pass consolidation.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "z1u2v3w4x5y6"
-down_revision: str | Sequence[str] | None = "a1b2c3d4e5f6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS observation_scopes JSONB")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS observation_scopes")

hindsight-api-slim/hindsight_api/engine/audit.py

@@ -1,209 +0,0 @@
-"""Audit logging for feature usage tracking.
-
-Provides fire-and-forget audit logging of all mutating and core operations
-(retain, recall, reflect, bank CRUD, etc.) across HTTP, MCP, and system transports.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import logging
-import uuid
-from collections.abc import Callable
-from contextlib import asynccontextmanager
-from dataclasses import dataclass, field
-from datetime import datetime, timezone
-from typing import Any
-
-import asyncpg
-
-from ..engine.db_utils import acquire_with_retry
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class AuditEntry:
-    """A single audit log entry."""
-
-    action: str
-    transport: str  # "http", "mcp", "system"
-    bank_id: str | None = None
-    started_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
-    ended_at: datetime | None = None
-    request: dict[str, Any] | None = None
-    response: dict[str, Any] | None = None
-    metadata: dict[str, Any] = field(default_factory=dict)
-
-
-def _json_default(obj: Any) -> str:
-    """JSON serializer for objects not serializable by default."""
-    if isinstance(obj, datetime):
-        return obj.isoformat()
-    if isinstance(obj, uuid.UUID):
-        return str(obj)
-    if isinstance(obj, bytes):
-        return "<bytes>"
-    if isinstance(obj, set):
-        return list(obj)
-    return str(obj)
-
-
-def _safe_json(data: Any) -> str | None:
-    """Serialize data to JSON string, returning None on failure."""
-    if data is None:
-        return None
-    try:
-        return json.dumps(data, default=_json_default)
-    except Exception:
-        logger.debug("Failed to serialize audit data", exc_info=True)
-        return None
-
-
-_SWEEP_INTERVAL_SECONDS = 3600  # Run retention sweep every hour
-
-
-class AuditLogger:
-    """Fire-and-forget audit log writer with optional retention sweep."""
-
-    def __init__(
-        self,
-        pool_getter: Callable[[], asyncpg.Pool | None],
-        schema_getter: Callable[[], str],
-        enabled: bool,
-        allowed_actions: list[str],
-        retention_days: int = -1,
-    ) -> None:
-        self._pool_getter = pool_getter
-        self._schema_getter = schema_getter
-        self._enabled = enabled
-        self._allowed_actions: frozenset[str] | None = frozenset(allowed_actions) if allowed_actions else None
-        self._retention_days = retention_days
-        self._sweep_task: asyncio.Task | None = None
-
-    def is_enabled(self, action: str) -> bool:
-        """Check if audit logging is enabled for this action."""
-        if not self._enabled:
-            return False
-        if self._allowed_actions is not None:
-            return action in self._allowed_actions
-        return True
-
-    def log_fire_and_forget(self, entry: AuditEntry) -> None:
-        """Schedule an audit write as a background task."""
-        if not self.is_enabled(entry.action):
-            return
-        try:
-            asyncio.create_task(self._safe_log(entry))
-        except RuntimeError:
-            # No running event loop (e.g. during shutdown)
-            logger.debug("Cannot schedule audit log write: no running event loop")
-
-    async def _safe_log(self, entry: AuditEntry) -> None:
-        """Write audit entry to DB. Errors are logged, never raised."""
-        pool = self._pool_getter()
-        if pool is None:
-            logger.debug("Audit log skipped: pool not available")
-            return
-        try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
-            async with acquire_with_retry(pool, max_retries=1) as conn:
-                await conn.execute(
-                    f"""
-                    INSERT INTO {table}
-                        (id, action, transport, bank_id, started_at, ended_at, request, response, metadata)
-                    VALUES
-                        ($1, $2, $3, $4, $5, $6, $7::jsonb, $8::jsonb, $9::jsonb)
-                    """,
-                    uuid.uuid4(),
-                    entry.action,
-                    entry.transport,
-                    entry.bank_id,
-                    entry.started_at,
-                    entry.ended_at,
-                    _safe_json(entry.request),
-                    _safe_json(entry.response),
-                    _safe_json(entry.metadata) or "{}",
-                )
-        except Exception as e:
-            logger.warning(f"Audit log write failed for action={entry.action}: {e}")
-
-    def start_retention_sweep(self) -> None:
-        """Start the periodic retention sweep if retention is configured."""
-        if self._retention_days <= 0 or not self._enabled:
-            return
-        try:
-            self._sweep_task = asyncio.create_task(self._sweep_loop())
-        except RuntimeError:
-            logger.debug("Cannot start retention sweep: no running event loop")
-
-    async def stop_retention_sweep(self) -> None:
-        """Stop the periodic retention sweep."""
-        if self._sweep_task and not self._sweep_task.done():
-            self._sweep_task.cancel()
-            try:
-                await self._sweep_task
-            except asyncio.CancelledError:
-                pass
-            self._sweep_task = None
-
-    async def _sweep_loop(self) -> None:
-        """Periodically delete audit log entries older than retention_days."""
-        while True:
-            await self._run_sweep()
-            await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
-
-    async def _run_sweep(self) -> None:
-        """Delete expired audit log entries. Concurrent-safe via row-level deletes."""
-        pool = self._pool_getter()
-        if pool is None:
-            return
-        try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
-            async with acquire_with_retry(pool, max_retries=1) as conn:
-                result = await conn.execute(
-                    f"DELETE FROM {table} WHERE started_at < NOW() - INTERVAL '{self._retention_days} days'"
-                )
-                if result and result != "DELETE 0":
-                    logger.info(f"Audit log retention sweep: {result}")
-        except Exception as e:
-            logger.warning(f"Audit log retention sweep failed: {e}")
-
-
-@asynccontextmanager
-async def audit_context(
-    audit_logger: AuditLogger | None,
-    action: str,
-    transport: str,
-    bank_id: str | None = None,
-    request: dict[str, Any] | None = None,
-    metadata: dict[str, Any] | None = None,
-):
-    """Async context manager that times the operation and writes audit on exit.
-
-    Usage:
-        async with audit_context(logger, "retain", "http", bank_id, request_dict) as entry:
-            result = await do_work()
-            entry.response = result_dict
-    """
-    if audit_logger is None or not audit_logger.is_enabled(action):
-        entry = AuditEntry(action=action, transport=transport, bank_id=bank_id)
-        yield entry
-        return
-
-    entry = AuditEntry(
-        action=action,
-        transport=transport,
-        bank_id=bank_id,
-        started_at=datetime.now(timezone.utc),
-        request=request,
-        metadata=metadata or {},
-    )
-    try:
-        yield entry
-    finally:
-        entry.ended_at = datetime.now(timezone.utc)
-        audit_logger.log_fire_and_forget(entry)

hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py

@@ -1,104 +0,0 @@
-"""Prompts for the consolidation engine."""
-
-# Default mission when no bank-specific mission is set
-_DEFAULT_MISSION = "Track every detail: names, numbers, dates, places, and relationships. Prefer specifics over abstractions, never generalise."
-
-# Processing rules — always present regardless of mission
-_PROCESSING_RULES = """Processing rules (always apply):
-
-1. ONE OBSERVATION PER DISTINCT FACET: each observation tracks exactly one specific facet — a count ("has 3 items"), a named entity ("has a dog named Rex"), a relationship ("works at Google"), etc. Never merge different facets into one observation.
-
-2. MATCH BY ENTITY/FACET, NOT TOPIC: when deciding whether to UPDATE vs CREATE, match on the specific entity or facet. "Sold item X" updates only the X observation. "Now has 5 items" updates only the count observation. Do not update observations about different entities just because they share a general topic.
-
-3. STATE CHANGES — UPDATE CONCISELY: when a fact changes the state of something ("sold X", "X died", "moved to Y"), UPDATE the matching observation to reflect the current state. Include dates when available. Keep it concise — only information about THAT specific facet. Example: "User owned a dog named Rex who died on March 15, 2025". Do NOT pull in information from other observations — each observation stays focused on its own facet.
-
-4. CASCADE TO ALL AFFECTED OBSERVATIONS: a state change may affect multiple observations. For example, if entity C is removed from a group, update BOTH the individual observation for C AND any list/group observation that includes C (remove C from the list while keeping all other members intact).
-
-5. NO COMPUTATION: you do not have the full picture — never calculate, derive, or adjust numeric values. If the user says "I have 2 dogs" and then "I have a dog named Rex", do NOT update the count to 3 — you don't know if Rex is one of the 2 or a new one. If the user says "I sold X", do NOT decrement a count. Only update a count when the user explicitly states a new count. Synthesize and consolidate what was stated, but never do arithmetic or logical deductions.
-
-6. SAME FACET → UPDATE, NOT CREATE: a new count supersedes the old count — UPDATE the existing count observation, don't create a second one. If there's an existing observation for the same specific facet, always UPDATE it rather than creating a duplicate.
-
-7. PRESERVE HISTORY: observations that record significant events (sold, died, moved, changed) are important history — never DELETE them. Only delete an observation when it is restated identically or truly meaningless. Be very conservative with deletes.
-
-8. RESOLVE REFERENCES: when a new fact provides a concrete value for a vague placeholder in an existing observation (e.g., "home country" → "Sweden"), UPDATE to embed the resolved value.
-
-9. NEVER merge observations about different people or unrelated topics."""
-
-# Data section — format placeholders {facts_text} and {observations_text} are substituted at call time
-_BATCH_DATA_SECTION = """
-NEW FACTS:
-{facts_text}
-
-EXISTING OBSERVATIONS (JSON array, pooled from recalls across all facts above):
-{observations_text}
-
-Each observation includes:
-- id: unique identifier for updating
-- text: the observation content
-- proof_count: number of supporting memories
-- occurred_start/occurred_end: temporal range of source facts
-- source_memories: array of supporting facts with their text and dates
-
-Compare the facts against existing observations:
-- Same facet as an existing observation → UPDATE it (observation_id + source_fact_ids)
-- New facet with durable knowledge → CREATE a new observation (source_fact_ids)
-- Cross-reference facts within the batch: a later fact may resolve a vague reference in an earlier one
-- Purely ephemeral facts → omit them unless the MISSION above explicitly targets such data (e.g. timestamped events, session state, screen content)"""
-
-# Output format — JSON braces escaped as {{ }} so .format() leaves them literal
-_BATCH_OUTPUT_FORMAT = """
-Output a JSON object with three arrays.
-
-## EXAMPLE
-
-Input facts:
-[a1b2c3d4-e5f6-7890-abcd-ef1234567890] Alice mentioned she works long hours, often past midnight | Involving: Alice (occurred_start=2024-01-15, mentioned_at=2024-01-15)
-[b2c3d4e5-f6a7-8901-bcde-f12345678901] Alice said she's exhausted from the project deadlines | Involving: Alice (occurred_start=2024-01-20, mentioned_at=2024-01-20)
-
-Good observation text — clean prose, no metadata, each fact tracked distinctly:
-  "Alice works long hours, often past midnight."
-  "Alice feels exhausted from project deadlines."
-
-Bad observation text — NEVER do this (verbatim copy of fact text with metadata):
-  "Alice mentioned she works long hours, often past midnight | Involving: Alice (occurred_start=2024-01-15, mentioned_at=2024-01-15)"
-
-Observation text rules:
-- Write clean prose — NEVER copy raw fact lines or their metadata (temporal fields, "Involving:", "When:" labels, UUIDs).
-- Parenthesized metadata like (occurred_start=...) and pipe-separated labels like "| Involving: ..." are fact formatting — strip them entirely from observation text.
-- How many observations to create and how much to aggregate is driven by the MISSION above.
-
-{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"]}}, {{"text": "Alice feels exhausted from project deadlines.", "source_fact_ids": ["b2c3d4e5-f6a7-8901-bcde-f12345678901"]}}],
-  "updates": [{{"text": "Alice works at Acme Corp as a senior engineer", "observation_id": "c3d4e5f6-a7b8-9012-cdef-123456789012", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"]}}],
-  "deletes": [{{"observation_id": "e5f6a7b8-c9d0-1234-efab-345678901234"}}]}}
-
-Rules:
-- "source_fact_ids": copy the EXACT UUID strings shown in brackets [uuid] from NEW FACTS — never use integers or positions.
-- "observation_id": copy the EXACT "id" UUID string from EXISTING OBSERVATIONS.
-- One create/update may reference multiple facts when they jointly support the observation.
-- "deletes": only when an observation is directly superseded or contradicted by new facts.
-- Do NOT include "tags" — handled automatically.
-- Return {{"creates": [], "updates": [], "deletes": []}} if nothing durable is found."""
-
-
-def build_batch_consolidation_prompt(
-    observations_mission: str | None = None,
-    observation_capacity_note: str | None = None,
-) -> str:
-    """
-    Build the consolidation prompt for batch mode (multiple facts per LLM call).
-
-    The mission defines *what* to track (customisable per bank).
-    Processing rules and output format are always present regardless of mission.
-    """
-    mission = observations_mission or _DEFAULT_MISSION
-
-    capacity_section = ""
-    if observation_capacity_note:
-        capacity_section = f"\n\n## CAPACITY CONSTRAINT\n{observation_capacity_note}"
-
-    return (
-        "You are a memory consolidation system. Synthesize facts into observations "
-        "and merge with existing observations when appropriate.\n\n"
-        f"## MISSION\n{mission}{capacity_section}\n\n"
-        f"{_PROCESSING_RULES}" + _BATCH_DATA_SECTION + _BATCH_OUTPUT_FORMAT
-    )

hindsight-api-slim/hindsight_api/engine/jina_mlx_reranker.py

@@ -1,144 +0,0 @@
-"""
-MLX implementation of jina-reranker-v3 for Apple Silicon.
-
-This file is adapted from the official model repository:
-  https://huggingface.co/jinaai/jina-reranker-v3-mlx/blob/main/rerank.py
-
-License: CC BY-NC 4.0 (contact Jina AI for commercial usage)
-
-Changes from upstream:
-- Removed the __main__ example block
-- Type annotations added to public methods
-- top_n parameter added to rerank() (upstream only exposed it implicitly)
-"""
-
-import numpy as np
-
-
-class _MLPProjector:
-    def __init__(self):
-        import mlx.nn as nn
-
-        self.linear1 = nn.Linear(1024, 512, bias=False)
-        self.linear2 = nn.Linear(512, 512, bias=False)
-
-    def __call__(self, x):
-        import mlx.nn as nn
-
-        x = self.linear1(x)
-        x = nn.relu(x)
-        x = self.linear2(x)
-        return x
-
-
-def _load_projector(projector_path: str) -> _MLPProjector:
-    import mlx.core as mx
-    from safetensors import safe_open
-
-    projector = _MLPProjector()
-    with safe_open(projector_path, framework="numpy") as f:
-        projector.linear1.weight = mx.array(f.get_tensor("linear1.weight"))
-        projector.linear2.weight = mx.array(f.get_tensor("linear2.weight"))
-    return projector
-
-
-def _sanitize(text: str, special_tokens: dict[str, str]) -> str:
-    for token in special_tokens.values():
-        text = text.replace(token, "")
-    return text
-
-
-def _format_prompt(query: str, docs: list[str], special_tokens: dict[str, str]) -> str:
-    query = _sanitize(query, special_tokens)
-    docs = [_sanitize(d, special_tokens) for d in docs]
-
-    doc_token = special_tokens["doc_embed_token"]
-    query_token = special_tokens["query_embed_token"]
-
-    prefix = (
-        "<|im_start|>system\n"
-        "You are a search relevance expert who can determine a ranking of the passages based on how relevant they are to the query. "
-        "If the query is a question, how relevant a passage is depends on how well it answers the question. "
-        "If not, try to analyze the intent of the query and assess how well each passage satisfies the intent. "
-        "If an instruction is provided, you should follow the instruction when determining the ranking."
-        "<|im_end|>\n<|im_start|>user\n"
-    )
-    suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
-
-    body = (
-        f"I will provide you with {len(docs)} passages, each indicated by a numerical identifier. "
-        f"Rank the passages based on their relevance to query: {query}\n"
-    )
-    body += "\n".join(f'<passage id="{i}">\n{doc}{doc_token}\n</passage>' for i, doc in enumerate(docs))
-    body += f"\n<query>\n{query}{query_token}\n</query>"
-    return prefix + body + suffix
-
-
-class MLXReranker:
-    """
-    MLX-accelerated jina-reranker-v3 for Apple Silicon.
-
-    Loads the model from a local directory (use huggingface_hub.snapshot_download
-    to fetch jinaai/jina-reranker-v3-mlx if you don't have it already).
-    """
-
-    _SPECIAL_TOKENS = {
-        "query_embed_token": "<|rerank_token|>",
-        "doc_embed_token": "<|embed_token|>",
-    }
-    _DOC_TOKEN_ID = 151670
-    _QUERY_TOKEN_ID = 151671
-
-    def __init__(self, model_path: str, projector_path: str):
-        from mlx_lm import load
-
-        self.model, self.tokenizer = load(model_path)
-        self.model.eval()
-        self.projector = _load_projector(projector_path)
-
-    def rerank(self, query: str, documents: list[str], top_n: int | None = None) -> list[dict]:
-        """
-        Rank documents by relevance to a query.
-
-        Returns a list of dicts with keys: document, relevance_score, index.
-        Sorted by descending relevance_score.
-        """
-        import mlx.core as mx
-
-        prompt = _format_prompt(query, documents, self._SPECIAL_TOKENS)
-        input_ids = self.tokenizer.encode(prompt)
-        hidden_states = self.model.model([input_ids])[0]  # [seq_len, hidden_size]
-
-        input_ids_np = np.array(input_ids)
-        query_positions = np.where(input_ids_np == self._QUERY_TOKEN_ID)[0]
-        doc_positions = np.where(input_ids_np == self._DOC_TOKEN_ID)[0]
-
-        if len(query_positions) == 0:
-            raise ValueError("Query embed token not found in prompt")
-        if len(doc_positions) == 0:
-            raise ValueError("Document embed tokens not found in prompt")
-
-        query_hidden = mx.expand_dims(hidden_states[int(query_positions[0])], axis=0)
-        doc_hidden = mx.stack([hidden_states[int(p)] for p in doc_positions])
-
-        query_emb = self.projector(query_hidden)  # [1, 512]
-        doc_emb = self.projector(doc_hidden)  # [num_docs, 512]
-
-        query_exp = mx.broadcast_to(mx.expand_dims(query_emb, 0), (1, len(documents), 512))
-        doc_exp = mx.expand_dims(doc_emb, 0)
-
-        scores = mx.sum(doc_exp * query_exp, axis=-1) / (
-            mx.sqrt(mx.sum(doc_exp * doc_exp, axis=-1)) * mx.sqrt(mx.sum(query_exp * query_exp, axis=-1))
-        )  # [1, num_docs]
-        scores_np = np.array(scores[0])
-
-        order = np.argsort(scores_np)[::-1]
-        n = min(top_n, len(documents)) if top_n is not None else len(documents)
-        return [
-            {
-                "document": documents[order[i]],
-                "relevance_score": float(scores_np[order[i]]),
-                "index": int(order[i]),
-            }
-            for i in range(n)
-        ]

hindsight-api-slim/hindsight_api/engine/operation_metadata.py

@@ -1,69 +0,0 @@
-"""
-Typed metadata models for async operations.
-
-These dataclasses define the structure of result_metadata for different operation types.
-The metadata is exposed in the API for debugging purposes and may change without notice.
-"""
-
-from dataclasses import asdict, dataclass
-from typing import Any
-
-
-@dataclass
-class BatchRetainParentMetadata:
-    """Metadata for parent batch_retain operations (when split into sub-batches)."""
-
-    items_count: int
-    total_tokens: int
-    num_sub_batches: int
-    is_parent: bool = True
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert to dict for JSON serialization."""
-        return asdict(self)
-
-
-@dataclass
-class BatchRetainChildMetadata:
-    """Metadata for child batch_retain operations (individual sub-batches)."""
-
-    items_count: int
-    parent_operation_id: str
-    sub_batch_index: int
-    total_sub_batches: int
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert to dict for JSON serialization."""
-        return asdict(self)
-
-
-@dataclass
-class RetainMetadata:
-    """Metadata for regular retain operations (non-batched, deprecated async path)."""
-
-    items_count: int
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert to dict for JSON serialization."""
-        return asdict(self)
-
-
-@dataclass
-class ConsolidationMetadata:
-    """Metadata for consolidation operations."""
-
-    # Currently empty, but structure for future fields
-    def to_dict(self) -> dict[str, Any]:
-        """Convert to dict for JSON serialization."""
-        return asdict(self)
-
-
-@dataclass
-class RefreshMentalModelMetadata:
-    """Metadata for mental model refresh operations."""
-
-    mental_model_id: str
-
-    def to_dict(self) -> dict[str, Any]:
-        """Convert to dict for JSON serialization."""
-        return asdict(self)

hindsight-api-slim/hindsight_api/engine/parsers/__init__.py

@@ -1,128 +0,0 @@
-"""File parser implementations."""
-
-import logging
-from dataclasses import dataclass
-
-from .base import FileParser, UnsupportedFileTypeError
-from .iris import IrisParser
-from .markitdown import MarkitdownParser
-
-__all__ = [
-    "FileParser",
-    "UnsupportedFileTypeError",
-    "IrisParser",
-    "MarkitdownParser",
-    "FileParserRegistry",
-    "ConvertResult",
-]
-
-
-@dataclass
-class ConvertResult:
-    """Result of a successful file conversion."""
-
-    content: str
-    parser_name: str
-
-
-logger = logging.getLogger(__name__)
-
-
-class FileParserRegistry:
-    """Registry for file parsers with auto-detection."""
-
-    def __init__(self):
-        """Initialize empty parser registry."""
-        self._parsers: dict[str, FileParser] = {}
-
-    def register(self, parser: FileParser):
-        """
-        Register a parser.
-
-        Args:
-            parser: FileParser instance
-        """
-        self._parsers[parser.name()] = parser
-
-    def get_parser(
-        self,
-        name: str | None,
-        filename: str,
-        content_type: str | None = None,
-    ) -> FileParser:
-        """
-        Get parser by name or auto-detect.
-
-        Args:
-            name: Parser name (e.g., "markitdown") or None for auto-detect
-            filename: File name for auto-detection
-            content_type: MIME type (optional)
-
-        Returns:
-            FileParser instance
-
-        Raises:
-            ValueError: If no suitable parser found
-        """
-        if name:
-            # Explicit parser requested — return it directly, let the parser
-            # raise UnsupportedFileTypeError from convert() if needed
-            if name not in self._parsers:
-                raise ValueError(f"Parser '{name}' not found. Available: {list(self._parsers.keys())}")
-            return self._parsers[name]
-
-        # Auto-detect parser
-        for parser in self._parsers.values():
-            if parser.supports(filename, content_type):
-                return parser
-
-        raise ValueError(f"No parser found for {filename}. Available parsers: {list(self._parsers.keys())}")
-
-    async def convert_with_fallback(
-        self,
-        parsers: list[str],
-        file_data: bytes,
-        filename: str,
-        content_type: str | None = None,
-    ) -> ConvertResult:
-        """
-        Try each parser in order, falling back on failure or empty content.
-
-        Moves to the next parser if the current one raises UnsupportedFileTypeError
-        or returns empty content. Any other exception (RuntimeError, network error,
-        etc.) also triggers a fallback so the chain is exhausted before failing.
-
-        Args:
-            parsers: Ordered list of parser names to try
-            file_data: Raw file bytes
-            filename: Original filename
-            content_type: MIME type (optional)
-
-        Returns:
-            ConvertResult with the parsed content and the name of the parser that succeeded
-
-        Raises:
-            ValueError: If a parser name is not registered
-            RuntimeError: If all parsers fail or return empty content
-        """
-        last_error: Exception | None = None
-        for name in parsers:
-            parser = self.get_parser(name, filename, content_type)
-            try:
-                content = await parser.convert(file_data, filename)
-                if content and content.strip():
-                    return ConvertResult(content=content, parser_name=name)
-                logger.warning(f"Parser '{name}' returned empty content for '{filename}', trying next")
-                last_error = RuntimeError(f"Parser '{name}' returned no content for '{filename}'")
-            except UnsupportedFileTypeError as e:
-                logger.warning(f"Parser '{name}' does not support '{filename}', trying next: {e}")
-                last_error = e
-            except Exception as e:
-                logger.warning(f"Parser '{name}' failed for '{filename}', trying next: {e}")
-                last_error = e
-
-        raise last_error or RuntimeError(f"No parsers available for '{filename}'")
-
-    def list_parsers(self) -> list[str]:
-        """Get list of registered parser names."""
-        return list(self._parsers.keys())

hindsight-api-slim/hindsight_api/engine/parsers/base.py

@@ -1,58 +0,0 @@
-"""Abstract base class for file parsers."""
-
-from abc import ABC, abstractmethod
-
-
-class UnsupportedFileTypeError(Exception):
-    """Raised by a parser when it does not support the given file type."""
-
-    pass
-
-
-class FileParser(ABC):
-    """Abstract base for file to markdown parsers."""
-
-    @abstractmethod
-    async def convert(self, file_data: bytes, filename: str) -> str:
-        """
-        Parse file to markdown.
-
-        Args:
-            file_data: Raw file bytes
-            filename: Original filename (used for format detection)
-
-        Returns:
-            Markdown content as string
-
-        Raises:
-            UnsupportedFileTypeError: If the file type is not supported by this parser
-            RuntimeError: If parsing fails for another reason
-        """
-        pass
-
-    def supports(self, filename: str, content_type: str | None = None) -> bool:
-        """
-        Check if parser supports this file type.
-
-        Override this for local/static extension-based filtering.
-        Parsers that delegate to a remote service should leave this as True
-        and raise UnsupportedFileTypeError from convert() instead.
-
-        Args:
-            filename: File name (used for extension check)
-            content_type: MIME type (optional)
-
-        Returns:
-            True if this parser can handle the file (default: True)
-        """
-        return True
-
-    @abstractmethod
-    def name(self) -> str:
-        """
-        Get parser name.
-
-        Returns:
-            Parser name (e.g., "markitdown")
-        """
-        pass

hindsight-api-slim/hindsight_api/engine/parsers/iris.py

@@ -1,138 +0,0 @@
-"""Iris parser implementation using the Vectorize Iris HTTP API."""
-
-import asyncio
-import logging
-import mimetypes
-import time
-
-import httpx
-
-from .base import FileParser, UnsupportedFileTypeError
-
-logger = logging.getLogger(__name__)
-
-_IRIS_BASE_URL = "https://api.vectorize.io/v1"
-_DEFAULT_POLL_INTERVAL = 2.0  # seconds
-_DEFAULT_TIMEOUT = 300.0  # seconds
-
-
-class IrisParser(FileParser):
-    """
-    Iris file parser using the Vectorize Iris cloud extraction service.
-
-    Uploads files to the Vectorize Iris API, starts an extraction job,
-    and polls until the text is ready. The API determines which file types
-    are supported — UnsupportedFileTypeError is raised if the file is rejected.
-
-    Authentication:
-        Requires HINDSIGHT_API_FILE_PARSER_IRIS_TOKEN and
-        HINDSIGHT_API_FILE_PARSER_IRIS_ORG_ID environment variables,
-        or pass them explicitly via the constructor.
-    """
-
-    def __init__(
-        self,
-        token: str,
-        org_id: str,
-        poll_interval: float = _DEFAULT_POLL_INTERVAL,
-        timeout: float = _DEFAULT_TIMEOUT,
-    ):
-        """
-        Initialize iris parser.
-
-        Args:
-            token: Vectorize API token
-            org_id: Vectorize organization ID
-            poll_interval: Seconds between status poll requests (default: 2)
-            timeout: Maximum seconds to wait for extraction (default: 300)
-        """
-        self._token = token
-        self._org_id = org_id
-        self._poll_interval = poll_interval
-        self._timeout = timeout
-        self._auth_headers = {"Authorization": f"Bearer {token}"}
-
-    async def convert(self, file_data: bytes, filename: str) -> str:
-        """
-        Parse file to text using the Vectorize Iris API.
-
-        Raises:
-            UnsupportedFileTypeError: If the Iris API rejects the file type (4xx)
-            RuntimeError: If extraction fails for another reason
-        """
-        content_type = mimetypes.guess_type(filename)[0] or "application/octet-stream"
-
-        async with httpx.AsyncClient(timeout=httpx.Timeout(30.0, read=120.0)) as client:
-            # Step 1: Request a presigned upload URL
-            init_resp = await client.post(
-                f"{_IRIS_BASE_URL}/org/{self._org_id}/files",
-                headers=self._auth_headers,
-                json={"name": filename, "contentType": content_type},
-            )
-            _raise_for_status(init_resp, filename, "file upload init")
-            init_data = init_resp.json()
-            file_id: str = init_data["fileId"]
-            upload_url: str = init_data["uploadUrl"]
-
-            # Step 2: Upload the file bytes to the presigned URL (no auth header)
-            # Ensure file_data is plain bytes (GCS storage may return obstore.Bytes)
-            upload_resp = await client.put(
-                upload_url,
-                content=bytes(file_data),
-                headers={"Content-Type": content_type},
-            )
-            _raise_for_status(upload_resp, filename, "file upload")
-
-            # Step 3: Start extraction
-            extract_resp = await client.post(
-                f"{_IRIS_BASE_URL}/org/{self._org_id}/extraction",
-                headers=self._auth_headers,
-                json={"fileId": file_id},
-            )
-            _raise_for_status(extract_resp, filename, "start extraction")
-            extraction_id: str = extract_resp.json()["extractionId"]
-
-            # Step 4: Poll until ready or timeout
-            deadline = time.monotonic() + self._timeout
-            while True:
-                status_resp = await client.get(
-                    f"{_IRIS_BASE_URL}/org/{self._org_id}/extraction/{extraction_id}",
-                    headers=self._auth_headers,
-                )
-                _raise_for_status(status_resp, filename, "poll extraction status")
-                status_data = status_resp.json()
-
-                if status_data.get("ready"):
-                    data = status_data.get("data", {})
-                    if not data.get("success"):
-                        error = data.get("error", "unknown error")
-                        raise RuntimeError(f"Iris extraction failed for '{filename}': {error}")
-                    text = data.get("text")
-                    if not text:
-                        raise RuntimeError(f"No content extracted from '{filename}'")
-                    return text
-
-                if time.monotonic() >= deadline:
-                    raise RuntimeError(f"Iris extraction timed out after {self._timeout}s for '{filename}'")
-
-                await asyncio.sleep(self._poll_interval)
-
-    def name(self) -> str:
-        """Get parser name."""
-        return "iris"
-
-
-def _raise_for_status(response: httpx.Response, filename: str, step: str) -> None:
-    """
-    Raise an appropriate error including the response body on HTTP errors.
-
-    Raises UnsupportedFileTypeError for 4xx responses (file rejected by the API),
-    RuntimeError for other HTTP errors.
-    """
-    if not response.is_error:
-        return
-    body = response.text or "<empty>"
-    msg = f"Iris API error during {step} for '{filename}': {response.status_code} {response.reason_phrase} — {body}"
-    if response.is_client_error:
-        raise UnsupportedFileTypeError(msg)
-    raise RuntimeError(msg)

hindsight-api-slim/hindsight_api/engine/parsers/markitdown.py

@@ -1,109 +0,0 @@
-"""Markitdown parser implementation."""
-
-import asyncio
-import logging
-import tempfile
-from pathlib import Path
-
-from .base import FileParser
-
-logger = logging.getLogger(__name__)
-
-
-class MarkitdownParser(FileParser):
-    """
-    Markitdown file parser.
-
-    Uses Microsoft's markitdown library to convert various file formats
-    to markdown including PDF, Office docs, images (via OCR), audio, HTML.
-
-    Supported formats:
-    - PDF (.pdf)
-    - Word (.docx, .doc)
-    - PowerPoint (.pptx, .ppt)
-    - Excel (.xlsx, .xls)
-    - Images (.jpg, .jpeg, .png) - with OCR
-    - HTML (.html, .htm)
-    - Text (.txt, .md)
-    - Audio (.mp3, .wav) - with transcription
-    """
-
-    def __init__(self):
-        """Initialize markitdown parser."""
-        # Lazy import to avoid requiring markitdown for all users
-        try:
-            from markitdown import MarkItDown
-
-            self._markitdown = MarkItDown()
-        except ImportError as e:
-            raise ImportError(
-                "markitdown package is required for file parsing. Install with: pip install markitdown"
-            ) from e
-
-    async def convert(self, file_data: bytes, filename: str) -> str:
-        """Parse file to markdown using markitdown."""
-        # markitdown is synchronous, so we run it in executor to avoid blocking
-        loop = asyncio.get_event_loop()
-        return await loop.run_in_executor(None, self._convert_sync, file_data, filename)
-
-    def _convert_sync(self, file_data: bytes, filename: str) -> str:
-        """Synchronous parsing (runs in thread pool)."""
-        # Write to temp file (markitdown requires file path)
-        with tempfile.NamedTemporaryFile(suffix=Path(filename).suffix, delete=False) as tmp:
-            tmp.write(file_data)
-            tmp_path = tmp.name
-
-        try:
-            # Parse using markitdown
-            result = self._markitdown.convert(tmp_path)
-
-            if not result or not result.text_content:
-                raise RuntimeError(f"No content extracted from '{filename}'")
-
-            return result.text_content
-
-        except Exception as e:
-            logger.error(f"Markitdown parsing failed for {filename}: {e}")
-            raise RuntimeError(f"Failed to parse '{filename}': {e}") from e
-
-        finally:
-            # Clean up temp file
-            try:
-                Path(tmp_path).unlink()
-            except Exception:
-                pass
-
-    def supports(self, filename: str, content_type: str | None = None) -> bool:
-        """Check if markitdown supports this file type."""
-        # Supported extensions (from markitdown docs)
-        supported_extensions = {
-            # Documents
-            ".pdf",
-            ".docx",
-            ".doc",
-            ".pptx",
-            ".ppt",
-            ".xlsx",
-            ".xls",
-            # Images (with OCR)
-            ".jpg",
-            ".jpeg",
-            ".png",
-            # Web
-            ".html",
-            ".htm",
-            # Text
-            ".txt",
-            ".md",
-            ".csv",
-            # Audio (with transcription)
-            ".mp3",
-            ".wav",
-        }
-
-        ext = Path(filename).suffix.lower()
-        return ext in supported_extensions
-
-    def name(self) -> str:
-        """Get parser name."""
-        return "markitdown"

hindsight-api-slim/hindsight_api/engine/providers/litellm_llm.py

@@ -1,385 +0,0 @@
-"""
-LiteLLM LLM provider for universal model support.
-
-This provider enables using 100+ LLM providers via the LiteLLM SDK, including:
-- AWS Bedrock (bedrock/anthropic.claude-3-5-sonnet-...)
-- Azure OpenAI (azure/gpt-4o)
-- Together AI (together_ai/meta-llama/...)
-- Any other LiteLLM-supported provider
-
-Uses litellm.acompletion() for async chat completions.
-Authentication for cloud providers (e.g., AWS Bedrock via boto3 credential chain)
-is handled automatically by LiteLLM.
-"""
-
-import asyncio
-import json
-import logging
-import time
-from typing import Any
-
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
-from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
-from hindsight_api.metrics import get_metrics_collector
-from hindsight_api.worker.stage import set_stage
-
-logger = logging.getLogger(__name__)
-
-
-class LiteLLMLLM(LLMInterface):
-    """
-    LLM provider using the LiteLLM SDK for universal model support.
-
-    Supports any model accessible via litellm.acompletion(), including AWS Bedrock,
-    Azure OpenAI, Together AI, Fireworks AI, and more.
-
-    Model names follow LiteLLM conventions with provider prefixes:
-    - bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
-    - azure/gpt-4o
-    - together_ai/meta-llama/Llama-3-70b-chat-hf
-    - fireworks_ai/accounts/fireworks/models/llama-v3p1-70b-instruct
-    """
-
-    def __init__(
-        self,
-        provider: str,
-        api_key: str,
-        base_url: str,
-        model: str,
-        reasoning_effort: str = "low",
-        timeout: float = 300.0,
-        **kwargs: Any,
-    ):
-        super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
-        self.timeout = timeout
-        self._litellm: Any = None
-
-        try:
-            import litellm
-
-            self._litellm = litellm
-            # Suppress LiteLLM's verbose logging
-            litellm.suppress_debug_info = True  # type: ignore[assignment]
-            # Drop unsupported params instead of raising errors (e.g. tool_choice on some Bedrock models)
-            litellm.drop_params = True  # type: ignore[assignment]
-            logging.getLogger("LiteLLM").setLevel(logging.WARNING)
-            logger.info(f"LiteLLM SDK initialized for model: {self.model}")
-        except ImportError as e:
-            raise RuntimeError("LiteLLM SDK not installed. Run: uv add litellm or pip install litellm") from e
-
-    async def verify_connection(self) -> None:
-        try:
-            test_messages = [{"role": "user", "content": "test"}]
-            await self.call(
-                messages=test_messages,
-                max_completion_tokens=50,
-                temperature=0.0,
-                scope="verification",
-                max_retries=0,
-            )
-            logger.info("LiteLLM connection verified successfully")
-        except OutputTooLongError:
-            # Truncation is fine for verification — it means the connection works
-            logger.info("LiteLLM connection verified successfully (response truncated)")
-        except Exception as e:
-            logger.error(f"LiteLLM connection verification failed: {e}")
-            raise RuntimeError(f"Failed to verify LiteLLM connection: {e}") from e
-
-    def _build_common_kwargs(
-        self,
-        messages: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-    ) -> dict[str, Any]:
-        """Build common kwargs for litellm calls."""
-        kwargs: dict[str, Any] = {
-            "model": self.model,
-            "messages": messages,
-            "timeout": self.timeout,
-        }
-
-        if self.api_key:
-            kwargs["api_key"] = self.api_key
-        if self.base_url:
-            kwargs["api_base"] = self.base_url
-        if max_completion_tokens is not None:
-            kwargs["max_completion_tokens"] = max_completion_tokens
-        if temperature is not None:
-            kwargs["temperature"] = temperature
-
-        return kwargs
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        start_time = time.time()
-
-        call_kwargs = self._build_common_kwargs(messages, max_completion_tokens, temperature)
-
-        # Add JSON schema response format if provided
-        if response_format is not None and hasattr(response_format, "model_json_schema"):
-            schema = response_format.model_json_schema()
-            call_kwargs["response_format"] = {
-                "type": "json_schema",
-                "json_schema": {
-                    "name": response_format.__name__ if hasattr(response_format, "__name__") else "response",
-                    "schema": schema,
-                    "strict": strict_schema,
-                },
-            }
-
-        last_exception = None
-
-        for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.litellm.{scope}.attempt={attempt + 1}/{max_retries + 1}")
-            try:
-                response = await self._litellm.acompletion(**call_kwargs)
-
-                content = response.choices[0].message.content or ""
-                finish_reason = response.choices[0].finish_reason
-
-                # Check for length-limited output
-                if finish_reason == "length":
-                    raise OutputTooLongError("LiteLLM response was truncated due to token limit")
-
-                if response_format is not None:
-                    # Strip markdown code fences if present
-                    clean_content = content
-                    if "```json" in content:
-                        clean_content = content.split("```json")[1].split("```")[0].strip()
-                    elif "```" in content:
-                        clean_content = content.split("```")[1].split("```")[0].strip()
-
-                    try:
-                        json_data = json.loads(clean_content)
-                    except json.JSONDecodeError:
-                        json_data = json.loads(content)
-
-                    if skip_validation:
-                        result = json_data
-                    else:
-                        result = response_format.model_validate(json_data)
-                else:
-                    result = content
-
-                # Extract usage
-                input_tokens = getattr(response.usage, "prompt_tokens", 0) or 0
-                output_tokens = getattr(response.usage, "completion_tokens", 0) or 0
-                total_tokens = input_tokens + output_tokens
-
-                # Record metrics
-                duration = time.time() - start_time
-                metrics = get_metrics_collector()
-                metrics.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    duration=duration,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    success=True,
-                )
-
-                # Record trace span
-                from hindsight_api.tracing import _serialize_for_span, get_span_recorder
-
-                span_recorder = get_span_recorder()
-                span_recorder.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    messages=messages,
-                    response_content=_serialize_for_span(result),
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    duration=duration,
-                    finish_reason=finish_reason,
-                    error=None,
-                )
-
-                if duration > 10.0:
-                    logger.info(
-                        f"slow llm call: scope={scope}, model={self.provider}/{self.model}, "
-                        f"input_tokens={input_tokens}, output_tokens={output_tokens}, "
-                        f"time={duration:.3f}s"
-                    )
-
-                if return_usage:
-                    token_usage = TokenUsage(
-                        input_tokens=input_tokens,
-                        output_tokens=output_tokens,
-                        total_tokens=total_tokens,
-                    )
-                    return result, token_usage
-                return result
-
-            except OutputTooLongError:
-                raise
-
-            except json.JSONDecodeError as e:
-                last_exception = e
-                if attempt < max_retries:
-                    logger.warning("LiteLLM returned invalid JSON, retrying...")
-                    backoff = min(initial_backoff * (2**attempt), max_backoff)
-                    await asyncio.sleep(backoff)
-                    continue
-                else:
-                    logger.error(f"LiteLLM returned invalid JSON after {max_retries + 1} attempts")
-                    raise
-
-            except Exception as e:
-                error_str = str(e).lower()
-                # Fast fail on auth errors
-                if "401" in error_str or "403" in error_str or "unauthorized" in error_str:
-                    logger.error(f"LiteLLM auth error, not retrying: {e}")
-                    raise
-
-                last_exception = e
-                if attempt < max_retries:
-                    # Retry on rate limits, connection errors, server errors
-                    is_retryable = any(
-                        keyword in error_str
-                        for keyword in ("rate", "limit", "timeout", "connection", "500", "502", "503", "529")
-                    )
-                    if is_retryable:
-                        backoff = min(initial_backoff * (2**attempt), max_backoff)
-                        jitter = backoff * 0.2 * (2 * (time.time() % 1) - 1)
-                        await asyncio.sleep(backoff + jitter)
-                        continue
-
-                logger.error(f"LiteLLM API error after {attempt + 1} attempts: {e}")
-                raise
-
-        if last_exception:
-            raise last_exception
-        raise RuntimeError("LiteLLM call failed after all retries")
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        start_time = time.time()
-
-        call_kwargs = self._build_common_kwargs(messages, max_completion_tokens, temperature)
-        call_kwargs["tools"] = tools
-        call_kwargs["tool_choice"] = tool_choice
-
-        last_exception = None
-        for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.litellm.tools.attempt={attempt + 1}/{max_retries + 1}")
-            try:
-                response = await self._litellm.acompletion(**call_kwargs)
-
-                message = response.choices[0].message
-                content = message.content
-                finish_reason = response.choices[0].finish_reason
-
-                # Extract tool calls
-                tool_calls: list[LLMToolCall] = []
-                if message.tool_calls:
-                    for tc in message.tool_calls:
-                        arguments = tc.function.arguments
-                        if isinstance(arguments, str):
-                            arguments = json.loads(arguments)
-                        tool_calls.append(
-                            LLMToolCall(
-                                id=tc.id,
-                                name=tc.function.name,
-                                arguments=arguments,
-                            )
-                        )
-
-                # Extract usage
-                input_tokens = getattr(response.usage, "prompt_tokens", 0) or 0
-                output_tokens = getattr(response.usage, "completion_tokens", 0) or 0
-
-                # Record metrics
-                duration = time.time() - start_time
-                metrics = get_metrics_collector()
-                metrics.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    duration=duration,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    success=True,
-                )
-
-                # Record trace span
-                from hindsight_api.tracing import get_span_recorder
-
-                span_recorder = get_span_recorder()
-                tool_calls_dict = (
-                    [{"id": tc.id, "name": tc.name, "arguments": tc.arguments} for tc in tool_calls]
-                    if tool_calls
-                    else None
-                )
-                span_recorder.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    messages=messages,
-                    response_content=content,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    duration=duration,
-                    finish_reason=finish_reason,
-                    error=None,
-                    tool_calls=tool_calls_dict,
-                )
-
-                return LLMToolCallResult(
-                    content=content,
-                    tool_calls=tool_calls,
-                    finish_reason=finish_reason or ("tool_calls" if tool_calls else "stop"),
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                )
-
-            except Exception as e:
-                error_str = str(e).lower()
-                if "401" in error_str or "403" in error_str or "unauthorized" in error_str:
-                    raise
-
-                last_exception = e
-                if attempt < max_retries:
-                    is_retryable = any(
-                        keyword in error_str
-                        for keyword in ("rate", "limit", "timeout", "connection", "500", "502", "503", "529")
-                    )
-                    if is_retryable:
-                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
-                        continue
-
-                logger.error(f"LiteLLM tool call error after {attempt + 1} attempts: {e}")
-                raise
-
-        if last_exception:
-            raise last_exception
-        raise RuntimeError("LiteLLM tool call failed after all retries")
-
-    async def cleanup(self) -> None:
-        """Clean up resources."""
-        pass

hindsight-api-slim/hindsight_api/engine/providers/llamacpp_llm.py

@@ -1,428 +0,0 @@
-"""
-Built-in llama.cpp LLM provider for fully offline operation.
-
-Manages a llama-cpp-python server as a subprocess, downloads GGUF models
-from HuggingFace on first use, and delegates inference to the OpenAI-compatible API.
-
-Usage:
-    HINDSIGHT_API_LLM_PROVIDER=llamacpp
-    HINDSIGHT_API_LLAMACPP_MODEL_PATH=~/.hindsight/models/gemma-4-E2B-it-Q4_K_M.gguf
-    HINDSIGHT_API_LLAMACPP_GPU_LAYERS=-1  # -1 = all layers on GPU
-    HINDSIGHT_API_LLAMACPP_CONTEXT_SIZE=8192
-"""
-
-import asyncio
-import logging
-import os
-import signal
-import socket
-import subprocess
-import sys
-import time
-from pathlib import Path
-from typing import Any
-
-from hindsight_api.engine.llm_interface import LLMInterface
-from hindsight_api.engine.response_models import LLMToolCallResult
-
-logger = logging.getLogger(__name__)
-
-# Default GGUF model for offline mode
-DEFAULT_LLAMACPP_HF_REPO = "bartowski/google_gemma-4-E2B-it-GGUF"
-DEFAULT_LLAMACPP_HF_FILENAME = "google_gemma-4-E2B-it-Q4_K_M.gguf"
-DEFAULT_LLAMACPP_MODEL_ALIAS = "gemma-4-e2b-it"
-
-MODELS_DIR = Path.home() / ".hindsight" / "models"
-
-# Singleton server instance — shared across all LlamaCppLLM instances
-# (retain, reflect, consolidation each create their own LLMProvider,
-# but they should all share one llama.cpp server process)
-_shared_server: "LlamaCppServer | None" = None
-_shared_server_lock = asyncio.Lock()
-
-
-def _find_free_port() -> int:
-    """Find a free TCP port on localhost."""
-    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-        s.bind(("127.0.0.1", 0))
-        return s.getsockname()[1]
-
-
-def _download_default_model() -> Path:
-    """Download the default GGUF model from HuggingFace if not already cached.
-
-    Returns:
-        Path to the downloaded GGUF file.
-    """
-    try:
-        from huggingface_hub import hf_hub_download
-    except ImportError:
-        raise ImportError(
-            "huggingface-hub is required for automatic model download. "
-            "Install with: pip install 'hindsight-api-slim[local-llm]'"
-        )
-
-    MODELS_DIR.mkdir(parents=True, exist_ok=True)
-    target = MODELS_DIR / DEFAULT_LLAMACPP_HF_FILENAME
-
-    if target.exists():
-        logger.info(f"Using cached model: {target}")
-        return target
-
-    logger.info(
-        f"Downloading {DEFAULT_LLAMACPP_HF_FILENAME} from {DEFAULT_LLAMACPP_HF_REPO} (~3.5 GB, first run only)..."
-    )
-
-    downloaded = hf_hub_download(
-        repo_id=DEFAULT_LLAMACPP_HF_REPO,
-        filename=DEFAULT_LLAMACPP_HF_FILENAME,
-        local_dir=str(MODELS_DIR),
-    )
-
-    logger.info(f"Model downloaded: {downloaded}")
-    return Path(downloaded)
-
-
-def _resolve_model_path(model_path: str | None) -> Path:
-    """Resolve the model path, downloading the default if needed.
-
-    Args:
-        model_path: Explicit path to a GGUF file, or None to use the default.
-
-    Returns:
-        Resolved Path to the GGUF file.
-    """
-    if model_path:
-        p = Path(model_path).expanduser()
-        if not p.exists():
-            raise FileNotFoundError(
-                f"GGUF model not found: {p}\n"
-                f"Set HINDSIGHT_API_LLAMACPP_MODEL_PATH to a valid .gguf file, "
-                f"or remove the setting to auto-download the default model."
-            )
-        return p
-
-    return _download_default_model()
-
-
-class LlamaCppServer:
-    """Manages a llama-cpp-python OpenAI-compatible server as a subprocess."""
-
-    def __init__(
-        self,
-        model_path: Path,
-        port: int,
-        gpu_layers: int = -1,
-        context_size: int = 8192,
-        chat_format: str | None = None,
-        extra_args: str | None = None,
-    ):
-        self.model_path = model_path
-        self.port = port
-        self.gpu_layers = gpu_layers
-        self.context_size = context_size
-        self.chat_format = chat_format
-        self.extra_args = extra_args
-        self._process: subprocess.Popen | None = None
-
-    @property
-    def base_url(self) -> str:
-        return f"http://127.0.0.1:{self.port}/v1"
-
-    async def start(self) -> None:
-        """Start the llama.cpp server subprocess."""
-        cmd = [
-            sys.executable,
-            "-m",
-            "llama_cpp.server",
-            "--model",
-            str(self.model_path),
-            "--host",
-            "127.0.0.1",
-            "--port",
-            str(self.port),
-            "--n_gpu_layers",
-            str(self.gpu_layers),
-            "--n_ctx",
-            str(self.context_size),
-            "--flash_attn",
-            "true",
-            "--n_batch",
-            "2048",
-            # Prompt cache: reuse KV cache for repeated system prompts
-            "--cache",
-            "true",
-        ]
-        # Only pass chat_format if explicitly set (most GGUF models have it embedded)
-        if self.chat_format:
-            cmd.extend(["--chat_format", self.chat_format])
-        # User-provided extra args (e.g. "--type_k 1 --type_v 1 --n_threads 8")
-        if self.extra_args:
-            cmd.extend(self.extra_args.split())
-
-        logger.info(f"Starting llama.cpp server: {' '.join(cmd)}")
-
-        # Write stderr to a log file to avoid pipe buffer deadlock
-        # (llama.cpp outputs a lot of model metadata on stderr during loading)
-        self._log_path = MODELS_DIR / "llamacpp_server.log"
-        self._log_file = open(self._log_path, "w")
-
-        self._process = subprocess.Popen(
-            cmd,
-            stdout=subprocess.DEVNULL,
-            stderr=self._log_file,
-            # Ensure the subprocess is killed when the parent exits
-            preexec_fn=os.setsid if hasattr(os, "setsid") else None,
-        )
-
-        # Wait for the server to be ready
-        await self._wait_for_ready()
-
-    async def _wait_for_ready(self, timeout: float = 120.0) -> None:
-        """Wait for the llama.cpp server to accept connections."""
-        import httpx
-
-        start = time.monotonic()
-        url = f"http://127.0.0.1:{self.port}/v1/models"
-        last_log = start
-
-        while time.monotonic() - start < timeout:
-            # Check if process died
-            if self._process and self._process.poll() is not None:
-                stderr = ""
-                try:
-                    stderr = self._log_path.read_text()[-2000:]
-                except Exception:
-                    pass
-                raise RuntimeError(f"llama.cpp server exited with code {self._process.returncode}.\nstderr: {stderr}")
-
-            try:
-                async with httpx.AsyncClient() as client:
-                    resp = await client.get(url, timeout=5.0)
-                    if resp.status_code == 200:
-                        logger.info(f"llama.cpp server ready on port {self.port}")
-                        return
-            except (httpx.ConnectError, httpx.TimeoutException, httpx.ConnectTimeout):
-                pass
-
-            # Log progress every 15s
-            now = time.monotonic()
-            if now - last_log > 15:
-                elapsed = int(now - start)
-                logger.info(f"Waiting for llama.cpp server to load model... ({elapsed}s)")
-                last_log = now
-
-            await asyncio.sleep(1.0)
-
-        # Timeout — read the log to help debug
-        stderr = ""
-        try:
-            stderr = self._log_path.read_text()[-2000:]
-        except Exception:
-            pass
-        raise TimeoutError(
-            f"llama.cpp server did not become ready within {timeout}s.\n"
-            f"Check model compatibility and available memory.\n"
-            f"Server log: {stderr}"
-        )
-
-    async def stop(self) -> None:
-        """Stop the llama.cpp server subprocess."""
-        if self._process is None:
-            return
-
-        logger.info("Stopping llama.cpp server...")
-        try:
-            # Send SIGTERM to the process group
-            if hasattr(os, "killpg"):
-                os.killpg(os.getpgid(self._process.pid), signal.SIGTERM)
-            else:
-                self._process.terminate()
-
-            # Wait up to 10s for graceful shutdown
-            try:
-                self._process.wait(timeout=10)
-            except subprocess.TimeoutExpired:
-                if hasattr(os, "killpg"):
-                    os.killpg(os.getpgid(self._process.pid), signal.SIGKILL)
-                else:
-                    self._process.kill()
-                self._process.wait(timeout=5)
-        except (ProcessLookupError, OSError):
-            pass  # Process already exited
-        finally:
-            self._process = None
-            if hasattr(self, "_log_file") and self._log_file:
-                self._log_file.close()
-                self._log_file = None
-            logger.info("llama.cpp server stopped")
-
-
-class LlamaCppLLM(LLMInterface):
-    """
-    Built-in llama.cpp provider.
-
-    Manages a llama-cpp-python server subprocess and delegates to OpenAICompatibleLLM
-    for actual inference calls. Handles model downloading and server lifecycle.
-    """
-
-    def __init__(
-        self,
-        provider: str,
-        api_key: str,
-        base_url: str,
-        model: str,
-        reasoning_effort: str = "low",
-        model_path: str | None = None,
-        gpu_layers: int = -1,
-        context_size: int = 8192,
-        chat_format: str | None = None,
-        no_grammar: bool = False,
-        extra_args: str | None = None,
-        **kwargs: Any,
-    ):
-        super().__init__(
-            provider=provider,
-            api_key=api_key or "llamacpp",
-            base_url=base_url or "",
-            model=model or DEFAULT_LLAMACPP_MODEL_ALIAS,
-            reasoning_effort=reasoning_effort,
-        )
-        self._model_path_str = model_path
-        self._gpu_layers = gpu_layers
-        self._context_size = context_size
-        self._chat_format = chat_format
-        self._no_grammar = no_grammar
-        self._extra_args = extra_args
-        self._server: LlamaCppServer | None = None
-        self._delegate: Any = None  # OpenAICompatibleLLM, created after server starts
-        self._initialized = False
-
-    async def _ensure_initialized(self) -> None:
-        """Lazy initialization: download model + start shared server on first use."""
-        if self._initialized:
-            return
-
-        global _shared_server
-
-        from .openai_compatible_llm import OpenAICompatibleLLM
-
-        async with _shared_server_lock:
-            if _shared_server is None:
-                # Resolve and potentially download the model
-                model_path = _resolve_model_path(self._model_path_str)
-                logger.info(f"Using GGUF model: {model_path}")
-
-                # Start the shared llama.cpp server
-                port = _find_free_port()
-                _shared_server = LlamaCppServer(
-                    model_path=model_path,
-                    port=port,
-                    gpu_layers=self._gpu_layers,
-                    context_size=self._context_size,
-                    chat_format=self._chat_format,
-                    extra_args=self._extra_args,
-                )
-                await _shared_server.start()
-
-        self._server = _shared_server
-
-        # Create the delegate that talks to the shared server's OpenAI-compatible API
-        if self._no_grammar:
-            logger.info("Grammar enforcement disabled (HINDSIGHT_API_LLAMACPP_NO_GRAMMAR=true)")
-        self._delegate = OpenAICompatibleLLM(
-            provider="llamacpp",
-            api_key="llamacpp",
-            base_url=self._server.base_url,
-            model=self.model,
-            reasoning_effort=self.reasoning_effort,
-        )
-
-        self._initialized = True
-
-    async def verify_connection(self) -> None:
-        """Verify the llama.cpp server is running and can generate text."""
-        await self._ensure_initialized()
-        # Make a simple test call to verify the model can actually generate
-        await self._delegate.call(
-            messages=[{"role": "user", "content": "Say 'ok'"}],
-            max_completion_tokens=10,
-            max_retries=2,
-            initial_backoff=0.5,
-            max_backoff=2.0,
-            scope="verification",
-        )
-        logger.info("llama.cpp LLM verification passed")
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        """Delegate call to the OpenAI-compatible API."""
-        await self._ensure_initialized()
-        return await self._delegate.call(
-            messages=messages,
-            response_format=response_format,
-            max_completion_tokens=max_completion_tokens,
-            temperature=temperature,
-            scope=scope,
-            max_retries=max_retries,
-            initial_backoff=initial_backoff,
-            max_backoff=max_backoff,
-            skip_validation=skip_validation,
-            strict_schema=strict_schema,
-            return_usage=return_usage,
-        )
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        """Delegate tool calls to the OpenAI-compatible API."""
-        await self._ensure_initialized()
-        return await self._delegate.call_with_tools(
-            messages=messages,
-            tools=tools,
-            max_completion_tokens=max_completion_tokens,
-            temperature=temperature,
-            scope=scope,
-            max_retries=max_retries,
-            initial_backoff=initial_backoff,
-            max_backoff=max_backoff,
-            tool_choice=tool_choice,
-        )
-
-    async def cleanup(self) -> None:
-        """Stop the shared llama.cpp server."""
-        global _shared_server
-
-        if self._delegate:
-            await self._delegate.cleanup()
-            self._delegate = None
-
-        # Stop the shared server (only the first cleanup call actually stops it)
-        async with _shared_server_lock:
-            if _shared_server is not None:
-                await _shared_server.stop()
-                _shared_server = None
-
-        self._server = None
-        self._initialized = False

hindsight-api-slim/hindsight_api/engine/providers/none_llm.py

@@ -1,78 +0,0 @@
-"""
-No-op LLM provider for chunk-only storage mode.
-
-When the LLM provider is set to "none", the system operates without any LLM dependency.
-Retain uses chunks mode (no fact extraction), and reflect/consolidation are disabled.
-This provider acts as a safety net — if any code path unexpectedly tries to call the LLM,
-it raises a clear error instead of a confusing connection failure.
-"""
-
-import logging
-from typing import Any
-
-from ..llm_interface import LLMInterface
-from ..response_models import LLMToolCallResult
-
-logger = logging.getLogger(__name__)
-
-
-class LLMNotAvailableError(Exception):
-    """Raised when an operation requires an LLM but the provider is set to 'none'."""
-
-    pass
-
-
-class NoneLLM(LLMInterface):
-    """
-    No-op LLM provider that rejects all LLM calls.
-
-    Used when HINDSIGHT_API_LLM_PROVIDER=none to run Hindsight as a chunk store
-    with semantic search but without LLM-based features (fact extraction, reflect,
-    consolidation).
-    """
-
-    async def verify_connection(self) -> None:
-        """No-op — no LLM connection to verify."""
-        logger.debug("NoneLLM: no LLM connection to verify (provider=none)")
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        """Raise LLMNotAvailableError — no LLM is configured."""
-        raise LLMNotAvailableError(
-            "LLM provider is set to 'none'. This operation requires an LLM. "
-            "Set HINDSIGHT_API_LLM_PROVIDER to a real provider (e.g., openai, anthropic, gemini)."
-        )
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        """Raise LLMNotAvailableError — no LLM is configured."""
-        raise LLMNotAvailableError(
-            "LLM provider is set to 'none'. This operation requires an LLM. "
-            "Set HINDSIGHT_API_LLM_PROVIDER to a real provider (e.g., openai, anthropic, gemini)."
-        )
-
-    async def cleanup(self) -> None:
-        """No-op — nothing to clean up."""
-        pass

hindsight-api-slim/hindsight_api/engine/reflect/delta_ops.py

@@ -1,307 +0,0 @@
-"""Delta operations for structured mental models.
-
-The LLM's job during a delta refresh is to emit a list of these operations,
-each targeting an existing section (by id) or referencing a position relative
-to one.  ``apply_operations`` validates and applies each op in turn against a
-copy of the document; invalid ops (unknown ``section_id``, out-of-range
-``block_index``, malformed payloads) are dropped with a debug-friendly reason.
-
-Sections and blocks not mentioned by any op are physically copied through
-unchanged — there is no LLM-mediated re-emission of unchanged text, so prose
-drift is structurally impossible.
-
-Why operations and not "output the new structured doc":
-- "Output the new doc" still asks the LLM to *generate* every section's
-  blocks, including ones it didn't intend to modify, which gives it the same
-  opportunity to drift.
-- Operations make the no-change case mechanical: zero ops → identical doc.
-- Operations are auditable: each refresh produces a log of exactly what
-  changed, useful for debugging the LLM's behaviour and explaining diffs.
-
-Failure modes are by design conservative: an operation list that fails to
-parse against the Pydantic schema, or an LLM that returns invalid ops, results
-in zero changes — the document stays as-is. The structure can only get better
-or stay the same per refresh, never get worse.
-"""
-
-from __future__ import annotations
-
-import logging
-from typing import Annotated, Any, Literal, Union
-
-from pydantic import BaseModel, ConfigDict, Field
-
-from .structured_doc import (
-    Block,
-    Section,
-    StructuredDocument,
-    make_unique_id,
-    slugify_heading,
-)
-
-logger = logging.getLogger(__name__)
-
-
-# Op payloads ---------------------------------------------------------------
-
-
-class _OpBase(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-
-
-class AppendBlockOp(_OpBase):
-    """Add a new block at the end of an existing section."""
-
-    op: Literal["append_block"] = "append_block"
-    section_id: str
-    block: Block
-
-
-class InsertBlockOp(_OpBase):
-    """Insert a new block at ``index`` in an existing section.
-
-    ``index`` may equal ``len(section.blocks)`` (append) but not be greater.
-    """
-
-    op: Literal["insert_block"] = "insert_block"
-    section_id: str
-    index: int = Field(ge=0)
-    block: Block
-
-
-class ReplaceBlockOp(_OpBase):
-    """Replace the block at ``index`` of an existing section."""
-
-    op: Literal["replace_block"] = "replace_block"
-    section_id: str
-    index: int = Field(ge=0)
-    block: Block
-
-
-class RemoveBlockOp(_OpBase):
-    """Remove the block at ``index`` of an existing section."""
-
-    op: Literal["remove_block"] = "remove_block"
-    section_id: str
-    index: int = Field(ge=0)
-
-
-class AddSectionOp(_OpBase):
-    """Add a brand-new section.
-
-    ``after_section_id`` is optional; when omitted the new section is appended
-    at the end. ``new_id`` is optional; when omitted we slugify the heading
-    and disambiguate against existing IDs.
-    """
-
-    op: Literal["add_section"] = "add_section"
-    heading: str
-    level: int = Field(default=2, ge=1, le=6)
-    blocks: list[Block] = Field(default_factory=list)
-    after_section_id: str | None = None
-    new_id: str | None = None
-
-
-class RemoveSectionOp(_OpBase):
-    """Remove an entire section by id."""
-
-    op: Literal["remove_section"] = "remove_section"
-    section_id: str
-
-
-class ReplaceSectionBlocksOp(_OpBase):
-    """Replace all blocks of a section in one go.
-
-    Used when most of a section's contents are stale and rebuilding it as a
-    unit is clearer than emitting many block-level ops. The section's heading
-    and id are preserved.
-    """
-
-    op: Literal["replace_section_blocks"] = "replace_section_blocks"
-    section_id: str
-    blocks: list[Block] = Field(default_factory=list)
-
-
-class RenameSectionOp(_OpBase):
-    """Rename a section's heading. The id is unchanged so future ops still resolve."""
-
-    op: Literal["rename_section"] = "rename_section"
-    section_id: str
-    new_heading: str
-
-
-Operation = Annotated[
-    Union[
-        AppendBlockOp,
-        InsertBlockOp,
-        ReplaceBlockOp,
-        RemoveBlockOp,
-        AddSectionOp,
-        RemoveSectionOp,
-        ReplaceSectionBlocksOp,
-        RenameSectionOp,
-    ],
-    Field(discriminator="op"),
-]
-
-
-class DeltaOperationList(BaseModel):
-    """Container for the operations produced by an LLM delta call."""
-
-    model_config = ConfigDict(extra="forbid")
-    operations: list[Operation] = Field(default_factory=list)
-
-
-# Application ---------------------------------------------------------------
-
-
-class AppliedDelta(BaseModel):
-    """Outcome of applying a list of operations to a document."""
-
-    model_config = ConfigDict(extra="forbid")
-
-    document: StructuredDocument
-    applied: list[dict[str, Any]] = Field(default_factory=list)
-    skipped: list[dict[str, Any]] = Field(default_factory=list)
-
-    @property
-    def changed(self) -> bool:
-        return len(self.applied) > 0
-
-
-def _op_summary(op: Operation) -> dict[str, Any]:
-    """Compact dict suitable for the audit trail."""
-    data = op.model_dump()
-    return {k: v for k, v in data.items() if k != "block" and k != "blocks"} | {
-        "op": data["op"],
-    }
-
-
-def apply_operations(
-    doc: StructuredDocument,
-    operations: list[Operation],
-) -> AppliedDelta:
-    """Apply a list of operations to a document, returning a new document.
-
-    The original document is never mutated. Invalid operations (unknown
-    section, out-of-range index, name collision when adding a section) are
-    skipped and recorded in ``skipped`` with a ``reason`` string.
-    """
-    new_doc = doc.model_copy(deep=True)
-    applied: list[dict[str, Any]] = []
-    skipped: list[dict[str, Any]] = []
-
-    def skip(op: Operation, reason: str) -> None:
-        entry = _op_summary(op)
-        entry["reason"] = reason
-        skipped.append(entry)
-        logger.debug(f"[STRUCTURED_DELTA] skipping op {entry}")
-
-    for op in operations:
-        if isinstance(op, AppendBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.blocks.append(op.block)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, InsertBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index > len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} > {len(section.blocks)}",
-                )
-                continue
-            section.blocks.insert(op.index, op.block)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, ReplaceBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index >= len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} >= {len(section.blocks)}",
-                )
-                continue
-            section.blocks[op.index] = op.block
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, RemoveBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index >= len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} >= {len(section.blocks)}",
-                )
-                continue
-            section.blocks.pop(op.index)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, AddSectionOp):
-            existing_ids = {s.id for s in new_doc.sections}
-            base_id = op.new_id or slugify_heading(op.heading)
-            section_id = make_unique_id(base_id, existing_ids)
-            new_section = Section(
-                id=section_id,
-                heading=op.heading,
-                level=op.level,
-                blocks=list(op.blocks),
-            )
-            if op.after_section_id is None:
-                new_doc.sections.append(new_section)
-            else:
-                idx = new_doc.section_index(op.after_section_id)
-                if idx is None:
-                    skip(op, f"unknown after_section_id: {op.after_section_id}")
-                    continue
-                new_doc.sections.insert(idx + 1, new_section)
-            entry = _op_summary(op)
-            entry["assigned_id"] = section_id
-            applied.append(entry)
-            continue
-
-        if isinstance(op, RemoveSectionOp):
-            idx = new_doc.section_index(op.section_id)
-            if idx is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            new_doc.sections.pop(idx)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, ReplaceSectionBlocksOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.blocks = list(op.blocks)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, RenameSectionOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.heading = op.new_heading
-            applied.append(_op_summary(op))
-            continue
-
-        skip(op, f"unhandled op type: {type(op).__name__}")  # pragma: no cover
-
-    return AppliedDelta(document=new_doc, applied=applied, skipped=skipped)

hindsight-api-slim/hindsight_api/engine/reflect/structured_doc.py

@@ -1,301 +0,0 @@
-"""Structured representation of a mental model document.
-
-Why this exists
----------------
-Storing mental models as raw markdown forces every refresh to round-trip prose
-through an LLM, which then drifts on stylistic details (numbered vs bulleted
-lists, casing, separator lines, paraphrasing) even when instructed to preserve
-content byte-for-byte. The intrinsic mechanism of an LLM is to *generate* the
-next token from a gestalt of the input — not to copy tokens verbatim — so any
-"preserve unchanged content" instruction is fundamentally a soft constraint.
-
-The fix is to give the LLM no opportunity to drift on unchanged content. We
-keep an authoritative structured representation of the document; the markdown
-shown to users is a deterministic render of that structure. Delta refreshes
-emit *operations* against the structure (see ``delta_ops.py``); sections and
-blocks not mentioned by any operation are physically untouched.
-
-Schema (v1)
------------
-A document is an ordered list of ``Section``s.  Each section has:
-- ``id``    : stable slug derived from ``heading`` (used as the operation
-              target across refreshes; surviving renames is a separate
-              concern handled by an explicit ``rename`` op).
-- ``heading``: the markdown heading text (without the ``#`` prefix).
-- ``level`` : 1 (``#``) … 6 (``######``).  Default 2.
-- ``blocks``: ordered list of typed blocks — paragraph, bullet_list,
-              ordered_list, code.
-
-The schema is intentionally narrow: it covers what real mental-model documents
-actually contain (the kind a coding agent writes for itself or a user writes as
-a "skill" doc).  Tables, images, and raw HTML are out of scope until needed.
-"""
-
-from __future__ import annotations
-
-import re
-from typing import Annotated, Literal, Union
-
-from pydantic import BaseModel, ConfigDict, Field
-
-# Blocks ---------------------------------------------------------------------
-
-
-class ParagraphBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["paragraph"] = "paragraph"
-    text: str
-
-
-class BulletListBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["bullet_list"] = "bullet_list"
-    items: list[str] = Field(default_factory=list)
-
-
-class OrderedListBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["ordered_list"] = "ordered_list"
-    items: list[str] = Field(default_factory=list)
-
-
-class CodeBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["code"] = "code"
-    language: str = ""
-    text: str
-
-
-Block = Annotated[
-    Union[ParagraphBlock, BulletListBlock, OrderedListBlock, CodeBlock],
-    Field(discriminator="type"),
-]
-
-
-# Section / Document ---------------------------------------------------------
-
-
-class Section(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    id: str
-    heading: str
-    level: int = Field(default=2, ge=1, le=6)
-    blocks: list[Block] = Field(default_factory=list)
-
-
-class StructuredDocument(BaseModel):
-    """Top-level structured representation of a mental model."""
-
-    model_config = ConfigDict(extra="forbid")
-    version: Literal[1] = 1
-    sections: list[Section] = Field(default_factory=list)
-
-    def section_by_id(self, section_id: str) -> Section | None:
-        for s in self.sections:
-            if s.id == section_id:
-                return s
-        return None
-
-    def section_index(self, section_id: str) -> int | None:
-        for i, s in enumerate(self.sections):
-            if s.id == section_id:
-                return i
-        return None
-
-
-# Slug helpers ---------------------------------------------------------------
-
-_SLUG_RX = re.compile(r"[^a-z0-9]+")
-
-
-def slugify_heading(heading: str) -> str:
-    """Stable, deterministic slug from a heading.
-
-    "Stop Conditions" -> "stop-conditions"
-    "Inputs and Context" -> "inputs-and-context"
-    """
-    slug = _SLUG_RX.sub("-", heading.strip().lower()).strip("-")
-    return slug or "section"
-
-
-def make_unique_id(base: str, existing: set[str]) -> str:
-    """Disambiguate by appending -2, -3, … if the slug is already in use."""
-    if base not in existing:
-        return base
-    i = 2
-    while f"{base}-{i}" in existing:
-        i += 1
-    return f"{base}-{i}"
-
-
-# Renderer -------------------------------------------------------------------
-
-
-def render_block(block: Block) -> str:
-    """Render a single block to markdown. No trailing newline."""
-    if isinstance(block, ParagraphBlock):
-        return block.text.rstrip()
-    if isinstance(block, BulletListBlock):
-        return "\n".join(f"- {item.rstrip()}" for item in block.items)
-    if isinstance(block, OrderedListBlock):
-        return "\n".join(f"{i + 1}. {item.rstrip()}" for i, item in enumerate(block.items))
-    if isinstance(block, CodeBlock):
-        fence_lang = block.language or ""
-        return f"```{fence_lang}\n{block.text}\n```"
-    raise TypeError(f"Unknown block type: {type(block)!r}")
-
-
-def render_section(section: Section) -> str:
-    """Render a section: heading + blank line + blocks separated by blank lines."""
-    parts = ["#" * section.level + " " + section.heading.strip()]
-    for block in section.blocks:
-        parts.append("")  # blank line before each block
-        parts.append(render_block(block))
-    return "\n".join(parts)
-
-
-def render_document(doc: StructuredDocument) -> str:
-    """Render the whole document. Sections separated by a single blank line.
-
-    The output is byte-stable: same structured input always produces the same
-    markdown, modulo the inherent ordering of sections/blocks/items.
-    """
-    if not doc.sections:
-        return ""
-    return "\n\n".join(render_section(s) for s in doc.sections) + "\n"
-
-
-# Parser ---------------------------------------------------------------------
-#
-# The parser is intentionally lenient: it accepts the markdown produced by
-# our own renderer (round-trip-safe) and the markdown an LLM tends to produce
-# for mental-model documents.  It is *not* a general CommonMark parser — it
-# does not need to be.  When it cannot classify a block it falls back to a
-# paragraph so that no content is silently dropped.
-
-_HEADING_RX = re.compile(r"^(#{1,6})\s+(.+?)\s*$")
-_BULLET_RX = re.compile(r"^\s*[-*+]\s+(.*)$")
-_ORDERED_RX = re.compile(r"^\s*\d+[.)]\s+(.*)$")
-_FENCE_RX = re.compile(r"^```([A-Za-z0-9_+-]*)\s*$")
-
-
-def _strip_separators(lines: list[str]) -> list[str]:
-    """Drop horizontal-rule lines (`---`, `***`) used as section separators.
-
-    Our renderer never emits these, but LLM output frequently includes them
-    between sections; treating them as blank lines avoids parsing them as
-    paragraphs.
-    """
-    return ["" if re.fullmatch(r"\s*([-*_])\1{2,}\s*", line) else line for line in lines]
-
-
-def _split_blocks(lines: list[str]) -> list[list[str]]:
-    """Group consecutive non-blank lines into block chunks."""
-    chunks: list[list[str]] = []
-    current: list[str] = []
-    in_fence = False
-    for line in lines:
-        if _FENCE_RX.match(line):
-            current.append(line)
-            in_fence = not in_fence
-            continue
-        if in_fence:
-            current.append(line)
-            continue
-        if line.strip() == "":
-            if current:
-                chunks.append(current)
-                current = []
-        else:
-            current.append(line)
-    if current:
-        chunks.append(current)
-    return chunks
-
-
-def _parse_block(chunk: list[str]) -> Block:
-    """Parse a single non-empty chunk into a block."""
-    if chunk and _FENCE_RX.match(chunk[0]):
-        m = _FENCE_RX.match(chunk[0])
-        lang = m.group(1) if m else ""
-        body_lines = chunk[1:]
-        if body_lines and _FENCE_RX.match(body_lines[-1]):
-            body_lines = body_lines[:-1]
-        return CodeBlock(language=lang, text="\n".join(body_lines))
-
-    if all(_BULLET_RX.match(line) for line in chunk):
-        items = []
-        for line in chunk:
-            m = _BULLET_RX.match(line)
-            assert m is not None
-            items.append(m.group(1).strip())
-        return BulletListBlock(items=items)
-
-    if all(_ORDERED_RX.match(line) for line in chunk):
-        items = []
-        for line in chunk:
-            m = _ORDERED_RX.match(line)
-            assert m is not None
-            items.append(m.group(1).strip())
-        return OrderedListBlock(items=items)
-
-    return ParagraphBlock(text=" ".join(line.strip() for line in chunk).strip())
-
-
-def parse_markdown(markdown: str) -> StructuredDocument:
-    """Best-effort parse of a markdown document into the structured schema.
-
-    Sections are introduced by ATX headings (``#``..``######``).  Anything
-    before the first heading is wrapped into an implicit "Overview" section
-    so we never silently drop user content.  Section IDs are unique slugs of
-    their headings.
-    """
-    raw_lines = (markdown or "").splitlines()
-    lines = _strip_separators(raw_lines)
-
-    sections: list[Section] = []
-    used_ids: set[str] = set()
-    pending: list[str] = []
-    current: Section | None = None
-
-    def flush_pending_into(section: Section) -> None:
-        if not pending:
-            return
-        for chunk in _split_blocks(pending):
-            section.blocks.append(_parse_block(chunk))
-        pending.clear()
-
-    for line in lines:
-        m = _HEADING_RX.match(line)
-        if m:
-            if current is not None:
-                flush_pending_into(current)
-                sections.append(current)
-            elif pending:
-                # Content before the first heading: wrap in implicit section.
-                base = "overview"
-                section_id = make_unique_id(base, used_ids)
-                used_ids.add(section_id)
-                implicit = Section(id=section_id, heading="Overview", level=2)
-                flush_pending_into(implicit)
-                sections.append(implicit)
-            level = len(m.group(1))
-            heading = m.group(2).strip()
-            section_id = make_unique_id(slugify_heading(heading), used_ids)
-            used_ids.add(section_id)
-            current = Section(id=section_id, heading=heading, level=level)
-        else:
-            pending.append(line)
-
-    if current is not None:
-        flush_pending_into(current)
-        sections.append(current)
-    elif pending:
-        base = "overview"
-        section_id = make_unique_id(base, used_ids)
-        used_ids.add(section_id)
-        implicit = Section(id=section_id, heading="Overview", level=2)
-        flush_pending_into(implicit)
-        sections.append(implicit)
-
-    return StructuredDocument(sections=sections)

hindsight-api-slim/hindsight_api/engine/retain/chunk_storage.py

@@ -1,144 +0,0 @@
-"""
-Chunk storage for retain pipeline.
-
-Handles storage of document chunks in the database.
-"""
-
-import hashlib
-import logging
-from dataclasses import dataclass
-
-from ..memory_engine import fq_table
-from .types import ChunkMetadata
-
-logger = logging.getLogger(__name__)
-
-
-def compute_chunk_hash(chunk_text: str) -> str:
-    """Compute SHA256 hash of chunk text for delta comparison."""
-    return hashlib.sha256(chunk_text.encode()).hexdigest()
-
-
-@dataclass
-class ExistingChunk:
-    """Represents a chunk already stored in the database."""
-
-    chunk_id: str
-    chunk_index: int
-    content_hash: str | None
-
-
-async def load_existing_chunks(conn, bank_id: str, document_id: str) -> list[ExistingChunk]:
-    """
-    Load existing chunk metadata for a document.
-
-    Returns list of ExistingChunk with chunk_id, chunk_index, and content_hash.
-    """
-    rows = await conn.fetch(
-        f"""
-        SELECT chunk_id, chunk_index, content_hash
-        FROM {fq_table("chunks")}
-        WHERE document_id = $1 AND bank_id = $2
-        ORDER BY chunk_index
-        """,
-        document_id,
-        bank_id,
-    )
-    return [
-        ExistingChunk(
-            chunk_id=row["chunk_id"],
-            chunk_index=row["chunk_index"],
-            content_hash=row["content_hash"],
-        )
-        for row in rows
-    ]
-
-
-async def delete_chunks_by_ids(conn, chunk_ids: list[str]) -> None:
-    """
-    Delete specific chunks by their IDs.
-
-    This cascades to memory_units (via FK with CASCADE delete)
-    and their links.
-    """
-    if not chunk_ids:
-        return
-    await conn.execute(
-        f"DELETE FROM {fq_table('chunks')} WHERE chunk_id = ANY($1::text[])",
-        chunk_ids,
-    )
-
-
-async def store_chunks_batch(conn, bank_id: str, document_id: str, chunks: list[ChunkMetadata]) -> dict[int, str]:
-    """
-    Store document chunks in the database.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        document_id: Document identifier
-        chunks: List of ChunkMetadata objects
-
-    Returns:
-        Dictionary mapping global chunk index to chunk_id
-    """
-    if not chunks:
-        return {}
-
-    # Prepare chunk data for batch insert
-    chunk_ids = []
-    chunk_texts = []
-    chunk_indices = []
-    content_hashes = []
-    chunk_id_map = {}
-
-    for chunk in chunks:
-        chunk_id = f"{bank_id}_{document_id}_{chunk.chunk_index}"
-        chunk_ids.append(chunk_id)
-        chunk_texts.append(chunk.chunk_text)
-        chunk_indices.append(chunk.chunk_index)
-        content_hashes.append(compute_chunk_hash(chunk.chunk_text))
-        chunk_id_map[chunk.chunk_index] = chunk_id
-
-    # Batch upsert all chunks. ON CONFLICT makes this idempotent: re-submitting
-    # a retain under the same document_id (the pattern in vectorize-io/hindsight#977)
-    # may produce chunk_ids that already exist when upstream cascade-delete or
-    # delta-retain paths don't run (or race with a concurrent task). Overwriting
-    # is the correct behavior per the document_id grouping semantics — the caller
-    # intends this chunk to hold the latest content at that (document_id, index).
-    await conn.execute(
-        f"""
-        INSERT INTO {fq_table("chunks")} (chunk_id, document_id, bank_id, chunk_text, chunk_index, content_hash)
-        SELECT * FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::integer[], $6::text[])
-        ON CONFLICT (chunk_id) DO UPDATE SET
-            chunk_text = EXCLUDED.chunk_text,
-            chunk_index = EXCLUDED.chunk_index,
-            content_hash = EXCLUDED.content_hash
-        """,
-        chunk_ids,
-        [document_id] * len(chunk_texts),
-        [bank_id] * len(chunk_texts),
-        chunk_texts,
-        chunk_indices,
-        content_hashes,
-    )
-
-    return chunk_id_map
-
-
-def map_facts_to_chunks(facts_chunk_indices: list[int], chunk_id_map: dict[int, str]) -> list[str | None]:
-    """
-    Map fact chunk indices to chunk IDs.
-
-    Args:
-        facts_chunk_indices: List of chunk indices for each fact
-        chunk_id_map: Dictionary mapping chunk index to chunk_id
-
-    Returns:
-        List of chunk_ids (same length as facts_chunk_indices)
-    """
-    chunk_ids = []
-    for chunk_idx in facts_chunk_indices:
-        chunk_id = chunk_id_map.get(chunk_idx)
-        chunk_ids.append(chunk_id)
-    return chunk_ids

hindsight-api-slim/hindsight_api/engine/retain/entity_labels.py

@@ -1,194 +0,0 @@
-"""
-Entity labels models and helpers for retain pipeline.
-
-Defines a controlled vocabulary of key:value classification labels
-(e.g., 'pedagogy:scaffolding', 'interest:active') that are extracted
-at retain time and stored as entities.
-"""
-
-from typing import Literal
-
-from pydantic import BaseModel, Field, create_model
-
-
-class LabelValue(BaseModel):
-    """A single allowed value for a label group."""
-
-    value: str
-    description: str = ""
-
-
-class LabelGroup(BaseModel):
-    """A label group (dimension) with its type and allowed values."""
-
-    key: str
-    description: str = ""
-    type: Literal["value", "multi-values", "text"] = "value"
-    optional: bool = True
-    tag: bool = False
-    values: list[LabelValue] = []
-
-
-class EntityLabelsConfig(BaseModel):
-    """Entity labels configuration for a bank (controlled vocabulary)."""
-
-    attributes: list[LabelGroup] = []
-
-
-def parse_entity_labels(raw: dict | list | None) -> EntityLabelsConfig | None:
-    """
-    Parse raw entity labels config into EntityLabelsConfig.
-
-    Accepts:
-    - None → returns None
-    - list → list of attribute dicts (each may use legacy free_values/multi_value or new type field)
-    - dict → {attributes: [...]}
-
-    Legacy migration (backward-compat):
-    - free_values=True          → type="text"
-    - multi_value=True          → type="multi-values"
-    - neither / free_values=False → type="value"
-
-    Args:
-        raw: Raw entity labels config from bank config
-
-    Returns:
-        EntityLabelsConfig or None if raw is None/empty
-    """
-    if raw is None:
-        return None
-
-    if isinstance(raw, list):
-        if not raw:
-            return None
-        attributes = [LabelGroup.model_validate(_migrate_label_group(a)) for a in raw]
-        return EntityLabelsConfig(attributes=attributes)
-
-    if isinstance(raw, dict):
-        attrs_raw = raw.get("attributes", [])
-        if not attrs_raw:
-            return None
-        attributes = [LabelGroup.model_validate(_migrate_label_group(a)) for a in attrs_raw]
-        return EntityLabelsConfig(attributes=attributes)
-
-    return None
-
-
-def _migrate_label_group(raw: dict) -> dict:
-    """Migrate legacy free_values/multi_value fields to the new type field."""
-    if not isinstance(raw, dict) or "type" in raw:
-        return raw
-    patched = dict(raw)
-    if patched.get("free_values"):
-        patched["type"] = "text"
-    elif patched.get("multi_value"):
-        patched["type"] = "multi-values"
-    else:
-        patched["type"] = "value"
-    # Remove legacy keys so Pydantic doesn't error on unknown fields
-    patched.pop("free_values", None)
-    patched.pop("multi_value", None)
-    return patched
-
-
-def build_labels_model(labels_cfg: EntityLabelsConfig) -> type[BaseModel] | None:
-    """
-    Build a dynamic Pydantic model for structured label extraction.
-
-    Each LabelGroup becomes a typed field based on its type:
-    - type="text"                        → str | None  (always optional)
-    - type="value",   optional=True      → Literal["v1","v2"] | None
-    - type="value",   optional=False     → Literal["v1","v2"]  (required)
-    - type="multi-values"                → list[Literal["v1","v2"]]
-
-    Args:
-        labels_cfg: Parsed EntityLabelsConfig
-
-    Returns:
-        Dynamic Pydantic model class, or None if no groups defined
-    """
-    fields: dict = {}
-    for group in labels_cfg.attributes:
-        if not group.key:
-            continue
-        description = group.description or group.key
-
-        if group.type == "text":
-            # Free-form: any string value accepted, always optional
-            fields[group.key] = (str | None, Field(default=None, description=description))
-        else:
-            # Enum-constrained: must have defined values
-            if not group.values:
-                continue
-            values = tuple(v.value for v in group.values if v.value)
-            if not values:
-                continue
-            # Literal[("v1", "v2")] is equivalent to Literal["v1", "v2"] in Python 3.11+
-            literal_type = Literal[values]  # type: ignore[valid-type]
-            if group.type == "multi-values":
-                fields[group.key] = (
-                    list[literal_type],  # type: ignore[valid-type]
-                    Field(default_factory=list, description=description),
-                )
-            elif group.optional:
-                fields[group.key] = (
-                    literal_type | None,  # type: ignore[valid-type]
-                    Field(default=None, description=description),
-                )
-            else:
-                fields[group.key] = (
-                    literal_type,  # type: ignore[valid-type]
-                    Field(description=description),
-                )
-
-    if not fields:
-        return None
-
-    return create_model("Labels", **fields)
-
-
-def is_label_entity(text: str, labels_cfg: EntityLabelsConfig, labels_lookup: set[str]) -> bool:
-    """
-    Return True if entity text belongs to any configured label group.
-
-    For enum groups: checks the pre-built lookup set.
-    For text groups: checks that the text starts with a known key prefix.
-    """
-    if text.lower() in labels_lookup:
-        return True
-    for group in labels_cfg.attributes:
-        if group.type == "text" and group.key and text.lower().startswith(f"{group.key.lower()}:"):
-            return True
-    return False
-
-
-def build_labels_lookup(labels_cfg: EntityLabelsConfig | list | None) -> set[str]:
-    """
-    Build a set of valid 'key:value' label strings (lowercase) for fast lookup.
-
-    Accepts either EntityLabelsConfig or raw list/None for backwards compatibility.
-
-    Args:
-        labels_cfg: EntityLabelsConfig, raw list of attribute dicts, or None
-
-    Returns:
-        Set of lowercase 'key:value' strings
-    """
-    if labels_cfg is None:
-        return set()
-
-    # Accept raw list/dict for backwards compatibility
-    if not isinstance(labels_cfg, EntityLabelsConfig):
-        parsed = parse_entity_labels(labels_cfg)
-        if parsed is None:
-            return set()
-        labels_cfg = parsed
-
-    valid = set()
-    for group in labels_cfg.attributes:
-        if group.type == "text":
-            continue  # No fixed vocabulary — all values accepted in post-processing
-        for v in group.values:
-            if group.key and v.value:
-                valid.add(f"{group.key}:{v.value}".lower())
-    return valid

hindsight-api-slim/hindsight_api/engine/retain/entity_processing.py

@@ -1,162 +0,0 @@
-"""
-Entity processing for retain pipeline.
-
-Handles entity extraction, resolution, and link creation for stored facts.
-"""
-
-import logging
-
-from . import link_utils
-from .types import EntityLink, ProcessedFact
-
-logger = logging.getLogger(__name__)
-
-
-def _prepare_facts_for_entity_processing(
-    facts: list[ProcessedFact],
-    user_entities_per_content: dict[int, list[dict]] | None = None,
-) -> tuple[list[str], list, list[list[dict]]]:
-    """
-    Extract fact texts, dates, and merged entity lists from ProcessedFact objects.
-
-    Returns:
-        Tuple of (fact_texts, fact_dates, entities_per_fact)
-    """
-    user_entities_per_content = user_entities_per_content or {}
-
-    fact_texts = [fact.fact_text for fact in facts]
-    fact_dates = [fact.occurred_start if fact.occurred_start is not None else fact.mentioned_at for fact in facts]
-
-    entities_per_fact = []
-    for fact in facts:
-        llm_entities = [{"text": entity.name, "type": "CONCEPT"} for entity in (fact.entities or [])]
-
-        user_entities = user_entities_per_content.get(fact.content_index, [])
-
-        seen_texts = {e["text"].lower() for e in llm_entities}
-        for user_entity in user_entities:
-            if user_entity["text"].lower() not in seen_texts:
-                llm_entities.append(
-                    {
-                        "text": user_entity["text"],
-                        "type": user_entity.get("type", "CONCEPT"),
-                    }
-                )
-                seen_texts.add(user_entity["text"].lower())
-
-        entities_per_fact.append(llm_entities)
-
-    return fact_texts, fact_dates, entities_per_fact
-
-
-async def resolve_entities(
-    entity_resolver,
-    conn,
-    bank_id: str,
-    unit_ids: list[str],
-    facts: list[ProcessedFact],
-    log_buffer: list[str] = None,
-    user_entities_per_content: dict[int, list[dict]] = None,
-    entity_labels: list | None = None,
-) -> tuple[list[str], list[tuple], dict[str, list[str]]]:
-    """
-    Phase 1: Resolve entity names to canonical IDs (read-heavy).
-
-    Should be called on a SEPARATE connection OUTSIDE the main write transaction
-    to avoid holding the transaction open during expensive trigram scans.
-
-    Args:
-        entity_resolver: EntityResolver instance
-        conn: Database connection (separate from the main write transaction)
-        bank_id: Bank identifier
-        unit_ids: Placeholder unit IDs (used only for grouping)
-        facts: List of ProcessedFact objects
-        log_buffer: Optional buffer for detailed logging
-        user_entities_per_content: Dict mapping content_index to user-provided entities
-        entity_labels: Optional entity label taxonomy
-
-    Returns:
-        Tuple of (resolved_entity_ids, entity_to_unit, unit_to_entity_ids)
-        to pass to build_entity_links().
-    """
-    if not unit_ids or not facts:
-        return [], [], {}
-
-    if len(unit_ids) != len(facts):
-        raise ValueError(f"Mismatch between unit_ids ({len(unit_ids)}) and facts ({len(facts)})")
-
-    fact_texts, fact_dates, entities_per_fact = _prepare_facts_for_entity_processing(facts, user_entities_per_content)
-
-    return await link_utils.resolve_entities_only(
-        entity_resolver,
-        conn,
-        bank_id,
-        unit_ids,
-        fact_texts,
-        "",  # context (not used in current implementation)
-        fact_dates,
-        entities_per_fact,
-        log_buffer,
-        entity_labels=entity_labels,
-    )
-
-
-async def build_entity_links(
-    entity_resolver,
-    conn,
-    bank_id: str,
-    unit_ids: list[str],
-    resolved_entity_ids: list[str],
-    entity_to_unit: list[tuple],
-    unit_to_entity_ids: dict[str, list[str]],
-    log_buffer: list[str] = None,
-    skip_unit_entities_insert: bool = False,
-) -> list[EntityLink]:
-    """
-    Build entity links for UI graph visualization.
-
-    Queries unit_entities to find shared entities between new and existing units,
-    then generates EntityLink objects. When called from Phase 3 (post-transaction),
-    set skip_unit_entities_insert=True since unit_entities were already inserted
-    in Phase 2.
-
-    Args:
-        entity_resolver: EntityResolver instance
-        conn: Database connection
-        bank_id: Bank identifier
-        unit_ids: Actual unit IDs (must already be inserted in the DB)
-        resolved_entity_ids: From resolve_entities()
-        entity_to_unit: From resolve_entities()
-        unit_to_entity_ids: From resolve_entities()
-        log_buffer: Optional buffer for detailed logging
-        skip_unit_entities_insert: Skip unit_entities INSERT (already done in Phase 2)
-
-    Returns:
-        List of EntityLink objects for batch insertion
-    """
-    return await link_utils.build_entity_links_from_resolved(
-        entity_resolver,
-        conn,
-        bank_id,
-        unit_ids,
-        resolved_entity_ids,
-        entity_to_unit,
-        unit_to_entity_ids,
-        log_buffer,
-        skip_unit_entities_insert=skip_unit_entities_insert,
-    )
-
-
-async def insert_entity_links_batch(conn, entity_links: list[EntityLink], bank_id: str) -> None:
-    """
-    Insert entity links in batch.
-
-    Args:
-        conn: Database connection
-        entity_links: List of EntityLink objects
-        bank_id: Bank identifier (stored directly on memory_links for fast filtering)
-    """
-    if not entity_links:
-        return
-
-    await link_utils.insert_entity_links_batch(conn, entity_links, bank_id)

hindsight-api-slim/hindsight_api/engine/retain/fact_storage.py

@@ -1,449 +0,0 @@
-"""
-Fact storage for retain pipeline.
-
-Handles insertion of facts into the database.
-"""
-
-import json
-import logging
-import uuid
-
-from ...config import get_config
-from ..memory_engine import fq_table
-from .bank_utils import DEFAULT_DISPOSITION, create_bank_vector_indexes
-from .fact_extraction import _sanitize_text
-from .types import ProcessedFact
-
-logger = logging.getLogger(__name__)
-
-
-async def get_document_content(
-    conn,
-    bank_id: str,
-    document_id: str,
-) -> str | None:
-    """Fetch the original_text of an existing document.
-
-    Returns None if the document does not exist.
-    """
-    row = await conn.fetchval(
-        f"SELECT original_text FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
-        document_id,
-        bank_id,
-    )
-    return row
-
-
-async def insert_facts_batch(
-    conn, bank_id: str, facts: list[ProcessedFact], document_id: str | None = None
-) -> list[str]:
-    """
-    Insert facts into the database in batch.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        facts: List of ProcessedFact objects to insert
-        document_id: Optional document ID to associate with facts
-
-    Returns:
-        List of unit IDs (UUIDs as strings) for the inserted facts
-    """
-    if not facts:
-        return []
-
-    # Prepare data for batch insert
-    fact_texts = []
-    embeddings = []
-    event_dates = []
-    occurred_starts = []
-    occurred_ends = []
-    mentioned_ats = []
-    contexts = []
-    fact_types = []
-    metadata_jsons = []
-    chunk_ids = []
-    document_ids = []
-    tags_list = []
-    observation_scopes_list = []
-    text_signals_list = []
-
-    for fact in facts:
-        fact_texts.append(_sanitize_text(fact.fact_text))
-        # Convert embedding to string for asyncpg vector type
-        embeddings.append(str(fact.embedding))
-        # event_date: Use occurred_start if available, otherwise use mentioned_at
-        # This maintains backward compatibility while handling None occurred_start
-        event_dates.append(fact.occurred_start if fact.occurred_start is not None else fact.mentioned_at)
-        occurred_starts.append(fact.occurred_start)
-        occurred_ends.append(fact.occurred_end)
-        mentioned_ats.append(fact.mentioned_at)
-        contexts.append(_sanitize_text(fact.context))
-        fact_types.append(fact.fact_type)
-        metadata_jsons.append(json.dumps(fact.metadata))
-        chunk_ids.append(fact.chunk_id)
-        # Use per-fact document_id if available, otherwise fallback to batch-level document_id
-        document_ids.append(fact.document_id if fact.document_id else document_id)
-        # Convert tags to JSON string for proper batch insertion (PostgreSQL unnest doesn't handle 2D arrays well)
-        tags_list.append(json.dumps(fact.tags if fact.tags else []))
-        # observation_scopes: stored as JSONB (string or 2D array), None if not provided
-        observation_scopes_list.append(
-            json.dumps(fact.observation_scopes) if fact.observation_scopes is not None else None
-        )
-        # Build text_signals: entity names + date tokens for enriched BM25 indexing
-        signal_parts = []
-        if fact.entities:
-            signal_parts.extend(e.name for e in fact.entities)
-        if fact.occurred_start:
-            try:
-                signal_parts.append(fact.occurred_start.strftime("%B %d %Y").lstrip("0").replace(" 0", " "))
-            except (ValueError, AttributeError):
-                pass
-        if fact.occurred_end and fact.occurred_end != fact.occurred_start:
-            try:
-                signal_parts.append(fact.occurred_end.strftime("%B %d %Y").lstrip("0").replace(" 0", " "))
-            except (ValueError, AttributeError):
-                pass
-        text_signals_list.append(" ".join(signal_parts) if signal_parts else None)
-
-    # Batch insert all facts
-    # Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
-    # Query varies based on text search backend
-    config = get_config()
-    if config.text_search_extension == "vchord":
-        # VectorChord: manually tokenize and insert search_vector
-        # text_signals (entity names etc.) are included in the tokenize input for enriched BM25
-        query = f"""
-            WITH input_data AS (
-                SELECT * FROM unnest(
-                    $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-                    $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
-                ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                       context, fact_type, metadata, chunk_id, document_id, tags_json,
-                       observation_scopes_json, text_signals)
-            )
-            INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                     context, fact_type, metadata, chunk_id, document_id, tags,
-                                     observation_scopes, text_signals, search_vector)
-            SELECT
-                $1,
-                text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                context, fact_type, metadata, chunk_id, document_id,
-                COALESCE(
-                    (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
-                    '{{}}'::varchar[]
-                ),
-                observation_scopes_json,
-                text_signals,
-                tokenize(
-                    COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, ''),
-                    'llmlingua2'
-                )::bm25_catalog.bm25vector
-            FROM input_data
-            RETURNING id
-        """
-    else:  # native or pg_textsearch
-        # Native PostgreSQL: search_vector is GENERATED ALWAYS (expression includes text_signals), don't include it
-        # pg_textsearch: indexes operate on base columns directly, don't populate search_vector
-        query = f"""
-            WITH input_data AS (
-                SELECT * FROM unnest(
-                    $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-                    $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
-                ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                       context, fact_type, metadata, chunk_id, document_id, tags_json,
-                       observation_scopes_json, text_signals)
-            )
-            INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                     context, fact_type, metadata, chunk_id, document_id, tags,
-                                     observation_scopes, text_signals)
-            SELECT
-                $1,
-                text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                context, fact_type, metadata, chunk_id, document_id,
-                COALESCE(
-                    (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
-                    '{{}}'::varchar[]
-                ),
-                observation_scopes_json,
-                text_signals
-            FROM input_data
-            RETURNING id
-        """
-
-    results = await conn.fetch(
-        query,
-        bank_id,
-        fact_texts,
-        embeddings,
-        event_dates,  # event_date: occurred_start if available, else mentioned_at
-        occurred_starts,
-        occurred_ends,
-        mentioned_ats,
-        contexts,
-        fact_types,
-        metadata_jsons,
-        chunk_ids,
-        document_ids,
-        tags_list,
-        observation_scopes_list,
-        text_signals_list,
-    )
-
-    unit_ids = [str(row["id"]) for row in results]
-    return unit_ids
-
-
-async def ensure_bank_exists(conn, bank_id: str) -> None:
-    """
-    Ensure bank exists in the database.
-
-    Creates bank with default values if it doesn't exist.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-    """
-    # Generate internal_id here so we control the value and can use it
-    # immediately for HNSW index creation without a RETURNING round-trip.
-    internal_id = uuid.uuid4()
-    inserted = await conn.fetchval(
-        f"""
-        INSERT INTO {fq_table("banks")} (bank_id, disposition, mission, internal_id)
-        VALUES ($1, $2::jsonb, $3, $4)
-        ON CONFLICT (bank_id) DO NOTHING
-        RETURNING bank_id
-        """,
-        bank_id,
-        json.dumps(DEFAULT_DISPOSITION),
-        "",
-        internal_id,
-    )
-    if inserted:
-        # Fresh insert — create per-bank vector indexes
-        await create_bank_vector_indexes(conn, bank_id, str(internal_id))
-
-
-async def delete_stale_observations_for_memories(
-    conn,
-    bank_id: str,
-    fact_ids: "list[str | uuid.UUID]",
-) -> int:
-    """Delete observations whose source memories are about to be removed.
-
-    Mirrors the cleanup performed by ``MemoryEngine.delete_document`` so that
-    every code path that removes ``memory_units`` also removes the
-    observations derived from them. Without this, ingesting a fresh version
-    of a document via the retain pipeline (which does a full-replace
-    ``DELETE FROM documents`` cascade) used to leave orphan observations
-    pointing at memory IDs that no longer existed.
-
-    For each observation referencing any of ``fact_ids``:
-    1. Delete the observation row (its text is stale once even one source
-       memory disappears).
-    2. Reset ``consolidated_at = NULL`` on the surviving source memories so
-       they get re-consolidated under fresh observations on the next run.
-
-    Must be called within an active transaction, before the source memories
-    are deleted.
-
-    Returns the number of observations deleted.
-    """
-    if not fact_ids:
-        return 0
-
-    fact_uuids = [uuid.UUID(str(fid)) if not isinstance(fid, uuid.UUID) else fid for fid in fact_ids]
-
-    affected_obs = await conn.fetch(
-        f"""
-        SELECT id, source_memory_ids
-        FROM {fq_table("memory_units")}
-        WHERE bank_id = $1
-          AND fact_type = 'observation'
-          AND source_memory_ids && $2::uuid[]
-        """,
-        bank_id,
-        fact_uuids,
-    )
-
-    if not affected_obs:
-        return 0
-
-    deleted_set = {str(uid) for uid in fact_uuids}
-    obs_ids = [obs["id"] for obs in affected_obs]
-    seen_remaining: set[str] = set()
-    remaining_source_ids: list[uuid.UUID] = []
-    for obs in affected_obs:
-        for src_id in obs["source_memory_ids"] or []:
-            src_str = str(src_id)
-            if src_str not in deleted_set and src_str not in seen_remaining:
-                remaining_source_ids.append(src_id)
-                seen_remaining.add(src_str)
-
-    await conn.execute(
-        f"DELETE FROM {fq_table('memory_units')} WHERE id = ANY($1::uuid[])",
-        obs_ids,
-    )
-
-    if remaining_source_ids:
-        await conn.execute(
-            f"""
-            UPDATE {fq_table("memory_units")}
-            SET consolidated_at = NULL
-            WHERE id = ANY($1::uuid[])
-              AND fact_type IN ('experience', 'world')
-            """,
-            remaining_source_ids,
-        )
-
-    logger.info(
-        f"[OBSERVATIONS] Deleted {len(obs_ids)} observations, reset {len(remaining_source_ids)} "
-        f"source memories for re-consolidation in bank {bank_id}"
-    )
-    return len(obs_ids)
-
-
-async def handle_document_tracking(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    is_first_batch: bool,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-) -> None:
-    """
-    Handle document tracking in the database (full-replace mode).
-
-    Deletes the existing document (cascading to all units and links) on the
-    first batch, then inserts the new document record.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        document_id: Document identifier
-        combined_content: Combined content text from all content items
-        is_first_batch: Whether this is the first batch (for chunked operations)
-        retain_params: Optional parameters passed during retain (context, event_date, etc.)
-        document_tags: Optional list of tags to associate with the document
-    """
-    import hashlib
-
-    # Sanitize and calculate content hash
-    combined_content = _sanitize_text(combined_content) or ""
-    content_hash = hashlib.sha256(combined_content.encode()).hexdigest()
-
-    # Delete old document first (cascades to units and links).
-    # Only delete on the first batch to avoid deleting data we just inserted.
-    # Before the cascade, fan out to delete observations derived from the
-    # outgoing memory_units — otherwise the FK ON DELETE CASCADE removes the
-    # source memory_units but leaves observation rows pointing at IDs that
-    # no longer exist (consolidated_at on co-source memories also stays
-    # frozen). Same cleanup the explicit ``delete_document`` API performs.
-    if is_first_batch:
-        existing_unit_rows = await conn.fetch(
-            f"""
-            SELECT id FROM {fq_table("memory_units")}
-            WHERE document_id = $1 AND fact_type IN ('experience', 'world')
-            """,
-            document_id,
-        )
-        existing_unit_ids = [row["id"] for row in existing_unit_rows]
-        if existing_unit_ids:
-            invalidated = await delete_stale_observations_for_memories(conn, bank_id, existing_unit_ids)
-            if invalidated:
-                logger.info(
-                    f"[RETAIN] Document {document_id} re-ingested: invalidated "
-                    f"{invalidated} observation(s) derived from {len(existing_unit_ids)} outgoing memory_units"
-                )
-        await conn.fetchval(
-            f"DELETE FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2 RETURNING id",
-            document_id,
-            bank_id,
-        )
-
-    # Insert document (or update if exists from concurrent operations)
-    await _upsert_document_row(conn, bank_id, document_id, combined_content, content_hash, retain_params, document_tags)
-
-
-async def upsert_document_metadata(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-) -> None:
-    """
-    Update document metadata without deleting existing facts/chunks.
-
-    Used by delta retain: the document row is upserted but chunks and
-    memory_units are managed separately at the chunk level.
-    """
-    import hashlib
-
-    combined_content = _sanitize_text(combined_content) or ""
-    content_hash = hashlib.sha256(combined_content.encode()).hexdigest()
-
-    await _upsert_document_row(conn, bank_id, document_id, combined_content, content_hash, retain_params, document_tags)
-
-
-async def _upsert_document_row(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    content_hash: str,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-) -> None:
-    """Insert or update a document row."""
-    await conn.execute(
-        f"""
-        INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, retain_params, tags)
-        VALUES ($1, $2, $3, $4, $5, $6)
-        ON CONFLICT (id, bank_id) DO UPDATE
-        SET original_text = EXCLUDED.original_text,
-            content_hash = EXCLUDED.content_hash,
-            retain_params = EXCLUDED.retain_params,
-            tags = EXCLUDED.tags,
-            updated_at = NOW()
-        """,
-        document_id,
-        bank_id,
-        combined_content,
-        content_hash,
-        json.dumps(retain_params) if retain_params else None,
-        document_tags or [],
-    )
-
-
-async def update_memory_units_tags(
-    conn,
-    bank_id: str,
-    document_id: str,
-    tags: list[str],
-) -> int:
-    """
-    Update tags on all memory_units belonging to a document.
-
-    Used during delta retain to propagate tag changes to unchanged facts.
-
-    Returns:
-        Number of memory units updated.
-    """
-    result = await conn.execute(
-        f"""
-        UPDATE {fq_table("memory_units")}
-        SET tags = $3, updated_at = NOW()
-        WHERE bank_id = $1 AND document_id = $2
-        """,
-        bank_id,
-        document_id,
-        tags or [],
-    )
-    # result is a status string like "UPDATE 5"
-    try:
-        return int(result.split()[-1])
-    except (ValueError, IndexError):
-        return 0