docs: update 0.4.15 blog cover image

.claude-plugin/marketplace.json

@@ -1,15 +0,0 @@
-{
-  "$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
-  "name": "hindsight",
-  "description": "Official Hindsight integrations for Claude Code",
-  "owner": {
-    "name": "vectorize-io"
-  },
-  "plugins": [
-    {
-      "name": "hindsight-memory",
-      "description": "Automatic long-term memory for Claude Code via Hindsight",
-      "source": "./hindsight-integrations/claude-code"
-    }
-  ]
-}

.claude/skills/code-review/SKILL.md

@@ -1,205 +0,0 @@
----
-name: code-review
-description: Review changed code against project standards. Checks for missing tests, dead code, type safety, lint issues, and coding conventions. Run after completing any implementation work.
-user_invocable: true
----
-
-# Code Review
-
-Review all changed code against the project's quality standards and coding conventions.
-
-## Code Standards
-
-Read and internalize these standards before writing code. The review steps below verify compliance.
-
-### Python Style
-- Python 3.11+, type hints required
-- Async throughout (asyncpg, async FastAPI)
-- Pydantic models for request/response
-- Ruff for linting (line-length 120)
-- No Python files at project root - maintain clean directory structure
-- **Never use multi-item tuple return values** — not even for internal/private functions. Always use a dataclass or Pydantic model. No exceptions, no "it's just two values" shortcuts. If a function returns more than one value, define a named type for it.
-
-### Type Safety with Pydantic Models
-**NEVER use raw `dict` types for structured data** — this applies to all code, including internal helpers and private functions. If the dict has known keys, it must be a dataclass or Pydantic model:
-- Use Pydantic `BaseModel` for all data structures passed between functions
-- Use `@dataclass` for lightweight internal data containers when Pydantic validation isn't needed
-- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
-- Avoid `dict.get()` patterns - use typed model attributes instead
-- Parse external data (JSON, API responses) into Pydantic models at the boundary
-- This catches type errors at parse time, not deep in business logic
-- The only acceptable `dict` usage is for truly dynamic/unknown keys (e.g., arbitrary metadata, JSON blobs with no fixed schema)
-
-```python
-# BAD - error-prone dict access
-def process(data: dict) -> str:
-    return data.get("name", "")  # No validation, silent failures
-
-# GOOD - typed and validated
-class UserData(BaseModel):
-    name: str
-    created_at: datetime
-
-    @field_validator("created_at", mode="before")
-    @classmethod
-    def ensure_tz_aware(cls, v):
-        if isinstance(v, str):
-            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
-        if v.tzinfo is None:
-            return v.replace(tzinfo=timezone.utc)
-        return v
-
-def process(data: UserData) -> str:
-    return data.name  # Type-safe, validated at construction
-```
-
-### TypeScript Style
-- Next.js App Router for control plane
-- Tailwind CSS with shadcn/ui components
-
-### Code Comments
-- **Always comment non-trivial technical decisions** with the reasoning behind the choice. If someone would ask "why is it done this way?", there should be a comment.
-- **Keep comments up to date with history** — when changing an approach, update the comment to explain what was tried before and why it was changed. Comments serve as a tracker of previous implementations that likely had problems.
-- Don't comment obvious code — only where the "why" isn't self-evident from the code itself.
-
-```python
-# BAD - no context for future readers
-results = await asyncio.gather(*tasks, return_exceptions=True)
-
-# GOOD - explains the non-obvious choice
-# Use return_exceptions=True to avoid cancelling sibling tasks on failure.
-# Previously we used TaskGroup but it cancelled all tasks when one failed,
-# causing partial writes that left orphaned entity links (see #412).
-results = await asyncio.gather(*tasks, return_exceptions=True)
-```
-
-### Branch Hygiene
-- **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
-- **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
-
-### General Principles
-- Don't add features, refactor code, or make "improvements" beyond what was asked
-- Don't add unnecessary error handling for impossible scenarios
-- Don't create helpers or abstractions for one-time operations
-- No backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
-- Three similar lines of code is better than a premature abstraction
-
-## Review Steps
-
-### 1. Check branch hygiene
-
-- Run `git log --oneline main..HEAD` to list all commits on the branch.
-- Verify every commit is relevant to the feature/PR. Flag any unrelated commits.
-- Check the branch is based on a recent `origin/main` (no stale base).
-
-### 2. Identify changed files
-
-Run `git diff --name-only HEAD` (unstaged) and `git diff --cached --name-only` (staged) to get all changed files. If there are no local changes, diff against the base branch using `git diff main...HEAD --name-only` and `git diff main...HEAD` to review all commits on the current branch.
-
-### 3. Run linters
-
-```bash
-./scripts/hooks/lint.sh
-```
-
-Report any failures. Do NOT fix them yourself — just report.
-
-### 4. Check for dead code
-
-For each changed Python file, check for:
-- Unused imports (Ruff should catch these, but verify)
-- Functions/methods/classes that were added but are never called from anywhere
-- Variables assigned but never read
-- Commented-out code blocks that should be removed
-
-For each changed TypeScript file, check for:
-- Unused imports
-- Unused variables or functions
-- Commented-out code
-
-### 5. Check type safety (Python)
-
-For each changed Python file, check for violations:
-- **No raw `dict` for structured data** — must use Pydantic model or dataclass, even for internal/private functions (only exception: truly dynamic/unknown keys)
-- **No multi-item tuple returns** — must use dataclass or Pydantic model, even for internal/private functions (no exceptions)
-- **Missing type hints** on function parameters and return types
-- **Missing `@field_validator`** for datetime fields that should be timezone-aware
-
-### 6. Check for missing tests
-
-For each new or significantly changed function/endpoint/class:
-- Check if there is a corresponding test addition or update
-- New API endpoints MUST have integration tests
-- New utility functions MUST have unit tests
-- Bug fixes SHOULD have a regression test
-
-Flag any new logic that lacks test coverage.
-
-### 7. Check API consistency
-
-If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
-- Were the OpenAPI specs regenerated? (`./scripts/generate-openapi.sh`)
-- Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
-- Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
-
-### 8. Check code comments
-
-For each non-trivial change:
-- **New non-obvious logic** — is there a comment explaining the reasoning?
-- **Changed approach** — does the comment include what was done before and why it changed?
-- **Stale comments** — do existing comments near the changed code still accurately describe the behavior?
-
-### 9. Check integration completeness
-
-If any files in `hindsight-integrations/` were added or changed, verify:
-- **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
-- **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
-- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
-- **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
-
-### 10. Check MCP tool registration completeness
-
-If any new MCP tools were added or existing tools renamed in `hindsight-api-slim/hindsight_api/mcp_tools.py`:
-- **`_ALL_TOOLS` set** in `mcp_tools.py` — must include the new tool name
-- **`tools_to_register` default set** in `register_mcp_tools()` in `mcp_tools.py` — must include the new tool name
-- **`_SINGLE_BANK_TOOLS` set** in `hindsight-api-slim/hindsight_api/api/mcp.py` — must include the new tool if it is bank-scoped (not a bank-management tool like `list_banks`/`create_bank`)
-- **`MCP_TOOL_GROUPS`** in `hindsight-control-plane/src/components/bank-config-view.tsx` — must include the new tool in the appropriate group for the UI tool selector
-- **Tool count assertions** in tests (e.g., `test_mcp_tools.py`) — must be updated to reflect the new count
-
-### 11. Review against other coding standards
-
-Check the diff for violations of the standards listed above:
-- Python files at project root (not allowed)
-- Missing async patterns (should be async throughout)
-- Pydantic models for request/response
-- Line length > 120 chars
-- New features/code beyond what was asked (over-engineering)
-- Unnecessary error handling for impossible scenarios
-- Premature abstractions or speculative helpers
-- Backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
-
-### 12. Report findings
-
-Present a clear summary organized by severity:
-
-**Must fix** — issues that will break CI or violate hard project rules:
-- Unrelated commits on the branch
-- Lint failures
-- Missing type hints on public functions
-- Raw dict usage for structured data (including internal code)
-- Multi-item tuple returns (including internal code)
-- Missing tests for new endpoints
-- New integration missing tests, CI job, or release-integration.sh entry
-
-**Should fix** — issues that hurt code quality:
-- Dead code / unused imports missed by linter
-- Missing tests for non-trivial utility functions
-- Over-engineering beyond the task scope
-
-**Note** — observations that may or may not need action:
-- API changes that might need client regeneration
-- Patterns that deviate from nearby code style
-
-For each finding, include the file path, line number, and a brief explanation.
-
-Do NOT auto-fix any issues. Report all findings and let the user decide what to address. If there are no findings, confirm the code looks good.

.env.example

@@ -2,7 +2,7 @@
 # Copy this file to .env and fill in your values
 
 # LLM Configuration (Required)
-# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, volcano
+# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai
 HINDSIGHT_API_LLM_PROVIDER=openai
 HINDSIGHT_API_LLM_API_KEY=your-api-key-here
 HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
@@ -20,11 +20,6 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
 # HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
 # HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/service-account-key.json  # Optional, uses ADC if not set
 
-# Example: MiniMax configuration (1M context window)
-# HINDSIGHT_API_LLM_PROVIDER=minimax
-# HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
-# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
-
 # Example: LM Studio local configuration (Qwen 2.5 32B recommended)
 # HINDSIGHT_API_LLM_PROVIDER=lmstudio
 # HINDSIGHT_API_LLM_API_KEY=lmstudio
@@ -44,7 +39,6 @@ HINDSIGHT_API_LOG_LEVEL=info
 
 # Database (Optional - uses embedded pg0 by default)
 # HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
-# HINDSIGHT_API_MIGRATION_DATABASE_URL=  # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
 # HINDSIGHT_API_DATABASE_SCHEMA=public  # PostgreSQL schema name (default: public)
 
 # Vector Extension (Optional - uses pgvector by default)

.github/dependabot.yml

@@ -1,6 +0,0 @@
-version: 2
-updates:
-  - package-ecosystem: "github-actions"
-    directory: "/"
-    schedule:
-      interval: "weekly"

.github/workflows/deploy-docs.yml

@@ -21,20 +21,20 @@ jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v6
-      - uses: actions/setup-node@v6
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
         with:
           node-version: 20
           cache: npm
           cache-dependency-path: package-lock.json
-      - uses: astral-sh/setup-uv@v7
+      - uses: astral-sh/setup-uv@v4
       - run: npm ci --workspace=hindsight-docs
       - run: uv run generate-llms-full
       - run: npm run build --workspace=hindsight-docs
         env:
           UMAMI_URL: https://analytics.hindsight.vectorize.io
           UMAMI_WEBSITE_ID: ${{ secrets.UMAMI_WEBSITE_ID }}
-      - uses: actions/upload-pages-artifact@v5
+      - uses: actions/upload-pages-artifact@v3
         with:
           path: hindsight-docs/build
   deploy:
@@ -44,5 +44,5 @@ jobs:
     runs-on: ubuntu-latest
     needs: build
     steps:
-      - uses: actions/deploy-pages@v5
+      - uses: actions/deploy-pages@v4
         id: deployment

.github/workflows/perf-test.yml

@@ -1,174 +0,0 @@
-name: Performance Tests
-
-on:
-  schedule:
-    # Run daily at 06:00 UTC
-    - cron: "0 6 * * *"
-  workflow_dispatch:
-    inputs:
-      scale:
-        description: "Test scale (perf-test)"
-        type: choice
-        options:
-          - tiny
-          - small
-          - medium
-          - large
-        default: large
-      suite:
-        description: "Perf-test suite to run (blank = all)"
-        type: choice
-        options:
-          - ""
-          - retain
-          - recall
-        default: ""
-      locomo_max_conversations:
-        description: "LoComo max conversations (0 = skip, blank = all)"
-        type: number
-        default: 0
-      locomo_skip:
-        description: "Skip LoComo job"
-        type: boolean
-        default: false
-      ref:
-        description: "Git ref to test (branch, tag, or SHA). Defaults to main."
-        type: string
-        default: ""
-
-concurrency:
-  group: perf-test
-  cancel-in-progress: true
-
-jobs:
-  perf-test:
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v6
-      with:
-        ref: ${{ inputs.ref || github.ref }}
-
-    - name: Install uv
-      uses: astral-sh/setup-uv@v7
-      with:
-        enable-cache: true
-        prune-cache: false
-
-    - name: Set up Python
-      uses: actions/setup-python@v6
-      with:
-        python-version-file: ".python-version"
-
-    - name: Cache HuggingFace models
-      uses: actions/cache@v5
-      with:
-        path: ~/.cache/huggingface
-        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
-        restore-keys: |
-          ${{ runner.os }}-huggingface-
-
-    - name: Pre-download models
-      working-directory: ./hindsight-api-slim
-      run: |
-        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
-        from sentence_transformers import SentenceTransformer
-        print('Downloading embedding model...')
-        SentenceTransformer('BAAI/bge-small-en-v1.5')
-        print('Model downloaded successfully')
-        "
-
-    - name: Install hindsight-dev dependencies
-      run: |
-        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
-
-    - name: Run perf tests
-      run: |
-        SUITE_ARG=""
-        if [ -n "${{ inputs.suite }}" ]; then
-          SUITE_ARG="--suite ${{ inputs.suite }}"
-        fi
-        ./scripts/benchmarks/run-perf-test.sh \
-          --scale ${{ inputs.scale || 'large' }} \
-          $SUITE_ARG \
-          --output perf-results.json
-
-    - name: Upload perf results
-      if: always()
-      uses: actions/upload-artifact@v4
-      with:
-        name: perf-results-${{ github.sha }}
-        path: hindsight-dev/perf-results.json
-        retention-days: 90
-
-  locomo:
-    if: inputs.locomo_skip != true
-    runs-on: ubuntu-latest
-    env:
-      HINDSIGHT_API_LLM_PROVIDER: vertexai
-      HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
-      HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
-      HINDSIGHT_API_JUDGE_LLM_PROVIDER: vertexai
-      HINDSIGHT_API_JUDGE_LLM_MODEL: google/gemini-2.5-flash-lite
-      HINDSIGHT_API_ANSWER_LLM_PROVIDER: vertexai
-      HINDSIGHT_API_ANSWER_LLM_MODEL: google/gemini-3.1-pro-preview
-    steps:
-    - uses: actions/checkout@v6
-      with:
-        ref: ${{ inputs.ref || github.ref }}
-
-    - name: Setup GCP credentials
-      run: |
-        printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
-        PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
-        echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
-
-    - name: Install uv
-      uses: astral-sh/setup-uv@v7
-      with:
-        enable-cache: true
-        prune-cache: false
-
-    - name: Set up Python
-      uses: actions/setup-python@v6
-      with:
-        python-version-file: ".python-version"
-
-    - name: Cache HuggingFace models
-      uses: actions/cache@v5
-      with:
-        path: ~/.cache/huggingface
-        key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
-        restore-keys: |
-          ${{ runner.os }}-huggingface-
-
-    - name: Pre-download models
-      working-directory: ./hindsight-api-slim
-      run: |
-        uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
-        from sentence_transformers import SentenceTransformer
-        print('Downloading embedding model...')
-        SentenceTransformer('BAAI/bge-small-en-v1.5')
-        print('Model downloaded successfully')
-        "
-
-    - name: Install hindsight-dev dependencies
-      run: |
-        cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
-
-    - name: Run LoComo benchmark
-      run: |
-        MAX_CONV_ARG=""
-        if [ "${{ inputs.locomo_max_conversations }}" != "0" ] && [ -n "${{ inputs.locomo_max_conversations }}" ]; then
-          MAX_CONV_ARG="--max-conversations ${{ inputs.locomo_max_conversations }}"
-        fi
-        uv run python hindsight-dev/benchmarks/locomo/locomo_benchmark.py \
-          --wait-consolidation \
-          $MAX_CONV_ARG
-
-    - name: Upload LoComo results
-      if: always()
-      uses: actions/upload-artifact@v4
-      with:
-        name: locomo-results-${{ github.sha }}
-        path: hindsight-dev/benchmarks/locomo/results/
-        retention-days: 90

.github/workflows/release-integration.yml

@@ -1,120 +0,0 @@
-name: Release Integration
-
-on:
-  push:
-    tags:
-      - 'integrations/**'
-
-jobs:
-  publish:
-    runs-on: ubuntu-latest
-    permissions:
-      id-token: write  # for PyPI trusted publishing
-
-    steps:
-    - uses: actions/checkout@v6
-
-    - name: Extract integration info
-      id: info
-      run: |
-        # refs/tags/integrations/litellm/v0.1.0 → integration=litellm, version=0.1.0
-        TAG="${GITHUB_REF#refs/tags/}"
-        INTEGRATION=$(echo "$TAG" | cut -d'/' -f2)
-        VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
-        echo "integration=$INTEGRATION" >> $GITHUB_OUTPUT
-        echo "version=$VERSION" >> $GITHUB_OUTPUT
-        echo "tag=$TAG" >> $GITHUB_OUTPUT
-        echo "Integration: $INTEGRATION, Version: $VERSION"
-
-    - name: Detect integration type
-      id: type
-      run: |
-        if [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/pyproject.toml" ]; then
-          echo "type=python" >> $GITHUB_OUTPUT
-        elif [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/package.json" ]; then
-          echo "type=typescript" >> $GITHUB_OUTPUT
-        else
-          echo "type=plugin" >> $GITHUB_OUTPUT
-        fi
-
-    # ── Python integrations (litellm, pydantic-ai, crewai) ──────────────────
-
-    - name: Install uv
-      if: steps.type.outputs.type == 'python'
-      uses: astral-sh/setup-uv@v7
-      with:
-        enable-cache: true
-
-    - name: Set up Python
-      if: steps.type.outputs.type == 'python'
-      uses: actions/setup-python@v6
-      with:
-        python-version-file: ".python-version"
-
-    - name: Build Python package
-      if: steps.type.outputs.type == 'python'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: uv build --out-dir dist
-
-    - name: Publish Python package to PyPI
-      if: steps.type.outputs.type == 'python'
-      uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        packages-dir: ./hindsight-integrations/${{ steps.info.outputs.integration }}/dist
-        skip-existing: true
-
-    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
-
-    # ── Plugin integrations (claude-code) — no package to publish ───────────
-
-    - name: Plugin release
-      if: steps.type.outputs.type == 'plugin'
-      run: |
-        echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
-        echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
-
-    # ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
-
-    - name: Set up Node.js
-      if: steps.type.outputs.type == 'typescript'
-      uses: actions/setup-node@v6
-      with:
-        node-version: '22'
-        registry-url: 'https://registry.npmjs.org'
-
-    # Guard: fail fast if the integration's lockfile resolves any dep from a
-    # monorepo workspace (link=true) or a relative file path. The release
-    # runner has no pre-built workspace `dist/` so `npm run build` would
-    # later fail at tsc with "Cannot find module". See:
-    # https://github.com/vectorize-io/hindsight/issues/… (0.6.0 openclaw retry)
-    - name: Check integration lockfile
-      if: steps.type.outputs.type == 'typescript'
-      run: ./scripts/check-integration-lockfiles.sh
-
-    - name: Install dependencies
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: npm ci
-
-    - name: Build TypeScript package
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: npm run build
-
-    - name: Publish TypeScript package to npm
-      if: steps.type.outputs.type == 'typescript'
-      working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
-      run: |
-        set +e
-        OUTPUT=$(npm publish --access public 2>&1)
-        EXIT_CODE=$?
-        echo "$OUTPUT"
-        if [ $EXIT_CODE -ne 0 ]; then
-          if echo "$OUTPUT" | grep -q "cannot publish over"; then
-            echo "Package version already published, skipping..."
-            exit 0
-          fi
-          exit $EXIT_CODE
-        fi
-      env:
-        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

.github/workflows/release.yml

@@ -13,15 +13,15 @@ jobs:
       id-token: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install uv
-      uses: astral-sh/setup-uv@v7
+      uses: astral-sh/setup-uv@v5
       with:
         enable-cache: true
 
     - name: Set up Python
-      uses: actions/setup-python@v6
+      uses: actions/setup-python@v5
       with:
         python-version-file: ".python-version"
 
@@ -30,39 +30,37 @@ jobs:
       working-directory: ./hindsight-clients/python
       run: uv build --out-dir dist
 
-    - name: Build hindsight-api-slim
-      working-directory: ./hindsight-api-slim
-      run: uv build --out-dir dist
-
     - name: Build hindsight-api
       working-directory: ./hindsight-api
       run: uv build --out-dir dist
 
     - name: Build hindsight-all
-      working-directory: ./hindsight-all
+      working-directory: ./hindsight
       run: uv build --out-dir dist
 
-    - name: Build hindsight-all-slim
-      working-directory: ./hindsight-all-slim
+    - name: Build hindsight-litellm
+      working-directory: ./hindsight-integrations/litellm
       run: uv build --out-dir dist
 
     - name: Build hindsight-embed
       working-directory: ./hindsight-embed
       run: uv build --out-dir dist
 
-    # Publish in order (client and api-slim first, then api/all wrappers which depend on them)
+    - name: Build hindsight-crewai
+      working-directory: ./hindsight-integrations/crewai
+      run: uv build --out-dir dist
+
+    - name: Build hindsight-pydantic-ai
+      working-directory: ./hindsight-integrations/pydantic-ai
+      run: uv build --out-dir dist
+
+    # Publish in order (client and api first, then hindsight-all which depends on them)
     - name: Publish hindsight-client to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         packages-dir: ./hindsight-clients/python/dist
         skip-existing: true
 
-    - name: Publish hindsight-api-slim to PyPI
-      uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        packages-dir: ./hindsight-api-slim/dist
-        skip-existing: true
-
     - name: Publish hindsight-api to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
@@ -72,13 +70,13 @@ jobs:
     - name: Publish hindsight-all to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight-all/dist
+        packages-dir: ./hindsight/dist
         skip-existing: true
 
-    - name: Publish hindsight-all-slim to PyPI
+    - name: Publish hindsight-litellm to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        packages-dir: ./hindsight-all-slim/dist
+        packages-dir: ./hindsight-integrations/litellm/dist
         skip-existing: true
 
     - name: Publish hindsight-embed to PyPI
@@ -87,18 +85,31 @@ jobs:
         packages-dir: ./hindsight-embed/dist
         skip-existing: true
 
+    - name: Publish hindsight-crewai to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        packages-dir: ./hindsight-integrations/crewai/dist
+        skip-existing: true
+
+    - name: Publish hindsight-pydantic-ai to PyPI
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        packages-dir: ./hindsight-integrations/pydantic-ai/dist
+        skip-existing: true
+
     # Upload artifacts for GitHub release
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: python-packages
         path: |
           hindsight-clients/python/dist/*
-          hindsight-api-slim/dist/*
           hindsight-api/dist/*
-          hindsight-all/dist/*
-          hindsight-all-slim/dist/*
+          hindsight/dist/*
+          hindsight-integrations/litellm/dist/*
           hindsight-embed/dist/*
+          hindsight-integrations/crewai/dist/*
+          hindsight-integrations/pydantic-ai/dist/*
         retention-days: 1
 
   release-typescript-client:
@@ -106,10 +117,10 @@ jobs:
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -144,35 +155,35 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: typescript-client
         path: hindsight-clients/typescript/*.tgz
         retention-days: 1
 
-  release-hindsight-all-npm:
+  release-openclaw-integration:
     runs-on: ubuntu-latest
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '22'
         registry-url: 'https://registry.npmjs.org'
-        cache: 'npm'
-        cache-dependency-path: package-lock.json
 
     - name: Install dependencies
-      run: npm ci --workspace=hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
+      run: npm ci
 
     - name: Build
-      run: npm run build --workspace=hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
+      run: npm run build
 
     - name: Publish to npm
-      working-directory: ./hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
       run: |
         set +e
         OUTPUT=$(npm publish --access public 2>&1)
@@ -189,14 +200,112 @@ jobs:
         NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
 
     - name: Pack for GitHub release
-      working-directory: ./hindsight-all-npm
+      working-directory: ./hindsight-integrations/openclaw
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
-        name: hindsight-all-npm
-        path: hindsight-all-npm/*.tgz
+        name: openclaw-integration
+        path: hindsight-integrations/openclaw/*.tgz
+        retention-days: 1
+
+  release-ai-sdk-integration:
+    runs-on: ubuntu-latest
+    environment: npm
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm ci
+
+    - name: Build
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm run build
+
+    - name: Publish to npm
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+    - name: Pack for GitHub release
+      working-directory: ./hindsight-integrations/ai-sdk
+      run: npm pack
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: ai-sdk-integration
+        path: hindsight-integrations/ai-sdk/*.tgz
+        retention-days: 1
+
+  release-chat-integration:
+    runs-on: ubuntu-latest
+    environment: npm
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '22'
+        registry-url: 'https://registry.npmjs.org'
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/chat
+      run: npm ci
+
+    - name: Build
+      working-directory: ./hindsight-integrations/chat
+      run: npm run build
+
+    - name: Publish to npm
+      working-directory: ./hindsight-integrations/chat
+      run: |
+        set +e
+        OUTPUT=$(npm publish --access public 2>&1)
+        EXIT_CODE=$?
+        echo "$OUTPUT"
+        if [ $EXIT_CODE -ne 0 ]; then
+          if echo "$OUTPUT" | grep -q "cannot publish over"; then
+            echo "Package version already published, skipping..."
+            exit 0
+          fi
+          exit $EXIT_CODE
+        fi
+      env:
+        NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+    - name: Pack for GitHub release
+      working-directory: ./hindsight-integrations/chat
+      run: npm pack
+
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v4
+      with:
+        name: chat-integration
+        path: hindsight-integrations/chat/*.tgz
         retention-days: 1
 
   release-control-plane:
@@ -204,10 +313,10 @@ jobs:
     environment: npm
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Set up Node.js
-      uses: actions/setup-node@v6
+      uses: actions/setup-node@v4
       with:
         node-version: '20'
         registry-url: 'https://registry.npmjs.org'
@@ -255,7 +364,7 @@ jobs:
       run: npm pack
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: control-plane
         path: hindsight-control-plane/*.tgz
@@ -278,13 +387,9 @@ jobs:
             target: aarch64-apple-darwin
             artifact_name: hindsight
             asset_name: hindsight-darwin-arm64
-          - os: ubuntu-24.04-arm
-            target: aarch64-unknown-linux-gnu
-            artifact_name: hindsight
-            asset_name: hindsight-linux-arm64
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install Rust
       uses: dtolnay/rust-toolchain@stable
@@ -302,7 +407,7 @@ jobs:
         chmod +x artifacts/${{ matrix.asset_name }}
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: rust-cli-${{ matrix.asset_name }}
         path: artifacts/${{ matrix.asset_name }}
@@ -343,7 +448,7 @@ jobs:
               PRELOAD_ML_MODELS=false
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Free Disk Space
       uses: jlumbroso/free-disk-space@main
@@ -357,13 +462,13 @@ jobs:
         swap-storage: true
 
     - name: Set up QEMU
-      uses: docker/setup-qemu-action@v4
+      uses: docker/setup-qemu-action@v3
 
     - name: Set up Docker Buildx
-      uses: docker/setup-buildx-action@v4
+      uses: docker/setup-buildx-action@v3
 
     - name: Log in to GitHub Container Registry
-      uses: docker/login-action@v4
+      uses: docker/login-action@v3
       with:
         registry: ghcr.io
         username: ${{ github.actor }}
@@ -375,7 +480,7 @@ jobs:
 
     - name: Extract metadata for release tags
       id: meta
-      uses: docker/metadata-action@v6
+      uses: docker/metadata-action@v5
       with:
         images: ghcr.io/${{ github.repository_owner }}/${{ matrix.image_name }}
         flavor: |
@@ -391,7 +496,7 @@ jobs:
     # # Step 1: Build for local testing (single platform, no push)
     # # This creates an identical image to what will be released, just for one platform
     # - name: Build image for testing
-    #   uses: docker/build-push-action@v7
+    #   uses: docker/build-push-action@v6
     #   with:
     #     context: .
     #     file: docker/standalone/Dockerfile
@@ -410,7 +515,7 @@ jobs:
 
     # Build multi-platform and push to release tags
     - name: Build and push release images
-      uses: docker/build-push-action@v7
+      uses: docker/build-push-action@v6
       with:
         context: .
         file: docker/standalone/Dockerfile
@@ -428,10 +533,10 @@ jobs:
       packages: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Install Helm
-      uses: azure/setup-helm@v5
+      uses: azure/setup-helm@v4
       with:
         version: 'latest'
 
@@ -448,7 +553,7 @@ jobs:
       run: helm push helm-packages/*.tgz oci://ghcr.io/${{ github.repository_owner }}/charts
 
     - name: Upload artifacts
-      uses: actions/upload-artifact@v7
+      uses: actions/upload-artifact@v4
       with:
         name: helm-chart
         path: helm-packages/*.tgz
@@ -456,61 +561,73 @@ jobs:
 
   create-github-release:
     runs-on: ubuntu-latest
-    needs: [release-python-packages, release-typescript-client, release-hindsight-all-npm, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
+    needs: [release-python-packages, release-typescript-client, release-openclaw-integration, release-ai-sdk-integration, release-chat-integration, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
     permissions:
       contents: write
 
     steps:
-    - uses: actions/checkout@v6
+    - uses: actions/checkout@v4
 
     - name: Extract version from tag
       id: get_version
       run: echo "VERSION=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT
 
     - name: Download Python packages
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: python-packages
         path: ./artifacts/python-packages
 
     - name: Download TypeScript client
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: typescript-client
         path: ./artifacts/typescript-client
 
+    - name: Download OpenClaw Integration
+      uses: actions/download-artifact@v4
+      with:
+        name: openclaw-integration
+        path: ./artifacts/openclaw-integration
+
+    - name: Download AI SDK Integration
+      uses: actions/download-artifact@v4
+      with:
+        name: ai-sdk-integration
+        path: ./artifacts/ai-sdk-integration
+
+    - name: Download Chat Integration
+      uses: actions/download-artifact@v4
+      with:
+        name: chat-integration
+        path: ./artifacts/chat-integration
+
     - name: Download Control Plane
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: control-plane
         path: ./artifacts/control-plane
 
-    - name: Download hindsight-embed npm wrapper
-      uses: actions/download-artifact@v8
-      with:
-        name: hindsight-all-npm
-        path: ./artifacts/hindsight-all-npm
-
     - name: Download Rust CLI (Linux)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-linux-amd64
         path: ./artifacts/rust-cli-linux
 
     - name: Download Rust CLI (macOS Intel)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-darwin-amd64
         path: ./artifacts/rust-cli-darwin-amd64
 
     - name: Download Rust CLI (macOS ARM)
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: rust-cli-hindsight-darwin-arm64
         path: ./artifacts/rust-cli-darwin-arm64
 
     - name: Download Helm chart
-      uses: actions/download-artifact@v8
+      uses: actions/download-artifact@v4
       with:
         name: helm-chart
         path: ./artifacts/helm-chart
@@ -520,15 +637,19 @@ jobs:
         mkdir -p release-assets
         # Python packages
         cp artifacts/python-packages/hindsight-clients/python/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-api-slim/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-api/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-all/dist/* release-assets/ || true
-        cp artifacts/python-packages/hindsight-all-slim/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-integrations/litellm/dist/* release-assets/ || true
+        cp artifacts/python-packages/hindsight-integrations/pydantic-ai/dist/* release-assets/ || true
         cp artifacts/python-packages/hindsight-embed/dist/* release-assets/ || true
         # TypeScript client
         cp artifacts/typescript-client/*.tgz release-assets/ || true
-        # hindsight-embed npm wrapper
-        cp artifacts/hindsight-all-npm/*.tgz release-assets/ || true
+        # OpenClaw Integration
+        cp artifacts/openclaw-integration/*.tgz release-assets/ || true
+        # AI SDK Integration
+        cp artifacts/ai-sdk-integration/*.tgz release-assets/ || true
+        # Chat Integration
+        cp artifacts/chat-integration/*.tgz release-assets/ || true
         # Control Plane
         cp artifacts/control-plane/*.tgz release-assets/ || true
         # Rust CLI binaries
@@ -540,7 +661,7 @@ jobs:
         ls -la release-assets/
 
     - name: Create GitHub Release
-      uses: softprops/action-gh-release@v3
+      uses: softprops/action-gh-release@v2
       with:
         files: release-assets/*
         generate_release_notes: true

.gitignore

@@ -50,8 +50,7 @@ hindsight-dev/benchmarks/perf/results/
 benchmarks/results/
 hindsight-cli/target
 hindsight-clients/rust/target
-.claude/*
-!.claude/skills/
+.claude
 whats-next.md
 TASK.md
 # Changelog is now tracked in hindsight-docs/src/pages/changelog.md

.prettierrc.json

@@ -1,7 +0,0 @@
-{
-  "semi": true,
-  "singleQuote": false,
-  "tabWidth": 2,
-  "trailingComma": "es5",
-  "printWidth": 100
-}

CLAUDE.md

@@ -11,32 +11,26 @@ Hindsight is an agent memory system that provides long-term memory for AI agents
 
 ## Development Commands
 
-### Local Development (API + UI)
-```bash
-# Start both API server and control plane UI
-./scripts/dev/start.sh
-```
-
 ### API Server (Python/FastAPI)
 ```bash
-# Start API server only (loads .env automatically)
+# Start API server (loads .env automatically)
 ./scripts/dev/start-api.sh
 
 # Run all tests (parallelized with pytest-xdist)
-cd hindsight-api-slim && uv run pytest tests/
+cd hindsight-api && uv run pytest tests/
 
 # Run specific test file
-cd hindsight-api-slim && uv run pytest tests/test_http_api_integration.py -v
+cd hindsight-api && uv run pytest tests/test_http_api_integration.py -v
 
 # Run single test function
-cd hindsight-api-slim && uv run pytest tests/test_retain.py::test_retain_simple -v
+cd hindsight-api && uv run pytest tests/test_retain.py::test_retain_simple -v
 
 # Lint and format
-cd hindsight-api-slim && uv run ruff check .
-cd hindsight-api-slim && uv run ruff format .
+cd hindsight-api && uv run ruff check .
+cd hindsight-api && uv run ruff format .
 
 # Type checking (uses ty - extremely fast type checker from Astral)
-cd hindsight-api-slim && uv run ty check hindsight_api/
+cd hindsight-api && uv run ty check hindsight_api/
 ```
 
 ### Control Plane (Next.js)
@@ -68,9 +62,8 @@ cd hindsight-control-plane && npm run dev
 ./scripts/benchmarks/run-locomo.sh
 
 # Performance benchmarks
-./scripts/benchmarks/run-perf-test.sh                      # System perf (mock LLM + pg0)
-./scripts/benchmarks/run-perf-test.sh --scale tiny          # Quick smoke test
 ./scripts/benchmarks/run-consolidation.sh
+./scripts/benchmarks/run-retain-perf.sh --document <path>  # Requires API server running
 
 # Results viewer
 ./scripts/benchmarks/start-visualizer.sh  # View results at localhost:8001
@@ -79,17 +72,18 @@ cd hindsight-control-plane && npm run dev
 ## Architecture
 
 ### Monorepo Structure
-- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)
+- **hindsight-api/**: Core FastAPI server with memory engine (Python, uv)
+- **hindsight/**: Embedded Python bundle (hindsight-all package)
 - **hindsight-control-plane/**: Admin UI (Next.js, npm)
 - **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)
 - **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)
 - **hindsight-docs/**: Docusaurus documentation site
-- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)
+- **hindsight-integrations/**: Framework integrations (LiteLLM, OpenAI)
 - **hindsight-dev/**: Development tools and benchmarks
 
-### Core Engine (hindsight-api-slim/hindsight_api/engine/)
-- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
-- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
+### Core Engine (hindsight-api/hindsight_api/engine/)
+- `memory_engine.py`: Main orchestrator (~170KB) for retain/recall/reflect operations
+- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, Groq, Ollama, LM Studio
 - `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
 - `cross_encoder.py`: Reranking (local or TEI)
 - `entity_resolver.py`: Entity extraction and normalization
@@ -102,13 +96,13 @@ cd hindsight-control-plane && npm run dev
 
 **search/**: Multi-strategy retrieval
 - `retrieval.py`: Main retrieval orchestrator
-- `graph_retrieval.py`: Graph retrieval abstract base class
-- `link_expansion_retrieval.py`: Link expansion graph retrieval
+- `graph_retrieval.py`: Entity/relationship graph traversal
+- `mpfp_retrieval.py`: Multi-Path Fact Propagation retrieval
 - `fusion.py`: Reciprocal rank fusion for combining results
 - `reranking.py`: Cross-encoder reranking
 
-### API Layer (hindsight-api-slim/hindsight_api/api/)
-- `http.py`: FastAPI HTTP routers for all REST endpoints
+### API Layer (hindsight-api/hindsight_api/api/)
+- `http.py`: FastAPI HTTP routers (~80KB) for all REST endpoints
 - `mcp.py`: Model Context Protocol server implementation
 
 Main operations:
@@ -117,13 +111,13 @@ Main operations:
 - **Reflect**: Disposition-aware reasoning using memories and mental models.
 
 ### Database
-PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api-slim/hindsight_api/alembic/`. Migrations run automatically on API startup.
+PostgreSQL with pgvector. Schema managed via Alembic migrations in `hindsight-api/hindsight_api/alembic/`. Migrations run automatically on API startup.
 
 Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 
 ### Adding Database Migrations
 
-1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:
+1. **Create a new migration file** in `hindsight-api/hindsight_api/alembic/versions/`:
    - File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)
    - Use a unique hex revision ID (12 chars)
    - Set `down_revision` to the previous migration's revision ID
@@ -160,7 +154,7 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 
 3. **Run migrations locally**:
    ```bash
-   # Set database URL and run migrations for the base schema plus all tenants
+   # Set database URL and run migrations
    uv run hindsight-admin run-db-migration
 
    # Run on a specific tenant schema
@@ -170,17 +164,11 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
 ## Key Conventions
 
 ### Code Quality
-
-**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).
-
 **Always run the lint script after making Python or TypeScript/Node changes:**
 ```bash
 ./scripts/hooks/lint.sh
 ```
-
-**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
-
-**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
+This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Prettier for TypeScript).
 
 ### Memory Banks
 - Each bank is an isolated memory store (like a "brain" for one user/agent)
@@ -212,20 +200,48 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
    - Update the client type definition in `lib/api.ts`
    - Update any UI components that need to use the new parameter
 
-### Adding New Integrations
+### Python Style
+- Python 3.11+, type hints required
+- Async throughout (asyncpg, async FastAPI)
+- Pydantic models for request/response
+- Ruff for linting (line-length 120)
+- No Python files at project root - maintain clean directory structure
+- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
 
-Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
+### Type Safety with Pydantic Models
+**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
+- Use Pydantic `BaseModel` for all data structures passed between functions
+- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
+- Avoid `dict.get()` patterns - use typed model attributes instead
+- Parse external data (JSON, API responses) into Pydantic models at the boundary
+- This catches type errors at parse time, not deep in business logic
 
-1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.
-2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.
-3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.
-4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).
+```python
+# BAD - error-prone dict access
+def process(data: dict) -> str:
+    return data.get("name", "")  # No validation, silent failures
 
-If any of these are missing, the integration is incomplete and must not be pushed or merged.
+# GOOD - typed and validated
+class UserData(BaseModel):
+    name: str
+    created_at: datetime
 
-### Changelogs
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def ensure_tz_aware(cls, v):
+        if isinstance(v, str):
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if v.tzinfo is None:
+            return v.replace(tzinfo=timezone.utc)
+        return v
 
-Never add "Unreleased" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.
+def process(data: UserData) -> str:
+    return data.name  # Type-safe, validated at construction
+```
+
+### TypeScript Style
+- Next.js App Router for control plane
+- Tailwind CSS with shadcn/ui components
 
 ### Adding New API Configuration Flags
 
@@ -235,24 +251,24 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
 
 #### Adding a New Configuration Field
 
-1. **config.py** (`hindsight-api-slim/hindsight_api/config.py`):
+1. **config.py** (`hindsight-api/hindsight_api/config.py`):
    - Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
    - Add `DEFAULT_*` constant for the default value
    - Add field to `HindsightConfig` dataclass with type annotation
-   - **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API
+   - **Mark as hierarchical or static** by adding to `_HIERARCHICAL_FIELDS` set (hierarchical) or leaving it out (static)
    - Add initialization in `from_env()` method
 
    ```python
-   # Configurable field (can be overridden per-tenant/bank via API)
-   _CONFIGURABLE_FIELDS = {
+   # Hierarchical field (can be overridden per-bank)
+   _HIERARCHICAL_FIELDS = {
        ...,
-       "my_setting",  # Add here for configurable
+       "my_setting",  # Add here for hierarchical
    }
 
-   # Static field - just don't add to _CONFIGURABLE_FIELDS
+   # Static field - just don't add to _HIERARCHICAL_FIELDS
    ```
 
-2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
+2. **main.py** (`hindsight-api/hindsight_api/main.py`):
    - Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
 
 3. **Use hierarchical config in MemoryEngine**:
@@ -292,14 +308,14 @@ cp .env.example .env
 # Edit .env with LLM API key
 
 # Python deps
-uv sync --directory hindsight-api-slim/
+uv sync --directory hindsight-api/
 
 # Node deps (uses npm workspaces)
 npm install
 ```
 
 Required env vars:
-- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, minimax, ollama, lmstudio
+- `HINDSIGHT_API_LLM_PROVIDER`: openai, anthropic, gemini, groq, ollama, lmstudio
 - `HINDSIGHT_API_LLM_API_KEY`: Your API key
 - `HINDSIGHT_API_LLM_MODEL`: Model name (e.g., gpt-4o-mini, claude-sonnet-4-20250514)
 

README.md

@@ -7,12 +7,10 @@
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
 [![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
-<br/>
 
-<a href="https://trendshift.io/repositories/15603" target="_blank"><img src="https://trendshift.io/api/badge/repositories/15603" alt="vectorize-io%2Fhindsight | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
+
 </div>
 
 ---
@@ -71,7 +69,7 @@ docker run --rm -it --pull always -p 8888:8888 -p 9999:9999 \
 >API: http://localhost:8888
 >UI: http://localhost:9999
 
-You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `minimax`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
+You can modify the LLM provider by setting `HINDSIGHT_API_LLM_PROVIDER`. Valid options are `openai`, `anthropic`, `gemini`, `groq`, `ollama`, and `lmstudio`. The documentation provides more details on [supported models](https://hindsight.vectorize.io/developer/models).
 
 
 

deno.lock

@@ -1,139 +0,0 @@
-{
-  "version": "5",
-  "specifiers": {
-    "jsr:@std/assert@^1.0.17": "1.0.19",
-    "jsr:@std/assert@^1.0.19": "1.0.19",
-    "jsr:@std/expect@*": "1.0.18",
-    "jsr:@std/internal@^1.0.12": "1.0.12",
-    "jsr:@std/path@^1.1.4": "1.1.4",
-    "jsr:@std/testing@*": "1.0.17"
-  },
-  "jsr": {
-    "@std/[email protected]": {
-      "integrity": "eaada96ee120cb980bc47e040f82814d786fe8162ecc53c91d8df60b8755991e",
-      "dependencies": [
-        "jsr:@std/internal"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "8566eab35200466f8609eb7e7aed062ed0db314e9a258d5d201b1b8997ce801a",
-      "dependencies": [
-        "jsr:@std/assert@^1.0.19",
-        "jsr:@std/internal",
-        "jsr:@std/path"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "972a634fd5bc34b242024402972cd5143eac68d8dffaca5eaa4dba30ce17b027"
-    },
-    "@std/[email protected]": {
-      "integrity": "1d2d43f39efb1b42f0b1882a25486647cb851481862dc7313390b2bb044314b5",
-      "dependencies": [
-        "jsr:@std/internal"
-      ]
-    },
-    "@std/[email protected]": {
-      "integrity": "87bdc2700fa98249d48a17cd72413352d3d3680dcfbdb64947fd0982d6bbf681",
-      "dependencies": [
-        "jsr:@std/assert@^1.0.17",
-        "jsr:@std/internal"
-      ]
-    }
-  },
-  "workspace": {
-    "members": {
-      "hindsight-clients/typescript": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@hey-api/[email protected]",
-            "npm:@types/jest@29",
-            "npm:@types/node@20",
-            "npm:jest@29",
-            "npm:ts-jest@29",
-            "npm:tsup@^8.5.1",
-            "npm:typescript@5"
-          ]
-        }
-      },
-      "hindsight-control-plane": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@eslint/eslintrc@^3.3.3",
-            "npm:@eslint/js@^9.39.2",
-            "npm:@radix-ui/react-alert-dialog@^1.1.15",
-            "npm:@radix-ui/react-checkbox@^1.3.3",
-            "npm:@radix-ui/react-dialog@^1.1.15",
-            "npm:@radix-ui/react-dropdown-menu@^2.1.16",
-            "npm:@radix-ui/react-label@^2.1.8",
-            "npm:@radix-ui/react-popover@^1.1.15",
-            "npm:@radix-ui/react-radio-group@^1.3.8",
-            "npm:@radix-ui/react-select@^2.2.6",
-            "npm:@radix-ui/react-slider@^1.3.6",
-            "npm:@radix-ui/react-slot@^1.2.4",
-            "npm:@radix-ui/react-switch@^1.2.6",
-            "npm:@radix-ui/react-tabs@^1.1.13",
-            "npm:@radix-ui/react-tooltip@^1.2.8",
-            "npm:@tailwindcss/postcss@^4.1.17",
-            "npm:@tailwindcss/typography@~0.5.19",
-            "npm:@types/cytoscape@^3.21.9",
-            "npm:@types/node@^24.10.0",
-            "npm:@types/react-dom@^19.2.2",
-            "npm:@types/react@^19.2.2",
-            "npm:autoprefixer@^10.4.21",
-            "npm:class-variance-authority@~0.7.1",
-            "npm:clsx@^2.1.1",
-            "npm:cmdk@^1.1.1",
-            "npm:cytoscape-fcose@^2.2.0",
-            "npm:cytoscape@^3.33.1",
-            "npm:eslint-config-next@^16.0.1",
-            "npm:eslint-plugin-react-hooks@^7.0.1",
-            "npm:eslint-plugin-react@^7.37.5",
-            "npm:eslint@^9.39.1",
-            "npm:[email protected]",
-            "npm:next-themes@~0.4.6",
-            "npm:next@^16.1.6",
-            "npm:postcss@^8.5.6",
-            "npm:prettier@^3.7.4",
-            "npm:react-chrono@^2.9.1",
-            "npm:react-dom@^19.2.0",
-            "npm:react-markdown@^10.1.0",
-            "npm:react18-json-view@~0.2.9",
-            "npm:react@^19.2.0",
-            "npm:recharts@^3.5.1",
-            "npm:remark-gfm@^4.0.1",
-            "npm:sonner@^2.0.7",
-            "npm:tailwind-merge@^3.4.0",
-            "npm:tailwindcss-animate@^1.0.7",
-            "npm:tailwindcss@^4.1.17",
-            "npm:[email protected]",
-            "npm:typescript-eslint@^8.50.0",
-            "npm:typescript@^5.9.3"
-          ]
-        }
-      },
-      "hindsight-docs": {
-        "packageJson": {
-          "dependencies": [
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/theme-common@^3.9.2",
-            "npm:@docusaurus/theme-mermaid@^3.9.2",
-            "npm:@docusaurus/[email protected]",
-            "npm:@docusaurus/[email protected]",
-            "npm:@easyops-cn/docusaurus-search-local@~0.52.2",
-            "npm:@mdx-js/react@3",
-            "npm:clsx@2",
-            "npm:prism-react-renderer@^2.3.0",
-            "npm:raw-loader@^4.0.2",
-            "npm:react-dom@19",
-            "npm:react-icons@^5.6.0",
-            "npm:react@19",
-            "npm:redocusaurus@^2.5.0",
-            "npm:typescript@~5.6.2"
-          ]
-        }
-      }
-    }
-  }
-}

docker/standalone/Dockerfile

@@ -42,22 +42,25 @@ RUN apt-get update && apt-get install -y \
     && pip install --no-cache-dir uv
 
 # Copy dependency files and README (required by pyproject.toml)
-COPY hindsight-api-slim/pyproject.toml ./api/
-COPY hindsight-api-slim/README.md ./api/
+COPY hindsight-api/pyproject.toml ./api/
+COPY hindsight-api/README.md ./api/
 
 WORKDIR /app/api
 
-# Sync dependencies using appropriate extras based on INCLUDE_LOCAL_MODELS
-# local-ml: torch, sentence-transformers, transformers, einops, flashrank, mlx (optional)
-# embedded-db: pg0-embedded (always included for embedded PostgreSQL support)
-RUN if [ "$INCLUDE_LOCAL_MODELS" = "true" ]; then \
-        uv sync --extra local-ml --extra embedded-db; \
-    else \
-        uv sync --extra embedded-db; \
+# Remove local ML model dependencies if INCLUDE_LOCAL_MODELS=false
+# This creates a smaller image when using external providers (TEI, OpenAI, Cohere)
+RUN if [ "$INCLUDE_LOCAL_MODELS" != "true" ]; then \
+        echo "Removing local-models dependencies (sentence-transformers, torch, transformers)..." && \
+        sed -i '/"sentence-transformers/d' pyproject.toml && \
+        sed -i '/"transformers/d' pyproject.toml && \
+        sed -i '/"torch/d' pyproject.toml; \
     fi
 
+# Sync dependencies (will create lock file if needed)
+RUN uv sync
+
 # Copy source code (alembic migrations are inside hindsight_api/)
-COPY hindsight-api-slim/hindsight_api ./hindsight_api
+COPY hindsight-api/hindsight_api ./hindsight_api
 
 # Install the local package (uv sync only installed dependencies, not the package itself)
 RUN uv pip install -e .

docker/standalone/start-all.sh

@@ -1,28 +1,6 @@
 #!/bin/bash
 set -e
 
-# =============================================================================
-# Embedded pg0 data integrity check (#675)
-#
-# When using embedded pg0, check if the data directory has existing PostgreSQL
-# data before starting. If the directory exists but appears empty/corrupt
-# (e.g., missing PG_VERSION file), log a warning. This helps diagnose data
-# loss scenarios where a container restart caused the data directory to be
-# wiped despite a volume mount being present.
-# =============================================================================
-PG0_DATA_DIR="${HOME}/.pg0"
-if [ -d "$PG0_DATA_DIR" ]; then
-    # Look for actual PostgreSQL data directories (pg0 creates subdirs per instance)
-    if compgen -G "$PG0_DATA_DIR"/*/PG_VERSION > /dev/null 2>&1; then
-        echo "✅ Existing pg0 data directory detected at $PG0_DATA_DIR"
-    elif [ "$(ls -A "$PG0_DATA_DIR" 2>/dev/null)" ]; then
-        echo "⚠️  WARNING: pg0 data directory exists at $PG0_DATA_DIR but no PG_VERSION found."
-        echo "   This may indicate data corruption or an incomplete previous shutdown."
-        echo "   If you see all migrations running from scratch after this, your data may have been lost."
-        echo "   See: https://github.com/vectorize-io/hindsight/issues/675"
-    fi
-fi
-
 # Service flags (default to true if not set)
 ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
 ENABLE_CP="${HINDSIGHT_ENABLE_CP:-true}"
@@ -93,95 +71,24 @@ if [ "${HINDSIGHT_WAIT_FOR_DEPS:-false}" = "true" ]; then
     done
 fi
 
-# =============================================================================
-# Graceful shutdown handler (#675)
-#
-# Docker sends SIGTERM on `docker stop`/`docker restart`. Without a trap, child
-# processes (hindsight-api + pg0, control-plane) are killed abruptly. For the
-# embedded pg0 database this can cause data loss when the data directory is on
-# a Docker volume that gets remounted after restart.
-#
-# The trap forwards SIGTERM to all tracked child PIDs so that:
-#   - hindsight-api receives the signal and can run its shutdown hooks
-#   - pg0 gets a clean PostgreSQL shutdown (checkpoint + WAL flush)
-#   - The control-plane Node.js process exits cleanly
-# =============================================================================
-# Guard against concurrent cleanup (e.g., child crash + SIGTERM arriving together)
-SHUTTING_DOWN=false
-
-cleanup() {
-    if $SHUTTING_DOWN; then return; fi
-    SHUTTING_DOWN=true
-
-    echo ""
-    echo "🛑 Received shutdown signal, stopping services gracefully..."
-    for pid in "${PIDS[@]}"; do
-        if kill -0 "$pid" 2>/dev/null; then
-            kill -TERM "$pid" 2>/dev/null
-        fi
-    done
-    # Give processes time to shut down cleanly (pg0 needs to flush WAL).
-    # NOTE: Docker's default stop_grace_period is 10s. If you use the default,
-    # either set stop_grace_period: 30s in your compose file / docker stop -t 30,
-    # or Docker will SIGKILL the container before this timeout expires.
-    local timeout=30
-    for ((i=1; i<=timeout; i++)); do
-        local all_stopped=true
-        for pid in "${PIDS[@]}"; do
-            if kill -0 "$pid" 2>/dev/null; then
-                all_stopped=false
-                break
-            fi
-        done
-        if $all_stopped; then
-            echo "✅ All services stopped cleanly"
-            exit 0
-        fi
-        sleep 1
-    done
-    # Force kill if still running after timeout
-    echo "⚠️  Timeout reached, forcing shutdown..."
-    for pid in "${PIDS[@]}"; do
-        if kill -0 "$pid" 2>/dev/null; then
-            kill -9 "$pid" 2>/dev/null
-        fi
-    done
-    exit 1
-}
-trap cleanup SIGTERM SIGINT
-
 # Track PIDs for wait
 PIDS=()
 
 # Start API if enabled
 if [ "$ENABLE_API" = "true" ]; then
     cd /app/api
-    API_HEALTH_URL="${HINDSIGHT_API_HEALTH_URL:-http://localhost:8888/health}"
-    API_STARTUP_WAIT_SECONDS="${HINDSIGHT_API_STARTUP_WAIT_SECONDS:-300}"
-
     # Run API directly - Python's PYTHONUNBUFFERED=1 handles output buffering
     hindsight-api &
     API_PID=$!
     PIDS+=($API_PID)
 
     # Wait for API to be ready
-    api_ready=false
-    for ((i=1; i<=API_STARTUP_WAIT_SECONDS; i++)); do
-        if ! kill -0 "$API_PID" 2>/dev/null; then
-            wait "$API_PID"
-            exit $?
-        fi
-        if curl -sf "$API_HEALTH_URL" &>/dev/null; then
-            api_ready=true
+    for i in {1..60}; do
+        if curl -sf http://localhost:8888/health &>/dev/null; then
             break
         fi
         sleep 1
     done
-
-    if [ "$api_ready" != "true" ]; then
-        echo "❌ API did not become healthy within ${API_STARTUP_WAIT_SECONDS}s"
-        exit 1
-    fi
 else
     echo "API disabled (HINDSIGHT_ENABLE_API=false)"
 fi
@@ -190,7 +97,6 @@ fi
 if [ "$ENABLE_CP" = "true" ]; then
     echo "🎛️  Starting Control Plane..."
     cd /app/control-plane
-    export HOSTNAME="${HINDSIGHT_CP_HOSTNAME:-0.0.0.0}"
     PORT="${HINDSIGHT_CP_PORT:-9999}" node server.js &
     CP_PID=$!
     PIDS+=($CP_PID)
@@ -217,21 +123,8 @@ if [ ${#PIDS[@]} -eq 0 ]; then
     exit 1
 fi
 
-# Wait for any process to exit (use wait -n with trap-safe loop)
-while true; do
-    # wait -n returns when any child exits; it also returns on signal delivery
-    # (the trap handler will run and exit, so this loop is just for robustness).
-    # `&& true` prevents `set -e` from killing the script when wait -n returns
-    # non-zero (child exited with error or no backgrounded children remain).
-    wait -n && true
-    # Check if any tracked PID has exited
-    for pid in "${PIDS[@]}"; do
-        if ! kill -0 "$pid" 2>/dev/null; then
-            wait "$pid" 2>/dev/null
-            exit_code=$?
-            echo "⚠️  Service (PID $pid) exited with code $exit_code"
-            # Trigger cleanup for remaining services
-            cleanup
-        fi
-    done
-done
+# Wait for any process to exit
+wait -n
+
+# Exit with status of first exited process
+exit $?

docker/test-image.sh

@@ -49,9 +49,6 @@
 
 set -euo pipefail
 
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-REPO_ROOT="$(dirname "$SCRIPT_DIR")"
-
 # Colors for output
 RED='\033[0;31m'
 GREEN='\033[0;32m'
@@ -181,21 +178,6 @@ for i in $(seq 1 "$TIMEOUT"); do
         echo "=== Health Response ==="
         curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}" | python3 -m json.tool 2>/dev/null || curl -s "http://localhost:${HEALTH_PORT}${HEALTH_PATH}"
         echo ""
-
-        # Run retain/recall smoke test for API targets
-        if [ "$TARGET" != "cp-only" ]; then
-            echo ""
-            echo "=== Retain/Recall Smoke Test ==="
-            if ! "$REPO_ROOT/scripts/smoke-test-slim.sh" "http://localhost:${HEALTH_PORT}"; then
-                echo ""
-                echo "=== Container Logs (last 50 lines) ==="
-                docker logs "$CONTAINER_NAME" 2>&1 | tail -50
-                echo ""
-                echo -e "${RED}Smoke test FAILED${NC}"
-                exit 1
-            fi
-        fi
-
         echo ""
         echo "=== Container Logs (last 50 lines) ==="
         docker logs "$CONTAINER_NAME" 2>&1 | tail -50

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.5.4
-appVersion: "0.5.4"
+version: 0.4.15
+appVersion: "0.4.15"
 keywords:
   - ai
   - memory

helm/hindsight/templates/api-deployment.yaml

@@ -95,27 +95,6 @@ spec:
           {{- toYaml .Values.api.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.api.resources | nindent 10 }}
-        {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumeMounts }}
-        volumeMounts:
-        {{- if .Values.api.persistence.modelCache.enabled }}
-        - name: model-cache
-          mountPath: /home/hindsight/.cache
-        {{- end }}
-        {{- with .Values.api.extraVolumeMounts }}
-        {{- toYaml . | nindent 8 }}
-        {{- end }}
-        {{- end }}
-      {{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumes }}
-      volumes:
-      {{- if .Values.api.persistence.modelCache.enabled }}
-      - name: model-cache
-        persistentVolumeClaim:
-          claimName: {{ include "hindsight.fullname" . }}-api-model-cache
-      {{- end }}
-      {{- with .Values.api.extraVolumes }}
-      {{- toYaml . | nindent 6 }}
-      {{- end }}
-      {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}

helm/hindsight/templates/api-model-cache-pvc.yaml

@@ -1,21 +0,0 @@
-{{- if and .Values.api.enabled .Values.api.persistence.modelCache.enabled }}
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: {{ include "hindsight.fullname" . }}-api-model-cache
-  labels:
-    {{- include "hindsight.api.labels" . | nindent 4 }}
-  {{- with .Values.api.persistence.modelCache.annotations }}
-  annotations:
-    {{- toYaml . | nindent 4 }}
-  {{- end }}
-spec:
-  accessModes:
-    {{- toYaml .Values.api.persistence.modelCache.accessModes | nindent 4 }}
-  {{- if .Values.api.persistence.modelCache.storageClass }}
-  storageClassName: {{ .Values.api.persistence.modelCache.storageClass }}
-  {{- end }}
-  resources:
-    requests:
-      storage: {{ .Values.api.persistence.modelCache.size }}
-{{- end }}

helm/hindsight/templates/worker-statefulset.yaml

@@ -95,16 +95,6 @@ spec:
           {{- toYaml .Values.worker.readinessProbe | nindent 10 }}
         resources:
           {{- toYaml .Values.worker.resources | nindent 10 }}
-        {{- if or .Values.worker.persistence.modelCache.enabled .Values.worker.extraVolumeMounts }}
-        volumeMounts:
-        {{- if .Values.worker.persistence.modelCache.enabled }}
-        - name: model-cache
-          mountPath: /home/hindsight/.cache
-        {{- end }}
-        {{- with .Values.worker.extraVolumeMounts }}
-        {{- toYaml . | nindent 8 }}
-        {{- end }}
-        {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -117,26 +107,4 @@ spec:
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.worker.extraVolumes }}
-      volumes:
-      {{- toYaml . | nindent 6 }}
-      {{- end }}
-  {{- if .Values.worker.persistence.modelCache.enabled }}
-  volumeClaimTemplates:
-  - metadata:
-      name: model-cache
-      {{- with .Values.worker.persistence.modelCache.annotations }}
-      annotations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
-    spec:
-      accessModes:
-        {{- toYaml .Values.worker.persistence.modelCache.accessModes | nindent 8 }}
-      {{- if .Values.worker.persistence.modelCache.storageClass }}
-      storageClassName: {{ .Values.worker.persistence.modelCache.storageClass }}
-      {{- end }}
-      resources:
-        requests:
-          storage: {{ .Values.worker.persistence.modelCache.size }}
-  {{- end }}
 {{- end }}

helm/hindsight/values.yaml

@@ -67,33 +67,6 @@ api:
   # Pod affinity/anti-affinity (overrides global affinity for this component)
   # affinity: {}
 
-  # Persistent volume for local model cache (reranker, embeddings)
-  # Models are downloaded to /home/hindsight/.cache on first use.
-  # Without persistence, models are re-downloaded on every pod restart.
-  persistence:
-    modelCache:
-      enabled: false
-      size: 5Gi
-      storageClass: ""
-      accessModes:
-        - ReadWriteOnce
-      annotations: {}
-
-  # Extra volume mounts for the api container
-  # e.g.
-  # extraVolumeMounts:
-  #   - name: my-volume
-  #     mountPath: /mnt/my-volume
-  extraVolumeMounts: []
-
-  # Extra volumes for the api pod
-  # e.g.
-  # extraVolumes:
-  #   - name: my-volume
-  #     configMap:
-  #       name: my-configmap
-  extraVolumes: []
-
   # Environment variables
   env:
     #HINDSIGHT_API_LLM_PROVIDER: "groq"
@@ -167,32 +140,6 @@ worker:
   # Pod affinity/anti-affinity (overrides global affinity for this component)
   # affinity: {}
 
-  # Persistent volume for local model cache (reranker, embeddings)
-  # Uses volumeClaimTemplates since worker is a StatefulSet.
-  persistence:
-    modelCache:
-      enabled: false
-      size: 5Gi
-      storageClass: ""
-      accessModes:
-        - ReadWriteOnce
-      annotations: {}
-
-  # Extra volume mounts for the worker container
-  # e.g.
-  # extraVolumeMounts:
-  #   - name: my-volume
-  #     mountPath: /mnt/my-volume
-  extraVolumeMounts: []
-
-  # Extra volumes for the worker pod
-  # e.g.
-  # extraVolumes:
-  #   - name: my-volume
-  #     configMap:
-  #       name: my-configmap
-  extraVolumes: []
-
   # Secret environment variables (inherited from api.secrets if not specified)
   secrets: {}
 

hindsight-all-npm/.gitignore

@@ -1,4 +0,0 @@
-node_modules
-dist
-*.tgz
-.DS_Store

hindsight-all-npm/README.md

@@ -1,80 +0,0 @@
-# @vectorize-io/hindsight-all
-
-Node.js equivalent of the Python [`hindsight-all`](https://pypi.org/project/hindsight-all/) package — programmatic lifecycle manager for a local Hindsight daemon. Use this when you want to embed Hindsight in a Node application without hand-rolling subprocess management.
-
-This package deliberately does **not** ship an HTTP client. Once the daemon is running, talk to it with [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client) against `server.getBaseUrl()`. The two packages compose — one owns the daemon process, the other owns the HTTP API surface.
-
-## Requirements
-
-- **Node.js >= 22** — uses global `fetch` and `AbortSignal.timeout`.
-- **`uv` / `uvx`** on `PATH` — used to download and run the underlying `hindsight-embed` daemon on first use. Install via <https://docs.astral.sh/uv/>.
-
-## Install
-
-```bash
-npm install @vectorize-io/hindsight-all @vectorize-io/hindsight-client
-```
-
-## Example
-
-```ts
-import { HindsightServer, consoleLogger } from "@vectorize-io/hindsight-all";
-import { HindsightClient } from "@vectorize-io/hindsight-client";
-
-const server = new HindsightServer({
-  profile: "my-app",
-  port: 9077,
-  env: {
-    HINDSIGHT_API_LLM_PROVIDER: "anthropic",
-    HINDSIGHT_API_LLM_API_KEY: process.env.ANTHROPIC_API_KEY,
-    HINDSIGHT_API_LLM_MODEL: "claude-sonnet-4-20250514",
-    HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT: "0",
-  },
-  logger: consoleLogger,
-});
-
-await server.start();
-
-const client = new HindsightClient({ baseUrl: server.getBaseUrl() });
-
-await client.retain("user-123", "User prefers dark mode and concise answers.", {
-  documentId: "pref-2026-04-01",
-});
-
-const recall = await client.recall("user-123", "what are the user preferences?");
-console.log(recall.results);
-
-await server.stop();
-```
-
-For a remote Hindsight API, skip `HindsightServer` entirely and just point `HindsightClient` at the remote URL.
-
-## Open config — forward-compatible with new daemon flags
-
-`HindsightServerOptions` is designed so every new environment variable or CLI flag in the underlying Hindsight daemon can be used without waiting for a wrapper release:
-
-- **`env`** accepts an arbitrary `Record<string, string>`. Every entry is exported into the daemon process and written into the profile config via `--env KEY=VALUE`.
-- **`extraProfileCreateArgs`** / **`extraDaemonStartArgs`** append raw args to the respective commands.
-
-## Development against a local checkout
-
-If you're hacking on the Python `hindsight-embed` package in the same monorepo, point the server at the local path — it'll use `uv run --directory <path>` instead of `uvx`:
-
-```ts
-new HindsightServer({
-  embedPackagePath: "/path/to/hindsight-embed",
-  // ...
-});
-```
-
-## API surface
-
-- `HindsightServer` — daemon lifecycle (`start`, `stop`, `checkHealth`, `getBaseUrl`, `getProfile`).
-- `Logger` interface plus `silentLogger` (default) and `consoleLogger` helpers.
-- `getEmbedCommand(opts)` — low-level helper that returns the `[cmd, ...args]` tuple used to invoke the underlying Python CLI.
-
-For memory operations (retain, recall, reflect, bank management, stats) use [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client).
-
-## License
-
-MIT

hindsight-all-npm/package.json

@@ -1,57 +0,0 @@
-{
-  "name": "@vectorize-io/hindsight-all",
-  "version": "0.5.4",
-  "description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
-  "main": "dist/index.js",
-  "types": "dist/index.d.ts",
-  "type": "module",
-  "exports": {
-    ".": {
-      "types": "./dist/index.d.ts",
-      "import": "./dist/index.js"
-    }
-  },
-  "keywords": [
-    "hindsight",
-    "hindsight-all",
-    "memory",
-    "ai",
-    "agent",
-    "long-term-memory",
-    "llm",
-    "embedded-server"
-  ],
-  "author": "Vectorize <[email protected]>",
-  "license": "MIT",
-  "repository": {
-    "type": "git",
-    "url": "https://github.com/vectorize-io/hindsight.git",
-    "directory": "hindsight-all-npm"
-  },
-  "files": [
-    "dist",
-    "README.md"
-  ],
-  "scripts": {
-    "build": "tsup",
-    "dev": "tsup --watch",
-    "clean": "rm -rf dist",
-    "test": "vitest run src",
-    "test:watch": "vitest src",
-    "prepublishOnly": "npm run clean && npm run build"
-  },
-  "devDependencies": {
-    "@types/node": "^22.0.0",
-    "tsup": "^8.5.1",
-    "typescript": "^5.7.0",
-    "vitest": "^4.1.2"
-  },
-  "engines": {
-    "node": ">=22"
-  },
-  "overrides": {
-    "rollup": "^4.59.0",
-    "picomatch": ">=2.3.2 <3.0.0 || >=4.0.4",
-    "vite": ">=8.0.5"
-  }
-}

hindsight-all-npm/src/command.test.ts

@@ -1,36 +0,0 @@
-import { describe, it, expect } from "vitest";
-import { getEmbedCommand } from "./command.js";
-
-describe("getEmbedCommand", () => {
-  it("defaults to uvx hindsight-embed@latest", () => {
-    expect(getEmbedCommand()).toEqual(["uvx", "hindsight-embed@latest"]);
-  });
-
-  it("honours an explicit version", () => {
-    expect(getEmbedCommand({ embedVersion: "0.5.0" })).toEqual(["uvx", "[email protected]"]);
-  });
-
-  it("treats an empty version as latest", () => {
-    expect(getEmbedCommand({ embedVersion: "" })).toEqual(["uvx", "hindsight-embed@latest"]);
-  });
-
-  it("uses uv run --directory when a local path is given", () => {
-    expect(getEmbedCommand({ embedPackagePath: "/abs/path" })).toEqual([
-      "uv",
-      "run",
-      "--directory",
-      "/abs/path",
-      "hindsight-embed",
-    ]);
-  });
-
-  it("local path takes precedence over version", () => {
-    expect(getEmbedCommand({ embedPackagePath: "/abs/path", embedVersion: "0.5.0" })).toEqual([
-      "uv",
-      "run",
-      "--directory",
-      "/abs/path",
-      "hindsight-embed",
-    ]);
-  });
-});

hindsight-all-npm/src/command.ts

@@ -1,25 +0,0 @@
-/**
- * Resolve the command that invokes the `hindsight-embed` Python CLI.
- *
- * - If `embedPackagePath` is set, runs the package from a local checkout via
- *   `uv run --directory <path> hindsight-embed`. Used for in-repo development.
- * - Otherwise runs it via `uvx hindsight-embed@<version>` so no global install
- *   is required.
- *
- * Returns the argv as `[command, ...baseArgs]` suitable for `spawn()` /
- * `execFile()` (never shell-interpolated).
- */
-export interface EmbedCommandOptions {
-  /** Version spec passed to uvx (e.g. "latest", "0.5.0"). Default: "latest". */
-  embedVersion?: string;
-  /** Local checkout path. When set, overrides `embedVersion` and uses `uv run`. */
-  embedPackagePath?: string;
-}
-
-export function getEmbedCommand(opts: EmbedCommandOptions = {}): string[] {
-  if (opts.embedPackagePath) {
-    return ["uv", "run", "--directory", opts.embedPackagePath, "hindsight-embed"];
-  }
-  const version = opts.embedVersion && opts.embedVersion.length > 0 ? opts.embedVersion : "latest";
-  return ["uvx", `hindsight-embed@${version}`];
-}

hindsight-all-npm/src/index.ts

@@ -1,7 +0,0 @@
-export { HindsightServer } from "./server.js";
-export { getEmbedCommand } from "./command.js";
-export { silentLogger, consoleLogger } from "./logger.js";
-
-export type { Logger } from "./logger.js";
-export type { EmbedCommandOptions } from "./command.js";
-export type { HindsightServerOptions } from "./types.js";

hindsight-all-npm/src/logger.ts

@@ -1,29 +0,0 @@
-/**
- * Pluggable logger interface.
- *
- * This package does not own any logging infrastructure — consumers inject
- * whatever they want (console, pino, openclaw's logger, a no-op). The default
- * is silent so embedding this package never adds noise to an unrelated app.
- */
-export interface Logger {
-  debug(msg: string): void;
-  info(msg: string): void;
-  warn(msg: string): void;
-  error(msg: string): void;
-}
-
-/** Logger that drops every call. Used when no logger is passed. */
-export const silentLogger: Logger = {
-  debug: () => {},
-  info: () => {},
-  warn: () => {},
-  error: () => {},
-};
-
-/** Logger that writes to the standard console. Handy for CLIs and tests. */
-export const consoleLogger: Logger = {
-  debug: (msg) => console.debug(msg),
-  info: (msg) => console.log(msg),
-  warn: (msg) => console.warn(msg),
-  error: (msg) => console.error(msg),
-};

hindsight-all-npm/src/server.test.ts

@@ -1,35 +0,0 @@
-import { describe, it, expect } from "vitest";
-import { HindsightServer } from "./server.js";
-
-describe("HindsightServer construction", () => {
-  it("defaults base URL to http://127.0.0.1:8888", () => {
-    const server = new HindsightServer();
-    expect(server.getBaseUrl()).toBe("http://127.0.0.1:8888");
-    expect(server.getProfile()).toBe("default");
-  });
-
-  it("honours custom profile, port, and host", () => {
-    const server = new HindsightServer({ profile: "app", port: 9077, host: "0.0.0.0" });
-    expect(server.getProfile()).toBe("app");
-    expect(server.getBaseUrl()).toBe("http://0.0.0.0:9077");
-  });
-
-  it("accepts open env pass-through without complaining about unknown keys", () => {
-    const server = new HindsightServer({
-      env: {
-        HINDSIGHT_API_LLM_PROVIDER: "openai",
-        HINDSIGHT_API_LLM_MODEL: "gpt-4o-mini",
-        // A field that does not exist today — should still be accepted
-        HINDSIGHT_FUTURE_FLAG: "enabled",
-      },
-    });
-    expect(server).toBeInstanceOf(HindsightServer);
-  });
-
-  it("exposes checkHealth that returns false when no daemon is running", async () => {
-    // Random high port that nothing is listening on.
-    const server = new HindsightServer({ port: 1, readyTimeoutMs: 100 });
-    const healthy = await server.checkHealth();
-    expect(healthy).toBe(false);
-  });
-});

hindsight-all-npm/src/server.ts

@@ -1,322 +0,0 @@
-import { spawn } from "child_process";
-import { getEmbedCommand } from "./command.js";
-import { silentLogger } from "./logger.js";
-import type { Logger } from "./logger.js";
-import type { HindsightServerOptions } from "./types.js";
-
-const DEFAULT_PORT = 8888;
-const DEFAULT_HOST = "127.0.0.1";
-const DEFAULT_PROFILE = "default";
-const DEFAULT_READY_TIMEOUT_MS = 30_000;
-const DEFAULT_READY_POLL_INTERVAL_MS = 1_000;
-
-/**
- * Manages the lifecycle of a local Hindsight daemon from a Node.js process.
- *
- * On {@link start}, this class:
- *   1. Resolves the `hindsight-embed` command (via `uvx` or a local `uv run`).
- *   2. Runs `profile create <name> --merge --port <port> [--env K=V ...]`
- *      with every entry in {@link HindsightServerOptions.env} forwarded as
- *      an `--env` flag.
- *   3. Runs `daemon --profile <name> start` and waits for the start command
- *      to exit.
- *   4. Polls `http://host:port/health` until it returns `200` or the
- *      `readyTimeoutMs` budget is exhausted.
- *
- * On {@link stop}, it runs `daemon --profile <name> stop` and returns once
- * the command exits (or after a short grace period).
- *
- * This is the Node.js equivalent of the Python `hindsight-all` package's
- * `HindsightServer`: a thin programmatic lifecycle wrapper around the
- * Hindsight daemon. It does NOT ship an HTTP client — once `start()`
- * resolves, use `@vectorize-io/hindsight-client` against `getBaseUrl()` for
- * retain / recall / reflect.
- *
- * The class is deliberately transparent about the daemon: new CLI flags or
- * environment variables never require a code change here — callers can pass
- * them via `env`, `extraProfileCreateArgs`, or `extraDaemonStartArgs`.
- */
-export class HindsightServer {
-  private readonly profile: string;
-  private readonly port: number;
-  private readonly host: string;
-  private readonly baseUrl: string;
-  private readonly embedVersion: string | undefined;
-  private readonly embedPackagePath: string | undefined;
-  private readonly userEnv: Record<string, string | undefined>;
-  private readonly extraProfileCreateArgs: string[];
-  private readonly extraDaemonStartArgs: string[];
-  private readonly platformCpuWorkaround: boolean;
-  private readonly readyTimeoutMs: number;
-  private readonly readyPollIntervalMs: number;
-  private readonly logger: Logger;
-
-  constructor(opts: HindsightServerOptions = {}) {
-    this.profile = opts.profile ?? DEFAULT_PROFILE;
-    this.port = opts.port ?? DEFAULT_PORT;
-    this.host = opts.host ?? DEFAULT_HOST;
-    this.baseUrl = `http://${this.host}:${this.port}`;
-    this.embedVersion = opts.embedVersion;
-    this.embedPackagePath = opts.embedPackagePath;
-    this.userEnv = opts.env ?? {};
-    this.extraProfileCreateArgs = opts.extraProfileCreateArgs ?? [];
-    this.extraDaemonStartArgs = opts.extraDaemonStartArgs ?? [];
-    this.platformCpuWorkaround = opts.platformCpuWorkaround ?? process.platform === "darwin";
-    this.readyTimeoutMs = opts.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
-    this.readyPollIntervalMs = opts.readyPollIntervalMs ?? DEFAULT_READY_POLL_INTERVAL_MS;
-    this.logger = opts.logger ?? silentLogger;
-  }
-
-  /** The base URL the daemon listens on (`http://host:port`). */
-  getBaseUrl(): string {
-    return this.baseUrl;
-  }
-
-  /** The profile name this server operates on. */
-  getProfile(): string {
-    return this.profile;
-  }
-
-  /**
-   * Ensure the daemon is configured and running. Idempotent — the underlying
-   * `profile create --merge` and `daemon start` commands tolerate re-runs.
-   */
-  async start(): Promise<void> {
-    this.logger.info(`[hindsight] starting daemon for profile "${this.profile}"`);
-
-    const env = this.buildEnv();
-    await this.configureProfile(env);
-    await this.startDaemon(env);
-    await this.waitForReady();
-
-    this.logger.info(`[hindsight] daemon ready at ${this.baseUrl}`);
-  }
-
-  /** Stop the daemon. Never throws — logs and resolves even on failure. */
-  async stop(): Promise<void> {
-    this.logger.info(`[hindsight] stopping daemon for profile "${this.profile}"`);
-
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const args = [...baseArgs, "daemon", "--profile", this.profile, "stop"];
-
-    const child = spawn(cmd, args, { stdio: "pipe" });
-    this.pipeOutput(child, "daemon.stop");
-
-    await new Promise<void>((resolve) => {
-      const timeout = setTimeout(() => {
-        this.logger.warn(`[hindsight] daemon stop timed out after 5s`);
-        resolve();
-      }, 5_000);
-      child.on("exit", () => {
-        clearTimeout(timeout);
-        this.logger.info(`[hindsight] daemon stopped`);
-        resolve();
-      });
-      child.on("error", (err) => {
-        clearTimeout(timeout);
-        this.logger.warn(`[hindsight] error stopping daemon: ${err.message}`);
-        resolve();
-      });
-    });
-  }
-
-  /** Probe `/health` once with a short timeout. */
-  async checkHealth(): Promise<boolean> {
-    try {
-      const res = await fetch(`${this.baseUrl}/health`, {
-        signal: AbortSignal.timeout(2_000),
-      });
-      return res.ok;
-    } catch {
-      return false;
-    }
-  }
-
-  // -------------------------------------------------------------------------
-  // Internal
-  // -------------------------------------------------------------------------
-
-  /**
-   * Merge the process env, the caller-supplied `env`, and (on macOS) the
-   * embeddings CPU workaround. Caller-supplied values always win over the
-   * workaround; undefined values are dropped.
-   */
-  private buildEnv(): NodeJS.ProcessEnv {
-    const merged: NodeJS.ProcessEnv = { ...process.env };
-
-    if (this.platformCpuWorkaround && process.platform === "darwin") {
-      merged["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1";
-      merged["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1";
-    }
-
-    for (const [key, value] of Object.entries(this.userEnv)) {
-      if (value !== undefined) {
-        merged[key] = value;
-      }
-    }
-
-    return merged;
-  }
-
-  /**
-   * Run `profile create <name> --merge --port <port> [--env K=V ...]`.
-   * Every entry in the merged env that was passed via {@link userEnv} (or
-   * auto-applied by the CPU workaround) is forwarded as `--env`.
-   */
-  private async configureProfile(env: NodeJS.ProcessEnv): Promise<void> {
-    this.logger.info(`[hindsight] configuring profile "${this.profile}"`);
-
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const createArgs = [
-      ...baseArgs,
-      "profile",
-      "create",
-      this.profile,
-      "--merge",
-      "--port",
-      String(this.port),
-    ];
-
-    // Forward every env var that the caller intended for the daemon as --env.
-    // We only forward keys the caller explicitly set (userEnv) plus the CPU
-    // workaround values — not the entire process.env, to avoid leaking random
-    // host state into profile config.
-    const envForProfile = this.collectProfileEnv(env);
-    for (const [key, value] of Object.entries(envForProfile)) {
-      createArgs.push("--env", `${key}=${value}`);
-    }
-
-    createArgs.push(...this.extraProfileCreateArgs);
-
-    await this.runCommand(cmd, createArgs, env, "profile.create");
-  }
-
-  /** Collect only the env vars that should be written into the profile file. */
-  private collectProfileEnv(env: NodeJS.ProcessEnv): Record<string, string> {
-    const out: Record<string, string> = {};
-
-    // 1. User-supplied env — always forwarded.
-    for (const [key, value] of Object.entries(this.userEnv)) {
-      if (value !== undefined) {
-        out[key] = value;
-      }
-    }
-
-    // 2. CPU workaround — only if auto-applied and not already overridden.
-    if (this.platformCpuWorkaround && process.platform === "darwin") {
-      const cpuKeys = [
-        "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU",
-        "HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU",
-      ];
-      for (const key of cpuKeys) {
-        if (!(key in out) && env[key] !== undefined) {
-          out[key] = env[key] as string;
-        }
-      }
-    }
-
-    return out;
-  }
-
-  private async startDaemon(env: NodeJS.ProcessEnv): Promise<void> {
-    const [cmd, ...baseArgs] = getEmbedCommand({
-      embedVersion: this.embedVersion,
-      embedPackagePath: this.embedPackagePath,
-    });
-    const args = [
-      ...baseArgs,
-      "daemon",
-      "--profile",
-      this.profile,
-      "start",
-      ...this.extraDaemonStartArgs,
-    ];
-
-    await this.runCommand(cmd, args, env, "daemon.start");
-  }
-
-  /**
-   * Spawn `cmd` with `args`, pipe its output through the logger, and resolve
-   * once it exits with code 0. Rejects on non-zero exit or spawn error.
-   */
-  private async runCommand(
-    cmd: string,
-    args: string[],
-    env: NodeJS.ProcessEnv,
-    label: string
-  ): Promise<void> {
-    const child = spawn(cmd, args, { stdio: "pipe", env });
-    let output = "";
-    child.stdout?.on("data", (data: Buffer) => {
-      const text = data.toString();
-      output += text;
-      for (const line of text.trimEnd().split("\n")) {
-        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
-      }
-    });
-    child.stderr?.on("data", (data: Buffer) => {
-      const text = data.toString();
-      output += text;
-      for (const line of text.trimEnd().split("\n")) {
-        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
-      }
-    });
-
-    await new Promise<void>((resolve, reject) => {
-      child.on("exit", (code) => {
-        if (code === 0) {
-          resolve();
-        } else {
-          reject(new Error(`${label} failed with code ${code}: ${output.trim()}`));
-        }
-      });
-      child.on("error", (err) => {
-        reject(new Error(`${label} failed to spawn: ${err.message}`, { cause: err }));
-      });
-    });
-  }
-
-  /** Stream a spawned child's stdout/stderr through the logger without blocking. */
-  private pipeOutput(child: ReturnType<typeof spawn>, label: string): void {
-    child.stdout?.on("data", (data: Buffer) => {
-      for (const line of data.toString().trimEnd().split("\n")) {
-        if (line) this.logger.info(`[hindsight:${label}] ${line}`);
-      }
-    });
-    child.stderr?.on("data", (data: Buffer) => {
-      for (const line of data.toString().trimEnd().split("\n")) {
-        if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
-      }
-    });
-  }
-
-  /** Poll `/health` until it succeeds or `readyTimeoutMs` elapses. */
-  private async waitForReady(): Promise<void> {
-    const deadline = Date.now() + this.readyTimeoutMs;
-    let attempt = 0;
-    while (Date.now() < deadline) {
-      attempt++;
-      try {
-        const res = await fetch(`${this.baseUrl}/health`, {
-          signal: AbortSignal.timeout(this.readyPollIntervalMs),
-        });
-        if (res.ok) {
-          this.logger.debug(`[hindsight] health check passed (attempt ${attempt})`);
-          return;
-        }
-      } catch {
-        // expected while the daemon is still booting
-      }
-      await new Promise((resolve) => setTimeout(resolve, this.readyPollIntervalMs));
-    }
-    throw new Error(
-      `Hindsight daemon did not become ready within ${this.readyTimeoutMs}ms at ${this.baseUrl}`
-    );
-  }
-}

hindsight-all-npm/src/types.ts

@@ -1,54 +0,0 @@
-import type { Logger } from "./logger.js";
-
-/**
- * Options for {@link HindsightServer}.
- *
- * The server is intentionally thin and pass-through: anything configurable
- * on the daemon side (env vars or CLI flags) can be set here without needing
- * a new dedicated option. Use {@link env} for `HINDSIGHT_*` / `OPENAI_API_KEY` /
- * custom provider settings, and the two `extra*` arrays to append raw CLI
- * args to `profile create` or `daemon start`.
- *
- * For talking to the daemon after `start()`, use `@vectorize-io/hindsight-client`
- * against `server.getBaseUrl()`. This package does not ship its own HTTP
- * client.
- */
-export interface HindsightServerOptions {
-  /** Profile name used for `--profile <name>` on every sub-command. Default: `"default"`. */
-  profile?: string;
-  /** TCP port the daemon listens on. Default: `8888`. */
-  port?: number;
-  /** Hostname the daemon binds to (for health checks). Default: `127.0.0.1`. */
-  host?: string;
-  /** Version of the underlying `hindsight-embed` PyPI package to run via `uvx`. Default: `"latest"`. */
-  embedVersion?: string;
-  /** Local path to a `hindsight-embed` checkout — takes precedence over `embedVersion`. */
-  embedPackagePath?: string;
-  /**
-   * Environment variables passed to the daemon process AND written into the
-   * profile via repeated `--env KEY=VALUE` flags. This is the preferred way
-   * to surface any `HINDSIGHT_API_*` / `HINDSIGHT_EMBED_*` setting — adding a
-   * new daemon env var never requires a wrapper update.
-   *
-   * Values of `undefined` are dropped (so you can spread conditionally).
-   */
-  env?: Record<string, string | undefined>;
-  /** Extra args appended verbatim to `hindsight-embed profile create <name> --merge ...`. */
-  extraProfileCreateArgs?: string[];
-  /** Extra args appended verbatim to `hindsight-embed daemon --profile <name> start ...`. */
-  extraDaemonStartArgs?: string[];
-  /**
-   * On macOS, automatically set
-   * `HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1` and
-   * `HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1` to avoid Metal/MPS crashes in
-   * daemon mode. Default: `true` on `darwin`, ignored elsewhere. Any value set
-   * explicitly in {@link env} wins over the auto-applied value.
-   */
-  platformCpuWorkaround?: boolean;
-  /** Max time (ms) to wait for `/health` to return 200. Default: `30_000`. */
-  readyTimeoutMs?: number;
-  /** Polling interval (ms) while waiting for `/health`. Default: `1_000`. */
-  readyPollIntervalMs?: number;
-  /** Optional pluggable logger. Default: silent. */
-  logger?: Logger;
-}

hindsight-all-npm/tsconfig.json

@@ -1,18 +0,0 @@
-{
-  "compilerOptions": {
-    "target": "ES2022",
-    "module": "ES2022",
-    "lib": ["ES2022"],
-    "moduleResolution": "node",
-    "declaration": true,
-    "outDir": "./dist",
-    "rootDir": "./src",
-    "strict": true,
-    "esModuleInterop": true,
-    "skipLibCheck": true,
-    "forceConsistentCasingInFileNames": true,
-    "resolveJsonModule": true
-  },
-  "include": ["src/**/*"],
-  "exclude": ["node_modules", "dist", "src/**/*.test.ts"]
-}

hindsight-all-npm/tsup.config.ts

@@ -1,11 +0,0 @@
-import { defineConfig } from "tsup";
-
-export default defineConfig({
-  entry: ["src/index.ts"],
-  format: ["esm"],
-  dts: true,
-  outDir: "dist",
-  clean: true,
-  sourcemap: true,
-  bundle: true,
-});

hindsight-all-npm/vitest.config.ts

@@ -1,8 +0,0 @@
-import { defineConfig } from "vitest/config";
-
-export default defineConfig({
-  test: {
-    include: ["src/**/*.test.ts"],
-    environment: "node",
-  },
-});

hindsight-all-slim/pyproject.toml

@@ -1,33 +0,0 @@
-[build-system]
-requires = ["setuptools>=61"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "hindsight-all-slim"
-version = "0.5.4"
-description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
-readme = "README.md"
-requires-python = ">=3.11"
-dependencies = [
-    "hindsight-api-slim>=0.4.17",
-    "hindsight-client>=0.0.7",
-    "hindsight-embed>=0.1.0",
-]
-
-[tool.uv.sources]
-hindsight-api-slim = { workspace = true }
-hindsight-client = { workspace = true }
-hindsight-embed = { workspace = true }
-
-[project.optional-dependencies]
-test = [
-    "pytest>=7.0.0",
-    "pytest-asyncio>=0.21.0",
-]
-
-[tool.setuptools]
-packages = []
-
-[tool.pytest.ini_options]
-asyncio_mode = "auto"
-asyncio_default_fixture_loop_scope = "function"

hindsight-all/README.md

@@ -1,48 +0,0 @@
-# hindsight-all
-
-All-in-one package for Hindsight - Agent Memory That Works Like Human Memory
-
-## Quick Start
-
-```python
-from hindsight import start_server, HindsightClient
-
-# Start server with embedded PostgreSQL
-server = start_server(
-    llm_provider="groq",
-    llm_api_key="your-api-key",
-    llm_model="openai/gpt-oss-120b"
-)
-
-# Create client
-client = HindsightClient(base_url=server.url)
-
-# Store memories
-client.put(agent_id="assistant", content="User prefers Python for data analysis")
-
-# Search memories
-results = client.search(agent_id="assistant", query="programming preferences")
-
-# Generate contextual response
-response = client.think(agent_id="assistant", query="What languages should I recommend?")
-
-# Stop server when done
-server.stop()
-```
-
-## Using Context Manager
-
-```python
-from hindsight import HindsightServer, HindsightClient
-
-with HindsightServer(llm_provider="groq", llm_api_key="...") as server:
-    client = HindsightClient(base_url=server.url)
-    # ... use client ...
-# Server automatically stops
-```
-
-## Installation
-
-```bash
-pip install hindsight-all
-```

hindsight-all/hindsight/client_wrapper.py

@@ -1,423 +0,0 @@
-"""
-Wrapper for Hindsight client that adds API namespaces.
-
-Provides organized access to different parts of the Hindsight API through
-namespaces like .banks, .mental_models, etc.
-"""
-
-from __future__ import annotations
-
-from typing import Any
-
-from hindsight_client import Hindsight
-
-
-class BanksAPI:
-    """Namespace for bank-related operations.
-    
-    Provides methods to create, delete, and manage memory banks.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str | None = None,
-        mission: str | None = None,
-        disposition: dict[str, Any] | None = None,
-    ) -> Any:
-        """Create a new bank.
-        
-        Args:
-            bank_id: Unique identifier for the bank.
-            name: Optional display name for the bank.
-            mission: Optional mission statement for the bank.
-            disposition: Optional disposition configuration dict.
-            
-        Returns:
-            Bank creation response from the API.
-        """
-        return self._client.create_bank(
-            bank_id=bank_id,
-            name=name,
-            mission=mission,
-            disposition=disposition,
-        )
-
-    def delete(self, bank_id: str) -> Any:
-        """Delete a bank.
-        
-        Args:
-            bank_id: The ID of the bank to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_bank(bank_id=bank_id)
-
-    def set_mission(self, bank_id: str, mission: str) -> Any:
-        """Set or update the mission for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mission: The mission statement to set.
-            
-        Returns:
-            API response confirming the update.
-        """
-        return self._client.set_mission(bank_id=bank_id, mission=mission)
-
-    def set_disposition(self, bank_id: str, disposition: dict[str, Any]) -> Any:
-        """Set or update the disposition for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            disposition: The disposition configuration dict.
-            
-        Returns:
-            API response confirming the update.
-        """
-        return self._client.set_disposition(bank_id=bank_id, disposition=disposition)
-
-    def list(self) -> Any:
-        """List all banks.
-        
-        Returns:
-            List of banks from the API.
-        """
-        from hindsight_client.hindsight_client import _run_async
-
-        return _run_async(self._client._banks_api.list_banks())
-
-
-class MentalModelsAPI:
-    """Namespace for mental model operations.
-    
-    Mental models are reusable knowledge structures that guide agent behavior.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str,
-        content: str,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Create a new mental model.
-        
-        Args:
-            bank_id: The ID of the bank to add the model to.
-            name: Name for the mental model.
-            content: The content/instructions for the mental model.
-            tags: Optional list of tags for categorization.
-            
-        Returns:
-            Creation response from the API.
-        """
-        return self._client.create_mental_model(
-            bank_id=bank_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
-        """List all mental models for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            tags: Optional filter by tags.
-            
-        Returns:
-            List of mental models.
-        """
-        return self._client.list_mental_models(bank_id=bank_id, tags=tags)
-
-    def get(self, bank_id: str, mental_model_id: str) -> Any:
-        """Get a specific mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model.
-            
-        Returns:
-            The mental model details.
-        """
-        return self._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-    def refresh(self, bank_id: str, mental_model_id: str) -> Any:
-        """Refresh a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to refresh.
-            
-        Returns:
-            Refresh response from the API.
-        """
-        return self._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-    def update(
-        self,
-        bank_id: str,
-        mental_model_id: str,
-        name: str | None = None,
-        content: str | None = None,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Update a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to update.
-            name: Optional new name.
-            content: Optional new content.
-            tags: Optional new tags list.
-            
-        Returns:
-            Update response from the API.
-        """
-        return self._client.update_mental_model(
-            bank_id=bank_id,
-            mental_model_id=mental_model_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def delete(self, bank_id: str, mental_model_id: str) -> Any:
-        """Delete a mental model.
-        
-        Args:
-            bank_id: The ID of the bank.
-            mental_model_id: The ID of the mental model to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
-
-
-class DirectivesAPI:
-    """Namespace for directive operations.
-    
-    Directives are explicit instructions that guide agent behavior.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def create(
-        self,
-        bank_id: str,
-        name: str,
-        content: str,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Create a new directive.
-        
-        Args:
-            bank_id: The ID of the bank to add the directive to.
-            name: Name for the directive.
-            content: The directive content/instructions.
-            tags: Optional list of tags for categorization.
-            
-        Returns:
-            Creation response from the API.
-        """
-        return self._client.create_directive(
-            bank_id=bank_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def list(self, bank_id: str, tags: list[str] | None = None) -> Any:
-        """List all directives for a bank.
-        
-        Args:
-            bank_id: The ID of the bank.
-            tags: Optional filter by tags.
-            
-        Returns:
-            List of directives.
-        """
-        return self._client.list_directives(bank_id=bank_id, tags=tags)
-
-    def get(self, bank_id: str, directive_id: str) -> Any:
-        """Get a specific directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive.
-            
-        Returns:
-            The directive details.
-        """
-        return self._client.get_directive(bank_id=bank_id, directive_id=directive_id)
-
-    def update(
-        self,
-        bank_id: str,
-        directive_id: str,
-        name: str | None = None,
-        content: str | None = None,
-        tags: list[str] | None = None,
-    ) -> Any:
-        """Update a directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive to update.
-            name: Optional new name.
-            content: Optional new content.
-            tags: Optional new tags list.
-            
-        Returns:
-            Update response from the API.
-        """
-        return self._client.update_directive(
-            bank_id=bank_id,
-            directive_id=directive_id,
-            name=name,
-            content=content,
-            tags=tags,
-        )
-
-    def delete(self, bank_id: str, directive_id: str) -> Any:
-        """Delete a directive.
-        
-        Args:
-            bank_id: The ID of the bank.
-            directive_id: The ID of the directive to delete.
-            
-        Returns:
-            Deletion response from the API.
-        """
-        return self._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
-
-
-class MemoriesAPI:
-    """Namespace for memory operations.
-    
-    Provides methods to query and retrieve stored memories.
-    """
-
-    def __init__(self, client: Hindsight):
-        self._client = client
-
-    def list(
-        self,
-        bank_id: str,
-        type: str | None = None,
-        search_query: str | None = None,
-        limit: int = 100,
-        offset: int = 0,
-    ) -> Any:
-        """List memories in a bank.
-        
-        Args:
-            bank_id: The ID of the bank to query.
-            type: Optional filter by memory type.
-            search_query: Optional search query for filtering.
-            limit: Maximum number of results to return (default: 100).
-            offset: Number of results to skip for pagination (default: 0).
-            
-        Returns:
-            List of memories matching the criteria.
-        """
-        return self._client.list_memories(
-            bank_id=bank_id,
-            type=type,
-            search_query=search_query,
-            limit=limit,
-            offset=offset,
-        )
-
-
-class HindsightClient(Hindsight):
-    """
-    Enhanced Hindsight client with organized API namespaces.
-
-    This wrapper extends the auto-generated Hindsight client with organized
-    access to different parts of the API through namespaces.
-
-    Example:
-        ```python
-        from hindsight import HindsightClient
-
-        client = HindsightClient(base_url="http://localhost:8888")
-
-        # Core operations (inherited from Hindsight)
-        client.retain(bank_id="test", content="Hello")
-        results = client.recall(bank_id="test", query="Hello")
-
-        # Organized API access through namespaces
-        client.banks.create(bank_id="test", name="Test Bank")
-        models = client.mental_models.list(bank_id="test")
-        directives = client.directives.list(bank_id="test")
-        memories = client.memories.list(bank_id="test")
-        ```
-        
-    Attributes:
-        banks: Namespace for bank management operations.
-        mental_models: Namespace for mental model operations.
-        directives: Namespace for directive operations.
-        memories: Namespace for memory listing operations.
-    """
-
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        super().__init__(*args, **kwargs)
-        self._banks_namespace: BanksAPI | None = None
-        self._mental_models_namespace: MentalModelsAPI | None = None
-        self._directives_namespace: DirectivesAPI | None = None
-        self._memories_namespace: MemoriesAPI | None = None
-
-    @property
-    def banks(self) -> BanksAPI:
-        """Access bank management operations.
-        
-        Returns:
-            BanksAPI instance for bank operations.
-        """
-        if self._banks_namespace is None:
-            self._banks_namespace = BanksAPI(self)
-        return self._banks_namespace
-
-    @property
-    def mental_models(self) -> MentalModelsAPI:
-        """Access mental model operations.
-        
-        Returns:
-            MentalModelsAPI instance for mental model operations.
-        """
-        if self._mental_models_namespace is None:
-            self._mental_models_namespace = MentalModelsAPI(self)
-        return self._mental_models_namespace
-
-    @property
-    def directives(self) -> DirectivesAPI:
-        """Access directive operations.
-        
-        Returns:
-            DirectivesAPI instance for directive operations.
-        """
-        if self._directives_namespace is None:
-            self._directives_namespace = DirectivesAPI(self)
-        return self._directives_namespace
-
-    @property
-    def memories(self) -> MemoriesAPI:
-        """Access memory listing operations.
-        
-        Returns:
-            MemoriesAPI instance for memory operations.
-        """
-        if self._memories_namespace is None:
-            self._memories_namespace = MemoriesAPI(self)
-        return self._memories_namespace

hindsight-all/tests/test_cleanup_timeout.py

@@ -1,56 +0,0 @@
-"""
-Unit test for _cleanup lock timeout behavior.
-
-Verifies that _cleanup completes even when the lock is held by another thread,
-instead of hanging indefinitely (fixes #952).
-"""
-
-import threading
-import time
-from unittest.mock import MagicMock, patch
-
-import pytest
-
-
-def test_cleanup_completes_when_lock_held():
-    """
-    _cleanup should complete (best-effort) even when self._lock is held
-    by another thread, e.g. during a long _ensure_started call.
-    """
-    with patch.dict("sys.modules", {
-        "hindsight_client": MagicMock(),
-        "hindsight_embed": MagicMock(),
-        "hindsight.api_namespaces": MagicMock(),
-    }):
-        from hindsight.embedded import HindsightEmbedded
-
-        client = HindsightEmbedded.__new__(HindsightEmbedded)
-        client.profile = "test"
-        client._lock = threading.Lock()
-        client._closed = False
-        client._client = None
-        client._started = False
-        client._ui = False
-
-        # Simulate another thread holding the lock
-        client._lock.acquire()
-
-        cleanup_done = threading.Event()
-
-        def run_cleanup():
-            client._cleanup()
-            cleanup_done.set()
-
-        t = threading.Thread(target=run_cleanup)
-        t.start()
-
-        # Cleanup should complete within the timeout (5s) + margin
-        assert cleanup_done.wait(timeout=8.0), (
-            "_cleanup hung instead of timing out on lock acquisition"
-        )
-
-        # Release the lock from the simulating thread
-        client._lock.release()
-        t.join(timeout=1.0)
-
-        assert client._closed, "Client should be marked as closed after cleanup"

hindsight-api-slim/README.md

@@ -1,137 +0,0 @@
-# Hindsight API
-
-**Memory System for AI Agents** — Temporal + Semantic + Entity Memory Architecture using PostgreSQL with pgvector.
-
-Hindsight gives AI agents persistent memory that works like human memory: it stores facts, tracks entities and relationships, handles temporal reasoning ("what happened last spring?"), and forms opinions based on configurable disposition traits.
-
-## Installation
-
-```bash
-pip install hindsight-api
-```
-
-## Quick Start
-
-### Run the Server
-
-```bash
-# Set your LLM provider
-export HINDSIGHT_API_LLM_PROVIDER=openai
-export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
-
-# Start the server (uses embedded PostgreSQL by default)
-hindsight-api
-```
-
-The server starts at http://localhost:8888 with:
-- REST API for memory operations
-- MCP server at `/mcp` for tool-use integration
-
-### Use the Python API
-
-```python
-from hindsight_api import MemoryEngine
-
-# Create and initialize the memory engine
-memory = MemoryEngine()
-await memory.initialize()
-
-# Create a memory bank for your agent
-bank = await memory.create_memory_bank(
-    name="my-assistant",
-    background="A helpful coding assistant"
-)
-
-# Store a memory
-await memory.retain(
-    memory_bank_id=bank.id,
-    content="The user prefers Python for data science projects"
-)
-
-# Recall memories
-results = await memory.recall(
-    memory_bank_id=bank.id,
-    query="What programming language does the user prefer?"
-)
-
-# Reflect with reasoning
-response = await memory.reflect(
-    memory_bank_id=bank.id,
-    query="Should I recommend Python or R for this ML project?"
-)
-```
-
-## CLI Options
-
-```bash
-hindsight-api --help
-
-# Common options
-hindsight-api --port 9000          # Custom port (default: 8888)
-hindsight-api --host 127.0.0.1     # Bind to localhost only
-hindsight-api --workers 4          # Multiple worker processes
-hindsight-api --log-level debug    # Verbose logging
-```
-
-## Configuration
-
-Configure via environment variables:
-
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `HINDSIGHT_API_DATABASE_URL` | PostgreSQL connection string | `pg0` (embedded) |
-| `HINDSIGHT_API_LLM_PROVIDER` | `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio` | `openai` |
-| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
-| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-4o-mini` |
-| `HINDSIGHT_API_HOST` | Server bind address | `0.0.0.0` |
-| `HINDSIGHT_API_PORT` | Server port | `8888` |
-
-### Example with External PostgreSQL
-
-```bash
-export HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@localhost:5432/hindsight
-export HINDSIGHT_API_LLM_PROVIDER=groq
-export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx
-
-hindsight-api
-```
-
-## Docker
-
-```bash
-docker run --rm -it -p 8888:8888 \
-  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
-  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
-  ghcr.io/vectorize-io/hindsight:latest
-```
-
-## MCP Server
-
-For local MCP integration without running the full API server:
-
-```bash
-hindsight-local-mcp
-```
-
-This runs a stdio-based MCP server that can be used directly with MCP-compatible clients.
-
-## Key Features
-
-- **Multi-Strategy Retrieval (TEMPR)** — Semantic, keyword, graph, and temporal search combined with RRF fusion
-- **Entity Graph** — Automatic entity extraction and relationship tracking
-- **Temporal Reasoning** — Native support for time-based queries
-- **Disposition Traits** — Configurable skepticism, literalism, and empathy influence opinion formation
-- **Three Memory Types** — World facts, bank actions, and formed opinions with confidence scores
-
-## Documentation
-
-Full documentation: [https://hindsight.vectorize.io](https://hindsight.vectorize.io)
-
-- [Installation Guide](https://hindsight.vectorize.io/developer/installation)
-- [Configuration Reference](https://hindsight.vectorize.io/developer/configuration)
-- [API Reference](https://hindsight.vectorize.io/api-reference)
-- [Python SDK](https://hindsight.vectorize.io/sdks/python)
-
-## License
-
-Apache 2.0

hindsight-api-slim/hindsight_api/alembic/versions/2eee35aa3cfc_case_insensitive_entities_trgm_index.py

@@ -1,45 +0,0 @@
-"""Recreate entities trigram index on LOWER(canonical_name) for case-insensitive matching
-
-The previous GIN trigram index on canonical_name was case-sensitive, causing
-"Alice" and "alice" to have different trigram sets. This recreates it on
-LOWER(canonical_name) so the % operator matches case-insensitively.
-
-Revision ID: 2eee35aa3cfc
-Revises: d6e7f8a9b0c1
-Create Date: 2026-03-31
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "2eee35aa3cfc"
-down_revision: str | Sequence[str] | None = "d6e7f8a9b0c1"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Drop the old case-sensitive trigram index
-    op.execute("DROP INDEX IF EXISTS entities_canonical_name_trgm_idx")
-    # Create case-insensitive trigram index on LOWER(canonical_name)
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS entities_canonical_name_lower_trgm_idx "
-        f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops)"
-    )
-
-
-def downgrade() -> None:
-    op.execute("DROP INDEX IF EXISTS entities_canonical_name_lower_trgm_idx")
-    schema = _get_schema_prefix()
-    # Restore original case-sensitive index
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS entities_canonical_name_trgm_idx "
-        f"ON {schema}entities USING GIN (canonical_name gin_trgm_ops)"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/8c6fa6f7230b_merge_v053_divergent_heads.py

@@ -1,40 +0,0 @@
-"""Merge divergent migration heads for v0.5.3
-
-v0.5.3 shipped with two migration heads that were never unified:
-
-  * ``c4x5y6z7a8b9`` — delta-refresh chain
-    (``add_last_refreshed_source_query`` ->
-     ``add_structured_content_to_mental_models`` ->
-     ``backsweep_orphan_observations_v2``)
-
-  * ``h3i4j5k6l7m8`` — per-bank vector indexes / audit log chain
-    (the ``merge_heads_and_add_unit_entities_index`` subtree)
-
-Both fork from ``z1u2v3w4x5y6``. Upgrades from v0.5.2 still succeed — the
-walker applies the three c4x5 revisions and leaves the database stamped at
-both heads — but the result is a split DAG: ``alembic upgrade head``
-(singular) is ambiguous, and any future migration has to pick one head as
-its parent, orphaning the other.
-
-This revision linearises the DAG into a single head. It has no schema
-effect.
-
-Revision ID: 8c6fa6f7230b
-Revises: c4x5y6z7a8b9, h3i4j5k6l7m8
-Create Date: 2026-04-18
-"""
-
-from collections.abc import Sequence
-
-revision: str = "8c6fa6f7230b"
-down_revision: str | Sequence[str] | None = ("c4x5y6z7a8b9", "h3i4j5k6l7m8")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def upgrade() -> None:
-    pass
-
-
-def downgrade() -> None:
-    pass

hindsight-api-slim/hindsight_api/alembic/versions/a2b3c4d5e6f8_add_gin_index_source_memory_ids.py

@@ -1,54 +0,0 @@
-"""Add GIN index on source_memory_ids for observation lookup performance
-
-Without this index, queries using the array overlap operator (&&) or array
-containment (@>) on source_memory_ids require a full sequential scan over all
-observation memory_units. At ~77k observations this was measured at 45ms per
-query, becoming a bottleneck during consolidation recall (57-64s timeouts) and
-user recall (18-27s average).
-
-The GIN index reduces these queries to index scans: 45ms → 0.049ms (927x
-speedup). Recall dropped from 18-27s to ~6s, and consolidation recall
-stabilised from timeout to ~15s.
-
-Created with CONCURRENTLY so the migration does not block reads or writes.
-CONCURRENTLY requires running outside a transaction block, so the migration
-emits an explicit COMMIT before the statement and uses IF NOT EXISTS for
-idempotency.
-
-Revision ID: a2b3c4d5e6f8
-Revises: f7g8h9i0j1k2
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a2b3c4d5e6f8"
-down_revision: str | Sequence[str] | None = "f7g8h9i0j1k2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
-    # Commit the current Alembic transaction first.
-    op.execute("COMMIT")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")

hindsight-api-slim/hindsight_api/alembic/versions/a2v3w4x5y6z7_add_last_refreshed_source_query.py

@@ -1,38 +0,0 @@
-"""Add last_refreshed_source_query column to mental_models
-
-Revision ID: a2v3w4x5y6z7
-Revises: z1u2v3w4x5y6
-Create Date: 2026-04-15
-
-Tracks the source_query that was used during the most recent refresh.
-Used by delta-mode refresh to detect when the query has changed: if it has,
-delta mode falls back to a full regeneration because the surgical-edit
-assumption (same topic, new facts) no longer holds.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a2v3w4x5y6z7"
-down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"""
-        ALTER TABLE {schema}mental_models
-        ADD COLUMN IF NOT EXISTS last_refreshed_source_query TEXT
-    """)
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS last_refreshed_source_query")

hindsight-api-slim/hindsight_api/alembic/versions/a3b4c5d6e7f8_add_consolidation_failed_at_to_memory_units.py

@@ -1,52 +0,0 @@
-"""Add consolidation_failed_at column to memory_units for tracking persistent LLM failures.
-
-When all LLM retries are exhausted on a single-memory batch, the memory is marked
-with consolidation_failed_at instead of consolidated_at, so it is not silently lost
-and can be retried later via the API.
-
-Revision ID: a3b4c5d6e7f8
-Revises: g7h8i9j0k1l2
-Create Date: 2026-03-17
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "a3b4c5d6e7f8"
-down_revision: str | Sequence[str] | None = "g7h8i9j0k1l2"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        ALTER TABLE {schema}memory_units
-        ADD COLUMN IF NOT EXISTS consolidation_failed_at TIMESTAMPTZ DEFAULT NULL
-        """
-    )
-
-    # Index to efficiently query memories that failed consolidation for a given bank
-    op.execute(
-        f"""
-        CREATE INDEX IF NOT EXISTS idx_memory_units_consolidation_failed
-        ON {schema}memory_units (bank_id, consolidation_failed_at)
-        WHERE consolidation_failed_at IS NOT NULL AND fact_type IN ('experience', 'world')
-        """
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_consolidation_failed")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidation_failed_at")

hindsight-api-slim/hindsight_api/alembic/versions/a4b5c6d7e8f9_fix_per_bank_vector_index_type.py

@@ -1,142 +0,0 @@
-"""Fix per-bank vector indexes to match configured extension
-
-Revision ID: a4b5c6d7e8f9
-Revises: 2eee35aa3cfc
-Create Date: 2026-04-01
-
-Migration d5e6f7a8b9c0 hardcoded HNSW when creating per-bank partial vector
-indexes, ignoring HINDSIGHT_API_VECTOR_EXTENSION. Banks that existed when that
-migration ran got HNSW indexes even when pgvectorscale (DiskANN) or vchord
-was configured.
-
-This migration detects the mismatch and recreates the affected indexes with
-the correct type. Skipped entirely when the configured extension is pgvector
-(the default), since those indexes are already correct.
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import text
-
-revision: str = "a4b5c6d7e8f9"
-down_revision: str | Sequence[str] | None = "2eee35aa3cfc"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_FACT_TYPES: dict[str, str] = {
-    "world": "worl",
-    "experience": "expr",
-    "observation": "obsv",
-}
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _target_index_type() -> str | None:
-    """Return the target index type, or None if pgvector (no fix needed)."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "diskann"
-    elif ext == "vchord":
-        return "vchordrq"
-    return None
-
-
-def _vector_index_using_clause() -> str:
-    """Return the USING clause based on the configured vector extension."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
-    elif ext == "vchord":
-        return "USING vchordrq (embedding vector_l2_ops)"
-    else:
-        return "USING hnsw (embedding vector_cosine_ops)"
-
-
-def upgrade() -> None:
-    target = _target_index_type()
-    if target is None:
-        # pgvector — indexes are already HNSW, nothing to fix
-        return
-
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    schema = _get_schema_prefix()
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-    using_clause = _vector_index_using_clause()
-    pg_schema = schema_name or "public"
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-
-            # Check if this index exists and what type it is
-            idx_info = bind.execute(
-                text("SELECT indexdef FROM pg_indexes WHERE schemaname = :schema AND indexname = :idx"),
-                {"schema": pg_schema, "idx": idx_name},
-            ).fetchone()
-
-            if idx_info is None:
-                # Index doesn't exist — create it with the correct type
-                bind.execute(
-                    text(
-                        f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                        f"ON {table_ref} {using_clause} "
-                        f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                    )
-                )
-                continue
-
-            indexdef = idx_info[0].lower()
-            if target in indexdef:
-                # Already the correct type
-                continue
-
-            # Wrong type — drop and recreate
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} {using_clause} "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )
-
-
-def downgrade() -> None:
-    # Downgrade recreates indexes as HNSW (the original hardcoded behavior)
-    target = _target_index_type()
-    if target is None:
-        return
-
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    schema = _get_schema_prefix()
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )

hindsight-api-slim/hindsight_api/alembic/versions/b3c4d5e6f7a8_add_content_hash_to_chunks.py

@@ -1,32 +0,0 @@
-"""add content_hash to chunks table for delta retain
-
-Revision ID: b3c4d5e6f7a8
-Revises: a3b4c5d6e7f8
-Create Date: 2026-03-25
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b3c4d5e6f7a8"
-down_revision: str | Sequence[str] | None = "a3b4c5d6e7f8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Add content_hash column to chunks table for delta comparison
-    op.execute(f"ALTER TABLE {schema}chunks ADD COLUMN IF NOT EXISTS content_hash TEXT")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}chunks DROP COLUMN IF EXISTS content_hash")

hindsight-api-slim/hindsight_api/alembic/versions/b3w4x5y6z7a8_add_structured_content_to_mental_models.py

@@ -1,44 +0,0 @@
-"""Add structured_content JSONB column to mental_models
-
-Revision ID: b3w4x5y6z7a8
-Revises: a2v3w4x5y6z7
-Create Date: 2026-04-16
-
-Stores the structured representation of a mental model document (sections,
-blocks). The plain ``content`` column remains the rendered markdown shown to
-users. ``structured_content`` is the source of truth for delta-mode refreshes:
-each refresh applies a list of typed operations to the structured doc, then
-re-renders to markdown — so unchanged sections come through byte-identical
-without an LLM round-trip.
-
-Nullable: existing markdown-only mental models continue to work in full mode;
-the column is populated lazily the first time a model is refreshed in delta
-mode.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "b3w4x5y6z7a8"
-down_revision: str | Sequence[str] | None = "a2v3w4x5y6z7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"""
-        ALTER TABLE {schema}mental_models
-        ADD COLUMN IF NOT EXISTS structured_content JSONB
-    """)
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS structured_content")

hindsight-api-slim/hindsight_api/alembic/versions/c2d3e4f5g6h7_add_audit_log_table.py

@@ -1,61 +0,0 @@
-"""Add audit_log table for feature usage tracking.
-
-Merge migration that combines the two existing heads (a3b4c5d6e7f8 + c8e5f2a3b4d1).
-
-Stores raw request/response as JSONB for expandability without future migrations.
-The metadata JSONB column allows adding arbitrary fields in the future.
-
-Revision ID: c2d3e4f5g6h7
-Revises: a3b4c5d6e7f8, c8e5f2a3b4d1
-Create Date: 2026-03-26
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c2d3e4f5g6h7"
-down_revision: str | Sequence[str] | None = ("a3b4c5d6e7f8", "c8e5f2a3b4d1")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}audit_log (
-            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-            action TEXT NOT NULL,
-            transport TEXT NOT NULL,
-            bank_id TEXT,
-            started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-            ended_at TIMESTAMPTZ,
-            request JSONB,
-            response JSONB,
-            metadata JSONB DEFAULT '{{}}'::jsonb
-        )
-        """
-    )
-
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_audit_log_action_started ON {schema}audit_log (action, started_at DESC)"
-    )
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_bank_started ON {schema}audit_log (bank_id, started_at DESC)")
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_started ON {schema}audit_log (started_at DESC)")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_bank_started")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_action_started")
-    op.execute(f"DROP TABLE IF EXISTS {schema}audit_log")

hindsight-api-slim/hindsight_api/alembic/versions/c3d4e5f6g7h8_add_history_to_mental_models.py

@@ -1,30 +0,0 @@
-"""Add history column to mental_models
-
-Revision ID: c3d4e5f6g7h8
-Revises: a2b3c4d5e6f7, a2b3c4d5e6f8
-Create Date: 2026-03-06
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c3d4e5f6g7h8"
-down_revision: str | Sequence[str] | None = ("a2b3c4d5e6f7", "a2b3c4d5e6f8")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")

hindsight-api-slim/hindsight_api/alembic/versions/c4x5y6z7a8b9_backsweep_orphan_observations_v2.py

@@ -1,66 +0,0 @@
-"""backsweep_orphan_observations_v2
-
-Re-run of Pass 2 from migration ``g7h8i9j0k1l2_backsweep_orphan_observations``
-to sweep observations that became orphaned between then and now.
-
-Why we need it again:
-  ``fact_storage.handle_document_tracking`` (the retain/upsert path) deleted
-  the existing document via the FK cascade — which removes the source
-  ``memory_units`` — but never invalidated the observations derived from
-  them.  Only the explicit ``MemoryEngine.delete_document`` API called
-  ``_delete_stale_observations_for_memories``.  Every document re-ingest
-  therefore left orphan observations whose ``source_memory_ids`` arrays
-  pointed at IDs that no longer existed in ``memory_units``.
-
-  ``handle_document_tracking`` now calls the same cleanup helper before the
-  cascade, so no new orphans will accumulate going forward.  This migration
-  cleans up the historical residue.
-
-Identical to Pass 2 of g7h8i9j0k1l2.  Pass 1 (memory_units whose bank is
-gone) is intentionally not re-run; that scenario has no fresh source.
-
-Revision ID: c4x5y6z7a8b9
-Revises: b3w4x5y6z7a8
-Create Date: 2026-04-16
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c4x5y6z7a8b9"
-down_revision: str | Sequence[str] | None = "b3w4x5y6z7a8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    mu = f"{schema}memory_units"
-
-    # Delete observations whose every source_memory_id refers to a now-deleted
-    # memory_unit (or the array is empty).  Observations with at least one
-    # surviving source are left alone — the consolidation engine will refresh
-    # their text on the next pass.
-    op.execute(
-        f"""
-        DELETE FROM {mu} orphan
-        WHERE orphan.fact_type = 'observation'
-          AND NOT EXISTS (
-              SELECT 1
-              FROM {mu} src
-              WHERE src.id = ANY(orphan.source_memory_ids)
-                AND src.bank_id = orphan.bank_id
-          )
-        """
-    )
-
-
-def downgrade() -> None:
-    # Deleted rows cannot be restored.
-    pass

hindsight-api-slim/hindsight_api/alembic/versions/c5d6e7f8a9b0_add_bank_id_to_memory_links.py

@@ -1,48 +0,0 @@
-"""Add bank_id column to memory_links for direct filtering
-
-The stats endpoint JOINs memory_links to memory_units just to filter by
-bank_id.  With millions of links this takes 18+ seconds.  Adding bank_id
-directly to memory_links lets Postgres push the filter down before the JOIN.
-
-Revision ID: c5d6e7f8a9b0
-Revises: b3c4d5e6f7a8
-Create Date: 2026-03-26
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "c5d6e7f8a9b0"
-down_revision: str | Sequence[str] | None = "b3c4d5e6f7a8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Add nullable column
-    op.execute(f"ALTER TABLE {schema}memory_links ADD COLUMN IF NOT EXISTS bank_id TEXT")
-
-    # 2. Backfill from memory_units
-    op.execute(f"""
-        UPDATE {schema}memory_links ml
-        SET bank_id = mu.bank_id
-        FROM {schema}memory_units mu
-        WHERE ml.from_unit_id = mu.id
-          AND ml.bank_id IS NULL
-    """)
-
-    # 3. Set NOT NULL
-    op.execute(f"ALTER TABLE {schema}memory_links ALTER COLUMN bank_id SET NOT NULL")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}memory_links DROP COLUMN IF EXISTS bank_id")

hindsight-api-slim/hindsight_api/alembic/versions/d4e5f6g7h8i9_gin_source_memory_ids_fastupdate_off.py

@@ -1,53 +0,0 @@
-"""Recreate idx_memory_units_source_memory_ids GIN index with fastupdate=off
-
-GIN indexes use a "fastupdate" pending list by default: small writes are
-buffered there and flushed to the main GIN tree in bulk. Flushing requires
-AccessExclusiveLock on the index. Under high insert concurrency (e.g. 8
-parallel pytest-xdist workers all calling retain_async) two transactions can
-each trigger a flush simultaneously and deadlock.
-
-Disabling fastupdate makes every insert write directly to the GIN tree
-(slightly slower per insert, but no pending-list lock cycles).
-
-Revision ID: d4e5f6g7h8i9
-Revises: d5e6f7a8b9c0
-Create Date: 2026-03-11
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "d4e5f6g7h8i9"
-down_revision: str | Sequence[str] | None = "d5e6f7a8b9c0"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # DROP + CREATE CONCURRENTLY must run outside a transaction block.
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WITH (fastupdate=off) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute("COMMIT")
-    op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
-    op.execute(
-        f"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_memory_units_source_memory_ids "
-        f"ON {schema}memory_units USING GIN (source_memory_ids) "
-        f"WHERE source_memory_ids IS NOT NULL"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/d5e6f7a8b9c0_add_bank_internal_id_and_per_bank_hnsw.py

@@ -1,139 +0,0 @@
-"""Add internal_id to banks and per-(bank, fact_type) partial vector indexes
-
-Revision ID: d5e6f7a8b9c0
-Revises: a3b4c5d6e7f8
-Create Date: 2026-03-11
-
-This migration:
-1. Adds internal_id UUID column to banks (stable identifier for index naming)
-2. Drops the global vector index (competes with per-bank partial indexes)
-3. Creates per-(bank_id, fact_type) partial vector indexes for all existing banks
-   using the configured vector extension (HNSW for pgvector, DiskANN for
-   pgvectorscale, vchordrq for vchord).
-   (new banks get indexes created at bank-creation time via bank_utils.create_bank_vector_indexes)
-
-Why per-(bank, fact_type) indexes:
-- fact_type-only partial indexes are never chosen by the planner when bank_id is in the WHERE
-  clause, because the idx_memory_units_bank_id B-tree index always wins at planning time.
-- Per-(bank, fact_type) partial indexes have both predicates matching → planner selects them.
-- The global vector index competes for larger partitions (world, observation) and must be dropped.
-"""
-
-import os
-from collections.abc import Sequence
-
-from alembic import context, op
-from sqlalchemy import text
-
-revision: str = "d5e6f7a8b9c0"
-down_revision: str | Sequence[str] | None = "c3d4e5f6g7h8"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-_FACT_TYPES: dict[str, str] = {
-    "world": "worl",
-    "experience": "expr",
-    "observation": "obsv",
-}
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def _vector_index_using_clause() -> str:
-    """Return the USING clause based on the configured vector extension."""
-    ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
-    if ext == "pgvectorscale":
-        return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
-    elif ext == "vchord":
-        return "USING vchordrq (embedding vector_l2_ops)"
-    else:
-        return "USING hnsw (embedding vector_cosine_ops)"
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Add internal_id column to banks
-    op.execute(
-        f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS internal_id UUID DEFAULT gen_random_uuid() NOT NULL"
-    )
-    op.execute(f"ALTER TABLE {schema}banks ADD CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)")
-
-    # 2. Drop any fact_type-only partial indexes that may exist from prior migrations
-    #    (bank_id B-tree always wins over them when bank_id is in the WHERE clause)
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_world")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_observation")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_experience")
-
-    # 4. Drop global vector index (competes with per-bank partial indexes)
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_embedding")
-
-    # 5. Create per-(bank, fact_type) partial vector indexes for all existing banks
-    #    using the configured extension (HNSW / DiskANN / vchordrq)
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-    using_clause = _vector_index_using_clause()
-
-    rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        bank_id = row[0]
-        internal_id = str(row[1]).replace("-", "")[:16]
-        escaped_bank_id = bank_id.replace("'", "''")
-        for ft, ft_short in _FACT_TYPES.items():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            # Index name is schema-unqualified (indexes live in the schema of their table)
-            bind.execute(
-                text(
-                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
-                    f"ON {table_ref} {using_clause} "
-                    f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
-                )
-            )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Drop per-bank HNSW indexes (iterate existing banks)
-    bind = op.get_bind()
-    schema_name = context.config.get_main_option("target_schema")
-    banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
-
-    rows = bind.execute(text(f"SELECT internal_id FROM {banks_ref}")).fetchall()  # noqa: S608
-    for row in rows:
-        internal_id = str(row[0]).replace("-", "")[:16]
-        for ft_short in _HNSW_FACT_TYPES.values():
-            idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
-            bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
-
-    # Restore the global HNSW index
-    table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_memory_units_embedding ON {table_ref} USING hnsw (embedding vector_cosine_ops)"
-    )
-
-    # Restore old fact_type-only partial indexes
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_world "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'world'"
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_observation "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'observation'"
-    )
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_mu_emb_experience "
-        f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
-        f"WHERE fact_type = 'experience'"
-    )
-
-    # Drop internal_id column
-    op.execute(f"ALTER TABLE {schema}banks DROP CONSTRAINT IF EXISTS banks_internal_id_unique")
-    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS internal_id")

hindsight-api-slim/hindsight_api/alembic/versions/d6e7f8a9b0c1_drop_documents_metadata_column.py

@@ -1,39 +0,0 @@
-"""Drop unused metadata column from documents table
-
-Revision ID: d6e7f8a9b0c1
-Revises: c2d3e4f5g6h7, c5d6e7f8a9b0
-Create Date: 2026-03-30
-
-The metadata column on documents was always stored as an empty dict {}.
-Actual document metadata is stored inside retain_params.metadata.
-
-This migration was originally shipped in v0.4.22, then its file was deleted
-in v0.5.0 (and its revision ID accidentally reused by 2eee35aa3cfc).
-Restoring the file so that databases stamped at this revision can upgrade
-cleanly to v0.5.x+.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "d6e7f8a9b0c1"
-down_revision: str | Sequence[str] | None = ("c2d3e4f5g6h7", "c5d6e7f8a9b0")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS metadata")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS metadata jsonb DEFAULT '{{}}'")

hindsight-api-slim/hindsight_api/alembic/versions/e4f5a6b7c8d9_add_webhooks_tables.py

@@ -1,62 +0,0 @@
-"""Add webhooks table and next_retry_at to async_operations.
-
-Webhook deliveries are handled as async_operations tasks (operation_type='webhook_delivery')
-rather than a dedicated webhook_deliveries table.
-
-Revision ID: e4f5a6b7c8d9
-Revises: d2e3f4a5b6c7
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "e4f5a6b7c8d9"
-down_revision: str | Sequence[str] | None = "d2e3f4a5b6c7"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    op.execute(
-        f"""
-        CREATE TABLE IF NOT EXISTS {schema}webhooks (
-            id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
-            bank_id TEXT,
-            url TEXT NOT NULL,
-            secret TEXT,
-            event_types TEXT[] NOT NULL DEFAULT '{{}}',
-            enabled BOOLEAN NOT NULL DEFAULT TRUE,
-            created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-            updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
-        )
-        """
-    )
-
-    # Index for bank-scoped webhook lookup
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_webhooks_bank_id ON {schema}webhooks(bank_id)")
-
-    # Add next_retry_at to async_operations for task-owned retry scheduling
-    op.execute(f"ALTER TABLE {schema}async_operations ADD COLUMN IF NOT EXISTS next_retry_at TIMESTAMPTZ NULL")
-
-    # Index for polling: status + next_retry_at
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_async_operations_status_retry "
-        f"ON {schema}async_operations(status, next_retry_at)"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_status_retry")
-    op.execute(f"ALTER TABLE {schema}async_operations DROP COLUMN IF EXISTS next_retry_at")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_webhooks_bank_id")
-    op.execute(f"DROP TABLE IF EXISTS {schema}webhooks")

hindsight-api-slim/hindsight_api/alembic/versions/e5f6g7h8i9j0_cascade_delete_ops_on_bank_delete.py

@@ -1,73 +0,0 @@
-"""Add CASCADE DELETE FK from async_operations and webhooks to banks.
-
-When a bank is deleted, all its async_operations and webhooks rows are
-automatically deleted by the database. This ensures that any in-flight
-worker tasks detect the deletion via _check_op_alive() and abort early.
-
-Revision ID: e5f6g7h8i9j0
-Revises: d4e5f6g7h8i9
-Create Date: 2026-03-11
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "e5f6g7h8i9j0"
-down_revision: str | Sequence[str] | None = "d4e5f6g7h8i9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Remove orphaned async_operations rows whose bank no longer exists
-    # (can happen because there was no FK before this migration).
-    op.execute(
-        f"""
-        DELETE FROM {schema}async_operations
-        WHERE bank_id IS NOT NULL
-          AND bank_id NOT IN (SELECT bank_id FROM {schema}banks)
-        """
-    )
-
-    # Remove orphaned webhooks rows whose bank no longer exists.
-    op.execute(
-        f"""
-        DELETE FROM {schema}webhooks
-        WHERE bank_id IS NOT NULL
-          AND bank_id NOT IN (SELECT bank_id FROM {schema}banks)
-        """
-    )
-
-    # Add FK with ON DELETE CASCADE so that deleting a bank automatically
-    # cleans up all its pending/processing operations and webhook configs.
-    op.execute(
-        f"""
-        ALTER TABLE {schema}async_operations
-            ADD CONSTRAINT fk_async_operations_bank_id
-            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id)
-            ON DELETE CASCADE
-        """
-    )
-
-    op.execute(
-        f"""
-        ALTER TABLE {schema}webhooks
-            ADD CONSTRAINT fk_webhooks_bank_id
-            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id)
-            ON DELETE CASCADE
-        """
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS fk_async_operations_bank_id")
-    op.execute(f"ALTER TABLE {schema}webhooks DROP CONSTRAINT IF EXISTS fk_webhooks_bank_id")

hindsight-api-slim/hindsight_api/alembic/versions/f6g7h8i9j0k1_chunk_fk_cascade_delete.py

@@ -1,57 +0,0 @@
-"""chunk_fk_cascade_delete
-
-Revision ID: f6g7h8i9j0k1
-Revises: e5f6g7h8i9j0
-Create Date: 2026-03-16 00:00:00.000000
-
-"""
-
-from collections.abc import Sequence
-
-from alembic import op
-
-# revision identifiers, used by Alembic.
-revision: str = "f6g7h8i9j0k1"
-down_revision: str | Sequence[str] | None = "e5f6g7h8i9j0"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def upgrade() -> None:
-    """Change memory_units.chunk_id FK from SET NULL to CASCADE.
-
-    When a document is deleted the CASCADE reaches chunks first; with SET NULL
-    the memory_units rows survived with chunk_id = NULL, leaving ghost records.
-    Switching to CASCADE ensures they are removed together with their chunk.
-    """
-    from alembic import context
-
-    schema = context.config.get_main_option("target_schema")
-    schema_prefix = f'"{schema}".' if schema else ""
-    # Use raw SQL with IF EXISTS so this is safe on schemas where the FK was
-    # already dropped or never existed under this name.
-    op.execute(f"ALTER TABLE {schema_prefix}memory_units DROP CONSTRAINT IF EXISTS memory_units_chunk_fkey")
-    # Use a DO block so the ADD is also idempotent: if the FK already exists (e.g.
-    # the schema was provisioned after the base migration already added it) the
-    # duplicate_object exception is swallowed rather than failing the migration.
-    op.execute(
-        f"""
-        DO $$ BEGIN
-            ALTER TABLE {schema_prefix}memory_units
-                ADD CONSTRAINT memory_units_chunk_fkey
-                FOREIGN KEY (chunk_id)
-                REFERENCES {schema_prefix}chunks (chunk_id)
-                ON DELETE CASCADE;
-        EXCEPTION
-            WHEN duplicate_object THEN NULL;
-        END $$;
-        """
-    )
-
-
-def downgrade() -> None:
-    """Revert to SET NULL behaviour."""
-    op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
-    op.create_foreign_key(
-        "memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="SET NULL"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/f7g8h9i0j1k2_add_webhook_http_config.py

@@ -1,33 +0,0 @@
-"""Add http_config JSONB column to webhooks table.
-
-Stores HTTP delivery configuration (method, timeout, headers, params) as a
-single JSONB column rather than separate columns.
-
-Revision ID: f7g8h9i0j1k2
-Revises: e4f5a6b7c8d9
-Create Date: 2026-03-04
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "f7g8h9i0j1k2"
-down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}webhooks ADD COLUMN IF NOT EXISTS http_config JSONB NOT NULL DEFAULT '{{}}'")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}webhooks DROP COLUMN IF EXISTS http_config")

hindsight-api-slim/hindsight_api/alembic/versions/g2h3i4j5k6l7_remove_opinion_fact_type.py

@@ -1,83 +0,0 @@
-"""remove_opinion_fact_type
-
-Revision ID: g2h3i4j5k6l7
-Revises: f1a2b3c4d5e6
-Create Date: 2026-04-02
-
-Remove the deprecated 'opinion' fact type: drop opinion-specific indexes,
-update CHECK constraints, delete any remaining opinion rows, and drop the
-confidence_score column (was only used for opinions, always NULL otherwise).
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-# revision identifiers, used by Alembic.
-revision: str = "g2h3i4j5k6l7"
-down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # 1. Delete any remaining opinion rows
-    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
-
-    # 2. Drop opinion-specific indexes
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_confidence")
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_date")
-
-    # 3. Drop confidence_score constraints and column (only used for opinions, always NULL otherwise)
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS confidence_score_fact_type_check")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_confidence_score_check")
-    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS confidence_score")
-
-    # 4. Replace fact_type CHECK constraint
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
-        f"CHECK (fact_type IN ('world', 'experience', 'observation'))"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-
-    # Restore confidence_score column
-    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS confidence_score float")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_confidence_score_check "
-        f"CHECK (confidence_score IS NULL OR (confidence_score >= 0.0 AND confidence_score <= 1.0))"
-    )
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT confidence_score_fact_type_check "
-        f"CHECK ((fact_type = 'opinion' AND confidence_score IS NOT NULL) OR "
-        f"(fact_type = 'observation') OR "
-        f"(fact_type NOT IN ('opinion', 'observation') AND confidence_score IS NULL))"
-    )
-
-    # Restore original fact_type CHECK constraint (with opinion)
-    op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
-    op.execute(
-        f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
-        f"CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))"
-    )
-
-    # Recreate opinion indexes
-    op.execute(
-        f"CREATE INDEX idx_memory_units_opinion_confidence ON {schema}memory_units "
-        f"(bank_id, confidence_score DESC) WHERE fact_type = 'opinion'"
-    )
-    op.execute(
-        f"CREATE INDEX idx_memory_units_opinion_date ON {schema}memory_units "
-        f"(bank_id, event_date DESC) WHERE fact_type = 'opinion'"
-    )

hindsight-api-slim/hindsight_api/alembic/versions/g7h8i9j0k1l2_backsweep_orphan_observations.py

@@ -1,71 +0,0 @@
-"""backsweep_orphan_memory_units
-
-Two-pass cleanup of memory_units rows that were never removed by earlier bugs:
-
-Pass 1 — any fact_type, bank gone:
-  memory_units whose bank_id no longer exists in banks.  These accumulate when
-  a bank is deleted without a proper cascade (no FK from memory_units to banks
-  exists in the schema).
-
-Pass 2 — observations only, all sources gone:
-  observation rows whose bank still exists but every source_memory_id points
-  to a deleted memory unit.  These were left behind before PR #580 fixed the
-  chunk FK cascade and before delete_document() called
-  _delete_stale_observations_for_memories.
-
-Revision ID: g7h8i9j0k1l2
-Revises: f6g7h8i9j0k1
-Create Date: 2026-03-16
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "g7h8i9j0k1l2"
-down_revision: str | Sequence[str] | None = "f6g7h8i9j0k1"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    mu = f"{schema}memory_units"
-    banks = f"{schema}banks"
-
-    # Pass 1: delete all memory_units (any fact_type) whose bank no longer exists.
-    # There is no FK from memory_units to banks, so these never cascade away.
-    op.execute(
-        f"""
-        DELETE FROM {mu}
-        WHERE NOT EXISTS (
-            SELECT 1 FROM {banks} b WHERE b.bank_id = {mu}.bank_id
-        )
-        """
-    )
-
-    # Pass 2: delete orphaned observations whose bank still exists but every
-    # source_memory_id refers to a now-deleted memory unit (or the array is
-    # empty).  Observations with at least one surviving source are left alone.
-    op.execute(
-        f"""
-        DELETE FROM {mu} orphan
-        WHERE orphan.fact_type = 'observation'
-          AND NOT EXISTS (
-              SELECT 1
-              FROM {mu} src
-              WHERE src.id = ANY(orphan.source_memory_ids)
-                AND src.bank_id = orphan.bank_id
-          )
-        """
-    )
-
-
-def downgrade() -> None:
-    # Deleted rows cannot be restored.
-    pass

hindsight-api-slim/hindsight_api/alembic/versions/h3i4j5k6l7m8_merge_heads_and_add_unit_entities_index.py

@@ -1,42 +0,0 @@
-"""Merge 3 migration heads and add unit_entities composite index
-
-Revision ID: h3i4j5k6l7m8
-Revises: a4b5c6d7e8f9, g2h3i4j5k6l7
-Create Date: 2026-04-07
-
-Merges three unmerged migration heads into one, and adds a composite index
-(entity_id, unit_id) on unit_entities for index-only scans in the LATERAL
-entity expansion query.
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "h3i4j5k6l7m8"
-down_revision: str | Sequence[str] | None = ("a4b5c6d7e8f9", "g2h3i4j5k6l7")
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    # Composite index enables index-only scans for entity_id -> unit_id lookups
-    op.execute(
-        f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity_unit ON {schema}unit_entities (entity_id, unit_id)"
-    )
-    # Drop the now-redundant single-column index
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity")
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity_unit")
-    # Restore the single-column index
-    op.execute(f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity ON {schema}unit_entities (entity_id)")

hindsight-api-slim/hindsight_api/alembic/versions/i4j5k6l7m8n9_add_cancelled_status_to_async_operations.py

@@ -1,39 +0,0 @@
-"""Add 'cancelled' to async_operations status check constraint
-
-Revision ID: i4j5k6l7m8n9
-Revises: 8c6fa6f7230b
-Create Date: 2026-04-23
-"""
-
-from collections.abc import Sequence
-
-from alembic import context, op
-
-revision: str = "i4j5k6l7m8n9"
-down_revision: str | Sequence[str] | None = "8c6fa6f7230b"
-branch_labels: str | Sequence[str] | None = None
-depends_on: str | Sequence[str] | None = None
-
-
-def _get_schema_prefix() -> str:
-    """Get schema prefix for table names (required for multi-tenant support)."""
-    schema = context.config.get_main_option("target_schema")
-    return f'"{schema}".' if schema else ""
-
-
-def upgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
-    op.execute(
-        f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
-        f"CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled'))"
-    )
-
-
-def downgrade() -> None:
-    schema = _get_schema_prefix()
-    op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
-    op.execute(
-        f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
-        f"CHECK (status IN ('pending', 'processing', 'completed', 'failed'))"
-    )

hindsight-api-slim/hindsight_api/db_url.py

@@ -1,57 +0,0 @@
-"""Database URL normalization.
-
-Hindsight accepts SQLAlchemy-style URLs like ``postgresql+asyncpg://...?ssl=require``
-for its async engine, but the same string cannot be handed directly to synchronous
-SQLAlchemy (psycopg2) or to :func:`asyncpg.create_pool`, which both expect a
-libpq-compatible URL (``postgresql://...?sslmode=require``).
-
-:func:`to_libpq_url` performs that translation. It is idempotent and safe to
-apply to URLs that are already libpq-compatible, to the ``pg0`` embedded-PG
-marker, or to any non-PostgreSQL string (returned unchanged).
-"""
-
-from __future__ import annotations
-
-from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
-
-_ASYNCPG_SCHEMES = ("postgresql+asyncpg", "postgres+asyncpg")
-_POSTGRES_SCHEMES = ("postgresql", "postgres") + _ASYNCPG_SCHEMES
-
-
-def to_libpq_url(url: str) -> str:
-    """Normalize a PostgreSQL URL for libpq-style consumers.
-
-    Accepts a SQLAlchemy URL (``postgresql+asyncpg://...``) or a plain libpq
-    URL and returns a form suitable for:
-
-    - :func:`sqlalchemy.create_engine` (sync / psycopg2)
-    - :func:`asyncpg.create_pool`
-
-    Transformations:
-
-    - ``postgresql+asyncpg`` / ``postgres+asyncpg`` / ``postgres`` → ``postgresql``
-    - Query param ``ssl=<mode>`` → ``sslmode=<mode>`` (SQLAlchemy's asyncpg
-      dialect uses ``ssl=``; libpq uses ``sslmode=``)
-
-    Any non-PostgreSQL input (e.g. the ``pg0`` embedded-PG marker, a sqlite
-    URL, an empty string) is returned unchanged. Already-normalized URLs are
-    returned unchanged.
-    """
-    if not url or "://" not in url:
-        return url
-
-    parts = urlsplit(url)
-    if parts.scheme not in _POSTGRES_SCHEMES:
-        return url
-
-    new_scheme = "postgresql"
-
-    new_query_pairs = [
-        ("sslmode", v) if k == "ssl" else (k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True)
-    ]
-    new_query = urlencode(new_query_pairs)
-
-    if new_scheme == parts.scheme and new_query == parts.query:
-        return url
-
-    return urlunsplit((new_scheme, parts.netloc, parts.path, new_query, parts.fragment))

hindsight-api-slim/hindsight_api/engine/audit.py

@@ -1,209 +0,0 @@
-"""Audit logging for feature usage tracking.
-
-Provides fire-and-forget audit logging of all mutating and core operations
-(retain, recall, reflect, bank CRUD, etc.) across HTTP, MCP, and system transports.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import json
-import logging
-import uuid
-from collections.abc import Callable
-from contextlib import asynccontextmanager
-from dataclasses import dataclass, field
-from datetime import datetime, timezone
-from typing import Any
-
-import asyncpg
-
-from ..engine.db_utils import acquire_with_retry
-
-logger = logging.getLogger(__name__)
-
-
-@dataclass
-class AuditEntry:
-    """A single audit log entry."""
-
-    action: str
-    transport: str  # "http", "mcp", "system"
-    bank_id: str | None = None
-    started_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
-    ended_at: datetime | None = None
-    request: dict[str, Any] | None = None
-    response: dict[str, Any] | None = None
-    metadata: dict[str, Any] = field(default_factory=dict)
-
-
-def _json_default(obj: Any) -> str:
-    """JSON serializer for objects not serializable by default."""
-    if isinstance(obj, datetime):
-        return obj.isoformat()
-    if isinstance(obj, uuid.UUID):
-        return str(obj)
-    if isinstance(obj, bytes):
-        return "<bytes>"
-    if isinstance(obj, set):
-        return list(obj)
-    return str(obj)
-
-
-def _safe_json(data: Any) -> str | None:
-    """Serialize data to JSON string, returning None on failure."""
-    if data is None:
-        return None
-    try:
-        return json.dumps(data, default=_json_default)
-    except Exception:
-        logger.debug("Failed to serialize audit data", exc_info=True)
-        return None
-
-
-_SWEEP_INTERVAL_SECONDS = 3600  # Run retention sweep every hour
-
-
-class AuditLogger:
-    """Fire-and-forget audit log writer with optional retention sweep."""
-
-    def __init__(
-        self,
-        pool_getter: Callable[[], asyncpg.Pool | None],
-        schema_getter: Callable[[], str],
-        enabled: bool,
-        allowed_actions: list[str],
-        retention_days: int = -1,
-    ) -> None:
-        self._pool_getter = pool_getter
-        self._schema_getter = schema_getter
-        self._enabled = enabled
-        self._allowed_actions: frozenset[str] | None = frozenset(allowed_actions) if allowed_actions else None
-        self._retention_days = retention_days
-        self._sweep_task: asyncio.Task | None = None
-
-    def is_enabled(self, action: str) -> bool:
-        """Check if audit logging is enabled for this action."""
-        if not self._enabled:
-            return False
-        if self._allowed_actions is not None:
-            return action in self._allowed_actions
-        return True
-
-    def log_fire_and_forget(self, entry: AuditEntry) -> None:
-        """Schedule an audit write as a background task."""
-        if not self.is_enabled(entry.action):
-            return
-        try:
-            asyncio.create_task(self._safe_log(entry))
-        except RuntimeError:
-            # No running event loop (e.g. during shutdown)
-            logger.debug("Cannot schedule audit log write: no running event loop")
-
-    async def _safe_log(self, entry: AuditEntry) -> None:
-        """Write audit entry to DB. Errors are logged, never raised."""
-        pool = self._pool_getter()
-        if pool is None:
-            logger.debug("Audit log skipped: pool not available")
-            return
-        try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
-            async with acquire_with_retry(pool, max_retries=1) as conn:
-                await conn.execute(
-                    f"""
-                    INSERT INTO {table}
-                        (id, action, transport, bank_id, started_at, ended_at, request, response, metadata)
-                    VALUES
-                        ($1, $2, $3, $4, $5, $6, $7::jsonb, $8::jsonb, $9::jsonb)
-                    """,
-                    uuid.uuid4(),
-                    entry.action,
-                    entry.transport,
-                    entry.bank_id,
-                    entry.started_at,
-                    entry.ended_at,
-                    _safe_json(entry.request),
-                    _safe_json(entry.response),
-                    _safe_json(entry.metadata) or "{}",
-                )
-        except Exception as e:
-            logger.warning(f"Audit log write failed for action={entry.action}: {e}")
-
-    def start_retention_sweep(self) -> None:
-        """Start the periodic retention sweep if retention is configured."""
-        if self._retention_days <= 0 or not self._enabled:
-            return
-        try:
-            self._sweep_task = asyncio.create_task(self._sweep_loop())
-        except RuntimeError:
-            logger.debug("Cannot start retention sweep: no running event loop")
-
-    async def stop_retention_sweep(self) -> None:
-        """Stop the periodic retention sweep."""
-        if self._sweep_task and not self._sweep_task.done():
-            self._sweep_task.cancel()
-            try:
-                await self._sweep_task
-            except asyncio.CancelledError:
-                pass
-            self._sweep_task = None
-
-    async def _sweep_loop(self) -> None:
-        """Periodically delete audit log entries older than retention_days."""
-        while True:
-            await self._run_sweep()
-            await asyncio.sleep(_SWEEP_INTERVAL_SECONDS)
-
-    async def _run_sweep(self) -> None:
-        """Delete expired audit log entries. Concurrent-safe via row-level deletes."""
-        pool = self._pool_getter()
-        if pool is None:
-            return
-        try:
-            schema = self._schema_getter()
-            table = f"{schema}.audit_log"
-            async with acquire_with_retry(pool, max_retries=1) as conn:
-                result = await conn.execute(
-                    f"DELETE FROM {table} WHERE started_at < NOW() - INTERVAL '{self._retention_days} days'"
-                )
-                if result and result != "DELETE 0":
-                    logger.info(f"Audit log retention sweep: {result}")
-        except Exception as e:
-            logger.warning(f"Audit log retention sweep failed: {e}")
-
-
-@asynccontextmanager
-async def audit_context(
-    audit_logger: AuditLogger | None,
-    action: str,
-    transport: str,
-    bank_id: str | None = None,
-    request: dict[str, Any] | None = None,
-    metadata: dict[str, Any] | None = None,
-):
-    """Async context manager that times the operation and writes audit on exit.
-
-    Usage:
-        async with audit_context(logger, "retain", "http", bank_id, request_dict) as entry:
-            result = await do_work()
-            entry.response = result_dict
-    """
-    if audit_logger is None or not audit_logger.is_enabled(action):
-        entry = AuditEntry(action=action, transport=transport, bank_id=bank_id)
-        yield entry
-        return
-
-    entry = AuditEntry(
-        action=action,
-        transport=transport,
-        bank_id=bank_id,
-        started_at=datetime.now(timezone.utc),
-        request=request,
-        metadata=metadata or {},
-    )
-    try:
-        yield entry
-    finally:
-        entry.ended_at = datetime.now(timezone.utc)
-        audit_logger.log_fire_and_forget(entry)

hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py

@@ -1,104 +0,0 @@
-"""Prompts for the consolidation engine."""
-
-# Default mission when no bank-specific mission is set
-_DEFAULT_MISSION = "Track every detail: names, numbers, dates, places, and relationships. Prefer specifics over abstractions, never generalise."
-
-# Processing rules — always present regardless of mission
-_PROCESSING_RULES = """Processing rules (always apply):
-
-1. ONE OBSERVATION PER DISTINCT FACET: each observation tracks exactly one specific facet — a count ("has 3 items"), a named entity ("has a dog named Rex"), a relationship ("works at Google"), etc. Never merge different facets into one observation.
-
-2. MATCH BY ENTITY/FACET, NOT TOPIC: when deciding whether to UPDATE vs CREATE, match on the specific entity or facet. "Sold item X" updates only the X observation. "Now has 5 items" updates only the count observation. Do not update observations about different entities just because they share a general topic.
-
-3. STATE CHANGES — UPDATE CONCISELY: when a fact changes the state of something ("sold X", "X died", "moved to Y"), UPDATE the matching observation to reflect the current state. Include dates when available. Keep it concise — only information about THAT specific facet. Example: "User owned a dog named Rex who died on March 15, 2025". Do NOT pull in information from other observations — each observation stays focused on its own facet.
-
-4. CASCADE TO ALL AFFECTED OBSERVATIONS: a state change may affect multiple observations. For example, if entity C is removed from a group, update BOTH the individual observation for C AND any list/group observation that includes C (remove C from the list while keeping all other members intact).
-
-5. NO COMPUTATION: you do not have the full picture — never calculate, derive, or adjust numeric values. If the user says "I have 2 dogs" and then "I have a dog named Rex", do NOT update the count to 3 — you don't know if Rex is one of the 2 or a new one. If the user says "I sold X", do NOT decrement a count. Only update a count when the user explicitly states a new count. Synthesize and consolidate what was stated, but never do arithmetic or logical deductions.
-
-6. SAME FACET → UPDATE, NOT CREATE: a new count supersedes the old count — UPDATE the existing count observation, don't create a second one. If there's an existing observation for the same specific facet, always UPDATE it rather than creating a duplicate.
-
-7. PRESERVE HISTORY: observations that record significant events (sold, died, moved, changed) are important history — never DELETE them. Only delete an observation when it is restated identically or truly meaningless. Be very conservative with deletes.
-
-8. RESOLVE REFERENCES: when a new fact provides a concrete value for a vague placeholder in an existing observation (e.g., "home country" → "Sweden"), UPDATE to embed the resolved value.
-
-9. NEVER merge observations about different people or unrelated topics."""
-
-# Data section — format placeholders {facts_text} and {observations_text} are substituted at call time
-_BATCH_DATA_SECTION = """
-NEW FACTS:
-{facts_text}
-
-EXISTING OBSERVATIONS (JSON array, pooled from recalls across all facts above):
-{observations_text}
-
-Each observation includes:
-- id: unique identifier for updating
-- text: the observation content
-- proof_count: number of supporting memories
-- occurred_start/occurred_end: temporal range of source facts
-- source_memories: array of supporting facts with their text and dates
-
-Compare the facts against existing observations:
-- Same facet as an existing observation → UPDATE it (observation_id + source_fact_ids)
-- New facet with durable knowledge → CREATE a new observation (source_fact_ids)
-- Cross-reference facts within the batch: a later fact may resolve a vague reference in an earlier one
-- Purely ephemeral facts → omit them unless the MISSION above explicitly targets such data (e.g. timestamped events, session state, screen content)"""
-
-# Output format — JSON braces escaped as {{ }} so .format() leaves them literal
-_BATCH_OUTPUT_FORMAT = """
-Output a JSON object with three arrays.
-
-## EXAMPLE
-
-Input facts:
-[a1b2c3d4-e5f6-7890-abcd-ef1234567890] Alice mentioned she works long hours, often past midnight | Involving: Alice (occurred_start=2024-01-15, mentioned_at=2024-01-15)
-[b2c3d4e5-f6a7-8901-bcde-f12345678901] Alice said she's exhausted from the project deadlines | Involving: Alice (occurred_start=2024-01-20, mentioned_at=2024-01-20)
-
-Good observation text — clean prose, no metadata, each fact tracked distinctly:
-  "Alice works long hours, often past midnight."
-  "Alice feels exhausted from project deadlines."
-
-Bad observation text — NEVER do this (verbatim copy of fact text with metadata):
-  "Alice mentioned she works long hours, often past midnight | Involving: Alice (occurred_start=2024-01-15, mentioned_at=2024-01-15)"
-
-Observation text rules:
-- Write clean prose — NEVER copy raw fact lines or their metadata (temporal fields, "Involving:", "When:" labels, UUIDs).
-- Parenthesized metadata like (occurred_start=...) and pipe-separated labels like "| Involving: ..." are fact formatting — strip them entirely from observation text.
-- How many observations to create and how much to aggregate is driven by the MISSION above.
-
-{{"creates": [{{"text": "Alice works long hours, often past midnight.", "source_fact_ids": ["a1b2c3d4-e5f6-7890-abcd-ef1234567890"]}}, {{"text": "Alice feels exhausted from project deadlines.", "source_fact_ids": ["b2c3d4e5-f6a7-8901-bcde-f12345678901"]}}],
-  "updates": [{{"text": "Alice works at Acme Corp as a senior engineer", "observation_id": "c3d4e5f6-a7b8-9012-cdef-123456789012", "source_fact_ids": ["d4e5f6a7-b8c9-0123-defa-234567890123"]}}],
-  "deletes": [{{"observation_id": "e5f6a7b8-c9d0-1234-efab-345678901234"}}]}}
-
-Rules:
-- "source_fact_ids": copy the EXACT UUID strings shown in brackets [uuid] from NEW FACTS — never use integers or positions.
-- "observation_id": copy the EXACT "id" UUID string from EXISTING OBSERVATIONS.
-- One create/update may reference multiple facts when they jointly support the observation.
-- "deletes": only when an observation is directly superseded or contradicted by new facts.
-- Do NOT include "tags" — handled automatically.
-- Return {{"creates": [], "updates": [], "deletes": []}} if nothing durable is found."""
-
-
-def build_batch_consolidation_prompt(
-    observations_mission: str | None = None,
-    observation_capacity_note: str | None = None,
-) -> str:
-    """
-    Build the consolidation prompt for batch mode (multiple facts per LLM call).
-
-    The mission defines *what* to track (customisable per bank).
-    Processing rules and output format are always present regardless of mission.
-    """
-    mission = observations_mission or _DEFAULT_MISSION
-
-    capacity_section = ""
-    if observation_capacity_note:
-        capacity_section = f"\n\n## CAPACITY CONSTRAINT\n{observation_capacity_note}"
-
-    return (
-        "You are a memory consolidation system. Synthesize facts into observations "
-        "and merge with existing observations when appropriate.\n\n"
-        f"## MISSION\n{mission}{capacity_section}\n\n"
-        f"{_PROCESSING_RULES}" + _BATCH_DATA_SECTION + _BATCH_OUTPUT_FORMAT
-    )

hindsight-api-slim/hindsight_api/engine/jina_mlx_reranker.py

@@ -1,144 +0,0 @@
-"""
-MLX implementation of jina-reranker-v3 for Apple Silicon.
-
-This file is adapted from the official model repository:
-  https://huggingface.co/jinaai/jina-reranker-v3-mlx/blob/main/rerank.py
-
-License: CC BY-NC 4.0 (contact Jina AI for commercial usage)
-
-Changes from upstream:
-- Removed the __main__ example block
-- Type annotations added to public methods
-- top_n parameter added to rerank() (upstream only exposed it implicitly)
-"""
-
-import numpy as np
-
-
-class _MLPProjector:
-    def __init__(self):
-        import mlx.nn as nn
-
-        self.linear1 = nn.Linear(1024, 512, bias=False)
-        self.linear2 = nn.Linear(512, 512, bias=False)
-
-    def __call__(self, x):
-        import mlx.nn as nn
-
-        x = self.linear1(x)
-        x = nn.relu(x)
-        x = self.linear2(x)
-        return x
-
-
-def _load_projector(projector_path: str) -> _MLPProjector:
-    import mlx.core as mx
-    from safetensors import safe_open
-
-    projector = _MLPProjector()
-    with safe_open(projector_path, framework="numpy") as f:
-        projector.linear1.weight = mx.array(f.get_tensor("linear1.weight"))
-        projector.linear2.weight = mx.array(f.get_tensor("linear2.weight"))
-    return projector
-
-
-def _sanitize(text: str, special_tokens: dict[str, str]) -> str:
-    for token in special_tokens.values():
-        text = text.replace(token, "")
-    return text
-
-
-def _format_prompt(query: str, docs: list[str], special_tokens: dict[str, str]) -> str:
-    query = _sanitize(query, special_tokens)
-    docs = [_sanitize(d, special_tokens) for d in docs]
-
-    doc_token = special_tokens["doc_embed_token"]
-    query_token = special_tokens["query_embed_token"]
-
-    prefix = (
-        "<|im_start|>system\n"
-        "You are a search relevance expert who can determine a ranking of the passages based on how relevant they are to the query. "
-        "If the query is a question, how relevant a passage is depends on how well it answers the question. "
-        "If not, try to analyze the intent of the query and assess how well each passage satisfies the intent. "
-        "If an instruction is provided, you should follow the instruction when determining the ranking."
-        "<|im_end|>\n<|im_start|>user\n"
-    )
-    suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
-
-    body = (
-        f"I will provide you with {len(docs)} passages, each indicated by a numerical identifier. "
-        f"Rank the passages based on their relevance to query: {query}\n"
-    )
-    body += "\n".join(f'<passage id="{i}">\n{doc}{doc_token}\n</passage>' for i, doc in enumerate(docs))
-    body += f"\n<query>\n{query}{query_token}\n</query>"
-    return prefix + body + suffix
-
-
-class MLXReranker:
-    """
-    MLX-accelerated jina-reranker-v3 for Apple Silicon.
-
-    Loads the model from a local directory (use huggingface_hub.snapshot_download
-    to fetch jinaai/jina-reranker-v3-mlx if you don't have it already).
-    """
-
-    _SPECIAL_TOKENS = {
-        "query_embed_token": "<|rerank_token|>",
-        "doc_embed_token": "<|embed_token|>",
-    }
-    _DOC_TOKEN_ID = 151670
-    _QUERY_TOKEN_ID = 151671
-
-    def __init__(self, model_path: str, projector_path: str):
-        from mlx_lm import load
-
-        self.model, self.tokenizer = load(model_path)
-        self.model.eval()
-        self.projector = _load_projector(projector_path)
-
-    def rerank(self, query: str, documents: list[str], top_n: int | None = None) -> list[dict]:
-        """
-        Rank documents by relevance to a query.
-
-        Returns a list of dicts with keys: document, relevance_score, index.
-        Sorted by descending relevance_score.
-        """
-        import mlx.core as mx
-
-        prompt = _format_prompt(query, documents, self._SPECIAL_TOKENS)
-        input_ids = self.tokenizer.encode(prompt)
-        hidden_states = self.model.model([input_ids])[0]  # [seq_len, hidden_size]
-
-        input_ids_np = np.array(input_ids)
-        query_positions = np.where(input_ids_np == self._QUERY_TOKEN_ID)[0]
-        doc_positions = np.where(input_ids_np == self._DOC_TOKEN_ID)[0]
-
-        if len(query_positions) == 0:
-            raise ValueError("Query embed token not found in prompt")
-        if len(doc_positions) == 0:
-            raise ValueError("Document embed tokens not found in prompt")
-
-        query_hidden = mx.expand_dims(hidden_states[int(query_positions[0])], axis=0)
-        doc_hidden = mx.stack([hidden_states[int(p)] for p in doc_positions])
-
-        query_emb = self.projector(query_hidden)  # [1, 512]
-        doc_emb = self.projector(doc_hidden)  # [num_docs, 512]
-
-        query_exp = mx.broadcast_to(mx.expand_dims(query_emb, 0), (1, len(documents), 512))
-        doc_exp = mx.expand_dims(doc_emb, 0)
-
-        scores = mx.sum(doc_exp * query_exp, axis=-1) / (
-            mx.sqrt(mx.sum(doc_exp * doc_exp, axis=-1)) * mx.sqrt(mx.sum(query_exp * query_exp, axis=-1))
-        )  # [1, num_docs]
-        scores_np = np.array(scores[0])
-
-        order = np.argsort(scores_np)[::-1]
-        n = min(top_n, len(documents)) if top_n is not None else len(documents)
-        return [
-            {
-                "document": documents[order[i]],
-                "relevance_score": float(scores_np[order[i]]),
-                "index": int(order[i]),
-            }
-            for i in range(n)
-        ]

hindsight-api-slim/hindsight_api/engine/parsers/__init__.py

@@ -1,130 +0,0 @@
-"""File parser implementations."""
-
-import logging
-from dataclasses import dataclass
-
-from .base import FileParser, UnsupportedFileTypeError
-from .iris import IrisParser
-from .llama_parse import LlamaParseParser
-from .markitdown import MarkitdownParser
-
-__all__ = [
-    "FileParser",
-    "UnsupportedFileTypeError",
-    "IrisParser",
-    "LlamaParseParser",
-    "MarkitdownParser",
-    "FileParserRegistry",
-    "ConvertResult",
-]
-
-
-@dataclass
-class ConvertResult:
-    """Result of a successful file conversion."""
-
-    content: str
-    parser_name: str
-
-
-logger = logging.getLogger(__name__)
-
-
-class FileParserRegistry:
-    """Registry for file parsers with auto-detection."""
-
-    def __init__(self):
-        """Initialize empty parser registry."""
-        self._parsers: dict[str, FileParser] = {}
-
-    def register(self, parser: FileParser):
-        """
-        Register a parser.
-
-        Args:
-            parser: FileParser instance
-        """
-        self._parsers[parser.name()] = parser
-
-    def get_parser(
-        self,
-        name: str | None,
-        filename: str,
-        content_type: str | None = None,
-    ) -> FileParser:
-        """
-        Get parser by name or auto-detect.
-
-        Args:
-            name: Parser name (e.g., "markitdown") or None for auto-detect
-            filename: File name for auto-detection
-            content_type: MIME type (optional)
-
-        Returns:
-            FileParser instance
-
-        Raises:
-            ValueError: If no suitable parser found
-        """
-        if name:
-            # Explicit parser requested — return it directly, let the parser
-            # raise UnsupportedFileTypeError from convert() if needed
-            if name not in self._parsers:
-                raise ValueError(f"Parser '{name}' not found. Available: {list(self._parsers.keys())}")
-            return self._parsers[name]
-
-        # Auto-detect parser
-        for parser in self._parsers.values():
-            if parser.supports(filename, content_type):
-                return parser
-
-        raise ValueError(f"No parser found for {filename}. Available parsers: {list(self._parsers.keys())}")
-
-    async def convert_with_fallback(
-        self,
-        parsers: list[str],
-        file_data: bytes,
-        filename: str,
-        content_type: str | None = None,
-    ) -> ConvertResult:
-        """
-        Try each parser in order, falling back on failure or empty content.
-
-        Moves to the next parser if the current one raises UnsupportedFileTypeError
-        or returns empty content. Any other exception (RuntimeError, network error,
-        etc.) also triggers a fallback so the chain is exhausted before failing.
-
-        Args:
-            parsers: Ordered list of parser names to try
-            file_data: Raw file bytes
-            filename: Original filename
-            content_type: MIME type (optional)
-
-        Returns:
-            ConvertResult with the parsed content and the name of the parser that succeeded
-
-        Raises:
-            ValueError: If a parser name is not registered
-            RuntimeError: If all parsers fail or return empty content
-        """
-        last_error: Exception | None = None
-        for name in parsers:
-            parser = self.get_parser(name, filename, content_type)
-            try:
-                content = await parser.convert(file_data, filename)
-                if content and content.strip():
-                    return ConvertResult(content=content, parser_name=name)
-                logger.warning(f"Parser '{name}' returned empty content for '{filename}', trying next")
-                last_error = RuntimeError(f"Parser '{name}' returned no content for '{filename}'")
-            except UnsupportedFileTypeError as e:
-                logger.warning(f"Parser '{name}' does not support '{filename}', trying next: {e}")
-                last_error = e
-            except Exception as e:
-                logger.warning(f"Parser '{name}' failed for '{filename}', trying next: {e}")
-                last_error = e
-
-        raise last_error or RuntimeError(f"No parsers available for '{filename}'")
-
-    def list_parsers(self) -> list[str]:
-        """Get list of registered parser names."""
-        return list(self._parsers.keys())

hindsight-api-slim/hindsight_api/engine/parsers/llama_parse.py

@@ -1,123 +0,0 @@
-"""LlamaParse parser implementation using the LlamaIndex Cloud parsing API."""
-
-import asyncio
-import logging
-import mimetypes
-import time
-
-import httpx
-
-from .base import FileParser, UnsupportedFileTypeError
-
-logger = logging.getLogger(__name__)
-
-_LLAMA_PARSE_BASE_URL = "https://api.cloud.llamaindex.ai/api/parsing"
-_DEFAULT_POLL_INTERVAL = 2.0  # seconds
-_DEFAULT_TIMEOUT = 300.0  # seconds
-
-
-class LlamaParseParser(FileParser):
-    """
-    LlamaParse file parser using LlamaIndex's hosted parsing service.
-
-    Uploads files to the LlamaParse API, polls until the parse job completes,
-    and returns the resulting markdown. The API determines which file types
-    are supported — UnsupportedFileTypeError is raised if the file is rejected.
-
-    Authentication:
-        Requires HINDSIGHT_API_FILE_PARSER_LLAMA_PARSE_API_KEY environment
-        variable, or pass the key explicitly via the constructor.
-    """
-
-    def __init__(
-        self,
-        api_key: str,
-        poll_interval: float = _DEFAULT_POLL_INTERVAL,
-        timeout: float = _DEFAULT_TIMEOUT,
-    ):
-        """
-        Initialize llama_parse parser.
-
-        Args:
-            api_key: LlamaCloud API key (typically starts with "llx-")
-            poll_interval: Seconds between status poll requests (default: 2)
-            timeout: Maximum seconds to wait for parsing (default: 300)
-        """
-        self._api_key = api_key
-        self._poll_interval = poll_interval
-        self._timeout = timeout
-        self._auth_headers = {"Authorization": f"Bearer {api_key}"}
-
-    async def convert(self, file_data: bytes, filename: str) -> str:
-        """
-        Parse file to markdown using the LlamaParse API.
-
-        Raises:
-            UnsupportedFileTypeError: If the LlamaParse API rejects the file type (4xx)
-            RuntimeError: If parsing fails for another reason
-        """
-        content_type = mimetypes.guess_type(filename)[0] or "application/octet-stream"
-
-        async with httpx.AsyncClient(timeout=httpx.Timeout(30.0, read=120.0)) as client:
-            # Step 1: Upload file and start parse job
-            # LlamaParse expects multipart/form-data with the file under "file"
-            upload_resp = await client.post(
-                f"{_LLAMA_PARSE_BASE_URL}/upload",
-                headers=self._auth_headers,
-                files={"file": (filename, bytes(file_data), content_type)},
-            )
-            _raise_for_status(upload_resp, filename, "upload")
-            job_id: str = upload_resp.json()["id"]
-
-            # Step 2: Poll job status until SUCCESS or ERROR
-            deadline = time.monotonic() + self._timeout
-            while True:
-                status_resp = await client.get(
-                    f"{_LLAMA_PARSE_BASE_URL}/job/{job_id}",
-                    headers=self._auth_headers,
-                )
-                _raise_for_status(status_resp, filename, "poll job status")
-                status_data = status_resp.json()
-                status = status_data.get("status")
-
-                if status == "SUCCESS":
-                    break
-                if status in ("ERROR", "CANCELLED"):
-                    error = status_data.get("error_code") or status_data.get("error") or "unknown error"
-                    raise RuntimeError(f"LlamaParse job failed for '{filename}': {error}")
-
-                if time.monotonic() >= deadline:
-                    raise RuntimeError(f"LlamaParse job timed out after {self._timeout}s for '{filename}'")
-
-                await asyncio.sleep(self._poll_interval)
-
-            # Step 3: Fetch the markdown result
-            result_resp = await client.get(
-                f"{_LLAMA_PARSE_BASE_URL}/job/{job_id}/result/markdown",
-                headers=self._auth_headers,
-            )
-            _raise_for_status(result_resp, filename, "fetch markdown result")
-            markdown = result_resp.json().get("markdown")
-            if not markdown:
-                raise RuntimeError(f"No content extracted from '{filename}'")
-            return markdown
-
-    def name(self) -> str:
-        """Get parser name."""
-        return "llama_parse"
-
-
-def _raise_for_status(response: httpx.Response, filename: str, step: str) -> None:
-    """
-    Raise an appropriate error including the response body on HTTP errors.
-
-    Raises UnsupportedFileTypeError for 4xx responses (file rejected by the API),
-    RuntimeError for other HTTP errors.
-    """
-    if not response.is_error:
-        return
-    body = response.text or "<empty>"
-    msg = f"LlamaParse API error during {step} for '{filename}': {response.status_code} {response.reason_phrase} — {body}"
-    if response.is_client_error:
-        raise UnsupportedFileTypeError(msg)
-    raise RuntimeError(msg)

hindsight-api-slim/hindsight_api/engine/providers/litellm_llm.py

@@ -1,385 +0,0 @@
-"""
-LiteLLM LLM provider for universal model support.
-
-This provider enables using 100+ LLM providers via the LiteLLM SDK, including:
-- AWS Bedrock (bedrock/anthropic.claude-3-5-sonnet-...)
-- Azure OpenAI (azure/gpt-4o)
-- Together AI (together_ai/meta-llama/...)
-- Any other LiteLLM-supported provider
-
-Uses litellm.acompletion() for async chat completions.
-Authentication for cloud providers (e.g., AWS Bedrock via boto3 credential chain)
-is handled automatically by LiteLLM.
-"""
-
-import asyncio
-import json
-import logging
-import time
-from typing import Any
-
-from hindsight_api.engine.llm_interface import LLMInterface, OutputTooLongError
-from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
-from hindsight_api.metrics import get_metrics_collector
-from hindsight_api.worker.stage import set_stage
-
-logger = logging.getLogger(__name__)
-
-
-class LiteLLMLLM(LLMInterface):
-    """
-    LLM provider using the LiteLLM SDK for universal model support.
-
-    Supports any model accessible via litellm.acompletion(), including AWS Bedrock,
-    Azure OpenAI, Together AI, Fireworks AI, and more.
-
-    Model names follow LiteLLM conventions with provider prefixes:
-    - bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
-    - azure/gpt-4o
-    - together_ai/meta-llama/Llama-3-70b-chat-hf
-    - fireworks_ai/accounts/fireworks/models/llama-v3p1-70b-instruct
-    """
-
-    def __init__(
-        self,
-        provider: str,
-        api_key: str,
-        base_url: str,
-        model: str,
-        reasoning_effort: str = "low",
-        timeout: float = 300.0,
-        **kwargs: Any,
-    ):
-        super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
-        self.timeout = timeout
-        self._litellm: Any = None
-
-        try:
-            import litellm
-
-            self._litellm = litellm
-            # Suppress LiteLLM's verbose logging
-            litellm.suppress_debug_info = True  # type: ignore[assignment]
-            # Drop unsupported params instead of raising errors (e.g. tool_choice on some Bedrock models)
-            litellm.drop_params = True  # type: ignore[assignment]
-            logging.getLogger("LiteLLM").setLevel(logging.WARNING)
-            logger.info(f"LiteLLM SDK initialized for model: {self.model}")
-        except ImportError as e:
-            raise RuntimeError("LiteLLM SDK not installed. Run: uv add litellm or pip install litellm") from e
-
-    async def verify_connection(self) -> None:
-        try:
-            test_messages = [{"role": "user", "content": "test"}]
-            await self.call(
-                messages=test_messages,
-                max_completion_tokens=50,
-                temperature=0.0,
-                scope="verification",
-                max_retries=0,
-            )
-            logger.info("LiteLLM connection verified successfully")
-        except OutputTooLongError:
-            # Truncation is fine for verification — it means the connection works
-            logger.info("LiteLLM connection verified successfully (response truncated)")
-        except Exception as e:
-            logger.error(f"LiteLLM connection verification failed: {e}")
-            raise RuntimeError(f"Failed to verify LiteLLM connection: {e}") from e
-
-    def _build_common_kwargs(
-        self,
-        messages: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-    ) -> dict[str, Any]:
-        """Build common kwargs for litellm calls."""
-        kwargs: dict[str, Any] = {
-            "model": self.model,
-            "messages": messages,
-            "timeout": self.timeout,
-        }
-
-        if self.api_key:
-            kwargs["api_key"] = self.api_key
-        if self.base_url:
-            kwargs["api_base"] = self.base_url
-        if max_completion_tokens is not None:
-            kwargs["max_completion_tokens"] = max_completion_tokens
-        if temperature is not None:
-            kwargs["temperature"] = temperature
-
-        return kwargs
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        start_time = time.time()
-
-        call_kwargs = self._build_common_kwargs(messages, max_completion_tokens, temperature)
-
-        # Add JSON schema response format if provided
-        if response_format is not None and hasattr(response_format, "model_json_schema"):
-            schema = response_format.model_json_schema()
-            call_kwargs["response_format"] = {
-                "type": "json_schema",
-                "json_schema": {
-                    "name": response_format.__name__ if hasattr(response_format, "__name__") else "response",
-                    "schema": schema,
-                    "strict": strict_schema,
-                },
-            }
-
-        last_exception = None
-
-        for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.litellm.{scope}.attempt={attempt + 1}/{max_retries + 1}")
-            try:
-                response = await self._litellm.acompletion(**call_kwargs)
-
-                content = response.choices[0].message.content or ""
-                finish_reason = response.choices[0].finish_reason
-
-                # Check for length-limited output
-                if finish_reason == "length":
-                    raise OutputTooLongError("LiteLLM response was truncated due to token limit")
-
-                if response_format is not None:
-                    # Strip markdown code fences if present
-                    clean_content = content
-                    if "```json" in content:
-                        clean_content = content.split("```json")[1].split("```")[0].strip()
-                    elif "```" in content:
-                        clean_content = content.split("```")[1].split("```")[0].strip()
-
-                    try:
-                        json_data = json.loads(clean_content)
-                    except json.JSONDecodeError:
-                        json_data = json.loads(content)
-
-                    if skip_validation:
-                        result = json_data
-                    else:
-                        result = response_format.model_validate(json_data)
-                else:
-                    result = content
-
-                # Extract usage
-                input_tokens = getattr(response.usage, "prompt_tokens", 0) or 0
-                output_tokens = getattr(response.usage, "completion_tokens", 0) or 0
-                total_tokens = input_tokens + output_tokens
-
-                # Record metrics
-                duration = time.time() - start_time
-                metrics = get_metrics_collector()
-                metrics.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    duration=duration,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    success=True,
-                )
-
-                # Record trace span
-                from hindsight_api.tracing import _serialize_for_span, get_span_recorder
-
-                span_recorder = get_span_recorder()
-                span_recorder.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    messages=messages,
-                    response_content=_serialize_for_span(result),
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    duration=duration,
-                    finish_reason=finish_reason,
-                    error=None,
-                )
-
-                if duration > 10.0:
-                    logger.info(
-                        f"slow llm call: scope={scope}, model={self.provider}/{self.model}, "
-                        f"input_tokens={input_tokens}, output_tokens={output_tokens}, "
-                        f"time={duration:.3f}s"
-                    )
-
-                if return_usage:
-                    token_usage = TokenUsage(
-                        input_tokens=input_tokens,
-                        output_tokens=output_tokens,
-                        total_tokens=total_tokens,
-                    )
-                    return result, token_usage
-                return result
-
-            except OutputTooLongError:
-                raise
-
-            except json.JSONDecodeError as e:
-                last_exception = e
-                if attempt < max_retries:
-                    logger.warning("LiteLLM returned invalid JSON, retrying...")
-                    backoff = min(initial_backoff * (2**attempt), max_backoff)
-                    await asyncio.sleep(backoff)
-                    continue
-                else:
-                    logger.error(f"LiteLLM returned invalid JSON after {max_retries + 1} attempts")
-                    raise
-
-            except Exception as e:
-                error_str = str(e).lower()
-                # Fast fail on auth errors
-                if "401" in error_str or "403" in error_str or "unauthorized" in error_str:
-                    logger.error(f"LiteLLM auth error, not retrying: {e}")
-                    raise
-
-                last_exception = e
-                if attempt < max_retries:
-                    # Retry on rate limits, connection errors, server errors
-                    is_retryable = any(
-                        keyword in error_str
-                        for keyword in ("rate", "limit", "timeout", "connection", "500", "502", "503", "529")
-                    )
-                    if is_retryable:
-                        backoff = min(initial_backoff * (2**attempt), max_backoff)
-                        jitter = backoff * 0.2 * (2 * (time.time() % 1) - 1)
-                        await asyncio.sleep(backoff + jitter)
-                        continue
-
-                logger.error(f"LiteLLM API error after {attempt + 1} attempts: {e}")
-                raise
-
-        if last_exception:
-            raise last_exception
-        raise RuntimeError("LiteLLM call failed after all retries")
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        start_time = time.time()
-
-        call_kwargs = self._build_common_kwargs(messages, max_completion_tokens, temperature)
-        call_kwargs["tools"] = tools
-        call_kwargs["tool_choice"] = tool_choice
-
-        last_exception = None
-        for attempt in range(max_retries + 1):
-            if attempt > 0:
-                set_stage(f"llm.litellm.tools.attempt={attempt + 1}/{max_retries + 1}")
-            try:
-                response = await self._litellm.acompletion(**call_kwargs)
-
-                message = response.choices[0].message
-                content = message.content
-                finish_reason = response.choices[0].finish_reason
-
-                # Extract tool calls
-                tool_calls: list[LLMToolCall] = []
-                if message.tool_calls:
-                    for tc in message.tool_calls:
-                        arguments = tc.function.arguments
-                        if isinstance(arguments, str):
-                            arguments = json.loads(arguments)
-                        tool_calls.append(
-                            LLMToolCall(
-                                id=tc.id,
-                                name=tc.function.name,
-                                arguments=arguments,
-                            )
-                        )
-
-                # Extract usage
-                input_tokens = getattr(response.usage, "prompt_tokens", 0) or 0
-                output_tokens = getattr(response.usage, "completion_tokens", 0) or 0
-
-                # Record metrics
-                duration = time.time() - start_time
-                metrics = get_metrics_collector()
-                metrics.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    duration=duration,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    success=True,
-                )
-
-                # Record trace span
-                from hindsight_api.tracing import get_span_recorder
-
-                span_recorder = get_span_recorder()
-                tool_calls_dict = (
-                    [{"id": tc.id, "name": tc.name, "arguments": tc.arguments} for tc in tool_calls]
-                    if tool_calls
-                    else None
-                )
-                span_recorder.record_llm_call(
-                    provider=self.provider,
-                    model=self.model,
-                    scope=scope,
-                    messages=messages,
-                    response_content=content,
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                    duration=duration,
-                    finish_reason=finish_reason,
-                    error=None,
-                    tool_calls=tool_calls_dict,
-                )
-
-                return LLMToolCallResult(
-                    content=content,
-                    tool_calls=tool_calls,
-                    finish_reason=finish_reason or ("tool_calls" if tool_calls else "stop"),
-                    input_tokens=input_tokens,
-                    output_tokens=output_tokens,
-                )
-
-            except Exception as e:
-                error_str = str(e).lower()
-                if "401" in error_str or "403" in error_str or "unauthorized" in error_str:
-                    raise
-
-                last_exception = e
-                if attempt < max_retries:
-                    is_retryable = any(
-                        keyword in error_str
-                        for keyword in ("rate", "limit", "timeout", "connection", "500", "502", "503", "529")
-                    )
-                    if is_retryable:
-                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
-                        continue
-
-                logger.error(f"LiteLLM tool call error after {attempt + 1} attempts: {e}")
-                raise
-
-        if last_exception:
-            raise last_exception
-        raise RuntimeError("LiteLLM tool call failed after all retries")
-
-    async def cleanup(self) -> None:
-        """Clean up resources."""
-        pass

hindsight-api-slim/hindsight_api/engine/providers/llamacpp_llm.py

@@ -1,428 +0,0 @@
-"""
-Built-in llama.cpp LLM provider for fully offline operation.
-
-Manages a llama-cpp-python server as a subprocess, downloads GGUF models
-from HuggingFace on first use, and delegates inference to the OpenAI-compatible API.
-
-Usage:
-    HINDSIGHT_API_LLM_PROVIDER=llamacpp
-    HINDSIGHT_API_LLAMACPP_MODEL_PATH=~/.hindsight/models/gemma-4-E2B-it-Q4_K_M.gguf
-    HINDSIGHT_API_LLAMACPP_GPU_LAYERS=-1  # -1 = all layers on GPU
-    HINDSIGHT_API_LLAMACPP_CONTEXT_SIZE=8192
-"""
-
-import asyncio
-import logging
-import os
-import signal
-import socket
-import subprocess
-import sys
-import time
-from pathlib import Path
-from typing import Any
-
-from hindsight_api.engine.llm_interface import LLMInterface
-from hindsight_api.engine.response_models import LLMToolCallResult
-
-logger = logging.getLogger(__name__)
-
-# Default GGUF model for offline mode
-DEFAULT_LLAMACPP_HF_REPO = "bartowski/google_gemma-4-E2B-it-GGUF"
-DEFAULT_LLAMACPP_HF_FILENAME = "google_gemma-4-E2B-it-Q4_K_M.gguf"
-DEFAULT_LLAMACPP_MODEL_ALIAS = "gemma-4-e2b-it"
-
-MODELS_DIR = Path.home() / ".hindsight" / "models"
-
-# Singleton server instance — shared across all LlamaCppLLM instances
-# (retain, reflect, consolidation each create their own LLMProvider,
-# but they should all share one llama.cpp server process)
-_shared_server: "LlamaCppServer | None" = None
-_shared_server_lock = asyncio.Lock()
-
-
-def _find_free_port() -> int:
-    """Find a free TCP port on localhost."""
-    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-        s.bind(("127.0.0.1", 0))
-        return s.getsockname()[1]
-
-
-def _download_default_model() -> Path:
-    """Download the default GGUF model from HuggingFace if not already cached.
-
-    Returns:
-        Path to the downloaded GGUF file.
-    """
-    try:
-        from huggingface_hub import hf_hub_download
-    except ImportError:
-        raise ImportError(
-            "huggingface-hub is required for automatic model download. "
-            "Install with: pip install 'hindsight-api-slim[local-llm]'"
-        )
-
-    MODELS_DIR.mkdir(parents=True, exist_ok=True)
-    target = MODELS_DIR / DEFAULT_LLAMACPP_HF_FILENAME
-
-    if target.exists():
-        logger.info(f"Using cached model: {target}")
-        return target
-
-    logger.info(
-        f"Downloading {DEFAULT_LLAMACPP_HF_FILENAME} from {DEFAULT_LLAMACPP_HF_REPO} (~3.5 GB, first run only)..."
-    )
-
-    downloaded = hf_hub_download(
-        repo_id=DEFAULT_LLAMACPP_HF_REPO,
-        filename=DEFAULT_LLAMACPP_HF_FILENAME,
-        local_dir=str(MODELS_DIR),
-    )
-
-    logger.info(f"Model downloaded: {downloaded}")
-    return Path(downloaded)
-
-
-def _resolve_model_path(model_path: str | None) -> Path:
-    """Resolve the model path, downloading the default if needed.
-
-    Args:
-        model_path: Explicit path to a GGUF file, or None to use the default.
-
-    Returns:
-        Resolved Path to the GGUF file.
-    """
-    if model_path:
-        p = Path(model_path).expanduser()
-        if not p.exists():
-            raise FileNotFoundError(
-                f"GGUF model not found: {p}\n"
-                f"Set HINDSIGHT_API_LLAMACPP_MODEL_PATH to a valid .gguf file, "
-                f"or remove the setting to auto-download the default model."
-            )
-        return p
-
-    return _download_default_model()
-
-
-class LlamaCppServer:
-    """Manages a llama-cpp-python OpenAI-compatible server as a subprocess."""
-
-    def __init__(
-        self,
-        model_path: Path,
-        port: int,
-        gpu_layers: int = -1,
-        context_size: int = 8192,
-        chat_format: str | None = None,
-        extra_args: str | None = None,
-    ):
-        self.model_path = model_path
-        self.port = port
-        self.gpu_layers = gpu_layers
-        self.context_size = context_size
-        self.chat_format = chat_format
-        self.extra_args = extra_args
-        self._process: subprocess.Popen | None = None
-
-    @property
-    def base_url(self) -> str:
-        return f"http://127.0.0.1:{self.port}/v1"
-
-    async def start(self) -> None:
-        """Start the llama.cpp server subprocess."""
-        cmd = [
-            sys.executable,
-            "-m",
-            "llama_cpp.server",
-            "--model",
-            str(self.model_path),
-            "--host",
-            "127.0.0.1",
-            "--port",
-            str(self.port),
-            "--n_gpu_layers",
-            str(self.gpu_layers),
-            "--n_ctx",
-            str(self.context_size),
-            "--flash_attn",
-            "true",
-            "--n_batch",
-            "2048",
-            # Prompt cache: reuse KV cache for repeated system prompts
-            "--cache",
-            "true",
-        ]
-        # Only pass chat_format if explicitly set (most GGUF models have it embedded)
-        if self.chat_format:
-            cmd.extend(["--chat_format", self.chat_format])
-        # User-provided extra args (e.g. "--type_k 1 --type_v 1 --n_threads 8")
-        if self.extra_args:
-            cmd.extend(self.extra_args.split())
-
-        logger.info(f"Starting llama.cpp server: {' '.join(cmd)}")
-
-        # Write stderr to a log file to avoid pipe buffer deadlock
-        # (llama.cpp outputs a lot of model metadata on stderr during loading)
-        self._log_path = MODELS_DIR / "llamacpp_server.log"
-        self._log_file = open(self._log_path, "w")
-
-        self._process = subprocess.Popen(
-            cmd,
-            stdout=subprocess.DEVNULL,
-            stderr=self._log_file,
-            # Ensure the subprocess is killed when the parent exits
-            preexec_fn=os.setsid if hasattr(os, "setsid") else None,
-        )
-
-        # Wait for the server to be ready
-        await self._wait_for_ready()
-
-    async def _wait_for_ready(self, timeout: float = 120.0) -> None:
-        """Wait for the llama.cpp server to accept connections."""
-        import httpx
-
-        start = time.monotonic()
-        url = f"http://127.0.0.1:{self.port}/v1/models"
-        last_log = start
-
-        while time.monotonic() - start < timeout:
-            # Check if process died
-            if self._process and self._process.poll() is not None:
-                stderr = ""
-                try:
-                    stderr = self._log_path.read_text()[-2000:]
-                except Exception:
-                    pass
-                raise RuntimeError(f"llama.cpp server exited with code {self._process.returncode}.\nstderr: {stderr}")
-
-            try:
-                async with httpx.AsyncClient() as client:
-                    resp = await client.get(url, timeout=5.0)
-                    if resp.status_code == 200:
-                        logger.info(f"llama.cpp server ready on port {self.port}")
-                        return
-            except (httpx.ConnectError, httpx.TimeoutException, httpx.ConnectTimeout):
-                pass
-
-            # Log progress every 15s
-            now = time.monotonic()
-            if now - last_log > 15:
-                elapsed = int(now - start)
-                logger.info(f"Waiting for llama.cpp server to load model... ({elapsed}s)")
-                last_log = now
-
-            await asyncio.sleep(1.0)
-
-        # Timeout — read the log to help debug
-        stderr = ""
-        try:
-            stderr = self._log_path.read_text()[-2000:]
-        except Exception:
-            pass
-        raise TimeoutError(
-            f"llama.cpp server did not become ready within {timeout}s.\n"
-            f"Check model compatibility and available memory.\n"
-            f"Server log: {stderr}"
-        )
-
-    async def stop(self) -> None:
-        """Stop the llama.cpp server subprocess."""
-        if self._process is None:
-            return
-
-        logger.info("Stopping llama.cpp server...")
-        try:
-            # Send SIGTERM to the process group
-            if hasattr(os, "killpg"):
-                os.killpg(os.getpgid(self._process.pid), signal.SIGTERM)
-            else:
-                self._process.terminate()
-
-            # Wait up to 10s for graceful shutdown
-            try:
-                self._process.wait(timeout=10)
-            except subprocess.TimeoutExpired:
-                if hasattr(os, "killpg"):
-                    os.killpg(os.getpgid(self._process.pid), signal.SIGKILL)
-                else:
-                    self._process.kill()
-                self._process.wait(timeout=5)
-        except (ProcessLookupError, OSError):
-            pass  # Process already exited
-        finally:
-            self._process = None
-            if hasattr(self, "_log_file") and self._log_file:
-                self._log_file.close()
-                self._log_file = None
-            logger.info("llama.cpp server stopped")
-
-
-class LlamaCppLLM(LLMInterface):
-    """
-    Built-in llama.cpp provider.
-
-    Manages a llama-cpp-python server subprocess and delegates to OpenAICompatibleLLM
-    for actual inference calls. Handles model downloading and server lifecycle.
-    """
-
-    def __init__(
-        self,
-        provider: str,
-        api_key: str,
-        base_url: str,
-        model: str,
-        reasoning_effort: str = "low",
-        model_path: str | None = None,
-        gpu_layers: int = -1,
-        context_size: int = 8192,
-        chat_format: str | None = None,
-        no_grammar: bool = False,
-        extra_args: str | None = None,
-        **kwargs: Any,
-    ):
-        super().__init__(
-            provider=provider,
-            api_key=api_key or "llamacpp",
-            base_url=base_url or "",
-            model=model or DEFAULT_LLAMACPP_MODEL_ALIAS,
-            reasoning_effort=reasoning_effort,
-        )
-        self._model_path_str = model_path
-        self._gpu_layers = gpu_layers
-        self._context_size = context_size
-        self._chat_format = chat_format
-        self._no_grammar = no_grammar
-        self._extra_args = extra_args
-        self._server: LlamaCppServer | None = None
-        self._delegate: Any = None  # OpenAICompatibleLLM, created after server starts
-        self._initialized = False
-
-    async def _ensure_initialized(self) -> None:
-        """Lazy initialization: download model + start shared server on first use."""
-        if self._initialized:
-            return
-
-        global _shared_server
-
-        from .openai_compatible_llm import OpenAICompatibleLLM
-
-        async with _shared_server_lock:
-            if _shared_server is None:
-                # Resolve and potentially download the model
-                model_path = _resolve_model_path(self._model_path_str)
-                logger.info(f"Using GGUF model: {model_path}")
-
-                # Start the shared llama.cpp server
-                port = _find_free_port()
-                _shared_server = LlamaCppServer(
-                    model_path=model_path,
-                    port=port,
-                    gpu_layers=self._gpu_layers,
-                    context_size=self._context_size,
-                    chat_format=self._chat_format,
-                    extra_args=self._extra_args,
-                )
-                await _shared_server.start()
-
-        self._server = _shared_server
-
-        # Create the delegate that talks to the shared server's OpenAI-compatible API
-        if self._no_grammar:
-            logger.info("Grammar enforcement disabled (HINDSIGHT_API_LLAMACPP_NO_GRAMMAR=true)")
-        self._delegate = OpenAICompatibleLLM(
-            provider="llamacpp",
-            api_key="llamacpp",
-            base_url=self._server.base_url,
-            model=self.model,
-            reasoning_effort=self.reasoning_effort,
-        )
-
-        self._initialized = True
-
-    async def verify_connection(self) -> None:
-        """Verify the llama.cpp server is running and can generate text."""
-        await self._ensure_initialized()
-        # Make a simple test call to verify the model can actually generate
-        await self._delegate.call(
-            messages=[{"role": "user", "content": "Say 'ok'"}],
-            max_completion_tokens=10,
-            max_retries=2,
-            initial_backoff=0.5,
-            max_backoff=2.0,
-            scope="verification",
-        )
-        logger.info("llama.cpp LLM verification passed")
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        """Delegate call to the OpenAI-compatible API."""
-        await self._ensure_initialized()
-        return await self._delegate.call(
-            messages=messages,
-            response_format=response_format,
-            max_completion_tokens=max_completion_tokens,
-            temperature=temperature,
-            scope=scope,
-            max_retries=max_retries,
-            initial_backoff=initial_backoff,
-            max_backoff=max_backoff,
-            skip_validation=skip_validation,
-            strict_schema=strict_schema,
-            return_usage=return_usage,
-        )
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        """Delegate tool calls to the OpenAI-compatible API."""
-        await self._ensure_initialized()
-        return await self._delegate.call_with_tools(
-            messages=messages,
-            tools=tools,
-            max_completion_tokens=max_completion_tokens,
-            temperature=temperature,
-            scope=scope,
-            max_retries=max_retries,
-            initial_backoff=initial_backoff,
-            max_backoff=max_backoff,
-            tool_choice=tool_choice,
-        )
-
-    async def cleanup(self) -> None:
-        """Stop the shared llama.cpp server."""
-        global _shared_server
-
-        if self._delegate:
-            await self._delegate.cleanup()
-            self._delegate = None
-
-        # Stop the shared server (only the first cleanup call actually stops it)
-        async with _shared_server_lock:
-            if _shared_server is not None:
-                await _shared_server.stop()
-                _shared_server = None
-
-        self._server = None
-        self._initialized = False

hindsight-api-slim/hindsight_api/engine/providers/none_llm.py

@@ -1,78 +0,0 @@
-"""
-No-op LLM provider for chunk-only storage mode.
-
-When the LLM provider is set to "none", the system operates without any LLM dependency.
-Retain uses chunks mode (no fact extraction), and reflect/consolidation are disabled.
-This provider acts as a safety net — if any code path unexpectedly tries to call the LLM,
-it raises a clear error instead of a confusing connection failure.
-"""
-
-import logging
-from typing import Any
-
-from ..llm_interface import LLMInterface
-from ..response_models import LLMToolCallResult
-
-logger = logging.getLogger(__name__)
-
-
-class LLMNotAvailableError(Exception):
-    """Raised when an operation requires an LLM but the provider is set to 'none'."""
-
-    pass
-
-
-class NoneLLM(LLMInterface):
-    """
-    No-op LLM provider that rejects all LLM calls.
-
-    Used when HINDSIGHT_API_LLM_PROVIDER=none to run Hindsight as a chunk store
-    with semantic search but without LLM-based features (fact extraction, reflect,
-    consolidation).
-    """
-
-    async def verify_connection(self) -> None:
-        """No-op — no LLM connection to verify."""
-        logger.debug("NoneLLM: no LLM connection to verify (provider=none)")
-
-    async def call(
-        self,
-        messages: list[dict[str, str]],
-        response_format: Any | None = None,
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "memory",
-        max_retries: int = 10,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 60.0,
-        skip_validation: bool = False,
-        strict_schema: bool = False,
-        return_usage: bool = False,
-    ) -> Any:
-        """Raise LLMNotAvailableError — no LLM is configured."""
-        raise LLMNotAvailableError(
-            "LLM provider is set to 'none'. This operation requires an LLM. "
-            "Set HINDSIGHT_API_LLM_PROVIDER to a real provider (e.g., openai, anthropic, gemini)."
-        )
-
-    async def call_with_tools(
-        self,
-        messages: list[dict[str, Any]],
-        tools: list[dict[str, Any]],
-        max_completion_tokens: int | None = None,
-        temperature: float | None = None,
-        scope: str = "tools",
-        max_retries: int = 5,
-        initial_backoff: float = 1.0,
-        max_backoff: float = 30.0,
-        tool_choice: str | dict[str, Any] = "auto",
-    ) -> LLMToolCallResult:
-        """Raise LLMNotAvailableError — no LLM is configured."""
-        raise LLMNotAvailableError(
-            "LLM provider is set to 'none'. This operation requires an LLM. "
-            "Set HINDSIGHT_API_LLM_PROVIDER to a real provider (e.g., openai, anthropic, gemini)."
-        )
-
-    async def cleanup(self) -> None:
-        """No-op — nothing to clean up."""
-        pass

hindsight-api-slim/hindsight_api/engine/reflect/delta_ops.py

@@ -1,307 +0,0 @@
-"""Delta operations for structured mental models.
-
-The LLM's job during a delta refresh is to emit a list of these operations,
-each targeting an existing section (by id) or referencing a position relative
-to one.  ``apply_operations`` validates and applies each op in turn against a
-copy of the document; invalid ops (unknown ``section_id``, out-of-range
-``block_index``, malformed payloads) are dropped with a debug-friendly reason.
-
-Sections and blocks not mentioned by any op are physically copied through
-unchanged — there is no LLM-mediated re-emission of unchanged text, so prose
-drift is structurally impossible.
-
-Why operations and not "output the new structured doc":
-- "Output the new doc" still asks the LLM to *generate* every section's
-  blocks, including ones it didn't intend to modify, which gives it the same
-  opportunity to drift.
-- Operations make the no-change case mechanical: zero ops → identical doc.
-- Operations are auditable: each refresh produces a log of exactly what
-  changed, useful for debugging the LLM's behaviour and explaining diffs.
-
-Failure modes are by design conservative: an operation list that fails to
-parse against the Pydantic schema, or an LLM that returns invalid ops, results
-in zero changes — the document stays as-is. The structure can only get better
-or stay the same per refresh, never get worse.
-"""
-
-from __future__ import annotations
-
-import logging
-from typing import Annotated, Any, Literal, Union
-
-from pydantic import BaseModel, ConfigDict, Field
-
-from .structured_doc import (
-    Block,
-    Section,
-    StructuredDocument,
-    make_unique_id,
-    slugify_heading,
-)
-
-logger = logging.getLogger(__name__)
-
-
-# Op payloads ---------------------------------------------------------------
-
-
-class _OpBase(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-
-
-class AppendBlockOp(_OpBase):
-    """Add a new block at the end of an existing section."""
-
-    op: Literal["append_block"] = "append_block"
-    section_id: str
-    block: Block
-
-
-class InsertBlockOp(_OpBase):
-    """Insert a new block at ``index`` in an existing section.
-
-    ``index`` may equal ``len(section.blocks)`` (append) but not be greater.
-    """
-
-    op: Literal["insert_block"] = "insert_block"
-    section_id: str
-    index: int = Field(ge=0)
-    block: Block
-
-
-class ReplaceBlockOp(_OpBase):
-    """Replace the block at ``index`` of an existing section."""
-
-    op: Literal["replace_block"] = "replace_block"
-    section_id: str
-    index: int = Field(ge=0)
-    block: Block
-
-
-class RemoveBlockOp(_OpBase):
-    """Remove the block at ``index`` of an existing section."""
-
-    op: Literal["remove_block"] = "remove_block"
-    section_id: str
-    index: int = Field(ge=0)
-
-
-class AddSectionOp(_OpBase):
-    """Add a brand-new section.
-
-    ``after_section_id`` is optional; when omitted the new section is appended
-    at the end. ``new_id`` is optional; when omitted we slugify the heading
-    and disambiguate against existing IDs.
-    """
-
-    op: Literal["add_section"] = "add_section"
-    heading: str
-    level: int = Field(default=2, ge=1, le=6)
-    blocks: list[Block] = Field(default_factory=list)
-    after_section_id: str | None = None
-    new_id: str | None = None
-
-
-class RemoveSectionOp(_OpBase):
-    """Remove an entire section by id."""
-
-    op: Literal["remove_section"] = "remove_section"
-    section_id: str
-
-
-class ReplaceSectionBlocksOp(_OpBase):
-    """Replace all blocks of a section in one go.
-
-    Used when most of a section's contents are stale and rebuilding it as a
-    unit is clearer than emitting many block-level ops. The section's heading
-    and id are preserved.
-    """
-
-    op: Literal["replace_section_blocks"] = "replace_section_blocks"
-    section_id: str
-    blocks: list[Block] = Field(default_factory=list)
-
-
-class RenameSectionOp(_OpBase):
-    """Rename a section's heading. The id is unchanged so future ops still resolve."""
-
-    op: Literal["rename_section"] = "rename_section"
-    section_id: str
-    new_heading: str
-
-
-Operation = Annotated[
-    Union[
-        AppendBlockOp,
-        InsertBlockOp,
-        ReplaceBlockOp,
-        RemoveBlockOp,
-        AddSectionOp,
-        RemoveSectionOp,
-        ReplaceSectionBlocksOp,
-        RenameSectionOp,
-    ],
-    Field(discriminator="op"),
-]
-
-
-class DeltaOperationList(BaseModel):
-    """Container for the operations produced by an LLM delta call."""
-
-    model_config = ConfigDict(extra="forbid")
-    operations: list[Operation] = Field(default_factory=list)
-
-
-# Application ---------------------------------------------------------------
-
-
-class AppliedDelta(BaseModel):
-    """Outcome of applying a list of operations to a document."""
-
-    model_config = ConfigDict(extra="forbid")
-
-    document: StructuredDocument
-    applied: list[dict[str, Any]] = Field(default_factory=list)
-    skipped: list[dict[str, Any]] = Field(default_factory=list)
-
-    @property
-    def changed(self) -> bool:
-        return len(self.applied) > 0
-
-
-def _op_summary(op: Operation) -> dict[str, Any]:
-    """Compact dict suitable for the audit trail."""
-    data = op.model_dump()
-    return {k: v for k, v in data.items() if k != "block" and k != "blocks"} | {
-        "op": data["op"],
-    }
-
-
-def apply_operations(
-    doc: StructuredDocument,
-    operations: list[Operation],
-) -> AppliedDelta:
-    """Apply a list of operations to a document, returning a new document.
-
-    The original document is never mutated. Invalid operations (unknown
-    section, out-of-range index, name collision when adding a section) are
-    skipped and recorded in ``skipped`` with a ``reason`` string.
-    """
-    new_doc = doc.model_copy(deep=True)
-    applied: list[dict[str, Any]] = []
-    skipped: list[dict[str, Any]] = []
-
-    def skip(op: Operation, reason: str) -> None:
-        entry = _op_summary(op)
-        entry["reason"] = reason
-        skipped.append(entry)
-        logger.debug(f"[STRUCTURED_DELTA] skipping op {entry}")
-
-    for op in operations:
-        if isinstance(op, AppendBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.blocks.append(op.block)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, InsertBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index > len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} > {len(section.blocks)}",
-                )
-                continue
-            section.blocks.insert(op.index, op.block)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, ReplaceBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index >= len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} >= {len(section.blocks)}",
-                )
-                continue
-            section.blocks[op.index] = op.block
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, RemoveBlockOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            if op.index >= len(section.blocks):
-                skip(
-                    op,
-                    f"index out of range: {op.index} >= {len(section.blocks)}",
-                )
-                continue
-            section.blocks.pop(op.index)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, AddSectionOp):
-            existing_ids = {s.id for s in new_doc.sections}
-            base_id = op.new_id or slugify_heading(op.heading)
-            section_id = make_unique_id(base_id, existing_ids)
-            new_section = Section(
-                id=section_id,
-                heading=op.heading,
-                level=op.level,
-                blocks=list(op.blocks),
-            )
-            if op.after_section_id is None:
-                new_doc.sections.append(new_section)
-            else:
-                idx = new_doc.section_index(op.after_section_id)
-                if idx is None:
-                    skip(op, f"unknown after_section_id: {op.after_section_id}")
-                    continue
-                new_doc.sections.insert(idx + 1, new_section)
-            entry = _op_summary(op)
-            entry["assigned_id"] = section_id
-            applied.append(entry)
-            continue
-
-        if isinstance(op, RemoveSectionOp):
-            idx = new_doc.section_index(op.section_id)
-            if idx is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            new_doc.sections.pop(idx)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, ReplaceSectionBlocksOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.blocks = list(op.blocks)
-            applied.append(_op_summary(op))
-            continue
-
-        if isinstance(op, RenameSectionOp):
-            section = new_doc.section_by_id(op.section_id)
-            if section is None:
-                skip(op, f"unknown section_id: {op.section_id}")
-                continue
-            section.heading = op.new_heading
-            applied.append(_op_summary(op))
-            continue
-
-        skip(op, f"unhandled op type: {type(op).__name__}")  # pragma: no cover
-
-    return AppliedDelta(document=new_doc, applied=applied, skipped=skipped)

hindsight-api-slim/hindsight_api/engine/reflect/structured_doc.py

@@ -1,301 +0,0 @@
-"""Structured representation of a mental model document.
-
-Why this exists
----------------
-Storing mental models as raw markdown forces every refresh to round-trip prose
-through an LLM, which then drifts on stylistic details (numbered vs bulleted
-lists, casing, separator lines, paraphrasing) even when instructed to preserve
-content byte-for-byte. The intrinsic mechanism of an LLM is to *generate* the
-next token from a gestalt of the input — not to copy tokens verbatim — so any
-"preserve unchanged content" instruction is fundamentally a soft constraint.
-
-The fix is to give the LLM no opportunity to drift on unchanged content. We
-keep an authoritative structured representation of the document; the markdown
-shown to users is a deterministic render of that structure. Delta refreshes
-emit *operations* against the structure (see ``delta_ops.py``); sections and
-blocks not mentioned by any operation are physically untouched.
-
-Schema (v1)
------------
-A document is an ordered list of ``Section``s.  Each section has:
-- ``id``    : stable slug derived from ``heading`` (used as the operation
-              target across refreshes; surviving renames is a separate
-              concern handled by an explicit ``rename`` op).
-- ``heading``: the markdown heading text (without the ``#`` prefix).
-- ``level`` : 1 (``#``) … 6 (``######``).  Default 2.
-- ``blocks``: ordered list of typed blocks — paragraph, bullet_list,
-              ordered_list, code.
-
-The schema is intentionally narrow: it covers what real mental-model documents
-actually contain (the kind a coding agent writes for itself or a user writes as
-a "skill" doc).  Tables, images, and raw HTML are out of scope until needed.
-"""
-
-from __future__ import annotations
-
-import re
-from typing import Annotated, Literal, Union
-
-from pydantic import BaseModel, ConfigDict, Field
-
-# Blocks ---------------------------------------------------------------------
-
-
-class ParagraphBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["paragraph"] = "paragraph"
-    text: str
-
-
-class BulletListBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["bullet_list"] = "bullet_list"
-    items: list[str] = Field(default_factory=list)
-
-
-class OrderedListBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["ordered_list"] = "ordered_list"
-    items: list[str] = Field(default_factory=list)
-
-
-class CodeBlock(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    type: Literal["code"] = "code"
-    language: str = ""
-    text: str
-
-
-Block = Annotated[
-    Union[ParagraphBlock, BulletListBlock, OrderedListBlock, CodeBlock],
-    Field(discriminator="type"),
-]
-
-
-# Section / Document ---------------------------------------------------------
-
-
-class Section(BaseModel):
-    model_config = ConfigDict(extra="forbid")
-    id: str
-    heading: str
-    level: int = Field(default=2, ge=1, le=6)
-    blocks: list[Block] = Field(default_factory=list)
-
-
-class StructuredDocument(BaseModel):
-    """Top-level structured representation of a mental model."""
-
-    model_config = ConfigDict(extra="forbid")
-    version: Literal[1] = 1
-    sections: list[Section] = Field(default_factory=list)
-
-    def section_by_id(self, section_id: str) -> Section | None:
-        for s in self.sections:
-            if s.id == section_id:
-                return s
-        return None
-
-    def section_index(self, section_id: str) -> int | None:
-        for i, s in enumerate(self.sections):
-            if s.id == section_id:
-                return i
-        return None
-
-
-# Slug helpers ---------------------------------------------------------------
-
-_SLUG_RX = re.compile(r"[^a-z0-9]+")
-
-
-def slugify_heading(heading: str) -> str:
-    """Stable, deterministic slug from a heading.
-
-    "Stop Conditions" -> "stop-conditions"
-    "Inputs and Context" -> "inputs-and-context"
-    """
-    slug = _SLUG_RX.sub("-", heading.strip().lower()).strip("-")
-    return slug or "section"
-
-
-def make_unique_id(base: str, existing: set[str]) -> str:
-    """Disambiguate by appending -2, -3, … if the slug is already in use."""
-    if base not in existing:
-        return base
-    i = 2
-    while f"{base}-{i}" in existing:
-        i += 1
-    return f"{base}-{i}"
-
-
-# Renderer -------------------------------------------------------------------
-
-
-def render_block(block: Block) -> str:
-    """Render a single block to markdown. No trailing newline."""
-    if isinstance(block, ParagraphBlock):
-        return block.text.rstrip()
-    if isinstance(block, BulletListBlock):
-        return "\n".join(f"- {item.rstrip()}" for item in block.items)
-    if isinstance(block, OrderedListBlock):
-        return "\n".join(f"{i + 1}. {item.rstrip()}" for i, item in enumerate(block.items))
-    if isinstance(block, CodeBlock):
-        fence_lang = block.language or ""
-        return f"```{fence_lang}\n{block.text}\n```"
-    raise TypeError(f"Unknown block type: {type(block)!r}")
-
-
-def render_section(section: Section) -> str:
-    """Render a section: heading + blank line + blocks separated by blank lines."""
-    parts = ["#" * section.level + " " + section.heading.strip()]
-    for block in section.blocks:
-        parts.append("")  # blank line before each block
-        parts.append(render_block(block))
-    return "\n".join(parts)
-
-
-def render_document(doc: StructuredDocument) -> str:
-    """Render the whole document. Sections separated by a single blank line.
-
-    The output is byte-stable: same structured input always produces the same
-    markdown, modulo the inherent ordering of sections/blocks/items.
-    """
-    if not doc.sections:
-        return ""
-    return "\n\n".join(render_section(s) for s in doc.sections) + "\n"
-
-
-# Parser ---------------------------------------------------------------------
-#
-# The parser is intentionally lenient: it accepts the markdown produced by
-# our own renderer (round-trip-safe) and the markdown an LLM tends to produce
-# for mental-model documents.  It is *not* a general CommonMark parser — it
-# does not need to be.  When it cannot classify a block it falls back to a
-# paragraph so that no content is silently dropped.
-
-_HEADING_RX = re.compile(r"^(#{1,6})\s+(.+?)\s*$")
-_BULLET_RX = re.compile(r"^\s*[-*+]\s+(.*)$")
-_ORDERED_RX = re.compile(r"^\s*\d+[.)]\s+(.*)$")
-_FENCE_RX = re.compile(r"^```([A-Za-z0-9_+-]*)\s*$")
-
-
-def _strip_separators(lines: list[str]) -> list[str]:
-    """Drop horizontal-rule lines (`---`, `***`) used as section separators.
-
-    Our renderer never emits these, but LLM output frequently includes them
-    between sections; treating them as blank lines avoids parsing them as
-    paragraphs.
-    """
-    return ["" if re.fullmatch(r"\s*([-*_])\1{2,}\s*", line) else line for line in lines]
-
-
-def _split_blocks(lines: list[str]) -> list[list[str]]:
-    """Group consecutive non-blank lines into block chunks."""
-    chunks: list[list[str]] = []
-    current: list[str] = []
-    in_fence = False
-    for line in lines:
-        if _FENCE_RX.match(line):
-            current.append(line)
-            in_fence = not in_fence
-            continue
-        if in_fence:
-            current.append(line)
-            continue
-        if line.strip() == "":
-            if current:
-                chunks.append(current)
-                current = []
-        else:
-            current.append(line)
-    if current:
-        chunks.append(current)
-    return chunks
-
-
-def _parse_block(chunk: list[str]) -> Block:
-    """Parse a single non-empty chunk into a block."""
-    if chunk and _FENCE_RX.match(chunk[0]):
-        m = _FENCE_RX.match(chunk[0])
-        lang = m.group(1) if m else ""
-        body_lines = chunk[1:]
-        if body_lines and _FENCE_RX.match(body_lines[-1]):
-            body_lines = body_lines[:-1]
-        return CodeBlock(language=lang, text="\n".join(body_lines))
-
-    if all(_BULLET_RX.match(line) for line in chunk):
-        items = []
-        for line in chunk:
-            m = _BULLET_RX.match(line)
-            assert m is not None
-            items.append(m.group(1).strip())
-        return BulletListBlock(items=items)
-
-    if all(_ORDERED_RX.match(line) for line in chunk):
-        items = []
-        for line in chunk:
-            m = _ORDERED_RX.match(line)
-            assert m is not None
-            items.append(m.group(1).strip())
-        return OrderedListBlock(items=items)
-
-    return ParagraphBlock(text=" ".join(line.strip() for line in chunk).strip())
-
-
-def parse_markdown(markdown: str) -> StructuredDocument:
-    """Best-effort parse of a markdown document into the structured schema.
-
-    Sections are introduced by ATX headings (``#``..``######``).  Anything
-    before the first heading is wrapped into an implicit "Overview" section
-    so we never silently drop user content.  Section IDs are unique slugs of
-    their headings.
-    """
-    raw_lines = (markdown or "").splitlines()
-    lines = _strip_separators(raw_lines)
-
-    sections: list[Section] = []
-    used_ids: set[str] = set()
-    pending: list[str] = []
-    current: Section | None = None
-
-    def flush_pending_into(section: Section) -> None:
-        if not pending:
-            return
-        for chunk in _split_blocks(pending):
-            section.blocks.append(_parse_block(chunk))
-        pending.clear()
-
-    for line in lines:
-        m = _HEADING_RX.match(line)
-        if m:
-            if current is not None:
-                flush_pending_into(current)
-                sections.append(current)
-            elif pending:
-                # Content before the first heading: wrap in implicit section.
-                base = "overview"
-                section_id = make_unique_id(base, used_ids)
-                used_ids.add(section_id)
-                implicit = Section(id=section_id, heading="Overview", level=2)
-                flush_pending_into(implicit)
-                sections.append(implicit)
-            level = len(m.group(1))
-            heading = m.group(2).strip()
-            section_id = make_unique_id(slugify_heading(heading), used_ids)
-            used_ids.add(section_id)
-            current = Section(id=section_id, heading=heading, level=level)
-        else:
-            pending.append(line)
-
-    if current is not None:
-        flush_pending_into(current)
-        sections.append(current)
-    elif pending:
-        base = "overview"
-        section_id = make_unique_id(base, used_ids)
-        used_ids.add(section_id)
-        implicit = Section(id=section_id, heading="Overview", level=2)
-        flush_pending_into(implicit)
-        sections.append(implicit)
-
-    return StructuredDocument(sections=sections)

hindsight-api-slim/hindsight_api/engine/retain/chunk_storage.py

@@ -1,144 +0,0 @@
-"""
-Chunk storage for retain pipeline.
-
-Handles storage of document chunks in the database.
-"""
-
-import hashlib
-import logging
-from dataclasses import dataclass
-
-from ..memory_engine import fq_table
-from .types import ChunkMetadata
-
-logger = logging.getLogger(__name__)
-
-
-def compute_chunk_hash(chunk_text: str) -> str:
-    """Compute SHA256 hash of chunk text for delta comparison."""
-    return hashlib.sha256(chunk_text.encode()).hexdigest()
-
-
-@dataclass
-class ExistingChunk:
-    """Represents a chunk already stored in the database."""
-
-    chunk_id: str
-    chunk_index: int
-    content_hash: str | None
-
-
-async def load_existing_chunks(conn, bank_id: str, document_id: str) -> list[ExistingChunk]:
-    """
-    Load existing chunk metadata for a document.
-
-    Returns list of ExistingChunk with chunk_id, chunk_index, and content_hash.
-    """
-    rows = await conn.fetch(
-        f"""
-        SELECT chunk_id, chunk_index, content_hash
-        FROM {fq_table("chunks")}
-        WHERE document_id = $1 AND bank_id = $2
-        ORDER BY chunk_index
-        """,
-        document_id,
-        bank_id,
-    )
-    return [
-        ExistingChunk(
-            chunk_id=row["chunk_id"],
-            chunk_index=row["chunk_index"],
-            content_hash=row["content_hash"],
-        )
-        for row in rows
-    ]
-
-
-async def delete_chunks_by_ids(conn, chunk_ids: list[str]) -> None:
-    """
-    Delete specific chunks by their IDs.
-
-    This cascades to memory_units (via FK with CASCADE delete)
-    and their links.
-    """
-    if not chunk_ids:
-        return
-    await conn.execute(
-        f"DELETE FROM {fq_table('chunks')} WHERE chunk_id = ANY($1::text[])",
-        chunk_ids,
-    )
-
-
-async def store_chunks_batch(conn, bank_id: str, document_id: str, chunks: list[ChunkMetadata]) -> dict[int, str]:
-    """
-    Store document chunks in the database.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        document_id: Document identifier
-        chunks: List of ChunkMetadata objects
-
-    Returns:
-        Dictionary mapping global chunk index to chunk_id
-    """
-    if not chunks:
-        return {}
-
-    # Prepare chunk data for batch insert
-    chunk_ids = []
-    chunk_texts = []
-    chunk_indices = []
-    content_hashes = []
-    chunk_id_map = {}
-
-    for chunk in chunks:
-        chunk_id = f"{bank_id}_{document_id}_{chunk.chunk_index}"
-        chunk_ids.append(chunk_id)
-        chunk_texts.append(chunk.chunk_text)
-        chunk_indices.append(chunk.chunk_index)
-        content_hashes.append(compute_chunk_hash(chunk.chunk_text))
-        chunk_id_map[chunk.chunk_index] = chunk_id
-
-    # Batch upsert all chunks. ON CONFLICT makes this idempotent: re-submitting
-    # a retain under the same document_id (the pattern in vectorize-io/hindsight#977)
-    # may produce chunk_ids that already exist when upstream cascade-delete or
-    # delta-retain paths don't run (or race with a concurrent task). Overwriting
-    # is the correct behavior per the document_id grouping semantics — the caller
-    # intends this chunk to hold the latest content at that (document_id, index).
-    await conn.execute(
-        f"""
-        INSERT INTO {fq_table("chunks")} (chunk_id, document_id, bank_id, chunk_text, chunk_index, content_hash)
-        SELECT * FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::integer[], $6::text[])
-        ON CONFLICT (chunk_id) DO UPDATE SET
-            chunk_text = EXCLUDED.chunk_text,
-            chunk_index = EXCLUDED.chunk_index,
-            content_hash = EXCLUDED.content_hash
-        """,
-        chunk_ids,
-        [document_id] * len(chunk_texts),
-        [bank_id] * len(chunk_texts),
-        chunk_texts,
-        chunk_indices,
-        content_hashes,
-    )
-
-    return chunk_id_map
-
-
-def map_facts_to_chunks(facts_chunk_indices: list[int], chunk_id_map: dict[int, str]) -> list[str | None]:
-    """
-    Map fact chunk indices to chunk IDs.
-
-    Args:
-        facts_chunk_indices: List of chunk indices for each fact
-        chunk_id_map: Dictionary mapping chunk index to chunk_id
-
-    Returns:
-        List of chunk_ids (same length as facts_chunk_indices)
-    """
-    chunk_ids = []
-    for chunk_idx in facts_chunk_indices:
-        chunk_id = chunk_id_map.get(chunk_idx)
-        chunk_ids.append(chunk_id)
-    return chunk_ids

hindsight-api-slim/hindsight_api/engine/retain/entity_processing.py

@@ -1,162 +0,0 @@
-"""
-Entity processing for retain pipeline.
-
-Handles entity extraction, resolution, and link creation for stored facts.
-"""
-
-import logging
-
-from . import link_utils
-from .types import EntityLink, ProcessedFact
-
-logger = logging.getLogger(__name__)
-
-
-def _prepare_facts_for_entity_processing(
-    facts: list[ProcessedFact],
-    user_entities_per_content: dict[int, list[dict]] | None = None,
-) -> tuple[list[str], list, list[list[dict]]]:
-    """
-    Extract fact texts, dates, and merged entity lists from ProcessedFact objects.
-
-    Returns:
-        Tuple of (fact_texts, fact_dates, entities_per_fact)
-    """
-    user_entities_per_content = user_entities_per_content or {}
-
-    fact_texts = [fact.fact_text for fact in facts]
-    fact_dates = [fact.occurred_start if fact.occurred_start is not None else fact.mentioned_at for fact in facts]
-
-    entities_per_fact = []
-    for fact in facts:
-        llm_entities = [{"text": entity.name, "type": "CONCEPT"} for entity in (fact.entities or [])]
-
-        user_entities = user_entities_per_content.get(fact.content_index, [])
-
-        seen_texts = {e["text"].lower() for e in llm_entities}
-        for user_entity in user_entities:
-            if user_entity["text"].lower() not in seen_texts:
-                llm_entities.append(
-                    {
-                        "text": user_entity["text"],
-                        "type": user_entity.get("type", "CONCEPT"),
-                    }
-                )
-                seen_texts.add(user_entity["text"].lower())
-
-        entities_per_fact.append(llm_entities)
-
-    return fact_texts, fact_dates, entities_per_fact
-
-
-async def resolve_entities(
-    entity_resolver,
-    conn,
-    bank_id: str,
-    unit_ids: list[str],
-    facts: list[ProcessedFact],
-    log_buffer: list[str] = None,
-    user_entities_per_content: dict[int, list[dict]] = None,
-    entity_labels: list | None = None,
-) -> tuple[list[str], list[tuple], dict[str, list[str]]]:
-    """
-    Phase 1: Resolve entity names to canonical IDs (read-heavy).
-
-    Should be called on a SEPARATE connection OUTSIDE the main write transaction
-    to avoid holding the transaction open during expensive trigram scans.
-
-    Args:
-        entity_resolver: EntityResolver instance
-        conn: Database connection (separate from the main write transaction)
-        bank_id: Bank identifier
-        unit_ids: Placeholder unit IDs (used only for grouping)
-        facts: List of ProcessedFact objects
-        log_buffer: Optional buffer for detailed logging
-        user_entities_per_content: Dict mapping content_index to user-provided entities
-        entity_labels: Optional entity label taxonomy
-
-    Returns:
-        Tuple of (resolved_entity_ids, entity_to_unit, unit_to_entity_ids)
-        to pass to build_entity_links().
-    """
-    if not unit_ids or not facts:
-        return [], [], {}
-
-    if len(unit_ids) != len(facts):
-        raise ValueError(f"Mismatch between unit_ids ({len(unit_ids)}) and facts ({len(facts)})")
-
-    fact_texts, fact_dates, entities_per_fact = _prepare_facts_for_entity_processing(facts, user_entities_per_content)
-
-    return await link_utils.resolve_entities_only(
-        entity_resolver,
-        conn,
-        bank_id,
-        unit_ids,
-        fact_texts,
-        "",  # context (not used in current implementation)
-        fact_dates,
-        entities_per_fact,
-        log_buffer,
-        entity_labels=entity_labels,
-    )
-
-
-async def build_entity_links(
-    entity_resolver,
-    conn,
-    bank_id: str,
-    unit_ids: list[str],
-    resolved_entity_ids: list[str],
-    entity_to_unit: list[tuple],
-    unit_to_entity_ids: dict[str, list[str]],
-    log_buffer: list[str] = None,
-    skip_unit_entities_insert: bool = False,
-) -> list[EntityLink]:
-    """
-    Build entity links for UI graph visualization.
-
-    Queries unit_entities to find shared entities between new and existing units,
-    then generates EntityLink objects. When called from Phase 3 (post-transaction),
-    set skip_unit_entities_insert=True since unit_entities were already inserted
-    in Phase 2.
-
-    Args:
-        entity_resolver: EntityResolver instance
-        conn: Database connection
-        bank_id: Bank identifier
-        unit_ids: Actual unit IDs (must already be inserted in the DB)
-        resolved_entity_ids: From resolve_entities()
-        entity_to_unit: From resolve_entities()
-        unit_to_entity_ids: From resolve_entities()
-        log_buffer: Optional buffer for detailed logging
-        skip_unit_entities_insert: Skip unit_entities INSERT (already done in Phase 2)
-
-    Returns:
-        List of EntityLink objects for batch insertion
-    """
-    return await link_utils.build_entity_links_from_resolved(
-        entity_resolver,
-        conn,
-        bank_id,
-        unit_ids,
-        resolved_entity_ids,
-        entity_to_unit,
-        unit_to_entity_ids,
-        log_buffer,
-        skip_unit_entities_insert=skip_unit_entities_insert,
-    )
-
-
-async def insert_entity_links_batch(conn, entity_links: list[EntityLink], bank_id: str) -> None:
-    """
-    Insert entity links in batch.
-
-    Args:
-        conn: Database connection
-        entity_links: List of EntityLink objects
-        bank_id: Bank identifier (stored directly on memory_links for fast filtering)
-    """
-    if not entity_links:
-        return
-
-    await link_utils.insert_entity_links_batch(conn, entity_links, bank_id)

hindsight-api-slim/hindsight_api/engine/retain/fact_storage.py

@@ -1,479 +0,0 @@
-"""
-Fact storage for retain pipeline.
-
-Handles insertion of facts into the database.
-"""
-
-import json
-import logging
-import uuid
-from datetime import datetime
-
-from ...config import get_config
-from ..memory_engine import fq_table
-from .bank_utils import DEFAULT_DISPOSITION, create_bank_vector_indexes
-from .fact_extraction import _sanitize_text
-from .types import ProcessedFact
-
-logger = logging.getLogger(__name__)
-
-
-async def get_document_content(
-    conn,
-    bank_id: str,
-    document_id: str,
-) -> str | None:
-    """Fetch the original_text of an existing document.
-
-    Returns None if the document does not exist.
-    """
-    row = await conn.fetchval(
-        f"SELECT original_text FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2",
-        document_id,
-        bank_id,
-    )
-    return row
-
-
-async def insert_facts_batch(
-    conn, bank_id: str, facts: list[ProcessedFact], document_id: str | None = None
-) -> list[str]:
-    """
-    Insert facts into the database in batch.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        facts: List of ProcessedFact objects to insert
-        document_id: Optional document ID to associate with facts
-
-    Returns:
-        List of unit IDs (UUIDs as strings) for the inserted facts
-    """
-    if not facts:
-        return []
-
-    # Prepare data for batch insert
-    fact_texts = []
-    embeddings = []
-    event_dates = []
-    occurred_starts = []
-    occurred_ends = []
-    mentioned_ats = []
-    contexts = []
-    fact_types = []
-    metadata_jsons = []
-    chunk_ids = []
-    document_ids = []
-    tags_list = []
-    observation_scopes_list = []
-    text_signals_list = []
-
-    for fact in facts:
-        fact_texts.append(_sanitize_text(fact.fact_text))
-        # Convert embedding to string for asyncpg vector type
-        embeddings.append(str(fact.embedding))
-        # event_date: Use occurred_start if available, otherwise use mentioned_at
-        # This maintains backward compatibility while handling None occurred_start
-        event_dates.append(fact.occurred_start if fact.occurred_start is not None else fact.mentioned_at)
-        occurred_starts.append(fact.occurred_start)
-        occurred_ends.append(fact.occurred_end)
-        mentioned_ats.append(fact.mentioned_at)
-        contexts.append(_sanitize_text(fact.context))
-        fact_types.append(fact.fact_type)
-        metadata_jsons.append(json.dumps(fact.metadata))
-        chunk_ids.append(fact.chunk_id)
-        # Use per-fact document_id if available, otherwise fallback to batch-level document_id
-        document_ids.append(fact.document_id if fact.document_id else document_id)
-        # Convert tags to JSON string for proper batch insertion (PostgreSQL unnest doesn't handle 2D arrays well)
-        tags_list.append(json.dumps(fact.tags if fact.tags else []))
-        # observation_scopes: stored as JSONB (string or 2D array), None if not provided
-        observation_scopes_list.append(
-            json.dumps(fact.observation_scopes) if fact.observation_scopes is not None else None
-        )
-        # Build text_signals: entity names + date tokens for enriched BM25 indexing
-        signal_parts = []
-        if fact.entities:
-            signal_parts.extend(e.name for e in fact.entities)
-        if fact.occurred_start:
-            try:
-                signal_parts.append(fact.occurred_start.strftime("%B %d %Y").lstrip("0").replace(" 0", " "))
-            except (ValueError, AttributeError):
-                pass
-        if fact.occurred_end and fact.occurred_end != fact.occurred_start:
-            try:
-                signal_parts.append(fact.occurred_end.strftime("%B %d %Y").lstrip("0").replace(" 0", " "))
-            except (ValueError, AttributeError):
-                pass
-        text_signals_list.append(" ".join(signal_parts) if signal_parts else None)
-
-    # Batch insert all facts
-    # Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
-    # Query varies based on text search backend
-    config = get_config()
-    if config.text_search_extension == "vchord":
-        # VectorChord: manually tokenize and insert search_vector
-        # text_signals (entity names etc.) are included in the tokenize input for enriched BM25
-        query = f"""
-            WITH input_data AS (
-                SELECT * FROM unnest(
-                    $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-                    $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
-                ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                       context, fact_type, metadata, chunk_id, document_id, tags_json,
-                       observation_scopes_json, text_signals)
-            )
-            INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                     context, fact_type, metadata, chunk_id, document_id, tags,
-                                     observation_scopes, text_signals, search_vector)
-            SELECT
-                $1,
-                text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                context, fact_type, metadata, chunk_id, document_id,
-                COALESCE(
-                    (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
-                    '{{}}'::varchar[]
-                ),
-                observation_scopes_json,
-                text_signals,
-                tokenize(
-                    COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, ''),
-                    'llmlingua2'
-                )::bm25_catalog.bm25vector
-            FROM input_data
-            RETURNING id
-        """
-    else:  # native or pg_textsearch
-        # Native PostgreSQL: search_vector is GENERATED ALWAYS (expression includes text_signals), don't include it
-        # pg_textsearch: indexes operate on base columns directly, don't populate search_vector
-        query = f"""
-            WITH input_data AS (
-                SELECT * FROM unnest(
-                    $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-                    $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
-                ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                       context, fact_type, metadata, chunk_id, document_id, tags_json,
-                       observation_scopes_json, text_signals)
-            )
-            INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                     context, fact_type, metadata, chunk_id, document_id, tags,
-                                     observation_scopes, text_signals)
-            SELECT
-                $1,
-                text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                context, fact_type, metadata, chunk_id, document_id,
-                COALESCE(
-                    (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
-                    '{{}}'::varchar[]
-                ),
-                observation_scopes_json,
-                text_signals
-            FROM input_data
-            RETURNING id
-        """
-
-    results = await conn.fetch(
-        query,
-        bank_id,
-        fact_texts,
-        embeddings,
-        event_dates,  # event_date: occurred_start if available, else mentioned_at
-        occurred_starts,
-        occurred_ends,
-        mentioned_ats,
-        contexts,
-        fact_types,
-        metadata_jsons,
-        chunk_ids,
-        document_ids,
-        tags_list,
-        observation_scopes_list,
-        text_signals_list,
-    )
-
-    unit_ids = [str(row["id"]) for row in results]
-    return unit_ids
-
-
-async def ensure_bank_exists(conn, bank_id: str) -> None:
-    """
-    Ensure bank exists in the database.
-
-    Creates bank with default values if it doesn't exist.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-    """
-    # Generate internal_id here so we control the value and can use it
-    # immediately for HNSW index creation without a RETURNING round-trip.
-    internal_id = uuid.uuid4()
-    inserted = await conn.fetchval(
-        f"""
-        INSERT INTO {fq_table("banks")} (bank_id, disposition, mission, internal_id)
-        VALUES ($1, $2::jsonb, $3, $4)
-        ON CONFLICT (bank_id) DO NOTHING
-        RETURNING bank_id
-        """,
-        bank_id,
-        json.dumps(DEFAULT_DISPOSITION),
-        "",
-        internal_id,
-    )
-    if inserted:
-        # Fresh insert — create per-bank vector indexes
-        await create_bank_vector_indexes(conn, bank_id, str(internal_id))
-
-
-async def delete_stale_observations_for_memories(
-    conn,
-    bank_id: str,
-    fact_ids: "list[str | uuid.UUID]",
-) -> int:
-    """Delete observations whose source memories are about to be removed.
-
-    Mirrors the cleanup performed by ``MemoryEngine.delete_document`` so that
-    every code path that removes ``memory_units`` also removes the
-    observations derived from them. Without this, ingesting a fresh version
-    of a document via the retain pipeline (which does a full-replace
-    ``DELETE FROM documents`` cascade) used to leave orphan observations
-    pointing at memory IDs that no longer existed.
-
-    For each observation referencing any of ``fact_ids``:
-    1. Delete the observation row (its text is stale once even one source
-       memory disappears).
-    2. Reset ``consolidated_at = NULL`` on the surviving source memories so
-       they get re-consolidated under fresh observations on the next run.
-
-    Must be called within an active transaction, before the source memories
-    are deleted.
-
-    Returns the number of observations deleted.
-    """
-    if not fact_ids:
-        return 0
-
-    fact_uuids = [uuid.UUID(str(fid)) if not isinstance(fid, uuid.UUID) else fid for fid in fact_ids]
-
-    affected_obs = await conn.fetch(
-        f"""
-        SELECT id, source_memory_ids
-        FROM {fq_table("memory_units")}
-        WHERE bank_id = $1
-          AND fact_type = 'observation'
-          AND source_memory_ids && $2::uuid[]
-        """,
-        bank_id,
-        fact_uuids,
-    )
-
-    if not affected_obs:
-        return 0
-
-    deleted_set = {str(uid) for uid in fact_uuids}
-    obs_ids = [obs["id"] for obs in affected_obs]
-    seen_remaining: set[str] = set()
-    remaining_source_ids: list[uuid.UUID] = []
-    for obs in affected_obs:
-        for src_id in obs["source_memory_ids"] or []:
-            src_str = str(src_id)
-            if src_str not in deleted_set and src_str not in seen_remaining:
-                remaining_source_ids.append(src_id)
-                seen_remaining.add(src_str)
-
-    await conn.execute(
-        f"DELETE FROM {fq_table('memory_units')} WHERE id = ANY($1::uuid[])",
-        obs_ids,
-    )
-
-    if remaining_source_ids:
-        await conn.execute(
-            f"""
-            UPDATE {fq_table("memory_units")}
-            SET consolidated_at = NULL
-            WHERE id = ANY($1::uuid[])
-              AND fact_type IN ('experience', 'world')
-            """,
-            remaining_source_ids,
-        )
-
-    logger.info(
-        f"[OBSERVATIONS] Deleted {len(obs_ids)} observations, reset {len(remaining_source_ids)} "
-        f"source memories for re-consolidation in bank {bank_id}"
-    )
-    return len(obs_ids)
-
-
-async def handle_document_tracking(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    is_first_batch: bool,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-) -> None:
-    """
-    Handle document tracking in the database (full-replace mode).
-
-    Deletes the existing document (cascading to all units and links) on the
-    first batch, then inserts the new document record.
-
-    Args:
-        conn: Database connection
-        bank_id: Bank identifier
-        document_id: Document identifier
-        combined_content: Combined content text from all content items
-        is_first_batch: Whether this is the first batch (for chunked operations)
-        retain_params: Optional parameters passed during retain (context, event_date, etc.)
-        document_tags: Optional list of tags to associate with the document
-    """
-    import hashlib
-
-    # Sanitize and calculate content hash
-    combined_content = _sanitize_text(combined_content) or ""
-    content_hash = hashlib.sha256(combined_content.encode()).hexdigest()
-
-    # Delete old document first (cascades to units and links).
-    # Only delete on the first batch to avoid deleting data we just inserted.
-    # Before the cascade, fan out to delete observations derived from the
-    # outgoing memory_units — otherwise the FK ON DELETE CASCADE removes the
-    # source memory_units but leaves observation rows pointing at IDs that
-    # no longer exist (consolidated_at on co-source memories also stays
-    # frozen). Same cleanup the explicit ``delete_document`` API performs.
-    preserved_created_at = None
-    if is_first_batch:
-        existing_unit_rows = await conn.fetch(
-            f"""
-            SELECT id FROM {fq_table("memory_units")}
-            WHERE document_id = $1 AND fact_type IN ('experience', 'world')
-            """,
-            document_id,
-        )
-        existing_unit_ids = [row["id"] for row in existing_unit_rows]
-        if existing_unit_ids:
-            invalidated = await delete_stale_observations_for_memories(conn, bank_id, existing_unit_ids)
-            if invalidated:
-                logger.info(
-                    f"[RETAIN] Document {document_id} re-ingested: invalidated "
-                    f"{invalidated} observation(s) derived from {len(existing_unit_ids)} outgoing memory_units"
-                )
-        # Explicitly delete memory_units by document_id BEFORE deleting the
-        # document row. The CASCADE from documents→chunks→memory_units only
-        # catches units that have a non-NULL chunk_id FK. Units with chunk_id=NULL
-        # (e.g. from partial writes or edge cases) would survive the cascade.
-        # This explicit delete ensures complete cleanup.
-        await conn.execute(
-            f"DELETE FROM {fq_table('memory_units')} WHERE document_id = $1 AND bank_id = $2",
-            document_id,
-            bank_id,
-        )
-        # Capture created_at before deletion so re-ingestion preserves it.
-        preserved_created_at = await conn.fetchval(
-            f"DELETE FROM {fq_table('documents')} WHERE id = $1 AND bank_id = $2 RETURNING created_at",
-            document_id,
-            bank_id,
-        )
-
-    # Insert document (or update if exists from concurrent operations)
-    await _upsert_document_row(
-        conn,
-        bank_id,
-        document_id,
-        combined_content,
-        content_hash,
-        retain_params,
-        document_tags,
-        preserved_created_at=preserved_created_at,
-    )
-
-
-async def upsert_document_metadata(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-) -> None:
-    """
-    Update document metadata without deleting existing facts/chunks.
-
-    Used by delta retain: the document row is upserted but chunks and
-    memory_units are managed separately at the chunk level.
-    """
-    import hashlib
-
-    combined_content = _sanitize_text(combined_content) or ""
-    content_hash = hashlib.sha256(combined_content.encode()).hexdigest()
-
-    await _upsert_document_row(conn, bank_id, document_id, combined_content, content_hash, retain_params, document_tags)
-
-
-async def _upsert_document_row(
-    conn,
-    bank_id: str,
-    document_id: str,
-    combined_content: str,
-    content_hash: str,
-    retain_params: dict | None = None,
-    document_tags: list[str] | None = None,
-    preserved_created_at: datetime | None = None,
-) -> None:
-    """Insert or update a document row.
-
-    When ``preserved_created_at`` is provided, it is used for ``created_at`` on
-    INSERT so that re-ingesting a document (which deletes + inserts the row)
-    keeps the original creation timestamp. ``updated_at`` is always set to
-    ``NOW()`` on both INSERT and the ON CONFLICT UPDATE branch.
-    """
-    await conn.execute(
-        f"""
-        INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, retain_params, tags, created_at, updated_at)
-        VALUES ($1, $2, $3, $4, $5, $6, COALESCE($7, NOW()), NOW())
-        ON CONFLICT (id, bank_id) DO UPDATE
-        SET original_text = EXCLUDED.original_text,
-            content_hash = EXCLUDED.content_hash,
-            retain_params = EXCLUDED.retain_params,
-            tags = EXCLUDED.tags,
-            updated_at = NOW()
-        """,
-        document_id,
-        bank_id,
-        combined_content,
-        content_hash,
-        json.dumps(retain_params) if retain_params else None,
-        document_tags or [],
-        preserved_created_at,
-    )
-
-
-async def update_memory_units_tags(
-    conn,
-    bank_id: str,
-    document_id: str,
-    tags: list[str],
-) -> int:
-    """
-    Update tags on all memory_units belonging to a document.
-
-    Used during delta retain to propagate tag changes to unchanged facts.
-
-    Returns:
-        Number of memory units updated.
-    """
-    result = await conn.execute(
-        f"""
-        UPDATE {fq_table("memory_units")}
-        SET tags = $3, updated_at = NOW()
-        WHERE bank_id = $1 AND document_id = $2
-        """,
-        bank_id,
-        document_id,
-        tags or [],
-    )
-    # result is a status string like "UPDATE 5"
-    try:
-        return int(result.split()[-1])
-    except (ValueError, IndexError):
-        return 0

hindsight-api-slim/hindsight_api/engine/search/graph_retrieval.py

@@ -1,70 +0,0 @@
-"""
-Graph retrieval strategies for memory recall.
-
-This module provides an abstraction for graph-based memory retrieval,
-allowing different algorithms to be swapped without changing the rest
-of the recall pipeline.
-"""
-
-import logging
-from abc import ABC, abstractmethod
-from datetime import datetime
-
-from .tags import TagGroup, TagsMatch
-from .types import GraphRetrievalTimings, RetrievalResult
-
-logger = logging.getLogger(__name__)
-
-
-class GraphRetriever(ABC):
-    """
-    Abstract base class for graph-based memory retrieval.
-
-    Implementations traverse the memory graph (entity links, temporal links,
-    causal links) to find relevant facts that might not be found by
-    semantic or keyword search alone.
-    """
-
-    @property
-    @abstractmethod
-    def name(self) -> str:
-        """Return identifier for this retrieval strategy (e.g., 'link_expansion')."""
-        pass
-
-    @abstractmethod
-    async def retrieve(
-        self,
-        pool,
-        query_embedding_str: str,
-        bank_id: str,
-        fact_type: str,
-        budget: int,
-        query_text: str | None = None,
-        semantic_seeds: list[RetrievalResult] | None = None,
-        temporal_seeds: list[RetrievalResult] | None = None,
-        adjacency=None,  # TypedAdjacency, optional pre-loaded graph
-        tags: list[str] | None = None,  # Visibility scope tags for filtering
-        tags_match: TagsMatch = "any",  # How to match tags: 'any' (OR) or 'all' (AND)
-        tag_groups: list[TagGroup] | None = None,  # Compound boolean tag filter groups
-        created_after: datetime | None = None,  # Only include memory_units created after this time
-        created_before: datetime | None = None,  # Only include memory_units created before this time
-    ) -> tuple[list[RetrievalResult], GraphRetrievalTimings | None]:
-        """
-        Retrieve relevant facts via graph traversal.
-
-        Args:
-            pool: Database connection pool
-            query_embedding_str: Query embedding as string (for finding entry points)
-            bank_id: Memory bank identifier
-            fact_type: Fact type to filter ('world', 'experience', 'observation')
-            budget: Maximum number of nodes to explore/return
-            query_text: Original query text (optional, for some strategies)
-            semantic_seeds: Pre-computed semantic entry points (from semantic retrieval)
-            temporal_seeds: Pre-computed temporal entry points (from temporal retrieval)
-            adjacency: Pre-loaded typed adjacency graph (optional)
-            tags: Optional list of tags for visibility filtering (OR matching)
-
-        Returns:
-            Tuple of (List of RetrievalResult with activation scores, optional timing info)
-        """
-        pass

hindsight-api-slim/hindsight_api/engine/search/reranking.py

@@ -1,249 +0,0 @@
-"""
-Cross-encoder neural reranking for search results.
-"""
-
-import math
-from datetime import datetime, timezone
-
-from .types import MergedCandidate, ScoredResult
-
-UTC = timezone.utc
-
-# Multiplicative boost alphas for recency and temporal proximity.
-# Each signal contributes at most ±(alpha/2) relative adjustment to the base CE score,
-# so the max combined boost is (1 + alpha/2)^2 ≈ +21% and min is (1 - alpha/2)^2 ≈ -19%.
-_RECENCY_ALPHA: float = 0.2
-_TEMPORAL_ALPHA: float = 0.2
-_PROOF_COUNT_ALPHA: float = 0.1  # Conservative: max ±5% for evidence strength
-
-
-def apply_combined_scoring(
-    scored_results: list[ScoredResult],
-    now: datetime,
-    recency_alpha: float = _RECENCY_ALPHA,
-    temporal_alpha: float = _TEMPORAL_ALPHA,
-    proof_count_alpha: float = _PROOF_COUNT_ALPHA,
-    is_passthrough_reranker: bool = False,
-) -> None:
-    """Apply combined scoring to a list of ScoredResults in-place.
-
-    Uses the cross-encoder score as the primary relevance signal, with recency,
-    temporal proximity, and proof count applied as multiplicative boosts. This
-    ensures the influence of these secondary signals is always proportional to
-    the base relevance score, regardless of the cross-encoder model's score
-    calibration.
-
-    Formula::
-
-        recency_boost     = 1 + recency_alpha     * (recency     - 0.5)   # in [1-α/2, 1+α/2]
-        temporal_boost    = 1 + temporal_alpha    * (temporal    - 0.5)   # in [1-α/2, 1+α/2]
-        proof_count_boost = 1 + proof_count_alpha * (proof_norm  - 0.5)   # in [1-α/2, 1+α/2]
-        combined_score    = CE_normalized * recency_boost * temporal_boost * proof_count_boost
-
-    proof_norm maps proof_count using a smooth logarithmic curve centered at 0.5,
-    clamped to [0, 1]:
-      proof_count=1 → 0.5 + 0 = 0.5 (neutral multiplier)
-      proof_count=150 → clamped to 1.0 (max +5% boost)
-
-    Temporal proximity is treated as neutral (0.5) when not set by temporal retrieval,
-    so temporal_boost collapses to 1.0 for non-temporal queries.
-
-    Proof count is treated as neutral (0.5) when not available (non-observation facts),
-    so proof_count_boost collapses to 1.0 for world/experience/opinion facts.
-
-    Args:
-        scored_results: Results from the cross-encoder reranker. Mutated in place.
-        now: Current UTC datetime for recency calculation.
-        recency_alpha: Max relative recency adjustment (default 0.2 → ±10%).
-        temporal_alpha: Max relative temporal adjustment (default 0.2 → ±10%).
-        proof_count_alpha: Max relative proof count adjustment (default 0.1 → ±5%).
-    """
-    if now.tzinfo is None:
-        now = now.replace(tzinfo=UTC)
-
-    # When the configured cross-encoder is a passthrough (e.g.
-    # RRFPassthroughCrossEncoder used by slim deployments), every
-    # cross_encoder_score_normalized is identical and provides no relevance
-    # signal. In that case the multiplicative recency / temporal / proof_count
-    # boosts below become the *only* ranking signal — making the final order a
-    # pure recency sort regardless of how relevant a candidate actually is.
-    #
-    # Detect that case and seed cross_encoder_score_normalized from the RRF
-    # rank instead, so the boosts modulate a meaningful base score rather than
-    # replacing it. This is a no-op for real cross-encoders, which produce
-    # diverse scores.
-    # When the reranker is a passthrough (e.g. RRFPassthroughCrossEncoder used
-    # by slim deployments), every cross_encoder_score_normalized is identical
-    # and provides no relevance signal. The multiplicative recency / temporal /
-    # proof_count boosts below would then become the *only* ranking signal,
-    # making the final order a pure recency sort regardless of how relevant a
-    # candidate actually is.
-    #
-    # Seed cross_encoder_score_normalized from the RRF rank instead, so the
-    # boosts modulate a meaningful base score. Caller passes is_passthrough
-    # explicitly because "all scores identical" is too fragile a heuristic —
-    # a real reranker can also tie scores (especially in tests with synthetic
-    # data) and we'd corrupt legitimate single-result reranks.
-    if is_passthrough_reranker and scored_results:
-        n = len(scored_results)
-        sorted_by_rrf = sorted(
-            scored_results,
-            key=lambda s: getattr(getattr(s, "candidate", None), "rrf_score", 0.0),
-            reverse=True,
-        )
-        denom = max(1, n - 1)
-        for new_rank, sr in enumerate(sorted_by_rrf):
-            # Map rank → [0.1, 1.0] so the recency boost can still nudge
-            # ordering between adjacent candidates without overpowering RRF.
-            sr.cross_encoder_score_normalized = 1.0 - (0.9 * new_rank / denom)
-
-    for sr in scored_results:
-        # Recency: linear decay over 365 days → [0.1, 1.0]; neutral 0.5 if no date.
-        sr.recency = 0.5
-        if sr.retrieval.occurred_start:
-            occurred = sr.retrieval.occurred_start
-            if occurred.tzinfo is None:
-                occurred = occurred.replace(tzinfo=UTC)
-            days_ago = (now - occurred).total_seconds() / 86400
-            sr.recency = max(0.1, min(1.0, 1.0 - (days_ago / 365)))
-
-        # Temporal proximity: meaningful only for temporal queries; neutral otherwise.
-        sr.temporal = sr.retrieval.temporal_proximity if sr.retrieval.temporal_proximity is not None else 0.5
-
-        # Proof count: log-normalized evidence strength; neutral for non-observations.
-        proof_count = sr.retrieval.proof_count
-        if proof_count is not None and proof_count >= 1:
-            # Clamp to [0, 1] so extreme counts stay within documented ±5% range
-            proof_norm = min(1.0, max(0.0, 0.5 + (math.log(proof_count) / 10.0)))
-        else:
-            # Neutral baseline is precisely 0.5, ensuring neutral multiplier (1.0)
-            proof_norm = 0.5
-
-        # RRF: kept at 0.0 for trace continuity but excluded from scoring.
-        # RRF is batch-relative (min-max normalised) and redundant after reranking.
-        sr.rrf_normalized = 0.0
-
-        recency_boost = 1.0 + recency_alpha * (sr.recency - 0.5)
-        temporal_boost = 1.0 + temporal_alpha * (sr.temporal - 0.5)
-        proof_count_boost = 1.0 + proof_count_alpha * (proof_norm - 0.5)
-        sr.combined_score = sr.cross_encoder_score_normalized * recency_boost * temporal_boost * proof_count_boost
-        sr.weight = sr.combined_score
-
-
-class CrossEncoderReranker:
-    """
-    Neural reranking using a cross-encoder model.
-
-    Configured via environment variables (see cross_encoder.py).
-    Default local model is cross-encoder/ms-marco-MiniLM-L-6-v2.
-    """
-
-    def __init__(self, cross_encoder=None):
-        """
-        Initialize cross-encoder reranker.
-
-        Args:
-            cross_encoder: CrossEncoderModel instance. If None, creates one from
-                          environment variables (defaults to local provider)
-        """
-        if cross_encoder is None:
-            from hindsight_api.engine.cross_encoder import create_cross_encoder_from_env
-
-            cross_encoder = create_cross_encoder_from_env()
-        self.cross_encoder = cross_encoder
-        self._initialized = False
-
-    async def ensure_initialized(self):
-        """Ensure the cross-encoder model is initialized (for lazy initialization)."""
-        if self._initialized:
-            return
-
-        import asyncio
-
-        cross_encoder = self.cross_encoder
-        # For local providers, run in thread pool to avoid blocking event loop
-        if cross_encoder.provider_name == "local":
-            loop = asyncio.get_event_loop()
-            await loop.run_in_executor(None, lambda: asyncio.run(cross_encoder.initialize()))
-        else:
-            await cross_encoder.initialize()
-        self._initialized = True
-
-    async def rerank(self, query: str, candidates: list[MergedCandidate]) -> list[ScoredResult]:
-        """
-        Rerank candidates using cross-encoder scores.
-
-        Args:
-            query: Search query
-            candidates: Merged candidates from RRF
-
-        Returns:
-            List of ScoredResult objects sorted by cross-encoder score
-        """
-        if not candidates:
-            return []
-
-        # Prepare query-document pairs with date information
-        pairs = []
-        for candidate in candidates:
-            retrieval = candidate.retrieval
-
-            # Use text + context for better ranking
-            doc_text = retrieval.text
-            if retrieval.context:
-                doc_text = f"{retrieval.context}: {doc_text}"
-
-            # Add formatted date information for temporal awareness
-            if retrieval.occurred_start:
-                occurred_start = retrieval.occurred_start
-
-                # Format in two styles for better model understanding
-                # 1. ISO format: YYYY-MM-DD
-                date_iso = occurred_start.strftime("%Y-%m-%d")
-
-                # 2. Human-readable: "June 5, 2022"
-                date_readable = occurred_start.strftime("%B %d, %Y")
-
-                # Prepend date to document text
-                doc_text = f"[Date: {date_readable} ({date_iso})] {doc_text}"
-
-            pairs.append([query, doc_text])
-
-        # Get cross-encoder scores
-        scores = await self.cross_encoder.predict(pairs)
-
-        # Normalize scores using sigmoid to [0, 1] range
-        # Cross-encoder returns logits which can be negative
-        import math
-
-        import numpy as np
-
-        def sigmoid(x):
-            return 1 / (1 + np.exp(-x))
-
-        normalized_scores = [sigmoid(score) for score in scores]
-
-        # Create ScoredResult objects with cross-encoder scores
-        scored_results = []
-        for candidate, raw_score, norm_score in zip(candidates, scores, normalized_scores):
-            # Sanitize NaN scores (cross-encoder can return NaN for certain inputs).
-            # NaN propagates through all downstream scoring and Pydantic serializes
-            # NaN as JSON null, which breaks clients expecting numeric values.
-            raw = float(raw_score)
-            norm = float(norm_score)
-            if math.isnan(raw):
-                raw = 0.0
-            if math.isnan(norm):
-                norm = 0.0
-            scored_result = ScoredResult(
-                candidate=candidate,
-                cross_encoder_score=raw,
-                cross_encoder_score_normalized=norm,
-                weight=norm,  # Initial weight is just cross-encoder score
-            )
-            scored_results.append(scored_result)
-
-        # Sort by cross-encoder score
-        scored_results.sort(key=lambda x: x.weight, reverse=True)
-
-        return scored_results

hindsight-api-slim/hindsight_api/engine/search/tags.py

@@ -1,390 +0,0 @@
-"""
-Tags filtering utilities for retrieval.
-
-Provides SQL building functions for filtering memories by tags.
-Supports four matching modes via TagsMatch enum:
-- "any": OR matching, includes untagged memories (default, backward compatible)
-- "all": AND matching, includes untagged memories
-- "any_strict": OR matching, excludes untagged memories
-- "all_strict": AND matching, excludes untagged memories
-
-OR matching (any/any_strict): Memory matches if ANY of its tags overlap with request tags
-AND matching (all/all_strict): Memory matches if ALL request tags are present in its tags
-"""
-
-from __future__ import annotations
-
-from typing import Annotated, Literal
-
-from pydantic import BaseModel, ConfigDict, Field
-
-TagsMatch = Literal["any", "all", "any_strict", "all_strict"]
-
-
-def _parse_tags_match(match: TagsMatch) -> tuple[str, bool]:
-    """
-    Parse TagsMatch into operator and include_untagged flag.
-
-    Returns:
-        Tuple of (operator, include_untagged)
-        - operator: "&&" for any/any_strict, "@>" for all/all_strict
-        - include_untagged: True for any/all, False for any_strict/all_strict
-    """
-    if match == "any":
-        return "&&", True
-    elif match == "all":
-        return "@>", True
-    elif match == "any_strict":
-        return "&&", False
-    elif match == "all_strict":
-        return "@>", False
-    else:
-        # Default to "any" behavior
-        return "&&", True
-
-
-def build_tags_where_clause(
-    tags: list[str] | None,
-    param_offset: int = 1,
-    table_alias: str = "",
-    match: TagsMatch = "any",
-) -> tuple[str, list, int]:
-    """
-    Build a SQL WHERE clause for filtering by tags.
-
-    Supports four matching modes:
-    - "any" (default): OR matching, includes untagged memories
-    - "all": AND matching, includes untagged memories
-    - "any_strict": OR matching, excludes untagged memories
-    - "all_strict": AND matching, excludes untagged memories
-
-    Args:
-        tags: List of tags to filter by. If None or empty, returns empty clause (no filtering).
-        param_offset: Starting parameter number for SQL placeholders (default 1).
-        table_alias: Optional table alias prefix (e.g., "mu." for "memory_units mu").
-        match: Matching mode. Defaults to "any".
-
-    Returns:
-        Tuple of (sql_clause, params, next_param_offset):
-        - sql_clause: SQL WHERE clause string
-        - params: List of parameter values to bind
-        - next_param_offset: Next available parameter number
-
-    Example:
-        >>> clause, params, next_offset = build_tags_where_clause(['user_a'], 3, 'mu.', 'any_strict')
-        >>> print(clause)  # "AND mu.tags IS NOT NULL AND mu.tags != '{}' AND mu.tags && $3"
-    """
-    if not tags:
-        return "", [], param_offset
-
-    column = f"{table_alias}tags" if table_alias else "tags"
-    operator, include_untagged = _parse_tags_match(match)
-
-    if include_untagged:
-        # Include untagged memories (NULL or empty array) OR matching tags
-        clause = f"AND ({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_offset})"
-    else:
-        # Strict: only memories with matching tags (exclude NULL and empty)
-        clause = f"AND {column} IS NOT NULL AND {column} != '{{}}' AND {column} {operator} ${param_offset}"
-
-    return clause, [tags], param_offset + 1
-
-
-def build_tags_where_clause_simple(
-    tags: list[str] | None,
-    param_num: int,
-    table_alias: str = "",
-    match: TagsMatch = "any",
-) -> str:
-    """
-    Build a simple SQL WHERE clause for tags filtering.
-
-    This is a convenience version that returns just the clause string,
-    assuming the caller will add the tags array to their params list.
-
-    Args:
-        tags: List of tags to filter by. If None or empty, returns empty string.
-        param_num: Parameter number to use in the clause.
-        table_alias: Optional table alias prefix.
-        match: Matching mode. Defaults to "any".
-
-    Returns:
-        SQL clause string or empty string.
-    """
-    if not tags:
-        return ""
-
-    column = f"{table_alias}tags" if table_alias else "tags"
-    operator, include_untagged = _parse_tags_match(match)
-
-    if include_untagged:
-        # Include untagged memories (NULL or empty array) OR matching tags
-        return f"AND ({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_num})"
-    else:
-        # Strict: only memories with matching tags (exclude NULL and empty)
-        return f"AND {column} IS NOT NULL AND {column} != '{{}}' AND {column} {operator} ${param_num}"
-
-
-def filter_results_by_tags(
-    results: list,
-    tags: list[str] | None,
-    match: TagsMatch = "any",
-) -> list:
-    """
-    Filter retrieval results by tags in Python (for post-processing).
-
-    Used when SQL filtering isn't possible (e.g., graph traversal results).
-
-    Args:
-        results: List of RetrievalResult objects with a 'tags' attribute.
-        tags: List of tags to filter by. If None or empty, returns all results.
-        match: Matching mode. Defaults to "any".
-
-    Returns:
-        Filtered list of results.
-    """
-    if not tags:
-        return results
-
-    _, include_untagged = _parse_tags_match(match)
-    is_any_match = match in ("any", "any_strict")
-
-    tags_set = set(tags)
-    filtered = []
-
-    for result in results:
-        result_tags = getattr(result, "tags", None)
-
-        # Check if untagged
-        is_untagged = result_tags is None or len(result_tags) == 0
-
-        if is_untagged:
-            if include_untagged:
-                filtered.append(result)
-            # else: skip untagged
-        else:
-            result_tags_set = set(result_tags)
-            if is_any_match:
-                # Any overlap
-                if result_tags_set & tags_set:
-                    filtered.append(result)
-            else:
-                # All tags must be present
-                if tags_set <= result_tags_set:
-                    filtered.append(result)
-
-    return filtered
-
-
-# =============================================================================
-# Compound tag group models (recursive boolean expressions)
-# =============================================================================
-
-
-class TagGroupLeaf(BaseModel):
-    """A leaf tag filter: matches memories by tag list and match mode."""
-
-    tags: list[str]
-    match: TagsMatch = "any_strict"
-
-
-class TagGroupAnd(BaseModel):
-    """Compound AND group: all child filters must match."""
-
-    model_config = ConfigDict(populate_by_name=True)
-    filters: list[TagGroup] = Field(alias="and")
-
-
-class TagGroupOr(BaseModel):
-    """Compound OR group: at least one child filter must match."""
-
-    model_config = ConfigDict(populate_by_name=True)
-    filters: list[TagGroup] = Field(alias="or")
-
-
-class TagGroupNot(BaseModel):
-    """Compound NOT group: child filter must NOT match."""
-
-    model_config = ConfigDict(populate_by_name=True)
-    filter: TagGroup = Field(alias="not")
-
-
-# TagGroup is a discriminated union; Pydantic will try left-to-right.
-# TagGroupLeaf is identified by the presence of 'tags'.
-# TagGroupAnd / TagGroupOr / TagGroupNot are compound (no 'tags' key).
-TagGroup = Annotated[
-    TagGroupLeaf | TagGroupAnd | TagGroupOr | TagGroupNot,
-    Field(union_mode="left_to_right"),
-]
-
-# Rebuild forward-reference models so recursive TagGroup is resolved.
-TagGroupAnd.model_rebuild()
-TagGroupOr.model_rebuild()
-TagGroupNot.model_rebuild()
-
-
-# =============================================================================
-# SQL builder for compound tag groups
-# =============================================================================
-
-
-def _build_group_clause(
-    group: TagGroup,
-    param_offset: int,
-    table_alias: str,
-) -> tuple[str, list, int]:
-    """
-    Recursively build an inner SQL clause (no leading AND/OR) for a single TagGroup.
-
-    Returns:
-        (inner_clause, params, next_param_offset)
-    """
-    if isinstance(group, TagGroupLeaf):
-        column = f"{table_alias}tags" if table_alias else "tags"
-        operator, include_untagged = _parse_tags_match(group.match)
-        if include_untagged:
-            clause = f"({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_offset})"
-        else:
-            clause = f"({column} IS NOT NULL AND {column} != '{{}}' AND {column} {operator} ${param_offset})"
-        return clause, [group.tags], param_offset + 1
-
-    elif isinstance(group, TagGroupAnd):
-        parts = []
-        params: list = []
-        offset = param_offset
-        for child in group.filters:
-            child_clause, child_params, offset = _build_group_clause(child, offset, table_alias)
-            parts.append(child_clause)
-            params.extend(child_params)
-        inner = " AND ".join(parts)
-        return f"({inner})", params, offset
-
-    elif isinstance(group, TagGroupOr):
-        parts = []
-        params = []
-        offset = param_offset
-        for child in group.filters:
-            child_clause, child_params, offset = _build_group_clause(child, offset, table_alias)
-            parts.append(child_clause)
-            params.extend(child_params)
-        inner = " OR ".join(parts)
-        return f"({inner})", params, offset
-
-    elif isinstance(group, TagGroupNot):
-        child_clause, child_params, next_offset = _build_group_clause(group.filter, param_offset, table_alias)
-        return f"NOT {child_clause}", child_params, next_offset
-
-    else:
-        # Should never happen with proper Pydantic validation
-        return "", [], param_offset
-
-
-def build_tag_groups_where_clause(
-    tag_groups: list[TagGroup] | None,
-    param_offset: int,
-    table_alias: str = "",
-) -> tuple[str, list, int]:
-    """
-    Build a SQL WHERE clause for compound tag group filtering.
-
-    Top-level groups are AND-ed together. Each group is a recursive boolean
-    expression (leaf, and, or, not).
-
-    Args:
-        tag_groups: List of TagGroup objects. If None or empty, returns empty clause.
-        param_offset: Starting parameter number for SQL placeholders.
-        table_alias: Optional table alias prefix (e.g., "mu." for "memory_units mu").
-
-    Returns:
-        Tuple of (sql_clause, params, next_param_offset):
-        - sql_clause: SQL WHERE clause string starting with "AND" (or empty string)
-        - params: List of parameter values to bind (one per leaf node)
-        - next_param_offset: Next available parameter number
-
-    Example:
-        >>> groups = [TagGroupLeaf(tags=["user:alice"], match="all_strict")]
-        >>> clause, params, next_offset = build_tag_groups_where_clause(groups, 3)
-        >>> print(clause)  # "AND (tags IS NOT NULL AND tags != '{}' AND tags @> $3)"
-    """
-    if not tag_groups:
-        return "", [], param_offset
-
-    all_params: list = []
-    all_clauses: list[str] = []
-    offset = param_offset
-
-    for group in tag_groups:
-        inner_clause, group_params, offset = _build_group_clause(group, offset, table_alias)
-        all_clauses.append(inner_clause)
-        all_params.extend(group_params)
-
-    combined = " AND ".join(all_clauses)
-    return f"AND {combined}", all_params, offset
-
-
-# =============================================================================
-# Python-side filter for compound tag groups (post-retrieval filtering)
-# =============================================================================
-
-
-def _match_group(result: object, group: TagGroup) -> bool:
-    """
-    Recursively evaluate a TagGroup against a retrieval result.
-
-    Args:
-        result: Any object with a 'tags' attribute (list[str] or None).
-        group: The TagGroup to evaluate.
-
-    Returns:
-        True if the result matches the group, False otherwise.
-    """
-    if isinstance(group, TagGroupLeaf):
-        result_tags = getattr(result, "tags", None)
-        is_untagged = result_tags is None or len(result_tags) == 0
-        _, include_untagged = _parse_tags_match(group.match)
-        is_any_match = group.match in ("any", "any_strict")
-        tags_set = set(group.tags)
-
-        if is_untagged:
-            return include_untagged
-        else:
-            result_tags_set = set(result_tags)
-            if is_any_match:
-                return bool(result_tags_set & tags_set)
-            else:
-                return tags_set <= result_tags_set
-
-    elif isinstance(group, TagGroupAnd):
-        return all(_match_group(result, child) for child in group.filters)
-
-    elif isinstance(group, TagGroupOr):
-        return any(_match_group(result, child) for child in group.filters)
-
-    elif isinstance(group, TagGroupNot):
-        return not _match_group(result, group.filter)
-
-    else:
-        return True
-
-
-def filter_results_by_tag_groups(
-    results: list,
-    tag_groups: list[TagGroup] | None,
-) -> list:
-    """
-    Filter retrieval results by compound tag groups in Python (for post-processing).
-
-    Used when SQL filtering isn't possible (e.g., graph traversal results).
-    Top-level groups are AND-ed together.
-
-    Args:
-        results: List of RetrievalResult objects with a 'tags' attribute.
-        tag_groups: List of TagGroup objects. If None or empty, returns all results.
-
-    Returns:
-        Filtered list of results where ALL top-level groups match.
-    """
-    if not tag_groups:
-        return results
-
-    return [r for r in results if all(_match_group(r, group) for group in tag_groups)]

hindsight-api-slim/hindsight_api/engine/storage/gcs.py

@@ -1,105 +0,0 @@
-"""Google Cloud Storage backend using obstore."""
-
-import logging
-import os
-from datetime import datetime, timedelta, timezone
-
-import obstore as obs
-from obstore.store import GCSStore
-
-from .base import FileStorage
-
-logger = logging.getLogger(__name__)
-
-
-def _make_google_auth_credential_provider():
-    """Create a credential provider using google.auth (supports all credential types).
-
-    obstore's built-in credential parsing only supports service_account and
-    authorized_user JSON types.  This provider uses the google-auth library
-    which additionally handles external_account (Workload Identity Federation),
-    impersonated credentials, and metadata-server credentials.
-    """
-    import google.auth
-    import google.auth.transport.requests
-
-    credentials, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
-    request = google.auth.transport.requests.Request()
-
-    def _provide():
-        credentials.refresh(request)
-        expiry = credentials.expiry
-        if expiry and expiry.tzinfo is None:
-            expiry = expiry.replace(tzinfo=timezone.utc)
-        return {"token": credentials.token, "expires_at": expiry}
-
-    return _provide
-
-
-class GCSFileStorage(FileStorage):
-    """
-    Google Cloud Storage backend.
-
-    Uses obstore (Rust-backed) for high-throughput async access to GCS.
-    Supports Application Default Credentials, service account keys, and explicit credentials.
-    """
-
-    def __init__(
-        self,
-        bucket: str,
-        service_account_key: str | None = None,
-    ):
-        kwargs: dict = {}
-        if service_account_key:
-            kwargs["service_account_key"] = service_account_key
-        else:
-            # Use google.auth credential provider for broad credential type support
-            # (service_account, authorized_user, external_account, metadata server, etc.)
-            try:
-                kwargs["credential_provider"] = _make_google_auth_credential_provider()
-                logger.info("Using google.auth credential provider for GCS")
-            except Exception as e:
-                logger.warning(
-                    f"Failed to create google.auth credential provider, falling back to obstore defaults: {e}"
-                )
-
-        # Workaround for https://github.com/developmentseed/obstore/issues/605
-        # obstore's Rust layer doesn't support external_account credentials (Workload
-        # Identity Federation) and eagerly parses GOOGLE_APPLICATION_CREDENTIALS even
-        # when credential_provider is given. Per the obstore maintainer's guidance,
-        # remove env vars so the Rust code doesn't try to authenticate itself.
-        # google.auth (used by credential_provider above) has already loaded credentials.
-        gac = os.environ.pop("GOOGLE_APPLICATION_CREDENTIALS", None)
-        try:
-            self._store = GCSStore(bucket, **kwargs)
-        finally:
-            if gac is not None:
-                os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = gac
-        logger.info(f"Initialized GCS file storage: bucket={bucket}")
-
-    async def store(self, file_data: bytes, key: str, metadata: dict[str, str] | None = None) -> str:
-        await obs.put_async(self._store, key, file_data)
-        logger.debug(f"Stored file {key} ({len(file_data)} bytes) in GCS")
-        return key
-
-    async def retrieve(self, key: str) -> bytes:
-        try:
-            response = await obs.get_async(self._store, key)
-            return await response.bytes_async()
-        except Exception as e:
-            if "not found" in str(e).lower():
-                raise FileNotFoundError(f"File not found: {key}") from e
-            raise
-
-    async def delete(self, key: str) -> None:
-        await obs.delete_async(self._store, key)
-
-    async def exists(self, key: str) -> bool:
-        try:
-            await obs.head_async(self._store, key)
-            return True
-        except Exception:
-            return False
-
-    async def get_download_url(self, key: str, expires_in: int = 3600) -> str:
-        return await obs.sign_async(self._store, "GET", key, timedelta(seconds=expires_in))

hindsight-api-slim/hindsight_api/main.py

@@ -1,302 +0,0 @@
-"""
-Command-line interface for Hindsight API.
-
-Run the server with:
-    hindsight-api
-
-Run as background daemon:
-    hindsight-api --daemon
-
-Stop with Ctrl+C.
-"""
-
-import argparse
-import asyncio
-import atexit
-import dataclasses
-import os
-import signal
-import sys
-import warnings
-
-import uvicorn
-
-from . import MemoryEngine, __version__
-from .api import create_app
-from .banner import print_banner
-from .config import DEFAULT_WORKERS, ENV_WORKERS, HindsightConfig, _get_raw_config
-from .daemon import (
-    DEFAULT_DAEMON_PORT,
-    DEFAULT_IDLE_TIMEOUT,
-    IdleTimeoutMiddleware,
-    daemonize,
-)
-from .extensions import DefaultExtensionContext, OperationValidatorExtension, TenantExtension, load_extension
-
-# Filter deprecation warnings from third-party libraries
-warnings.filterwarnings("ignore", message="websockets.legacy is deprecated")
-warnings.filterwarnings("ignore", message="websockets.server.WebSocketServerProtocol is deprecated")
-
-# Disable tokenizers parallelism to avoid warnings
-os.environ["TOKENIZERS_PARALLELISM"] = "false"
-
-# Global reference for cleanup
-_memory: MemoryEngine | None = None
-
-
-def _cleanup():
-    """Synchronous cleanup function to stop resources on exit."""
-    global _memory
-    if _memory is not None and _memory._pg0 is not None:
-        try:
-            loop = asyncio.new_event_loop()
-            loop.run_until_complete(_memory._pg0.stop())
-            loop.close()
-            print("\npg0 stopped.")
-        except Exception as e:
-            print(f"\nError stopping pg0: {e}")
-
-
-def _signal_handler(signum, frame):
-    """Handle SIGINT/SIGTERM to ensure cleanup."""
-    print(f"\nReceived signal {signum}, shutting down...")
-    _cleanup()
-    sys.exit(0)
-
-
-def main():
-    """Main entry point for the CLI."""
-    global _memory
-
-    # Load configuration from environment (for CLI args defaults)
-    config = _get_raw_config()
-
-    parser = argparse.ArgumentParser(
-        prog="hindsight-api",
-        description="Hindsight API Server",
-    )
-
-    # Server options
-    parser.add_argument(
-        "--host", default=config.host, help=f"Host to bind to (default: {config.host}, env: HINDSIGHT_API_HOST)"
-    )
-    parser.add_argument(
-        "--port",
-        type=int,
-        default=config.port,
-        help=f"Port to bind to (default: {config.port}, env: HINDSIGHT_API_PORT)",
-    )
-    parser.add_argument(
-        "--log-level",
-        default=config.log_level,
-        choices=["critical", "error", "warning", "info", "debug", "trace"],
-        help=f"Log level (default: {config.log_level}, env: HINDSIGHT_API_LOG_LEVEL)",
-    )
-
-    # Development options
-    parser.add_argument("--reload", action="store_true", help="Enable auto-reload on code changes (development only)")
-    parser.add_argument(
-        "--workers",
-        type=int,
-        default=int(os.getenv(ENV_WORKERS, str(DEFAULT_WORKERS))),
-        help=f"Number of worker processes (env: {ENV_WORKERS}, default: {DEFAULT_WORKERS})",
-    )
-
-    # Access log options
-    parser.add_argument("--access-log", action="store_true", help="Enable access log")
-    parser.add_argument("--no-access-log", dest="access_log", action="store_false", help="Disable access log (default)")
-    parser.set_defaults(access_log=False)
-
-    # Proxy options
-    parser.add_argument(
-        "--proxy-headers", action="store_true", help="Enable X-Forwarded-Proto, X-Forwarded-For headers"
-    )
-    parser.add_argument(
-        "--forwarded-allow-ips", default=None, help="Comma separated list of IPs to trust with proxy headers"
-    )
-
-    # SSL options
-    parser.add_argument("--ssl-keyfile", default=None, help="SSL key file")
-    parser.add_argument("--ssl-certfile", default=None, help="SSL certificate file")
-
-    # Daemon mode options
-    parser.add_argument(
-        "--daemon",
-        action="store_true",
-        help=f"Run as background daemon (uses port {DEFAULT_DAEMON_PORT}, auto-exits after idle)",
-    )
-    parser.add_argument(
-        "--idle-timeout",
-        type=int,
-        default=DEFAULT_IDLE_TIMEOUT,
-        help=f"Idle timeout in seconds before auto-exit in daemon mode (default: {DEFAULT_IDLE_TIMEOUT})",
-    )
-
-    args = parser.parse_args()
-
-    # Daemon mode handling
-    if args.daemon:
-        # Use port from args (may be custom for profiles)
-        if args.port == config.port:  # No custom port specified
-            args.port = DEFAULT_DAEMON_PORT
-        args.host = "127.0.0.1"  # Only bind to localhost for security
-
-        # Fork into background
-        # No lockfile needed - port binding prevents duplicate daemons
-        daemonize()
-
-    # Print banner (not in daemon mode)
-    if not args.daemon:
-        print()
-        print_banner()
-
-    # Configure Python logging based on log level
-    # Update config with CLI override if provided
-    if args.log_level != config.log_level:
-        config = dataclasses.replace(config, host=args.host, port=args.port, log_level=args.log_level)
-    config.configure_logging()
-    if not args.daemon:
-        config.log_config()
-
-    # Register cleanup handlers
-    atexit.register(_cleanup)
-    signal.signal(signal.SIGINT, _signal_handler)
-    signal.signal(signal.SIGTERM, _signal_handler)
-
-    # Load operation validator extension if configured
-    operation_validator = load_extension("OPERATION_VALIDATOR", OperationValidatorExtension)
-    if operation_validator:
-        import logging
-
-        logging.info(f"Loaded operation validator: {operation_validator.__class__.__name__}")
-
-    # Load tenant extension if configured
-    tenant_extension = load_extension("TENANT", TenantExtension)
-    if tenant_extension:
-        import logging
-
-        logging.info(f"Loaded tenant extension: {tenant_extension.__class__.__name__}")
-
-    # Create MemoryEngine (reads configuration from environment)
-    _memory = MemoryEngine(
-        operation_validator=operation_validator,
-        tenant_extension=tenant_extension,
-        run_migrations=config.run_migrations_on_startup,
-    )
-
-    # Set extension context on tenant extension (needed for schema provisioning)
-    if tenant_extension:
-        extension_context = DefaultExtensionContext(
-            database_url=config.database_url,
-            memory_engine=_memory,
-        )
-        tenant_extension.set_context(extension_context)
-        logging.info("Extension context set on tenant extension")
-
-    # Create FastAPI app
-    app = create_app(
-        memory=_memory,
-        http_api_enabled=True,
-        mcp_api_enabled=config.mcp_enabled,
-        mcp_mount_path="/mcp",
-        initialize_memory=True,
-    )
-
-    # Wrap with idle timeout middleware in daemon mode
-    idle_middleware = None
-    if args.daemon:
-        idle_middleware = IdleTimeoutMiddleware(app, idle_timeout=args.idle_timeout)
-        app = idle_middleware
-
-    # Prepare uvicorn config
-    # When using workers or reload, we must use import string so each worker can import the app
-    use_import_string = args.workers > 1 or args.reload
-    # Check for uvloop/winloop availability
-    import sys
-
-    loop_impl = "asyncio"
-    if sys.platform == "win32":
-        try:
-            import winloop
-
-            winloop.install()  # Patches asyncio globally — uvicorn uses "asyncio" but gets winloop
-            loop_impl = "asyncio"  # Tell uvicorn "asyncio" — it's now winloop underneath
-            print("winloop installed as asyncio event loop policy (Windows uvloop port)")
-        except ImportError:
-            print("winloop not installed, using default asyncio event loop")
-    else:
-        try:
-            import uvloop  # noqa: F401
-
-            loop_impl = "uvloop"
-            print("uvloop available, will use for event loop")
-        except ImportError:
-            print("uvloop not installed, using default asyncio event loop")
-
-    uvicorn_config = {
-        "app": "hindsight_api.server:app" if use_import_string else app,
-        "host": args.host,
-        "port": args.port,
-        "log_level": args.log_level,
-        "access_log": args.access_log,
-        "proxy_headers": args.proxy_headers,
-        "ws": "wsproto",  # Use wsproto instead of websockets to avoid deprecation warnings
-        "loop": loop_impl,  # Explicitly set event loop implementation
-        "timeout_keep_alive": 30,  # Exceed aiohttp's 15s client timeout so the client always closes first
-        "timeout_graceful_shutdown": 5,  # Cap graceful shutdown at 5s; also enables force-kill on second Ctrl+C
-    }
-
-    # Add optional parameters if provided
-    if args.reload:
-        uvicorn_config["reload"] = True
-    if args.workers > 1:
-        uvicorn_config["workers"] = args.workers
-    if args.forwarded_allow_ips:
-        uvicorn_config["forwarded_allow_ips"] = args.forwarded_allow_ips
-    if args.ssl_keyfile:
-        uvicorn_config["ssl_keyfile"] = args.ssl_keyfile
-    if args.ssl_certfile:
-        uvicorn_config["ssl_certfile"] = args.ssl_certfile
-
-    # Print startup info (not in daemon mode)
-    if not args.daemon:
-        from .banner import print_startup_info
-
-        print_startup_info(
-            host=args.host,
-            port=args.port,
-            database_url=config.database_url,
-            llm_provider=config.llm_provider,
-            llm_model=config.llm_model,
-            embeddings_provider=config.embeddings_provider,
-            reranker_provider=config.reranker_provider,
-            mcp_enabled=config.mcp_enabled,
-            version=__version__,
-            vector_extension=config.vector_extension,
-            text_search_extension=config.text_search_extension,
-        )
-
-    # Start idle checker in daemon mode
-    if idle_middleware is not None:
-        # Start the idle checker in a background thread with its own event loop
-        import logging
-        import threading
-
-        def run_idle_checker():
-            import time
-
-            time.sleep(2)  # Wait for uvicorn to start
-            try:
-                loop = asyncio.new_event_loop()
-                asyncio.set_event_loop(loop)
-                loop.run_until_complete(idle_middleware._check_idle())
-            except Exception as e:
-                logging.error(f"Idle checker error: {e}", exc_info=True)
-
-        threading.Thread(target=run_idle_checker, daemon=True).start()
-
-    uvicorn.run(**uvicorn_config)
-
-
-if __name__ == "__main__":
-    main()

hindsight-api-slim/hindsight_api/webhooks/__init__.py

@@ -1,13 +0,0 @@
-"""Webhook system for Hindsight API event notifications."""
-
-from .manager import WebhookManager
-from .models import ConsolidationEventData, RetainEventData, WebhookConfig, WebhookEvent, WebhookEventType
-
-__all__ = [
-    "WebhookManager",
-    "WebhookConfig",
-    "WebhookEvent",
-    "WebhookEventType",
-    "ConsolidationEventData",
-    "RetainEventData",
-]

hindsight-api-slim/hindsight_api/webhooks/manager.py

@@ -1,242 +0,0 @@
-"""Webhook manager for delivering event notifications."""
-
-import hashlib
-import hmac
-import json
-import logging
-import uuid
-from datetime import datetime, timezone
-from typing import TYPE_CHECKING
-
-import asyncpg
-
-from .models import WebhookConfig, WebhookEvent, WebhookHttpConfig
-
-if TYPE_CHECKING:
-    from hindsight_api.extensions.tenant import TenantExtension
-
-logger = logging.getLogger(__name__)
-
-# Retry delay schedule in seconds: 5 retries after the first attempt.
-# Fast early retries catch transient failures; later retries handle longer outages.
-RETRY_DELAYS = [5, 300, 1800, 7200, 18000]
-MAX_ATTEMPTS = len(RETRY_DELAYS) + 1  # first attempt + len(RETRY_DELAYS) retries
-
-
-def _fq_table(table: str, schema: str | None = None) -> str:
-    """Get fully-qualified table name with optional schema prefix."""
-    if schema:
-        return f'"{schema}".{table}'
-    return table
-
-
-def _parse_http_config(value: str | dict | None) -> WebhookHttpConfig:
-    """Parse http_config column value (JSONB returned as text or dict) into a model."""
-    if value is None:
-        return WebhookHttpConfig()
-    if isinstance(value, str):
-        return WebhookHttpConfig.model_validate_json(value)
-    return WebhookHttpConfig.model_validate(value)
-
-
-class WebhookManager:
-    """
-    Manages webhook registration and event firing.
-
-    Supports both global webhooks (configured via env vars) and per-bank
-    webhooks stored in the database. Deliveries are queued as async_operations
-    tasks (operation_type='webhook_delivery') and picked up by the worker poller.
-    """
-
-    def __init__(
-        self,
-        pool: asyncpg.Pool,
-        global_webhooks: list[WebhookConfig],
-        tenant_extension: "TenantExtension | None" = None,
-    ):
-        self._pool = pool
-        self._global_webhooks = global_webhooks
-        self._tenant_extension = tenant_extension
-
-    def _sign_payload(self, secret: str, payload_bytes: bytes) -> str:
-        """Compute HMAC-SHA256 signature for a payload."""
-        return "sha256=" + hmac.new(secret.encode(), payload_bytes, hashlib.sha256).hexdigest()
-
-    async def fire_event(self, event: WebhookEvent, schema: str | None = None) -> None:
-        """
-        Queue webhook deliveries for an event as async_operations tasks.
-
-        Loads per-bank and global webhooks, inserts pending webhook_delivery tasks for
-        any webhook whose event_types list matches the fired event type. The worker
-        poller picks these up and calls MemoryEngine._handle_webhook_delivery().
-
-        Args:
-            event: The event to deliver.
-            schema: Database schema (for multi-tenant). None = default schema.
-        """
-        webhook_table = _fq_table("webhooks", schema)
-        ops_table = _fq_table("async_operations", schema)
-        now = datetime.now(timezone.utc)
-        payload_str = event.model_dump_json()
-
-        try:
-            # Load per-bank webhooks from DB (bank-specific + global NULL rows)
-            rows = await self._pool.fetch(
-                f"""
-                SELECT id, bank_id, url, secret, event_types, enabled, http_config::text
-                FROM {webhook_table}
-                WHERE (bank_id = $1 OR bank_id IS NULL) AND enabled = true
-                """,
-                event.bank_id,
-            )
-
-            db_webhooks = [
-                WebhookConfig(
-                    id=str(row["id"]),
-                    bank_id=row["bank_id"],
-                    url=row["url"],
-                    secret=row["secret"],
-                    event_types=list(row["event_types"]) if row["event_types"] else [],
-                    enabled=row["enabled"],
-                    http_config=_parse_http_config(row["http_config"]),
-                )
-                for row in rows
-            ]
-
-            # Merge with global webhooks from env config
-            all_webhooks = self._global_webhooks + db_webhooks
-            matched = 0
-
-            for webhook in all_webhooks:
-                if not webhook.enabled:
-                    continue
-                if event.event.value not in webhook.event_types:
-                    continue
-
-                operation_id = uuid.uuid4()
-                webhook_id = webhook.id if webhook.id else None
-
-                task_payload = json.dumps(
-                    {
-                        "type": "webhook_delivery",
-                        "operation_id": str(operation_id),
-                        "bank_id": event.bank_id,
-                        "url": webhook.url,
-                        "secret": webhook.secret,
-                        "event_type": event.event.value,
-                        "payload": payload_str,
-                        "webhook_id": webhook_id,
-                        "http_config": webhook.http_config.model_dump(),
-                    }
-                )
-
-                await self._pool.execute(
-                    f"""
-                    INSERT INTO {ops_table}
-                      (operation_id, bank_id, operation_type, status, task_payload, result_metadata, created_at, updated_at)
-                    VALUES ($1, $2, 'webhook_delivery', 'pending', $3::jsonb, '{{}}'::jsonb, $4, $4)
-                    """,
-                    operation_id,
-                    event.bank_id,
-                    task_payload,
-                    now,
-                )
-                matched += 1
-
-            logger.debug(f"Fired webhook event {event.event} for bank {event.bank_id}: {matched} delivery(ies) queued")
-
-        except Exception as e:
-            logger.error(f"Failed to queue webhook deliveries for event {event.event}: {e}")
-
-    async def fire_event_with_conn(
-        self, event: WebhookEvent, conn: asyncpg.Connection, schema: str | None = None
-    ) -> None:
-        """
-        Queue webhook deliveries within an existing database connection/transaction.
-
-        Identical to fire_event() but uses the provided connection instead of acquiring
-        one from the pool. Use this to atomically insert delivery tasks in the same
-        transaction as the primary operation (transactional outbox pattern).
-
-        Args:
-            event: The event to deliver.
-            conn: Existing asyncpg connection (may be inside an active transaction).
-            schema: Database schema (for multi-tenant). None = default schema.
-        """
-        webhook_table = _fq_table("webhooks", schema)
-        ops_table = _fq_table("async_operations", schema)
-        now = datetime.now(timezone.utc)
-        payload_str = event.model_dump_json()
-
-        try:
-            rows = await conn.fetch(
-                f"""
-                SELECT id, bank_id, url, secret, event_types, enabled, http_config::text
-                FROM {webhook_table}
-                WHERE (bank_id = $1 OR bank_id IS NULL) AND enabled = true
-                """,
-                event.bank_id,
-            )
-
-            db_webhooks = [
-                WebhookConfig(
-                    id=str(row["id"]),
-                    bank_id=row["bank_id"],
-                    url=row["url"],
-                    secret=row["secret"],
-                    event_types=list(row["event_types"]) if row["event_types"] else [],
-                    enabled=row["enabled"],
-                    http_config=_parse_http_config(row["http_config"]),
-                )
-                for row in rows
-            ]
-
-            all_webhooks = self._global_webhooks + db_webhooks
-            matched = 0
-
-            for webhook in all_webhooks:
-                if not webhook.enabled:
-                    continue
-                if event.event.value not in webhook.event_types:
-                    continue
-
-                operation_id = uuid.uuid4()
-                webhook_id = webhook.id if webhook.id else None
-
-                task_payload = json.dumps(
-                    {
-                        "type": "webhook_delivery",
-                        "operation_id": str(operation_id),
-                        "bank_id": event.bank_id,
-                        "url": webhook.url,
-                        "secret": webhook.secret,
-                        "event_type": event.event.value,
-                        "payload": payload_str,
-                        "webhook_id": webhook_id,
-                        "http_config": webhook.http_config.model_dump(),
-                    }
-                )
-
-                await conn.execute(
-                    f"""
-                    INSERT INTO {ops_table}
-                      (operation_id, bank_id, operation_type, status, task_payload, result_metadata, created_at, updated_at)
-                    VALUES ($1, $2, 'webhook_delivery', 'pending', $3::jsonb, '{{}}'::jsonb, $4, $4)
-                    """,
-                    operation_id,
-                    event.bank_id,
-                    task_payload,
-                    now,
-                )
-                matched += 1
-
-            logger.debug(
-                f"Fired webhook event {event.event} for bank {event.bank_id}: {matched} delivery(ies) queued (in-transaction)"
-            )
-
-        except Exception as e:
-            logger.error(
-                f"Failed to queue webhook deliveries (in-transaction) for event {event.event}: {e}. "
-                "CRITICAL: The enclosing database transaction is now aborted and will roll back all changes."
-            )
-            raise

hindsight-api-slim/hindsight_api/webhooks/models.py

@@ -1,51 +0,0 @@
-"""Pydantic models for the webhook system."""
-
-from datetime import datetime
-from enum import StrEnum
-
-from pydantic import BaseModel, Field
-
-
-class WebhookEventType(StrEnum):
-    CONSOLIDATION_COMPLETED = "consolidation.completed"
-    RETAIN_COMPLETED = "retain.completed"
-
-
-class ConsolidationEventData(BaseModel):
-    observations_created: int | None = None
-    observations_updated: int | None = None
-    observations_deleted: int | None = None
-    error_message: str | None = None
-
-
-class RetainEventData(BaseModel):
-    document_id: str | None = None
-    tags: list[str] | None = None
-
-
-class WebhookEvent(BaseModel):
-    event: WebhookEventType
-    bank_id: str
-    operation_id: str
-    status: str  # "completed" or "failed"
-    timestamp: datetime
-    data: ConsolidationEventData | RetainEventData
-
-
-class WebhookHttpConfig(BaseModel):
-    """HTTP delivery configuration for a webhook."""
-
-    method: str = Field(default="POST", description="HTTP method: GET or POST")
-    timeout_seconds: int = Field(default=30, description="HTTP request timeout in seconds")
-    headers: dict[str, str] = Field(default_factory=dict, description="Custom HTTP headers")
-    params: dict[str, str] = Field(default_factory=dict, description="Custom HTTP query parameters")
-
-
-class WebhookConfig(BaseModel):
-    id: str
-    bank_id: str | None
-    url: str
-    secret: str | None
-    event_types: list[str]
-    enabled: bool
-    http_config: WebhookHttpConfig = Field(default_factory=WebhookHttpConfig)

hindsight-api-slim/hindsight_api/worker/exceptions.py

@@ -1,30 +0,0 @@
-from datetime import datetime
-
-
-class RetryTaskAt(Exception):
-    """Raise from a task handler to schedule a retry at a specific time."""
-
-    def __init__(self, retry_at: datetime, message: str = ""):
-        self.retry_at = retry_at
-        super().__init__(message)
-
-
-class DeferOperation(Exception):
-    """Raise from an extension hook (or task handler) to requeue the
-    operation for execution at a later time, without counting as a retry.
-
-    Unlike `RetryTaskAt`, this is not a failure: `retry_count` is not
-    incremented and `error_message` is not written. Use this for
-    backpressure / "not yet, try later" decisions made before or during
-    task execution (e.g. quota windows, warming dependencies, upstream
-    rate limits).
-
-    Worker-only: raising this from a hook called in HTTP request context
-    (e.g. `validate_recall` for a synchronous recall) will surface as an
-    unhandled 500 — there is no queue to defer to.
-    """
-
-    def __init__(self, exec_date: datetime, reason: str = ""):
-        self.exec_date = exec_date
-        self.reason = reason
-        super().__init__(reason)

hindsight-api-slim/hindsight_api/worker/stage.py

@@ -1,59 +0,0 @@
-"""Stage breadcrumbs for in-flight worker tasks.
-
-The worker poller binds a `StageHolder` to each task's contextvar scope.
-Engine code calls `set_stage("retain.facts.llm")` at phase boundaries; the
-poller reads the holder periodically to surface what each in-flight task is
-currently doing in `WORKER_STATS` / `WORKER_TASK` log lines.
-
-Outside a worker context the contextvar is unset and `set_stage` is a no-op,
-so engine code is safe to call from sync HTTP requests, tests, or the CLI
-without any setup.
-"""
-
-from __future__ import annotations
-
-import time
-from contextvars import ContextVar
-from dataclasses import dataclass, field
-
-
-@dataclass
-class StageHolder:
-    """Mutable container for the current task's stage label."""
-
-    stage: str = "init"
-    updated_at: float = field(default_factory=time.monotonic)
-
-
-_current_holder: ContextVar[StageHolder | None] = ContextVar("hindsight_stage_holder", default=None)
-
-
-def bind_holder(holder: StageHolder):
-    """Bind a holder to the current async context.
-
-    Must be called from inside the task coroutine itself (not from the
-    spawning code) so the binding lives in the task's own contextvar scope.
-
-    Returns the token that can be passed to `_current_holder.reset()` if
-    the binding ever needs to be unwound.
-    """
-    return _current_holder.set(holder)
-
-
-def set_stage(name: str) -> None:
-    """Update the current task's stage label.
-
-    No-op when called outside a worker task context (e.g. from a sync HTTP
-    request, a test, or the CLI). Cheap enough to call per-phase.
-    """
-    holder = _current_holder.get()
-    if holder is None:
-        return
-    holder.stage = name
-    holder.updated_at = time.monotonic()
-
-
-def get_stage() -> str | None:
-    """Return the current stage label, or None if no holder is bound."""
-    holder = _current_holder.get()
-    return holder.stage if holder is not None else None

hindsight-api-slim/pyproject.toml

@@ -1,220 +0,0 @@
-[build-system]
-requires = ["hatchling"]
-build-backend = "hatchling.build"
-
-[project]
-name = "hindsight-api-slim"
-version = "0.5.4"
-description = "Hindsight: Agent Memory That Works Like Human Memory"
-readme = "README.md"
-requires-python = ">=3.11"
-dependencies = [
-    "asyncpg>=0.29.0",
-    "python-dotenv>=1.0.0",
-    "openai>=1.0.0",
-    "pydantic>=2.0.0",
-    "rich>=13.0.0",
-    "langchain-text-splitters>=0.3.0",
-    "fastapi[standard]>=0.120.3",
-    "uvicorn>=0.38.0",
-    "wsproto>=1.0.0",
-    "sqlalchemy>=2.0.44",
-    "alembic>=1.17.1",
-    "pgvector>=0.4.1",
-    "greenlet>=3.2.4,<3.4.0",  # 3.4.0 lacks arm64 wheels for manylinux_2_41
-    "psycopg2-binary>=2.9.11",
-    "tiktoken>=0.12.0",
-    "httpx>=0.27.0",
-    "PyJWT[crypto]>=2.8.0",
-    "fastmcp>=3.2.0", # SSRF/path traversal, OAuth confused deputy, command injection fixes
-    "python-dateutil>=2.8.0",
-    "opentelemetry-api>=1.20.0",
-    "opentelemetry-sdk>=1.20.0",
-    "opentelemetry-instrumentation-fastapi>=0.41b0",
-    "opentelemetry-exporter-prometheus>=0.41b0",
-    "opentelemetry-exporter-otlp-proto-http>=1.20.0",
-    "opentelemetry-semantic-conventions>=0.41b0",
-    "dateparser>=1.2.2",
-    "google-genai>=1.0.0",
-    "google-auth>=2.0.0",
-    "anthropic>=0.40.0",
-    "typer>=0.9.0",
-    "cohere>=5.0.0",
-    "litellm>=1.83.0", # 1.82.7/1.82.8 had a supply chain compromise (yanked); 1.83.0+ also fixes GHSA-jjhc-v7c2-5hh6 / GHSA-53mr-6c8q-9789
-    "markitdown[pdf,docx,pptx,xlsx,xls]>=0.1.4", # File to markdown conversion
-    "obstore>=0.4.0", # S3/GCS/Azure object storage client (Rust-backed)
-    "winloop>=0.1.0; sys_platform == 'win32'",
-    "uvloop>=0.22.1; sys_platform != 'win32'",
-    # Transitive dependency security fixes
-    "pyasn1>=0.6.3", # DoS vulnerability fix
-    "urllib3>=2.6.3", # Decompression-bomb safeguards bypass fix
-    "langchain-core>=1.2.22", # Path traversal in legacy load_prompt functions fix
-    "langsmith>=0.6.3", # SSRF via tracing header injection fix
-    "protobuf>=6.33.5", # JSON recursion depth bypass fix
-    "pillow>=12.1.1", # Out-of-bounds write in PSD image loading fix
-    "cryptography>=46.0.6", # Incomplete DNS name constraint enforcement fix
-    "filelock>=3.20.1", # TOCTOU race condition fix
-    "authlib>=1.6.9", # Account takeover/JWS header injection vulnerability fix
-    "pyjwt>=2.12.0", # Accepts unknown crit header extensions fix
-    "orjson>=3.11.6", # Unbounded recursion DoS fix
-    "python-multipart>=0.0.22", # Arbitrary file write via non-default configuration fix
-    "tornado>=6.5.5", # DoS multipart/incomplete cookie validation fix
-    "aiohttp>=3.13.3", # Multiple DoS vulnerabilities
-    "pygments>=2.20.0", # ReDoS via inefficient GUID regex fix
-    "claude-agent-sdk>=0.1.27",
-    "boto3>=1.42.74",
-]
-
-[project.optional-dependencies]
-local-ml = [
-    # Local ML models for embeddings/reranking
-    "sentence-transformers>=3.3.0",
-    "transformers>=4.53.0", # Security fixes for ReDoS vulnerabilities
-    "torch>=2.6.0", # CVE fix for remote code execution
-    "einops>=0.8.2",
-    "flashrank>=0.2.0",
-    # Apple Silicon local inference — mlx publishes wheels only for
-    # macOS/Linux, not Windows, so gate on platform to let `uv sync
-    # --all-extras` resolve on win_amd64 runners.
-    "mlx>=0.31.0; sys_platform != 'win32'",
-    "mlx-lm>=0.31.1; sys_platform != 'win32'",
-    "safetensors>=0.6.2",
-]
-local-llm = [
-    # Built-in llama.cpp inference for fully offline operation
-    "llama-cpp-python[server]>=0.3.0",
-    "huggingface-hub>=0.20.0",
-]
-embedded-db = [
-    "pg0-embedded>=0.13.0",
-]
-all = [
-    "hindsight-api-slim[local-ml,embedded-db]",
-]
-test = [
-    "pytest>=7.0.0",
-    "pytest-asyncio>=0.21.0",
-    "pytest-timeout>=2.4.0",
-    "pytest-xdist>=3.0.0",
-    "filelock>=3.20.1",  # TOCTOU race condition fix
-    "testcontainers>=4.0.0",
-]
-
-[project.scripts]
-hindsight-api = "hindsight_api.main:main"
-hindsight-worker = "hindsight_api.worker.main:main"
-hindsight-local-mcp = "hindsight_api.mcp_local:main"
-hindsight-admin = "hindsight_api.admin.cli:main"
-
-[tool.hatch.build.targets.wheel]
-packages = ["hindsight_api"]
-
-[tool.hatch.build.targets.wheel.sources]
-"hindsight_api" = "hindsight_api"
-
-[tool.hatch.build.targets.sdist]
-include = [
-    "hindsight_api/**/*",
-]
-
-[tool.hatch.build]
-include = [
-    "hindsight_api/**/*.py",
-    "hindsight_api/alembic/**/*",
-]
-
-[tool.pytest.ini_options]
-log_cli = true
-log_cli_level = "INFO"
-log_cli_format = "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
-log_cli_date_format = "%Y-%m-%d %H:%M:%S"
-addopts = "--timeout 300 -n 8 --dist loadgroup --durations=10 -v"
-asyncio_mode = "auto"
-asyncio_default_fixture_loop_scope = "function"
-log_auto_indent = true
-filterwarnings = [
-    "ignore:The @wait_container_is_ready decorator is deprecated:DeprecationWarning",
-    "ignore::RuntimeWarning:asyncio",
-]
-
-[dependency-groups]
-dev = [
-    "pytest>=9.0.0",
-    "pytest-asyncio>=1.3.0",
-    "pytest-timeout>=2.4.0",
-    "pytest-xdist>=3.8.0",
-    "pytest-rerunfailures>=15.0",
-    "python-dotenv>=1.2.1",
-    "filelock>=3.20.1",  # TOCTOU race condition fix
-    "ruff>=0.8.0",
-    "ty>=0.0.1",
-    "testcontainers>=4.0.0",
-]
-
-[tool.ruff]
-line-length = 120
-target-version = "py311"
-exclude = [
-    "tests/",
-    "**/tests/",
-]
-
-[tool.ruff.lint]
-select = [
-    "E",      # pycodestyle errors
-    "W",      # pycodestyle warnings
-    "F",      # Pyflakes
-    "I",      # isort
-]
-ignore = [
-    "E501",   # line too long (handled by formatter)
-    "E402",   # module import not at top of file
-    "F401",   # unused import (too noisy during development)
-    "F841",   # unused variable (too noisy during development)
-    "F811",   # redefined while unused
-    "F821",   # undefined name (forward references in type hints)
-]
-
-[tool.ruff.lint.isort]
-known-third-party = ["alembic"]
-
-[tool.ruff.format]
-quote-style = "double"
-indent-style = "space"
-
-[tool.uv]
-# Use explicit index for PyTorch to prevent the pytorch index from serving
-# non-pytorch packages (e.g. markupsafe) with incompatible wheels
-[[tool.uv.index]]
-name = "pytorch-cpu"
-url = "https://download.pytorch.org/whl/cpu"
-explicit = true
-
-[tool.uv.sources]
-# Route torch to the CPU-only PyTorch index; everything else uses PyPI
-torch = { index = "pytorch-cpu" }
-
-[tool.ty]
-# Type checking configuration
-# ty is an extremely fast Python type checker from Astral (same team as ruff/uv)
-
-[tool.ty.environment]
-python-version = "3.11"
-
-[tool.ty.src]
-exclude = [
-    "tests/",
-    "hindsight_api/alembic/",
-]
-
-[tool.ty.rules]
-# Disable noisy rules while keeping important ones
-invalid-argument-type = "ignore"    # False positives with **kwargs patterns
-invalid-return-type = "ignore"      # Often intentional in async code
-invalid-parameter-default = "ignore" # Optional params with None default
-possibly-missing-attribute = "ignore" # Common with Optional types
-invalid-raise = "ignore"            # False positives with exception tracking
-call-non-callable = "ignore"        # False positives with Optional types
-invalid-key = "ignore"              # Pydantic ConfigDict not understood
-invalid-method-override = "ignore"  # Intentional signature differences
-unresolved-reference = "ignore"     # Forward references not always resolved

hindsight-api-slim/tests/test_alembic_dag.py

@@ -1,47 +0,0 @@
-"""Graph-level sanity checks for the Alembic migration DAG.
-
-These tests do not touch a database; they only parse the revision files on
-disk, so they are cheap to run in CI and catch DAG accidents (divergent
-heads, unreachable revisions) at merge time instead of at deploy time.
-"""
-
-from pathlib import Path
-
-from alembic.config import Config
-from alembic.script import ScriptDirectory
-
-
-def _script_directory() -> ScriptDirectory:
-    cfg = Config()
-    script_location = Path(__file__).parent.parent / "hindsight_api" / "alembic"
-    cfg.set_main_option("script_location", str(script_location))
-    return ScriptDirectory.from_config(cfg)
-
-
-def test_single_head() -> None:
-    """The DAG must have exactly one head.
-
-    A second head means a branch was added without a merge revision, which
-    makes ``alembic upgrade head`` (singular) ambiguous and forces the next
-    migration author to orphan whichever head they don't pick as parent.
-    v0.5.3 shipped in exactly that state; this test would have caught it.
-
-    Fix for a new head: ``alembic merge heads -m "<reason>"``.
-    """
-    script = _script_directory()
-    heads = script.get_heads()
-    assert len(heads) == 1, (
-        f"Alembic has {len(heads)} heads ({heads}); expected exactly 1. "
-        "Unify them with ``alembic merge heads -m '<reason>'``."
-    )
-
-
-def test_single_base() -> None:
-    """The DAG must have exactly one base (the initial schema).
-
-    Multiple bases mean disconnected migration trees, which can only happen
-    through manual file edits.
-    """
-    script = _script_directory()
-    bases = script.get_bases()
-    assert len(bases) == 1, f"Alembic has {len(bases)} bases ({bases}); expected exactly 1."

hindsight-api-slim/tests/test_async_batch_retain.py

@@ -1,831 +0,0 @@
-"""Test async batch retain with smart batching and parent-child operations."""
-
-import asyncio
-import json
-import uuid
-
-import pytest
-
-from hindsight_api.extensions import RequestContext
-
-# These tests submit async operations and rely on the engine-owned worker to
-# drain them. test_worker.py drives its own WorkerPoller.claim_batch() against
-# the same pool, so running the two files on different xdist workers causes
-# them to steal each other's pending rows. Share the "worker_tests" group so
-# they serialize on the same xdist process.
-pytestmark = pytest.mark.xdist_group("worker_tests")
-
-
-async def _ensure_bank(pool, bank_id: str) -> None:
-    """Upsert a minimal bank row so FK on async_operations passes."""
-    await pool.execute(
-        "INSERT INTO banks (bank_id, name) VALUES ($1, $2) ON CONFLICT DO NOTHING",
-        bank_id,
-        bank_id,
-    )
-
-
-@pytest.mark.asyncio
-async def test_duplicate_document_ids_rejected_async(memory, request_context):
-    """Test that async retain rejects batches with duplicate document_ids."""
-    bank_id = "test_duplicate_async"
-    contents = [
-        {"content": "First item", "document_id": "doc1"},
-        {"content": "Second item", "document_id": "doc2"},
-        {"content": "Third item", "document_id": "doc1"},  # Duplicate!
-    ]
-
-    # Should raise ValueError due to duplicate document_ids
-    with pytest.raises(ValueError, match="duplicate document_ids.*doc1"):
-        await memory.submit_async_retain(
-            bank_id=bank_id,
-            contents=contents,
-            request_context=request_context,
-        )
-
-
-@pytest.mark.asyncio
-async def test_duplicate_document_ids_rejected_sync(memory, request_context):
-    """Test that sync retain also rejects batches with duplicate document_ids."""
-    bank_id = "test_duplicate_sync"
-    contents = [
-        {"content": "First item", "document_id": "doc1"},
-        {"content": "Second item", "document_id": "doc1"},  # Duplicate!
-    ]
-
-    # Should raise ValueError due to duplicate document_ids
-    with pytest.raises(ValueError, match="duplicate document_ids.*doc1"):
-        await memory.retain_batch_async(
-            bank_id=bank_id,
-            contents=contents,
-            request_context=request_context,
-        )
-
-
-@pytest.mark.asyncio
-async def test_small_async_batch_no_splitting(memory, request_context):
-    """Test that small async batches create parent with single child (simplified code path)."""
-    bank_id = "test_small_async"
-    contents = [{"content": "Alice works at Google", "document_id": f"doc{i}"} for i in range(5)]
-
-    # Calculate total chars (should be well under threshold)
-    total_chars = sum(len(item["content"]) for item in contents)
-    assert total_chars < 10_000, "Test batch should be small"
-
-    # Submit async retain
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-
-    # Verify we got an operation_id back
-    assert "operation_id" in result
-    assert "items_count" in result
-    assert result["items_count"] == 5
-
-    operation_id = result["operation_id"]
-
-    # Wait for task to complete (SyncTaskBackend executes immediately)
-    await asyncio.sleep(0.1)
-
-    # Check operation status
-    status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=operation_id,
-        request_context=request_context,
-    )
-
-    # Should be a parent operation with single child (simplified code path)
-    assert status["status"] == "completed"
-    assert status["operation_type"] == "batch_retain"
-    assert "child_operations" in status
-    assert status["result_metadata"]["num_sub_batches"] == 1  # Single sub-batch
-    assert len(status["child_operations"]) == 1
-    assert status["child_operations"][0]["status"] == "completed"
-
-
-@pytest.mark.asyncio
-async def test_large_async_batch_auto_splits(memory, request_context):
-    """Test that large async batches automatically split into sub-batches with parent operation."""
-    from hindsight_api.engine.memory_engine import count_tokens
-
-    bank_id = "test_large_async"
-
-    # Create a large batch that exceeds the threshold (10k tokens default)
-    # Repeating "A"s gets heavily compressed by tokenizer, use varied content
-    # Use ~22k chars per item = ~5.5k tokens per item, 2 items = ~11k tokens total (exceeds 10k)
-    large_content = "The quick brown fox jumps over the lazy dog. " * 500  # ~22k chars = ~5.5k tokens
-    contents = [{"content": large_content + f" item {i}", "document_id": f"doc{i}"} for i in range(2)]
-
-    # Calculate total tokens (should exceed threshold)
-    total_tokens = sum(count_tokens(item["content"]) for item in contents)
-    assert total_tokens > 10_000, "Test batch should exceed threshold"
-
-    # Submit async retain
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-
-    # Verify we got an operation_id back
-    assert "operation_id" in result
-    assert "items_count" in result
-    assert result["items_count"] == 2
-
-    parent_operation_id = result["operation_id"]
-
-    # Wait for tasks to complete
-    await asyncio.sleep(0.5)
-
-    # Check parent operation status
-    parent_status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=parent_operation_id,
-        request_context=request_context,
-    )
-
-    # Should be a parent operation with children
-    assert parent_status["operation_type"] == "batch_retain"
-    assert "child_operations" in parent_status
-    assert "num_sub_batches" in parent_status["result_metadata"]
-    assert parent_status["result_metadata"]["num_sub_batches"] >= 2  # Should split into at least 2 batches
-    assert parent_status["result_metadata"]["items_count"] == 2
-
-    # Verify child operations
-    child_ops = parent_status["child_operations"]
-    assert len(child_ops) >= 2, "Should have at least 2 child operations"
-
-    # All children should be completed (SyncTaskBackend executes immediately)
-    for child in child_ops:
-        assert child["status"] == "completed"
-        assert child["sub_batch_index"] is not None
-        assert child["items_count"] > 0
-
-    # Parent status should be aggregated as "completed"
-    assert parent_status["status"] == "completed"
-
-
-@pytest.mark.asyncio
-async def test_parent_operation_status_aggregation_pending(memory, request_context):
-    """Test that parent operation shows 'pending' when children are pending."""
-    bank_id = "test_parent_pending"
-    pool = await memory._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    # Manually create a parent operation
-    parent_id = uuid.uuid4()
-    async with pool.acquire() as conn:
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            parent_id,
-            bank_id,
-            "batch_retain",
-            json.dumps({"items_count": 20, "num_sub_batches": 2, "is_parent": True}),
-            "pending",
-        )
-
-        # Create 2 child operations - one completed, one pending
-        child1_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child1_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 1,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "completed",
-        )
-
-        child2_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child2_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 2,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "pending",
-        )
-
-    # Check parent status
-    parent_status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=str(parent_id),
-        request_context=request_context,
-    )
-
-    # Parent should aggregate as "pending" since one child is still pending
-    assert parent_status["status"] == "pending"
-    assert len(parent_status["child_operations"]) == 2
-
-
-@pytest.mark.asyncio
-async def test_parent_operation_status_aggregation_failed(memory, request_context):
-    """Test that parent operation shows 'failed' when any child fails."""
-    bank_id = "test_parent_failed"
-    pool = await memory._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    # Manually create a parent operation
-    parent_id = uuid.uuid4()
-    async with pool.acquire() as conn:
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            parent_id,
-            bank_id,
-            "batch_retain",
-            json.dumps({"items_count": 20, "num_sub_batches": 2, "is_parent": True}),
-            "pending",
-        )
-
-        # Create 2 child operations - one completed, one failed
-        child1_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child1_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 1,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "completed",
-        )
-
-        child2_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status, error_message)
-            VALUES ($1, $2, $3, $4, $5, $6)
-            """,
-            child2_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 2,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "failed",
-            "Test error",
-        )
-
-    # Check parent status
-    parent_status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=str(parent_id),
-        request_context=request_context,
-    )
-
-    # Parent should aggregate as "failed" since one child failed
-    assert parent_status["status"] == "failed"
-    assert len(parent_status["child_operations"]) == 2
-
-    # Verify child with error is included
-    failed_child = [c for c in parent_status["child_operations"] if c["status"] == "failed"][0]
-    assert failed_child["error_message"] == "Test error"
-
-
-@pytest.mark.asyncio
-async def test_parent_operation_status_aggregation_completed(memory, request_context):
-    """Test that parent operation shows 'completed' when all children are completed."""
-    bank_id = "test_parent_completed"
-    pool = await memory._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    # Manually create a parent operation
-    parent_id = uuid.uuid4()
-    async with pool.acquire() as conn:
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            parent_id,
-            bank_id,
-            "batch_retain",
-            json.dumps({"items_count": 20, "num_sub_batches": 2, "is_parent": True}),
-            "pending",
-        )
-
-        # Create 2 child operations - both completed
-        child1_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child1_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 1,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "completed",
-        )
-
-        child2_id = uuid.uuid4()
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child2_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {
-                    "items_count": 10,
-                    "parent_operation_id": str(parent_id),
-                    "sub_batch_index": 2,
-                    "total_sub_batches": 2,
-                }
-            ),
-            "completed",
-        )
-
-    # Check parent status
-    parent_status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=str(parent_id),
-        request_context=request_context,
-    )
-
-    # Parent should aggregate as "completed" since all children are completed
-    assert parent_status["status"] == "completed"
-    assert len(parent_status["child_operations"]) == 2
-    assert all(c["status"] == "completed" for c in parent_status["child_operations"])
-
-
-@pytest.mark.asyncio
-async def test_config_retain_batch_tokens_respected(memory, request_context):
-    """Test that the retain_batch_tokens config setting is respected."""
-    from hindsight_api.config import get_config
-    from hindsight_api.engine.memory_engine import count_tokens
-
-    bank_id = "test_config_batch_tokens"
-    config = get_config()
-
-    # Check that config has the retain_batch_tokens setting
-    assert hasattr(config, "retain_batch_tokens")
-    assert config.retain_batch_tokens > 0
-
-    # Create a batch that's just under the threshold
-    # Use content that produces roughly half the token limit per item
-    content_size = config.retain_batch_tokens * 2  # chars (rough estimate: 1 token ~= 4 chars)
-    contents = [{"content": "A" * content_size, "document_id": f"doc{i}"} for i in range(2)]
-
-    total_tokens = sum(count_tokens(item["content"]) for item in contents)
-    # Should be equal to threshold (boundary case, no splitting since we use > not >=)
-    assert total_tokens <= config.retain_batch_tokens
-
-    # Submit - should NOT split
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-
-    # Wait for completion
-    await asyncio.sleep(0.1)
-
-    # Check status - should be a parent with single child (even for small batches)
-    status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=result["operation_id"],
-        request_context=request_context,
-    )
-
-    # Even small batches use parent-child pattern now (simpler code path)
-    assert "child_operations" in status
-    assert status["result_metadata"]["num_sub_batches"] == 1
-
-
-async def _child_metadata(memory, bank_id: str, parent_operation_id: str, request_context):
-    """Fetch the first child operation's result_metadata for a parent batch_retain."""
-    parent = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=parent_operation_id,
-        request_context=request_context,
-    )
-    assert parent["status"] == "completed", parent
-    assert parent["child_operations"], "expected at least one child operation"
-    child_id = parent["child_operations"][0]["operation_id"]
-    child = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=child_id,
-        request_context=request_context,
-    )
-    return child["result_metadata"]
-
-
-@pytest.mark.asyncio
-async def test_retain_records_user_provided_document_ids(memory, request_context):
-    """User-supplied document_ids land in child op result_metadata.document_ids."""
-    bank_id = "test_doc_ids_user_supplied"
-    d1 = str(uuid.uuid4())
-    d2 = str(uuid.uuid4())
-    contents = [
-        {"content": "User-supplied doc one content.", "document_id": d1},
-        {"content": "User-supplied doc two content.", "document_id": d2},
-    ]
-
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-    await asyncio.sleep(0.2)
-
-    meta = await _child_metadata(memory, bank_id, result["operation_id"], request_context)
-    assert "document_ids" in meta, meta
-    assert set(meta["document_ids"]) == {d1, d2}
-
-
-@pytest.mark.asyncio
-async def test_retain_records_generated_document_id(memory, request_context):
-    """With no document_ids supplied, retain records the single generated id."""
-    bank_id = "test_doc_ids_generated"
-    contents = [
-        {"content": "Generated doc item one."},
-        {"content": "Generated doc item two."},
-    ]
-
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-    await asyncio.sleep(0.2)
-
-    meta = await _child_metadata(memory, bank_id, result["operation_id"], request_context)
-    assert "document_ids" in meta, meta
-    assert isinstance(meta["document_ids"], list)
-    assert len(meta["document_ids"]) == 1
-    # Must be a valid UUID string (generated by the orchestrator)
-    uuid.UUID(meta["document_ids"][0])
-
-
-@pytest.mark.asyncio
-async def test_retain_records_shared_document_id_once(memory, request_context):
-    """Items sharing one document_id record it exactly once (idempotent set-append)."""
-    bank_id = "test_doc_ids_shared"
-    shared = str(uuid.uuid4())
-    # Duplicate per-item doc_ids are rejected up front, so shared-doc mode
-    # is exercised by a single item carrying the id.
-    contents = [{"content": "Shared doc, chunk A.", "document_id": shared}]
-
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-    await asyncio.sleep(0.2)
-
-    meta = await _child_metadata(memory, bank_id, result["operation_id"], request_context)
-    assert meta.get("document_ids") == [shared]
-
-
-@pytest.mark.asyncio
-async def test_get_operation_status_include_payload(memory, request_context):
-    """include_payload=True returns the original submission payload; default omits it."""
-    bank_id = "test_include_payload"
-    contents = [{"content": "Payload roundtrip test item."}]
-
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=contents,
-        request_context=request_context,
-    )
-    await asyncio.sleep(0.2)
-
-    parent = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=result["operation_id"],
-        request_context=request_context,
-    )
-    child_id = parent["child_operations"][0]["operation_id"]
-
-    # Default: no payload
-    without = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=child_id,
-        request_context=request_context,
-    )
-    assert without.get("task_payload") is None
-
-    # With flag: payload populated
-    with_payload = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=child_id,
-        request_context=request_context,
-        include_payload=True,
-    )
-    payload = with_payload.get("task_payload")
-    assert payload is not None, with_payload
-    assert payload.get("bank_id") == bank_id
-    assert payload.get("contents")
-    assert payload["contents"][0]["content"] == "Payload roundtrip test item."
-
-
-@pytest.mark.asyncio
-async def test_operation_status_exposes_retry_count_and_next_retry_at(memory, request_context):
-    """get_operation_status and list_operations return retry_count and next_retry_at.
-
-    Consumers need these to distinguish a freshly-queued pending task from
-    one that's parked for a future retry (e.g. because an extension raised
-    DeferOperation). Without them, "pending" is ambiguous and callers can't
-    render a helpful "deferred until X" state.
-    """
-    from datetime import datetime, timedelta, timezone
-
-    bank_id = "test_retry_fields"
-    result = await memory.submit_async_retain(
-        bank_id=bank_id,
-        contents=[{"content": "retry-fields test item"}],
-        request_context=request_context,
-    )
-    await asyncio.sleep(0.1)
-    parent_id = result["operation_id"]
-    child_id = None
-
-    # Get the child op (the batch_retain parent holds a single child in the
-    # sync/simplified path used by SyncTaskBackend tests).
-    parent_status = await memory.get_operation_status(
-        bank_id=bank_id,
-        operation_id=parent_id,
-        request_context=request_context,
-    )
-    assert "retry_count" in parent_status
-    assert "next_retry_at" in parent_status
-    assert parent_status["retry_count"] == 0
-    # Completed tasks should have next_retry_at cleared on the row (or the
-    # status field doesn't include it meaningfully), so we don't assert a
-    # specific value here — only that the key is present.
-    if parent_status.get("child_operations"):
-        child_id = parent_status["child_operations"][0]["operation_id"]
-
-    # list_operations also exposes both fields
-    listed = await memory.list_operations(
-        bank_id=bank_id,
-        request_context=request_context,
-        limit=10,
-        offset=0,
-    )
-    assert listed["operations"], listed
-    for op in listed["operations"]:
-        assert "retry_count" in op
-        assert "next_retry_at" in op
-        assert isinstance(op["retry_count"], int)
-
-    # Simulate a deferred op: set next_retry_at to 15 min in the future for
-    # the child row directly in the DB, then fetch via the API and confirm
-    # the value round-trips as an ISO-8601 string.
-    if child_id:
-        pool = await memory._get_pool()
-        future = datetime.now(timezone.utc) + timedelta(minutes=15)
-        await pool.execute(
-            "UPDATE async_operations SET status = 'pending', next_retry_at = $1, retry_count = 2 WHERE operation_id = $2",
-            future,
-            uuid.UUID(child_id),
-        )
-        fetched = await memory.get_operation_status(
-            bank_id=bank_id,
-            operation_id=child_id,
-            request_context=request_context,
-        )
-        assert fetched["retry_count"] == 2
-        assert fetched["next_retry_at"] is not None
-        # Round-trip tolerance: within 1 second.
-        parsed = datetime.fromisoformat(fetched["next_retry_at"])
-        assert abs((parsed - future).total_seconds()) < 1.0
-
-
-@pytest.mark.asyncio
-async def test_list_operations_exclude_parents(memory, request_context):
-    """list_operations with exclude_parents=True hides parent batch operations."""
-    bank_id = "test_exclude_parents"
-    pool = await memory._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    # Create a parent operation (is_parent=True)
-    parent_id = uuid.uuid4()
-    child_id = uuid.uuid4()
-    standalone_id = uuid.uuid4()
-
-    async with pool.acquire() as conn:
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            parent_id,
-            bank_id,
-            "batch_retain",
-            json.dumps({"items_count": 10, "num_sub_batches": 1, "is_parent": True}),
-            "completed",
-        )
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            child_id,
-            bank_id,
-            "retain",
-            json.dumps(
-                {"items_count": 10, "parent_operation_id": str(parent_id), "sub_batch_index": 1, "total_sub_batches": 1}
-            ),
-            "completed",
-        )
-        await conn.execute(
-            """
-            INSERT INTO async_operations (operation_id, bank_id, operation_type, result_metadata, status)
-            VALUES ($1, $2, $3, $4, $5)
-            """,
-            standalone_id,
-            bank_id,
-            "consolidation",
-            json.dumps({}),
-            "completed",
-        )
-
-    # Without exclude_parents: all 3 operations visible
-    all_ops = await memory.list_operations(
-        bank_id=bank_id,
-        request_context=request_context,
-        limit=10,
-        offset=0,
-    )
-    all_ids = {op["id"] for op in all_ops["operations"]}
-    assert str(parent_id) in all_ids
-    assert str(child_id) in all_ids
-    assert str(standalone_id) in all_ids
-    assert all_ops["total"] == 3
-
-    # With exclude_parents: parent is hidden
-    filtered_ops = await memory.list_operations(
-        bank_id=bank_id,
-        request_context=request_context,
-        limit=10,
-        offset=0,
-        exclude_parents=True,
-    )
-    filtered_ids = {op["id"] for op in filtered_ops["operations"]}
-    assert str(parent_id) not in filtered_ids
-    assert str(child_id) in filtered_ids
-    assert str(standalone_id) in filtered_ids
-    assert filtered_ops["total"] == 2
-
-
-@pytest.mark.asyncio
-async def test_request_context_retry_count_propagated_to_validator(memory_no_llm_verify, request_context):
-    """_handle_batch_retain forwards the task's _retry_count as
-    RequestContext.retry_count, so validator extensions can compute
-    exponential backoff without querying async_operations themselves.
-    """
-    from hindsight_api.extensions import (
-        OperationValidatorExtension,
-        RecallContext,
-        ReflectContext,
-        RetainContext,
-        ValidationResult,
-    )
-
-    captured: dict[str, int] = {"retry_count": -1}
-
-    class CapturingValidator(OperationValidatorExtension):
-        def __init__(self):
-            super().__init__({})
-
-        async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
-            captured["retry_count"] = ctx.request_context.retry_count
-            return ValidationResult.accept()
-
-        async def validate_recall(self, ctx: RecallContext) -> ValidationResult:
-            return ValidationResult.accept()
-
-        async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
-            return ValidationResult.accept()
-
-    memory_no_llm_verify._operation_validator = CapturingValidator()
-
-    bank_id = f"test-retry-propagate-{uuid.uuid4().hex[:8]}"
-    pool = await memory_no_llm_verify._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    task_dict = {
-        "type": "batch_retain",
-        "bank_id": bank_id,
-        "contents": [{"content": "retry-propagate test"}],
-        "_tenant_id": "default",
-        "_retry_count": 3,  # simulate 3rd retry
-    }
-    await memory_no_llm_verify._handle_batch_retain(task_dict)
-
-    assert captured["retry_count"] == 3, (
-        f"Validator should see retry_count=3 from task_dict['_retry_count']; got {captured['retry_count']}"
-    )
-
-    # Default (missing _retry_count key) must surface as 0, not raise.
-    captured["retry_count"] = -1
-    task_dict_no_retry = {
-        "type": "batch_retain",
-        "bank_id": bank_id,
-        "contents": [{"content": "retry-propagate default test"}],
-        "_tenant_id": "default",
-    }
-    await memory_no_llm_verify._handle_batch_retain(task_dict_no_retry)
-    assert captured["retry_count"] == 0
-
-
-@pytest.mark.asyncio
-async def test_submit_async_operation_leaves_claimable_row_when_submit_task_fails(memory):
-    """Regression for the crash-window orphan bug fixed in #1091.
-
-    Previously, _submit_async_operation INSERTed the async_operations row without
-    task_payload, then called submit_task as a separate step to fill it in. If
-    submit_task failed (crash, timeout, dropped connection) after the INSERT
-    committed, the row was left with task_payload IS NULL and became permanently
-    stuck because the worker claim query filters on task_payload IS NOT NULL.
-
-    With the atomic INSERT, even if submit_task raises afterwards the row is born
-    claimable. This test simulates the crash by forcing submit_task to raise.
-    """
-    bank_id = f"test_orphan_prevention_{uuid.uuid4().hex[:8]}"
-    pool = await memory._get_pool()
-    await _ensure_bank(pool, bank_id)
-
-    async def failing_submit_task(_task_dict):
-        raise RuntimeError("Simulated crash between INSERT and submit_task")
-
-    memory._task_backend.submit_task = failing_submit_task  # type: ignore[method-assign]
-
-    with pytest.raises(RuntimeError, match="Simulated crash"):
-        await memory._submit_async_operation(
-            bank_id=bank_id,
-            operation_type="retain",
-            task_type="batch_retain",
-            task_payload={"contents": [{"content": "hello", "document_id": "d1"}]},
-        )
-
-    rows = await pool.fetch(
-        """
-        SELECT status, task_payload
-        FROM async_operations
-        WHERE bank_id = $1 AND operation_type = 'retain'
-        """,
-        bank_id,
-    )
-    assert len(rows) == 1, f"Expected exactly one retain row for bank_id={bank_id}, got {len(rows)}"
-    row = rows[0]
-    assert row["status"] == "pending"
-    assert row["task_payload"] is not None, (
-        "task_payload must be set atomically by the INSERT — a NULL here means "
-        "the worker claim query (task_payload IS NOT NULL) will never pick this row up"
-    )
-    payload = json.loads(row["task_payload"])
-    assert payload["type"] == "batch_retain"
-    assert payload["bank_id"] == bank_id
-    assert payload["contents"] == [{"content": "hello", "document_id": "d1"}]

hindsight-api-slim/tests/test_audit_log.py

@@ -1,449 +0,0 @@
-"""
-Tests for the audit log feature.
-
-Tests the audit log list, stats, filtering, and pagination endpoints.
-Verifies that audit entries are created for operations when audit logging is enabled.
-"""
-
-import asyncio
-
-import httpx
-import pytest
-import pytest_asyncio
-
-from hindsight_api.api import create_app
-from hindsight_api.config import get_config
-
-
-@pytest_asyncio.fixture
-async def audit_api_client(memory):
-    """Create a test client with audit logging enabled."""
-    # Enable audit logging on the memory engine's audit logger
-    memory._audit_logger._enabled = True
-    memory._audit_logger._allowed_actions = None  # All actions
-
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        yield client
-
-
-@pytest.fixture
-def bank_id():
-    """Provide a unique bank ID for audit tests."""
-    from datetime import datetime
-
-    return f"audit_test_{datetime.now().timestamp()}"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_list_empty(audit_api_client, bank_id):
-    """Test listing audit logs for a bank with no entries returns empty."""
-    # Create the bank first
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    # Small delay for fire-and-forget audit writes
-    await asyncio.sleep(0.5)
-
-    response = await audit_api_client.get(f"/v1/default/banks/{bank_id}/audit-logs")
-    assert response.status_code == 200
-    data = response.json()
-    assert data["bank_id"] == bank_id
-    assert "total" in data
-    assert "items" in data
-    assert "limit" in data
-    assert "offset" in data
-    assert isinstance(data["items"], list)
-
-
-@pytest.mark.asyncio
-async def test_audit_log_created_for_retain(audit_api_client, bank_id):
-    """Test that a retain operation creates an audit log entry."""
-    # Create bank
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    # Perform a retain
-    response = await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories",
-        json={
-            "items": [{"content": "Alice likes cats", "context": "preferences"}],
-        },
-    )
-    assert response.status_code == 200
-
-    # Wait for fire-and-forget audit writes
-    await asyncio.sleep(1.0)
-
-    # List audit logs - should have entries for create_bank and retain
-    response = await audit_api_client.get(f"/v1/default/banks/{bank_id}/audit-logs")
-    assert response.status_code == 200
-    data = response.json()
-    assert data["total"] >= 1
-
-    actions = [item["action"] for item in data["items"]]
-    assert "retain" in actions, f"Expected 'retain' in audit actions, got: {actions}"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_entry_fields(audit_api_client, bank_id):
-    """Test that audit log entries have all expected fields."""
-    # Create bank + recall to generate entries
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories/recall",
-        json={"query": "test query"},
-    )
-
-    await asyncio.sleep(1.0)
-
-    response = await audit_api_client.get(f"/v1/default/banks/{bank_id}/audit-logs")
-    assert response.status_code == 200
-    data = response.json()
-    assert data["total"] >= 1
-
-    # Check the recall entry has all fields
-    recall_entries = [item for item in data["items"] if item["action"] == "recall"]
-    assert len(recall_entries) >= 1, f"Expected recall entry, got actions: {[i['action'] for i in data['items']]}"
-
-    entry = recall_entries[0]
-    assert entry["id"] is not None
-    assert entry["action"] == "recall"
-    assert entry["transport"] == "http"
-    assert entry["bank_id"] == bank_id
-    assert entry["started_at"] is not None
-    assert entry["ended_at"] is not None
-    # Request should contain the recall parameters
-    assert entry["request"] is not None
-    assert "query" in entry["request"]
-    # Response should contain the recall results
-    assert entry["response"] is not None
-
-
-@pytest.mark.asyncio
-async def test_audit_log_filter_by_action(audit_api_client, bank_id):
-    """Test filtering audit logs by action type."""
-    # Create bank and do retain + recall
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-    await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories",
-        json={"items": [{"content": "test content", "context": "test"}]},
-    )
-    await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories/recall",
-        json={"query": "test"},
-    )
-
-    await asyncio.sleep(1.0)
-
-    # Filter by retain only
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"action": "retain"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    for item in data["items"]:
-        assert item["action"] == "retain"
-
-    # Filter by recall only
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"action": "recall"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    for item in data["items"]:
-        assert item["action"] == "recall"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_filter_by_transport(audit_api_client, bank_id):
-    """Test filtering audit logs by transport type."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await asyncio.sleep(0.5)
-
-    # Filter by http transport
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"transport": "http"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    for item in data["items"]:
-        assert item["transport"] == "http"
-
-    # Filter by mcp transport - should be empty (no MCP calls in this test)
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"transport": "mcp"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    assert data["total"] == 0
-
-
-@pytest.mark.asyncio
-async def test_audit_log_filter_by_date_range(audit_api_client, bank_id):
-    """Test filtering audit logs by date range."""
-    from datetime import datetime, timedelta, timezone
-
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await asyncio.sleep(0.5)
-
-    now = datetime.now(timezone.utc)
-
-    # Filter with start_date in the past - should include entries
-    past = (now - timedelta(hours=1)).isoformat()
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"start_date": past},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    assert data["total"] >= 1
-
-    # Filter with start_date in the future - should be empty
-    future = (now + timedelta(hours=1)).isoformat()
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"start_date": future},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    assert data["total"] == 0
-
-
-@pytest.mark.asyncio
-async def test_audit_log_pagination(audit_api_client, bank_id):
-    """Test audit log pagination with limit and offset."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    # Generate multiple audit entries
-    for i in range(5):
-        await audit_api_client.post(
-            f"/v1/default/banks/{bank_id}/memories/recall",
-            json={"query": f"test query {i}"},
-        )
-
-    await asyncio.sleep(1.5)
-
-    # Get first page
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"limit": 2, "offset": 0},
-    )
-    assert response.status_code == 200
-    page1 = response.json()
-    assert len(page1["items"]) == 2
-    assert page1["limit"] == 2
-    assert page1["offset"] == 0
-    assert page1["total"] >= 5  # At least 5 recall + 1 create_bank
-
-    # Get second page
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs",
-        params={"limit": 2, "offset": 2},
-    )
-    assert response.status_code == 200
-    page2 = response.json()
-    assert len(page2["items"]) == 2
-    assert page2["offset"] == 2
-
-    # Entries should be different between pages
-    page1_ids = {item["id"] for item in page1["items"]}
-    page2_ids = {item["id"] for item in page2["items"]}
-    assert page1_ids.isdisjoint(page2_ids), "Pages should not overlap"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_stats(audit_api_client, bank_id):
-    """Test the audit log stats endpoint returns correct structure."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories/recall",
-        json={"query": "stats test"},
-    )
-
-    await asyncio.sleep(1.0)
-
-    # Get stats for last 24h
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs/stats",
-        params={"period": "1d"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    assert data["bank_id"] == bank_id
-    assert data["period"] == "1d"
-    assert data["trunc"] == "day"
-    assert "buckets" in data
-    assert isinstance(data["buckets"], list)
-
-    # Should have at least one bucket with our operations
-    assert len(data["buckets"]) >= 1
-    bucket = data["buckets"][0]
-    assert "time" in bucket
-    assert "actions" in bucket
-    assert "total" in bucket
-    assert bucket["total"] >= 1
-
-
-@pytest.mark.asyncio
-async def test_audit_log_stats_filter_by_action(audit_api_client, bank_id):
-    """Test stats endpoint filters by action."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await audit_api_client.post(
-        f"/v1/default/banks/{bank_id}/memories/recall",
-        json={"query": "test"},
-    )
-
-    await asyncio.sleep(1.0)
-
-    # Stats filtered by recall
-    response = await audit_api_client.get(
-        f"/v1/default/banks/{bank_id}/audit-logs/stats",
-        params={"period": "1d", "action": "recall"},
-    )
-    assert response.status_code == 200
-    data = response.json()
-    for bucket in data["buckets"]:
-        # All actions in buckets should be "recall" only
-        for action_name in bucket["actions"]:
-            assert action_name == "recall"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_stats_periods(audit_api_client, bank_id):
-    """Test stats endpoint supports different periods."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Audit Test Bank"},
-    )
-
-    await asyncio.sleep(0.5)
-
-    for period, expected_trunc in [("1d", "day"), ("7d", "day"), ("30d", "day")]:
-        response = await audit_api_client.get(
-            f"/v1/default/banks/{bank_id}/audit-logs/stats",
-            params={"period": period},
-        )
-        assert response.status_code == 200
-        data = response.json()
-        assert data["period"] == period
-        assert data["trunc"] == expected_trunc
-
-
-@pytest.mark.asyncio
-async def test_audit_log_disabled(memory):
-    """Test that no audit logs are created when audit logging is disabled."""
-    # Ensure audit logging is disabled
-    memory._audit_logger._enabled = False
-
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        from datetime import datetime
-
-        bid = f"audit_disabled_test_{datetime.now().timestamp()}"
-
-        await client.put(f"/v1/default/banks/{bid}", json={"name": "No Audit"})
-        await client.post(
-            f"/v1/default/banks/{bid}/memories/recall",
-            json={"query": "test"},
-        )
-
-        await asyncio.sleep(0.5)
-
-        response = await client.get(f"/v1/default/banks/{bid}/audit-logs")
-        assert response.status_code == 200
-        data = response.json()
-        assert data["total"] == 0, "No audit entries should exist when audit logging is disabled"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_action_allowlist(memory):
-    """Test that only allowed actions are audited when allowlist is set."""
-    memory._audit_logger._enabled = True
-    memory._audit_logger._allowed_actions = frozenset({"recall"})  # Only audit recall
-
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        from datetime import datetime
-
-        bid = f"audit_allowlist_test_{datetime.now().timestamp()}"
-
-        # create_bank should NOT be audited
-        await client.put(f"/v1/default/banks/{bid}", json={"name": "Allowlist Test"})
-        # recall should be audited
-        await client.post(
-            f"/v1/default/banks/{bid}/memories/recall",
-            json={"query": "allowlist test"},
-        )
-
-        await asyncio.sleep(1.0)
-
-        response = await client.get(f"/v1/default/banks/{bid}/audit-logs")
-        assert response.status_code == 200
-        data = response.json()
-        actions = [item["action"] for item in data["items"]]
-        assert "recall" in actions, "recall should be audited"
-        assert "create_bank" not in actions, "create_bank should NOT be audited (not in allowlist)"
-
-
-@pytest.mark.asyncio
-async def test_audit_log_ordered_by_most_recent(audit_api_client, bank_id):
-    """Test that audit logs are returned ordered by most recent first."""
-    await audit_api_client.put(
-        f"/v1/default/banks/{bank_id}",
-        json={"name": "Order Test Bank"},
-    )
-
-    for i in range(3):
-        await audit_api_client.post(
-            f"/v1/default/banks/{bank_id}/memories/recall",
-            json={"query": f"order test {i}"},
-        )
-        await asyncio.sleep(0.2)  # Small gap between requests
-
-    await asyncio.sleep(1.0)
-
-    response = await audit_api_client.get(f"/v1/default/banks/{bank_id}/audit-logs")
-    assert response.status_code == 200
-    data = response.json()
-
-    # Check descending order by started_at
-    timestamps = [item["started_at"] for item in data["items"] if item["started_at"]]
-    assert timestamps == sorted(timestamps, reverse=True), "Audit logs should be ordered most recent first"

hindsight-api-slim/tests/test_bank_stats.py

@@ -1,248 +0,0 @@
-"""
-Tests for the bank stats endpoint and the memories-timeseries endpoint.
-
-Covers the new fields exposed by GET /v1/default/banks/{bank_id}/stats
-(operations_by_status) and the new endpoint
-GET /v1/default/banks/{bank_id}/stats/memories-timeseries.
-"""
-import uuid
-from datetime import datetime
-
-import httpx
-import pytest
-import pytest_asyncio
-
-from hindsight_api.api import create_app
-
-
-@pytest_asyncio.fixture
-async def api_client(memory):
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        yield client
-
-
-@pytest.fixture
-def test_bank_id():
-    return f"stats_test_{datetime.now().timestamp()}"
-
-
-async def _insert_memory(memory, bank_id: str, text: str, *, failed: bool = False) -> str:
-    """Insert a single experience memory, optionally marked as consolidation-failed."""
-    mem_id = uuid.uuid4()
-    async with memory._pool.acquire() as conn:
-        await conn.execute(
-            """
-            INSERT INTO memory_units (id, bank_id, text, fact_type, created_at, consolidation_failed_at)
-            VALUES ($1, $2, $3, 'experience', now(), CASE WHEN $4 THEN now() ELSE NULL END)
-            """,
-            mem_id,
-            bank_id,
-            text,
-            failed,
-        )
-    return str(mem_id)
-
-
-@pytest.mark.asyncio
-async def test_bank_stats_exposes_operations_by_status(api_client, test_bank_id):
-    """/stats should return operations_by_status with all finished operations."""
-    try:
-        # Kick off a retain so at least one completed operation exists.
-        response = await api_client.post(
-            f"/v1/default/banks/{test_bank_id}/memories",
-            json={"items": [{"content": "Alice is a software engineer.", "context": "team"}]},
-        )
-        assert response.status_code == 200
-
-        response = await api_client.get(f"/v1/default/banks/{test_bank_id}/stats")
-        assert response.status_code == 200
-        stats = response.json()
-
-        assert "operations_by_status" in stats
-        assert isinstance(stats["operations_by_status"], dict)
-        # A synchronous retain finishes as "completed".
-        assert stats["operations_by_status"].get("completed", 0) >= 1
-        # pending/failed counters should still be present as scalar mirrors.
-        assert stats["pending_operations"] == stats["operations_by_status"].get("pending", 0)
-        assert stats["failed_operations"] == stats["operations_by_status"].get("failed", 0)
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-@pytest.mark.parametrize(
-    "period,expected_count,expected_trunc",
-    [
-        ("1h", 60, "minute"),
-        ("12h", 12, "hour"),
-        ("1d", 24, "hour"),
-        ("7d", 7, "day"),
-        ("30d", 30, "day"),
-        ("90d", 90, "day"),
-    ],
-)
-async def test_memories_timeseries_periods(
-    api_client, test_bank_id, period, expected_count, expected_trunc
-):
-    """Every period must return the full expected bucket count and trunc."""
-    try:
-        response = await api_client.post(
-            f"/v1/default/banks/{test_bank_id}/memories",
-            json={"items": [{"content": "Bob works on infrastructure.", "context": "team"}]},
-        )
-        assert response.status_code == 200
-
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/stats/memories-timeseries",
-            params={"period": period},
-        )
-        assert response.status_code == 200
-        body = response.json()
-
-        assert body["bank_id"] == test_bank_id
-        assert body["period"] == period
-        assert body["trunc"] == expected_trunc
-        assert len(body["buckets"]) == expected_count
-
-        for bucket in body["buckets"]:
-            assert "time" in bucket
-            # Bucket `time` must serialize as a tz-aware ISO (ending in `+00:00` or `Z`).
-            # A naive ISO (`2026-04-18T00:00:00`) would be parsed as local time by
-            # `new Date()` per ECMA-262, shifting the chart by the browser's timezone.
-            assert bucket["time"].endswith("+00:00") or bucket["time"].endswith("Z"), (
-                f"bucket time must include UTC offset, got {bucket['time']!r}"
-            )
-            assert bucket["world"] >= 0
-            assert bucket["experience"] >= 0
-            assert bucket["observation"] >= 0
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_memories_timeseries_invalid_period_falls_back(api_client, test_bank_id):
-    """An unknown period must fall back to the 7d default."""
-    try:
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/stats/memories-timeseries",
-            params={"period": "nonsense"},
-        )
-        assert response.status_code == 200
-        body = response.json()
-        assert body["period"] == "7d"
-        assert body["trunc"] == "day"
-        assert len(body["buckets"]) == 7
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_memories_timeseries_empty_bank_returns_zero_filled_buckets(
-    api_client, test_bank_id
-):
-    """A bank with no memories must still return the full zero-filled bucket set."""
-    try:
-        # Ensure the bank exists.
-        response = await api_client.get(f"/v1/default/banks/{test_bank_id}/profile")
-        assert response.status_code == 200
-
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/stats/memories-timeseries",
-            params={"period": "7d"},
-        )
-        assert response.status_code == 200
-        body = response.json()
-        assert len(body["buckets"]) == 7
-        for bucket in body["buckets"]:
-            assert bucket["world"] == 0
-            assert bucket["experience"] == 0
-            assert bucket["observation"] == 0
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_memories_timeseries_reflects_retained_memories(api_client, test_bank_id):
-    """Freshly-retained memories must show up in today's bucket counts."""
-    try:
-        response = await api_client.post(
-            f"/v1/default/banks/{test_bank_id}/memories",
-            json={
-                "items": [
-                    {"content": "Alice is a software engineer.", "context": "team"},
-                    {"content": "Bob works on infrastructure.", "context": "team"},
-                ]
-            },
-        )
-        assert response.status_code == 200
-
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/stats/memories-timeseries",
-            params={"period": "7d"},
-        )
-        assert response.status_code == 200
-        body = response.json()
-        totals = sum(b["world"] + b["experience"] + b["observation"] for b in body["buckets"])
-        assert totals >= 2, "expected at least two memories across all buckets"
-
-        # Those memories should land in the most-recent bucket.
-        latest = body["buckets"][-1]
-        assert latest["world"] + latest["experience"] + latest["observation"] >= 2
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_bank_stats_reports_failed_consolidation(api_client, memory, test_bank_id):
-    """/stats must surface the count of memories with consolidation_failed_at set."""
-    try:
-        await _insert_memory(memory, test_bank_id, "Alice failed 1.", failed=True)
-        await _insert_memory(memory, test_bank_id, "Alice failed 2.", failed=True)
-        await _insert_memory(memory, test_bank_id, "Alice pending.", failed=False)
-
-        response = await api_client.get(f"/v1/default/banks/{test_bank_id}/stats")
-        assert response.status_code == 200
-        stats = response.json()
-
-        assert stats["failed_consolidation"] == 2
-        # The two failed memories also count as "not-yet-consolidated".
-        assert stats["pending_consolidation"] >= 3
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_list_memories_filter_by_consolidation_state_failed(api_client, memory, test_bank_id):
-    """?consolidation_state=failed returns only memories with consolidation_failed_at set."""
-    try:
-        failed_id = await _insert_memory(memory, test_bank_id, "Broken item.", failed=True)
-        await _insert_memory(memory, test_bank_id, "Healthy item.", failed=False)
-
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/memories/list",
-            params={"consolidation_state": "failed"},
-        )
-        assert response.status_code == 200
-        body = response.json()
-
-        ids = [item["id"] for item in body["items"]]
-        assert failed_id in ids
-        assert body["total"] == 1
-        assert body["items"][0]["consolidation_failed_at"] is not None
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")
-
-
-@pytest.mark.asyncio
-async def test_list_memories_filter_by_consolidation_state_rejects_unknown(api_client, test_bank_id):
-    """An invalid consolidation_state value must return a 400 (not 500)."""
-    try:
-        response = await api_client.get(
-            f"/v1/default/banks/{test_bank_id}/memories/list",
-            params={"consolidation_state": "bogus"},
-        )
-        assert response.status_code == 400
-    finally:
-        await api_client.delete(f"/v1/default/banks/{test_bank_id}")

hindsight-api-slim/tests/test_bank_template_configurable_fields.py

@@ -1,115 +0,0 @@
-"""Verify that BankTemplateConfig exposes every hierarchical field that
-_CONFIGURABLE_FIELDS already accepts at the engine layer.
-
-This test guards the fix for the gap described in the upstream PR title
-"fix(bank-template): align BankTemplateConfig with _CONFIGURABLE_FIELDS".
-Each new field is POSTed through /v1/default/banks/{id}/import and then
-read back via the bank-config endpoint; assertion is that the applied
-value round-trips through the engine.
-
-Runs via: uv run pytest tests/test_bank_template_configurable_fields.py -v
-
-The api_client fixture (shared with tests/test_bank_templates.py) wraps
-create_app(memory, initialize_memory=False) in an httpx.ASGITransport
-with base_url http://test — in-process, no network, no tenant extension.
-Copy the fixture inline here so the test file does not depend on a
-conftest we do not ship in the patch.
-"""
-
-from __future__ import annotations
-
-from datetime import datetime
-
-import httpx
-import pytest
-import pytest_asyncio
-
-from hindsight_api.api import create_app
-from hindsight_api.api.http import BankTemplateConfig
-
-# Each tuple is (field_name, applied_value). Values chosen to differ
-# visibly from defaults so round-trip bugs surface.
-NEW_FIELDS: list[tuple[str, object]] = [
-    ("retain_default_strategy", "strategy-a"),
-    ("retain_strategies", {"strategy-a": {"mode": "concise", "max_tokens": 512}}),
-    ("retain_chunk_batch_size", 7),
-    ("mcp_enabled_tools", ["list_banks", "get_bank_profile"]),
-    ("consolidation_llm_batch_size", 11),
-    ("consolidation_source_facts_max_tokens", 2048),
-    ("consolidation_source_facts_max_tokens_per_observation", 256),
-    ("max_observations_per_scope", 13),
-    ("reflect_source_facts_max_tokens", 4096),
-    ("llm_gemini_safety_settings", [{"category": "HARM_CATEGORY_HARASSMENT", "threshold": "BLOCK_NONE"}]),
-    ("recall_budget_function", "adaptive"),
-    ("recall_budget_fixed_low", 50),
-    ("recall_budget_fixed_mid", 250),
-    ("recall_budget_fixed_high", 800),
-    ("recall_budget_adaptive_low", 0.05),
-    ("recall_budget_adaptive_mid", 0.1),
-    ("recall_budget_adaptive_high", 0.4),
-    ("recall_budget_min", 30),
-    ("recall_budget_max", 1500),
-]
-
-
-@pytest_asyncio.fixture
-async def api_client(memory):
-    """Matches the fixture in tests/test_bank_templates.py — in-process
-    ASGI test client, no tenant extension, no auth."""
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        yield client
-
-
-@pytest.fixture
-def bank_id():
-    return f"tmpl_config_{datetime.now().timestamp()}"
-
-
-def test_bank_template_config_declares_every_configurable_field():
-    """Pydantic-level guard: every field in NEW_FIELDS must be a declared
-    attribute of BankTemplateConfig so get_config_updates() picks it up."""
-    declared = set(BankTemplateConfig.model_fields.keys())
-    missing = [name for name, _ in NEW_FIELDS if name not in declared]
-    assert not missing, f"BankTemplateConfig missing fields: {missing}"
-
-
-@pytest.mark.asyncio
-@pytest.mark.parametrize("field_name,applied_value", NEW_FIELDS, ids=[n for n, _ in NEW_FIELDS])
-async def test_new_field_round_trips_through_import(
-    api_client: httpx.AsyncClient,
-    bank_id: str,
-    field_name: str,
-    applied_value: object,
-):
-    """POST a minimal manifest with one new field set, then read bank
-    config back and assert the value made it through.
-
-    Bank config response shape per upstream's test_import_applies_config:
-    top-level keys are resolved hierarchical config; per-bank overrides
-    live under config["overrides"][<field>]. Assert on the override slot.
-    """
-    unique_bank_id = f"{bank_id}_{field_name}"
-    manifest = {
-        "version": "1",
-        "bank": {field_name: applied_value},
-    }
-
-    resp = await api_client.post(
-        f"/v1/default/banks/{unique_bank_id}/import",
-        json=manifest,
-    )
-    assert resp.status_code == 200, resp.text
-
-    # Read bank config back — field must reflect the applied value
-    # under the "overrides" slot, matching upstream's own test shape.
-    read = await api_client.get(f"/v1/default/banks/{unique_bank_id}/config")
-    assert read.status_code == 200, read.text
-    config = read.json()
-    overrides = config.get("overrides", {})
-    assert overrides.get(field_name) == applied_value, (
-        f"round-trip mismatch for {field_name}: "
-        f"sent {applied_value!r}, got {overrides.get(field_name)!r} "
-        f"(full overrides: {overrides!r})"
-    )

hindsight-api-slim/tests/test_bank_templates.py

@@ -1,779 +0,0 @@
-"""Integration tests for bank template import/export endpoints."""
-
-import pytest
-import pytest_asyncio
-import httpx
-from datetime import datetime
-from hindsight_api.api import create_app
-
-
-@pytest_asyncio.fixture
-async def api_client(memory):
-    """Create an async test client for the FastAPI app."""
-    app = create_app(memory, initialize_memory=False)
-    transport = httpx.ASGITransport(app=app)
-    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
-        yield client
-
-
-@pytest.fixture
-def bank_id():
-    return f"template_test_{datetime.now().timestamp()}"
-
-
-@pytest.fixture
-def sample_template():
-    return {
-        "version": "1",
-        "bank": {
-            "reflect_mission": "Test mission for reflect",
-            "retain_mission": "Extract test data carefully",
-            "retain_extraction_mode": "verbose",
-            "disposition_empathy": 5,
-            "disposition_skepticism": 2,
-            "enable_observations": True,
-            "observations_mission": "Track test patterns",
-        },
-        "mental_models": [
-            {
-                "id": "test-model-one",
-                "name": "Test Model One",
-                "source_query": "What are the key patterns?",
-                "tags": ["test"],
-                "max_tokens": 1024,
-                "trigger": {"refresh_after_consolidation": True},
-            },
-            {
-                "id": "test-model-two",
-                "name": "Test Model Two",
-                "source_query": "What are the common issues?",
-            },
-        ],
-        "directives": [
-            {
-                "name": "Be concise",
-                "content": "Always respond concisely.",
-                "priority": 10,
-            },
-            {
-                "name": "Use examples",
-                "content": "Include examples when explaining concepts.",
-                "tags": ["style"],
-            },
-        ],
-    }
-
-
-class TestImportValidation:
-    """Test template manifest validation."""
-
-    @pytest.mark.asyncio
-    async def test_import_dry_run_valid(self, api_client, bank_id, sample_template):
-        """dry_run=true with a valid manifest returns what would happen."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import?dry_run=true",
-            json=sample_template,
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["dry_run"] is True
-        assert data["config_applied"] is True
-        assert set(data["mental_models_created"]) == {"test-model-one", "test-model-two"}
-        assert set(data["directives_created"]) == {"Be concise", "Use examples"}
-
-    @pytest.mark.asyncio
-    async def test_import_invalid_version(self, api_client, bank_id):
-        """Reject manifest with unsupported version."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={"version": "999"},
-        )
-        assert resp.status_code == 400
-
-    @pytest.mark.asyncio
-    async def test_import_invalid_extraction_mode(self, api_client, bank_id):
-        """Semantic validation catches bad extraction mode."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {"retain_extraction_mode": "invalid_mode"},
-            },
-        )
-        assert resp.status_code == 400
-        assert "retain_extraction_mode" in resp.json()["detail"]
-
-    @pytest.mark.asyncio
-    async def test_import_custom_instructions_without_custom_mode(self, api_client, bank_id):
-        """Validate that custom_instructions requires extraction_mode=custom."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {
-                    "retain_extraction_mode": "verbose",
-                    "retain_custom_instructions": "some custom prompt",
-                },
-            },
-        )
-        assert resp.status_code == 400
-        assert "retain_custom_instructions" in resp.json()["detail"]
-
-    @pytest.mark.asyncio
-    async def test_import_duplicate_mental_model_ids(self, api_client, bank_id):
-        """Reject manifest with duplicate mental model IDs."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {"id": "dup-id", "name": "First", "source_query": "q1"},
-                    {"id": "dup-id", "name": "Second", "source_query": "q2"},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-
-    @pytest.mark.asyncio
-    async def test_import_duplicate_directive_names(self, api_client, bank_id):
-        """Reject manifest with duplicate directive names."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {"name": "Same Name", "content": "First"},
-                    {"name": "Same Name", "content": "Second"},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-
-    @pytest.mark.asyncio
-    async def test_import_missing_mental_model_id(self, api_client, bank_id):
-        """Mental model without id is rejected."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {"name": "No ID Model", "source_query": "test query"},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-
-    @pytest.mark.asyncio
-    async def test_import_invalid_mental_model_id_format(self, api_client, bank_id):
-        """Mental model with invalid ID format is rejected."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {"id": "UPPERCASE-NOT-ALLOWED", "name": "Bad", "source_query": "q"},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-
-    @pytest.mark.asyncio
-    async def test_import_empty_manifest(self, api_client, bank_id):
-        """Import with no bank or mental_models is valid (no-op)."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={"version": "1"},
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is False
-        assert data["mental_models_created"] == []
-        assert data["directives_created"] == []
-
-    @pytest.mark.asyncio
-    async def test_import_empty_mental_model_name(self, api_client, bank_id):
-        """Semantic validation catches empty mental model name."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {"id": "test-mm", "name": "  ", "source_query": "q"},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-        assert "name" in resp.json()["detail"]
-
-    @pytest.mark.asyncio
-    async def test_import_empty_directive_content(self, api_client, bank_id):
-        """Semantic validation catches empty directive content."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {"name": "Bad Directive", "content": "  "},
-                ],
-            },
-        )
-        assert resp.status_code == 400
-        assert "content" in resp.json()["detail"]
-
-
-class TestImportApply:
-    """Test that import actually applies config, mental models, and directives."""
-
-    @pytest.mark.asyncio
-    async def test_import_applies_config(self, api_client, bank_id):
-        """Import with bank config applies config overrides on a new bank."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {
-                    "reflect_mission": "Imported mission",
-                    "disposition_empathy": 4,
-                },
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is True
-        assert data["dry_run"] is False
-
-        # Verify config was actually applied
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        assert config_resp.status_code == 200
-        config = config_resp.json()
-        assert config["overrides"]["reflect_mission"] == "Imported mission"
-        assert config["overrides"]["disposition_empathy"] == 4
-
-    @pytest.mark.asyncio
-    async def test_import_into_existing_bank(self, api_client, bank_id):
-        """Import into an already-existing bank applies config and creates resources."""
-        # Pre-create the bank
-        await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {"reflect_mission": "Existing bank mission"},
-                "mental_models": [
-                    {"id": "existing-bank-mm", "name": "MM", "source_query": "q"},
-                ],
-                "directives": [
-                    {"name": "Existing Bank Directive", "content": "Be helpful"},
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is True
-        assert "existing-bank-mm" in data["mental_models_created"]
-        assert "Existing Bank Directive" in data["directives_created"]
-
-        # Verify everything exists
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        assert config_resp.json()["overrides"]["reflect_mission"] == "Existing bank mission"
-
-        mm_resp = await api_client.get(f"/v1/default/banks/{bank_id}/mental-models/existing-bank-mm")
-        assert mm_resp.status_code == 200
-
-        dir_resp = await api_client.get(f"/v1/default/banks/{bank_id}/directives")
-        assert dir_resp.status_code == 200
-        names = [d["name"] for d in dir_resp.json()["items"]]
-        assert "Existing Bank Directive" in names
-
-    @pytest.mark.asyncio
-    async def test_import_creates_mental_models(self, api_client, bank_id):
-        """Import creates mental models and returns operation IDs."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {
-                        "id": "import-mm-1",
-                        "name": "Imported Model",
-                        "source_query": "What patterns exist?",
-                        "tags": ["imported"],
-                    },
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert "import-mm-1" in data["mental_models_created"]
-        assert len(data["operation_ids"]) == 1
-
-        # Verify mental model exists
-        mm_resp = await api_client.get(f"/v1/default/banks/{bank_id}/mental-models/import-mm-1")
-        assert mm_resp.status_code == 200
-        mm = mm_resp.json()
-        assert mm["name"] == "Imported Model"
-        assert mm["source_query"] == "What patterns exist?"
-        assert mm["tags"] == ["imported"]
-
-    @pytest.mark.asyncio
-    async def test_import_updates_existing_mental_models(self, api_client, bank_id):
-        """Re-importing updates existing mental models matched by ID."""
-        # First import
-        await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {
-                        "id": "reusable-mm",
-                        "name": "Original Name",
-                        "source_query": "Original query",
-                    },
-                ],
-            },
-        )
-
-        # Second import with same ID but different content
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {
-                        "id": "reusable-mm",
-                        "name": "Updated Name",
-                        "source_query": "Updated query",
-                    },
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert "reusable-mm" in data["mental_models_updated"]
-        assert data["mental_models_created"] == []
-
-        # Verify update
-        mm_resp = await api_client.get(f"/v1/default/banks/{bank_id}/mental-models/reusable-mm")
-        assert mm_resp.status_code == 200
-        mm = mm_resp.json()
-        assert mm["name"] == "Updated Name"
-        assert mm["source_query"] == "Updated query"
-
-    @pytest.mark.asyncio
-    async def test_import_creates_directives(self, api_client, bank_id):
-        """Import creates directives."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {
-                        "name": "Test Directive",
-                        "content": "Always be helpful and precise.",
-                        "priority": 5,
-                        "tags": ["test"],
-                    },
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert "Test Directive" in data["directives_created"]
-        assert data["directives_updated"] == []
-
-        # Verify directive exists
-        dir_resp = await api_client.get(f"/v1/default/banks/{bank_id}/directives")
-        assert dir_resp.status_code == 200
-        items = dir_resp.json()["items"]
-        assert len(items) == 1
-        assert items[0]["name"] == "Test Directive"
-        assert items[0]["content"] == "Always be helpful and precise."
-        assert items[0]["priority"] == 5
-        assert items[0]["tags"] == ["test"]
-
-    @pytest.mark.asyncio
-    async def test_import_updates_existing_directives(self, api_client, bank_id):
-        """Re-importing updates existing directives matched by name."""
-        # First import
-        await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {"name": "Reusable Directive", "content": "Original content", "priority": 1},
-                ],
-            },
-        )
-
-        # Second import with same name but different content
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {"name": "Reusable Directive", "content": "Updated content", "priority": 10},
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert "Reusable Directive" in data["directives_updated"]
-        assert data["directives_created"] == []
-
-        # Verify update
-        dir_resp = await api_client.get(f"/v1/default/banks/{bank_id}/directives")
-        items = dir_resp.json()["items"]
-        directive = [d for d in items if d["name"] == "Reusable Directive"][0]
-        assert directive["content"] == "Updated content"
-        assert directive["priority"] == 10
-
-    @pytest.mark.asyncio
-    async def test_import_config_only(self, api_client, bank_id):
-        """Import with only bank config (no mental_models or directives) works."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {"retain_extraction_mode": "verbose"},
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is True
-        assert data["mental_models_created"] == []
-        assert data["directives_created"] == []
-        assert data["operation_ids"] == []
-
-    @pytest.mark.asyncio
-    async def test_import_mental_models_only(self, api_client, bank_id):
-        """Import with only mental_models works."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "mental_models": [
-                    {"id": "mm-only", "name": "MM Only", "source_query": "test"},
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is False
-        assert "mm-only" in data["mental_models_created"]
-        assert data["directives_created"] == []
-
-    @pytest.mark.asyncio
-    async def test_import_directives_only(self, api_client, bank_id):
-        """Import with only directives works."""
-        resp = await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "directives": [
-                    {"name": "Dir Only", "content": "test directive"},
-                ],
-            },
-        )
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["config_applied"] is False
-        assert data["mental_models_created"] == []
-        assert "Dir Only" in data["directives_created"]
-
-
-class TestExport:
-    """Test bank template export."""
-
-    @pytest.mark.asyncio
-    async def test_export_empty_bank(self, api_client, bank_id):
-        """Export a bank with no overrides returns minimal manifest."""
-        # Create bank
-        await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-
-        resp = await api_client.get(f"/v1/default/banks/{bank_id}/export")
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["version"] == "1"
-        assert data["bank"] is None
-        assert data["mental_models"] is None
-        assert data["directives"] is None
-
-    @pytest.mark.asyncio
-    async def test_export_after_import(self, api_client, bank_id):
-        """Export after import returns the imported config, mental models, and directives."""
-        template = {
-            "version": "1",
-            "bank": {
-                "reflect_mission": "Roundtrip mission",
-                "disposition_empathy": 3,
-            },
-            "mental_models": [
-                {
-                    "id": "roundtrip-mm",
-                    "name": "Roundtrip Model",
-                    "source_query": "What happened?",
-                    "tags": ["roundtrip"],
-                    "max_tokens": 512,
-                },
-            ],
-            "directives": [
-                {
-                    "name": "Roundtrip Directive",
-                    "content": "Be thorough.",
-                    "priority": 3,
-                    "tags": ["roundtrip"],
-                },
-            ],
-        }
-
-        # Import
-        import_resp = await api_client.post(f"/v1/default/banks/{bank_id}/import", json=template)
-        assert import_resp.status_code == 200
-
-        # Export
-        resp = await api_client.get(f"/v1/default/banks/{bank_id}/export")
-        assert resp.status_code == 200
-        data = resp.json()
-
-        assert data["version"] == "1"
-        assert data["bank"]["reflect_mission"] == "Roundtrip mission"
-        assert data["bank"]["disposition_empathy"] == 3
-
-        assert len(data["mental_models"]) == 1
-        mm = data["mental_models"][0]
-        assert mm["id"] == "roundtrip-mm"
-        assert mm["name"] == "Roundtrip Model"
-        assert mm["source_query"] == "What happened?"
-        assert mm["tags"] == ["roundtrip"]
-        assert mm["max_tokens"] == 512
-
-        assert len(data["directives"]) == 1
-        d = data["directives"][0]
-        assert d["name"] == "Roundtrip Directive"
-        assert d["content"] == "Be thorough."
-        assert d["priority"] == 3
-        assert d["tags"] == ["roundtrip"]
-
-    @pytest.mark.asyncio
-    async def test_export_reimport_roundtrip(self, api_client, bank_id):
-        """Exported manifest can be re-imported into a new bank."""
-        # Set up source bank
-        await api_client.post(
-            f"/v1/default/banks/{bank_id}/import",
-            json={
-                "version": "1",
-                "bank": {"retain_mission": "Roundtrip test"},
-                "mental_models": [
-                    {"id": "rt-mm", "name": "RT Model", "source_query": "test query"},
-                ],
-                "directives": [
-                    {"name": "RT Directive", "content": "test directive"},
-                ],
-            },
-        )
-
-        # Export
-        export_resp = await api_client.get(f"/v1/default/banks/{bank_id}/export")
-        assert export_resp.status_code == 200
-        exported = export_resp.json()
-
-        # Import into a new bank
-        new_bank_id = f"{bank_id}_clone"
-        import_resp = await api_client.post(
-            f"/v1/default/banks/{new_bank_id}/import",
-            json=exported,
-        )
-        assert import_resp.status_code == 200
-        data = import_resp.json()
-        assert data["config_applied"] is True
-        assert "rt-mm" in data["mental_models_created"]
-        assert "RT Directive" in data["directives_created"]
-
-    @pytest.mark.asyncio
-    async def test_export_nonexistent_bank(self, api_client):
-        """Export from a nonexistent bank returns the bank with defaults (auto-created)."""
-        resp = await api_client.get("/v1/default/banks/nonexistent-export-test/export")
-        # get_bank_profile auto-creates, so this returns a valid empty manifest
-        assert resp.status_code == 200
-        data = resp.json()
-        assert data["version"] == "1"
-
-
-class TestDefaultBankTemplateEnvVar:
-    """Tests for HINDSIGHT_API_DEFAULT_BANK_TEMPLATE — a server-level env var
-    whose manifest is applied automatically to every newly-created bank."""
-
-    @pytest.fixture
-    def default_template(self):
-        return {
-            "version": "1",
-            "bank": {
-                "reflect_mission": "default-env-mission",
-                "retain_extraction_mode": "verbose",
-                "disposition_empathy": 5,
-                "disposition_skepticism": 1,
-            },
-            "mental_models": [
-                {
-                    "id": "default-env-model",
-                    "name": "Default Env Model",
-                    "source_query": "What is the default?",
-                },
-            ],
-            "directives": [
-                {
-                    "name": "Default Env Directive",
-                    "content": "Follow the default behavior.",
-                    "priority": 7,
-                },
-            ],
-        }
-
-    @pytest.fixture
-    def _patched_default_template(self, monkeypatch, default_template):
-        """Install the default template on the already-initialized global config.
-
-        We can't rely on env-var resolution here: MemoryEngine (and its
-        ConfigResolver) snapshot the global config at fixture init time.
-        Patching the field directly keeps the test deterministic while still
-        exercising the same code path that reads `get_config().default_bank_template`.
-        """
-        from hindsight_api.config import _get_raw_config
-
-        raw = _get_raw_config()
-        monkeypatch.setattr(raw, "default_bank_template", default_template)
-        yield default_template
-
-    @pytest.mark.asyncio
-    async def test_default_template_applied_on_new_bank(
-        self, api_client, bank_id, _patched_default_template
-    ):
-        """Creating a new bank applies the default template (config + mental models + directives)."""
-        # Trigger bank auto-creation via GET profile
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        assert resp.status_code == 200
-
-        # Config from template should be present as bank overrides
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        assert config_resp.status_code == 200
-        overrides = config_resp.json()["overrides"]
-        assert overrides["reflect_mission"] == "default-env-mission"
-        assert overrides["retain_extraction_mode"] == "verbose"
-        assert overrides["disposition_empathy"] == 5
-        assert overrides["disposition_skepticism"] == 1
-
-        # Mental model from template should exist
-        mm_resp = await api_client.get(f"/v1/default/banks/{bank_id}/mental-models/default-env-model")
-        assert mm_resp.status_code == 200
-        assert mm_resp.json()["name"] == "Default Env Model"
-
-        # Directive from template should exist
-        dir_resp = await api_client.get(f"/v1/default/banks/{bank_id}/directives")
-        assert dir_resp.status_code == 200
-        names = [d["name"] for d in dir_resp.json()["items"]]
-        assert "Default Env Directive" in names
-
-    @pytest.mark.asyncio
-    async def test_default_template_overrides_env_config_defaults(
-        self, api_client, bank_id, monkeypatch, default_template
-    ):
-        """Fields set by the default template override server-level env-var defaults.
-
-        We point both HINDSIGHT_API_RETAIN_EXTRACTION_MODE (env) and the
-        default template at different values, then confirm the template wins
-        via the per-bank config overrides layer (highest precedence).
-        """
-        from hindsight_api.config import _get_raw_config
-
-        raw = _get_raw_config()
-        # Simulate an env-level default of "concise", overridden by a template that sets "verbose".
-        monkeypatch.setattr(raw, "retain_extraction_mode", "concise")
-        monkeypatch.setattr(raw, "default_bank_template", default_template)
-
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        assert resp.status_code == 200
-
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        overrides = config_resp.json()["overrides"]
-        # Template value wins at the bank-override layer.
-        assert overrides["retain_extraction_mode"] == "verbose"
-
-    @pytest.mark.asyncio
-    async def test_default_template_not_reapplied_on_existing_bank(
-        self, api_client, bank_id, _patched_default_template
-    ):
-        """Template only applies on FIRST creation; subsequent puts are no-ops."""
-        # First hit creates the bank and applies the template
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        assert resp.status_code == 200
-
-        # User explicitly overrides a template-set field
-        patch_resp = await api_client.patch(
-            f"/v1/default/banks/{bank_id}/config",
-            json={"updates": {"reflect_mission": "user-override"}},
-        )
-        assert patch_resp.status_code == 200
-
-        # Second put — template must NOT be reapplied (would clobber the override)
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        assert resp.status_code == 200
-
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        assert config_resp.json()["overrides"]["reflect_mission"] == "user-override"
-
-    @pytest.mark.asyncio
-    async def test_default_template_unset_is_noop(self, api_client, bank_id):
-        """With the env var unset (fixture default), bank creation behaves as before."""
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        assert resp.status_code == 200
-
-        # No template = no overrides
-        config_resp = await api_client.get(f"/v1/default/banks/{bank_id}/config")
-        assert config_resp.json()["overrides"] == {}
-
-    @pytest.mark.asyncio
-    async def test_default_template_malformed_is_swallowed(
-        self, api_client, bank_id, monkeypatch
-    ):
-        """A malformed default template is logged and ignored — bank creation still succeeds."""
-        from hindsight_api.config import _get_raw_config
-
-        raw = _get_raw_config()
-        # Wrong version number fails Pydantic validation.
-        monkeypatch.setattr(raw, "default_bank_template", {"version": "999"})
-
-        resp = await api_client.put(f"/v1/default/banks/{bank_id}", json={})
-        # Bank creation must not fail even though the template is broken.
-        assert resp.status_code == 200
-
-    def test_parse_default_bank_template_valid_json(self, monkeypatch):
-        """_parse_default_bank_template parses a valid JSON object env var."""
-        from hindsight_api.config import _parse_default_bank_template
-
-        parsed = _parse_default_bank_template('{"version": "1", "bank": {"disposition_empathy": 4}}')
-        assert parsed == {"version": "1", "bank": {"disposition_empathy": 4}}
-
-    def test_parse_default_bank_template_none_or_empty(self):
-        """Unset / empty env var resolves to None."""
-        from hindsight_api.config import _parse_default_bank_template
-
-        assert _parse_default_bank_template(None) is None
-        assert _parse_default_bank_template("") is None
-        assert _parse_default_bank_template("   ") is None
-
-    def test_parse_default_bank_template_invalid_json_raises(self):
-        """Invalid JSON fails fast with a clear error."""
-        from hindsight_api.config import _parse_default_bank_template
-
-        with pytest.raises(ValueError, match="HINDSIGHT_API_DEFAULT_BANK_TEMPLATE"):
-            _parse_default_bank_template("not-json")
-
-    def test_parse_default_bank_template_non_object_raises(self):
-        """Non-object JSON (e.g. array, string) fails fast."""
-        from hindsight_api.config import _parse_default_bank_template
-
-        with pytest.raises(ValueError, match="expected a JSON object"):
-            _parse_default_bank_template("[1, 2, 3]")
-        with pytest.raises(ValueError, match="expected a JSON object"):
-            _parse_default_bank_template('"just a string"')

hindsight-api-slim/tests/test_chunk_storage_upsert.py

@@ -1,149 +0,0 @@
-"""
-Regression tests for chunk_storage.store_chunks_batch idempotency.
-
-Covers vectorize-io/hindsight#977: re-submitting a retain under the same
-document_id must not fail with ``UniqueViolationError`` on ``pk_chunks``.
-The upstream retain paths (cascade delete on first batch, delta retain)
-should usually prevent a chunk_id collision, but any bug in those paths
-used to surface as a raw Postgres constraint violation. ``store_chunks_batch``
-is now idempotent: inserting the same ``chunk_id`` twice overwrites the
-existing row rather than raising.
-"""
-
-from datetime import datetime, timezone
-
-import pytest
-
-from hindsight_api.engine.retain import chunk_storage
-from hindsight_api.engine.retain.types import ChunkMetadata
-
-
-def _ts() -> float:
-    return datetime.now(timezone.utc).timestamp()
-
-
-async def _seed_bank_and_document(conn, bank_id: str, document_id: str) -> None:
-    """Insert the minimum rows required for the chunks FK to pass."""
-    await conn.execute(
-        "INSERT INTO banks (bank_id, name) VALUES ($1, $2) ON CONFLICT DO NOTHING",
-        bank_id,
-        bank_id,
-    )
-    await conn.execute(
-        """
-        INSERT INTO documents (id, bank_id, original_text, content_hash)
-        VALUES ($1, $2, $3, $4)
-        ON CONFLICT (id, bank_id) DO NOTHING
-        """,
-        document_id,
-        bank_id,
-        "seed",
-        "seed-hash",
-    )
-
-
-@pytest.mark.asyncio
-async def test_store_chunks_batch_is_idempotent_for_same_chunk_id(memory):
-    """
-    Regression for #977.
-
-    Directly exercises the chunk insert path: inserting a ChunkMetadata with
-    a chunk_index that already exists (i.e., the same chunk_id) must not
-    raise. The new content should overwrite the old one.
-    """
-    bank_id = f"test_chunk_upsert_{_ts()}"
-    document_id = "doc-upsert-regression"
-
-    pool = await memory._get_pool()
-    try:
-        async with pool.acquire() as conn:
-            await _seed_bank_and_document(conn, bank_id, document_id)
-
-            # First insert — fresh chunks at indices 0, 1, 2.
-            v1 = [
-                ChunkMetadata(chunk_text="alpha", fact_count=1, content_index=0, chunk_index=0),
-                ChunkMetadata(chunk_text="beta", fact_count=1, content_index=0, chunk_index=1),
-                ChunkMetadata(chunk_text="gamma", fact_count=1, content_index=0, chunk_index=2),
-            ]
-            v1_map = await chunk_storage.store_chunks_batch(conn, bank_id, document_id, v1)
-            assert set(v1_map.keys()) == {0, 1, 2}
-
-            # Second insert — overlapping chunk_index (1 and 2) with new text,
-            # plus a fresh chunk at index 3. Before the fix this raised
-            # asyncpg.exceptions.UniqueViolationError on pk_chunks; after the
-            # fix the conflicting rows are overwritten and the new one is
-            # inserted.
-            v2 = [
-                ChunkMetadata(chunk_text="beta-updated", fact_count=1, content_index=0, chunk_index=1),
-                ChunkMetadata(chunk_text="gamma-updated", fact_count=1, content_index=0, chunk_index=2),
-                ChunkMetadata(chunk_text="delta", fact_count=1, content_index=0, chunk_index=3),
-            ]
-            v2_map = await chunk_storage.store_chunks_batch(conn, bank_id, document_id, v2)
-            assert set(v2_map.keys()) == {1, 2, 3}
-
-            # Verify the stored state matches the upserted content.
-            rows = await conn.fetch(
-                """
-                SELECT chunk_index, chunk_text, content_hash
-                FROM chunks
-                WHERE document_id = $1 AND bank_id = $2
-                ORDER BY chunk_index
-                """,
-                document_id,
-                bank_id,
-            )
-            by_index = {row["chunk_index"]: row for row in rows}
-
-            assert set(by_index.keys()) == {0, 1, 2, 3}, (
-                "Expected four chunks total after upsert (0 untouched, 1-2 overwritten, 3 new)"
-            )
-            assert by_index[0]["chunk_text"] == "alpha", "Untouched chunk must be preserved"
-            assert by_index[1]["chunk_text"] == "beta-updated", "Conflicting chunk must be overwritten"
-            assert by_index[2]["chunk_text"] == "gamma-updated", "Conflicting chunk must be overwritten"
-            assert by_index[3]["chunk_text"] == "delta", "New chunk must be inserted"
-
-            # content_hash should reflect the new text, not the original.
-            assert by_index[1]["content_hash"] == chunk_storage.compute_chunk_hash("beta-updated")
-            assert by_index[2]["content_hash"] == chunk_storage.compute_chunk_hash("gamma-updated")
-    finally:
-        async with pool.acquire() as conn:
-            await conn.execute("DELETE FROM chunks WHERE bank_id = $1", bank_id)
-            await conn.execute("DELETE FROM documents WHERE bank_id = $1", bank_id)
-            await conn.execute("DELETE FROM banks WHERE bank_id = $1", bank_id)
-
-
-@pytest.mark.asyncio
-async def test_store_chunks_batch_second_call_with_identical_payload(memory):
-    """
-    The exact #977 shape: ``store_chunks_batch`` called twice with the same
-    chunks must succeed both times (the second call is a no-op in terms of
-    stored content, but must not raise).
-    """
-    bank_id = f"test_chunk_upsert_identical_{_ts()}"
-    document_id = "doc-upsert-identical"
-
-    pool = await memory._get_pool()
-    try:
-        async with pool.acquire() as conn:
-            await _seed_bank_and_document(conn, bank_id, document_id)
-
-            chunks = [
-                ChunkMetadata(chunk_text=f"chunk-{i}", fact_count=1, content_index=0, chunk_index=i)
-                for i in range(5)
-            ]
-
-            await chunk_storage.store_chunks_batch(conn, bank_id, document_id, chunks)
-            # Second call with identical chunks — must not raise.
-            await chunk_storage.store_chunks_batch(conn, bank_id, document_id, chunks)
-
-            count = await conn.fetchval(
-                "SELECT COUNT(*) FROM chunks WHERE document_id = $1 AND bank_id = $2",
-                document_id,
-                bank_id,
-            )
-            assert count == 5, "Second identical insert should not duplicate rows"
-    finally:
-        async with pool.acquire() as conn:
-            await conn.execute("DELETE FROM chunks WHERE bank_id = $1", bank_id)
-            await conn.execute("DELETE FROM documents WHERE bank_id = $1", bank_id)
-            await conn.execute("DELETE FROM banks WHERE bank_id = $1", bank_id)