Compare commits

..
Author SHA1 Message Date
Nicolò Boschi 4ed5d169a7 fix: add spacing between Fact Types label and pills, rename to Exclude all mental models 2026-03-19 16:41:44 +01:00
Nicolò Boschi 3629b350af feat: pill-style toggle buttons for fact type filter (blue/emerald/amber per type) 2026-03-19 16:33:12 +01:00
Nicolò Boschi 7788927966 refactor: shared FactTypeFilter component, tabbed mental model dialogs use General tab, clean up labels 2026-03-19 16:20:03 +01:00
Nicolò Boschi 5005669497 feat: tabbed mental model dialogs (Basic / Options tabs) 2026-03-19 16:08:57 +01:00
Nicolò Boschi f404f31ae2 fix: add missing trigger fields to local MentalModel interface in mental-models-view 2026-03-19 14:16:56 +01:00
Nicolò Boschi 7e191ecd9f fix: add missing trigger fields to MentalModel type in control plane api.ts 2026-03-19 14:04:23 +01:00
Nicolò Boschi 1232c79652 feat: expose fact_types, exclude_mental_models, exclude_mental_model_ids in control plane UI 2026-03-19 13:51:16 +01:00
Nicolò Boschi b6dedd5e4b chore: merge main, fix lint formatting and update skills openapi.json 2026-03-19 13:00:05 +01:00
Nicolò Boschi 11d7a9a5a5 Merge remote-tracking branch 'origin/main' into reflect 2026-03-19 12:59:24 +01:00
Nicolò Boschi e5186129fb fix: filter hallucinated tool calls before trace to prevent disabled tools appearing in results 2026-03-19 12:41:53 +01:00
Nicolò Boschi f828ae46d9 fix: add missing ReflectRequest fields in Rust CLI struct initializers 2026-03-19 12:26:33 +01:00
Nicolò Boschi 00280e65bf fix: guard against disabled-tool hallucination and regenerate clients
- Add enabled_tools guard in reflect agent: if an LLM calls a tool that
  was excluded (e.g. recall when fact_types=["observation"]), return an
  error result instead of executing it
- Regenerate OpenAPI spec and all SDK clients (Go, Python, TypeScript)
  to include new fact_types / exclude_mental_models fields
2026-03-19 12:08:43 +01:00
Nicolò Boschi 226b0b5fba feat: add fact_types and mental model exclusion filters to reflect and mental models
Adds three new filtering options to both the reflect endpoint and mental model creation/refresh:

- `fact_types`: restrict which fact types (world, experience, observation) are retrieved.
  Disables irrelevant agent tools entirely (no wasted tokens).
- `exclude_mental_models`: skip the search_mental_models tool altogether.
- `exclude_mental_model_ids`: exclude specific mental models by ID (merged with the
  existing self-exclusion logic during mental model refresh).

For mental models, options are persisted in the existing `trigger` JSONB column so they
are automatically applied on every refresh. The `UpdateMentalModelRequest` already
proxies `trigger`, so no extra endpoint changes are needed.

Also fixes the test fixture (`pg0_db_url` in conftest.py) to correctly resolve pg0://
URLs and run migrations before tests, which was causing all DB-dependent tests to fail
with "relation public.banks does not exist" when HINDSIGHT_API_DATABASE_URL=pg0://uuuu.
2026-03-19 11:51:45 +01:00
1606 changed files with 31448 additions and 222248 deletions
-15
View File
@@ -1,15 +0,0 @@
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "hindsight",
"description": "Official Hindsight integrations for Claude Code",
"owner": {
"name": "vectorize-io"
},
"plugins": [
{
"name": "hindsight-memory",
"description": "Automatic long-term memory for Claude Code via Hindsight",
"source": "./hindsight-integrations/claude-code"
}
]
}
-205
View File
@@ -1,205 +0,0 @@
---
name: code-review
description: Review changed code against project standards. Checks for missing tests, dead code, type safety, lint issues, and coding conventions. Run after completing any implementation work.
user_invocable: true
---
# Code Review
Review all changed code against the project's quality standards and coding conventions.
## Code Standards
Read and internalize these standards before writing code. The review steps below verify compliance.
### Python Style
- Python 3.11+, type hints required
- Async throughout (asyncpg, async FastAPI)
- Pydantic models for request/response
- Ruff for linting (line-length 120)
- No Python files at project root - maintain clean directory structure
- **Never use multi-item tuple return values** — not even for internal/private functions. Always use a dataclass or Pydantic model. No exceptions, no "it's just two values" shortcuts. If a function returns more than one value, define a named type for it.
### Type Safety with Pydantic Models
**NEVER use raw `dict` types for structured data** — this applies to all code, including internal helpers and private functions. If the dict has known keys, it must be a dataclass or Pydantic model:
- Use Pydantic `BaseModel` for all data structures passed between functions
- Use `@dataclass` for lightweight internal data containers when Pydantic validation isn't needed
- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
- Avoid `dict.get()` patterns - use typed model attributes instead
- Parse external data (JSON, API responses) into Pydantic models at the boundary
- This catches type errors at parse time, not deep in business logic
- The only acceptable `dict` usage is for truly dynamic/unknown keys (e.g., arbitrary metadata, JSON blobs with no fixed schema)
```python
# BAD - error-prone dict access
def process(data: dict) -> str:
return data.get("name", "") # No validation, silent failures
# GOOD - typed and validated
class UserData(BaseModel):
name: str
created_at: datetime
@field_validator("created_at", mode="before")
@classmethod
def ensure_tz_aware(cls, v):
if isinstance(v, str):
v = datetime.fromisoformat(v.replace("Z", "+00:00"))
if v.tzinfo is None:
return v.replace(tzinfo=timezone.utc)
return v
def process(data: UserData) -> str:
return data.name # Type-safe, validated at construction
```
### TypeScript Style
- Next.js App Router for control plane
- Tailwind CSS with shadcn/ui components
### Code Comments
- **Always comment non-trivial technical decisions** with the reasoning behind the choice. If someone would ask "why is it done this way?", there should be a comment.
- **Keep comments up to date with history** — when changing an approach, update the comment to explain what was tried before and why it was changed. Comments serve as a tracker of previous implementations that likely had problems.
- Don't comment obvious code — only where the "why" isn't self-evident from the code itself.
```python
# BAD - no context for future readers
results = await asyncio.gather(*tasks, return_exceptions=True)
# GOOD - explains the non-obvious choice
# Use return_exceptions=True to avoid cancelling sibling tasks on failure.
# Previously we used TaskGroup but it cancelled all tasks when one failed,
# causing partial writes that left orphaned entity links (see #412).
results = await asyncio.gather(*tasks, return_exceptions=True)
```
### Branch Hygiene
- **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
- **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
### General Principles
- Don't add features, refactor code, or make "improvements" beyond what was asked
- Don't add unnecessary error handling for impossible scenarios
- Don't create helpers or abstractions for one-time operations
- No backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
- Three similar lines of code is better than a premature abstraction
## Review Steps
### 1. Check branch hygiene
- Run `git log --oneline main..HEAD` to list all commits on the branch.
- Verify every commit is relevant to the feature/PR. Flag any unrelated commits.
- Check the branch is based on a recent `origin/main` (no stale base).
### 2. Identify changed files
Run `git diff --name-only HEAD` (unstaged) and `git diff --cached --name-only` (staged) to get all changed files. If there are no local changes, diff against the base branch using `git diff main...HEAD --name-only` and `git diff main...HEAD` to review all commits on the current branch.
### 3. Run linters
```bash
./scripts/hooks/lint.sh
```
Report any failures. Do NOT fix them yourself — just report.
### 4. Check for dead code
For each changed Python file, check for:
- Unused imports (Ruff should catch these, but verify)
- Functions/methods/classes that were added but are never called from anywhere
- Variables assigned but never read
- Commented-out code blocks that should be removed
For each changed TypeScript file, check for:
- Unused imports
- Unused variables or functions
- Commented-out code
### 5. Check type safety (Python)
For each changed Python file, check for violations:
- **No raw `dict` for structured data** — must use Pydantic model or dataclass, even for internal/private functions (only exception: truly dynamic/unknown keys)
- **No multi-item tuple returns** — must use dataclass or Pydantic model, even for internal/private functions (no exceptions)
- **Missing type hints** on function parameters and return types
- **Missing `@field_validator`** for datetime fields that should be timezone-aware
### 6. Check for missing tests
For each new or significantly changed function/endpoint/class:
- Check if there is a corresponding test addition or update
- New API endpoints MUST have integration tests
- New utility functions MUST have unit tests
- Bug fixes SHOULD have a regression test
Flag any new logic that lacks test coverage.
### 7. Check API consistency
If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
- Were the OpenAPI specs regenerated? (`./scripts/generate-openapi.sh`)
- Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
- Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
### 8. Check code comments
For each non-trivial change:
- **New non-obvious logic** — is there a comment explaining the reasoning?
- **Changed approach** — does the comment include what was done before and why it changed?
- **Stale comments** — do existing comments near the changed code still accurately describe the behavior?
### 9. Check integration completeness
If any files in `hindsight-integrations/` were added or changed, verify:
- **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
- **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
- **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
### 10. Check MCP tool registration completeness
If any new MCP tools were added or existing tools renamed in `hindsight-api-slim/hindsight_api/mcp_tools.py`:
- **`_ALL_TOOLS` set** in `mcp_tools.py` — must include the new tool name
- **`tools_to_register` default set** in `register_mcp_tools()` in `mcp_tools.py` — must include the new tool name
- **`_SINGLE_BANK_TOOLS` set** in `hindsight-api-slim/hindsight_api/api/mcp.py` — must include the new tool if it is bank-scoped (not a bank-management tool like `list_banks`/`create_bank`)
- **`MCP_TOOL_GROUPS`** in `hindsight-control-plane/src/components/bank-config-view.tsx` — must include the new tool in the appropriate group for the UI tool selector
- **Tool count assertions** in tests (e.g., `test_mcp_tools.py`) — must be updated to reflect the new count
### 11. Review against other coding standards
Check the diff for violations of the standards listed above:
- Python files at project root (not allowed)
- Missing async patterns (should be async throughout)
- Pydantic models for request/response
- Line length > 120 chars
- New features/code beyond what was asked (over-engineering)
- Unnecessary error handling for impossible scenarios
- Premature abstractions or speculative helpers
- Backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
### 12. Report findings
Present a clear summary organized by severity:
**Must fix** — issues that will break CI or violate hard project rules:
- Unrelated commits on the branch
- Lint failures
- Missing type hints on public functions
- Raw dict usage for structured data (including internal code)
- Multi-item tuple returns (including internal code)
- Missing tests for new endpoints
- New integration missing tests, CI job, or release-integration.sh entry
**Should fix** — issues that hurt code quality:
- Dead code / unused imports missed by linter
- Missing tests for non-trivial utility functions
- Over-engineering beyond the task scope
**Note** — observations that may or may not need action:
- API changes that might need client regeneration
- Patterns that deviate from nearby code style
For each finding, include the file path, line number, and a brief explanation.
Do NOT auto-fix any issues. Report all findings and let the user decide what to address. If there are no findings, confirm the code looks good.
+1 -7
View File
@@ -2,7 +2,7 @@
# Copy this file to .env and fill in your values
# LLM Configuration (Required)
# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, volcano
# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax
HINDSIGHT_API_LLM_PROVIDER=openai
HINDSIGHT_API_LLM_API_KEY=your-api-key-here
HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
@@ -25,11 +25,6 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
# HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
# Example: DeepSeek configuration (https://api.deepseek.com)
# HINDSIGHT_API_LLM_PROVIDER=deepseek
# HINDSIGHT_API_LLM_API_KEY=your-deepseek-api-key
# HINDSIGHT_API_LLM_MODEL=deepseek-v4-flash # or deepseek-v4-pro / deepseek-chat / deepseek-reasoner
# Example: LM Studio local configuration (Qwen 2.5 32B recommended)
# HINDSIGHT_API_LLM_PROVIDER=lmstudio
# HINDSIGHT_API_LLM_API_KEY=lmstudio
@@ -49,7 +44,6 @@ HINDSIGHT_API_LOG_LEVEL=info
# Database (Optional - uses embedded pg0 by default)
# HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
# HINDSIGHT_API_MIGRATION_DATABASE_URL= # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
# HINDSIGHT_API_DATABASE_SCHEMA=public # PostgreSQL schema name (default: public)
# Vector Extension (Optional - uses pgvector by default)
+2 -2
View File
@@ -34,7 +34,7 @@ jobs:
env:
UMAMI_URL: https://analytics.hindsight.vectorize.io
UMAMI_WEBSITE_ID: ${{ secrets.UMAMI_WEBSITE_ID }}
- uses: actions/upload-pages-artifact@v5
- uses: actions/upload-pages-artifact@v4
with:
path: hindsight-docs/build
deploy:
@@ -44,5 +44,5 @@ jobs:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/deploy-pages@v5
- uses: actions/deploy-pages@v4
id: deployment
-197
View File
@@ -1,197 +0,0 @@
name: Performance Tests
on:
schedule:
# Run daily at 06:00 UTC
- cron: "0 6 * * *"
workflow_dispatch:
inputs:
scale:
description: "Test scale (perf-test)"
type: choice
options:
- tiny
- small
- medium
- large
default: large
suite:
description: "Perf-test suite to run (blank = all)"
type: choice
options:
- ""
- retain
- recall
- recall-with-observations
- consolidation
default: ""
locomo_max_conversations:
description: "LoComo max conversations (0 = skip, blank = all)"
type: number
default: 0
locomo_skip:
description: "Skip LoComo job"
type: boolean
default: false
ref:
description: "Git ref to test (branch, tag, or SHA). Defaults to main."
type: string
default: ""
concurrency:
group: perf-test
cancel-in-progress: true
jobs:
perf-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true
prune-cache: false
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version-file: ".python-version"
- name: Cache HuggingFace models
uses: actions/cache@v5
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-huggingface-
- name: Pre-download models
working-directory: ./hindsight-api-slim
run: |
uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
from sentence_transformers import SentenceTransformer
print('Downloading embedding model...')
SentenceTransformer('BAAI/bge-small-en-v1.5')
print('Model downloaded successfully')
"
- name: Install hindsight-dev dependencies
run: |
cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
- name: "Suite: retain"
if: inputs.suite == '' || inputs.suite == 'retain'
run: |
./scripts/benchmarks/run-perf-test.sh \
--scale ${{ inputs.scale || 'large' }} \
--suite retain \
--output perf-results-retain.json
- name: "Suite: recall"
if: inputs.suite == '' || inputs.suite == 'recall'
run: |
./scripts/benchmarks/run-perf-test.sh \
--scale ${{ inputs.scale || 'large' }} \
--suite recall \
--output perf-results-recall.json
- name: "Suite: recall-with-observations"
if: inputs.suite == '' || inputs.suite == 'recall-with-observations'
run: |
./scripts/benchmarks/run-perf-test.sh \
--scale ${{ inputs.scale || 'large' }} \
--suite recall-with-observations \
--output perf-results-recall-with-observations.json
- name: "Suite: consolidation"
if: inputs.suite == '' || inputs.suite == 'consolidation'
run: |
./scripts/benchmarks/run-perf-test.sh \
--scale ${{ inputs.scale || 'large' }} \
--suite consolidation \
--output perf-results-consolidation.json
- name: Upload perf results
if: always()
uses: actions/upload-artifact@v7
with:
name: perf-results-${{ github.sha }}
path: hindsight-dev/perf-results-*.json
retention-days: 90
locomo:
if: inputs.locomo_skip != true
runs-on: ubuntu-latest
env:
HINDSIGHT_API_LLM_PROVIDER: vertexai
HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
HINDSIGHT_API_JUDGE_LLM_PROVIDER: vertexai
HINDSIGHT_API_JUDGE_LLM_MODEL: google/gemini-2.5-flash-lite
HINDSIGHT_API_ANSWER_LLM_PROVIDER: vertexai
HINDSIGHT_API_ANSWER_LLM_MODEL: google/gemini-2.5-flash
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- name: Setup GCP credentials
run: |
printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
enable-cache: true
prune-cache: false
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version-file: ".python-version"
- name: Cache HuggingFace models
uses: actions/cache@v5
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-huggingface-
- name: Pre-download models
working-directory: ./hindsight-api-slim
run: |
uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
from sentence_transformers import SentenceTransformer
print('Downloading embedding model...')
SentenceTransformer('BAAI/bge-small-en-v1.5')
print('Model downloaded successfully')
"
- name: Install hindsight-dev dependencies
run: |
cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
- name: Run LoComo benchmark
run: |
MAX_CONV_ARG=""
if [ "${{ inputs.locomo_max_conversations }}" != "0" ] && [ -n "${{ inputs.locomo_max_conversations }}" ]; then
MAX_CONV_ARG="--max-conversations ${{ inputs.locomo_max_conversations }}"
fi
uv run python hindsight-dev/benchmarks/locomo/locomo_benchmark.py \
--wait-consolidation \
$MAX_CONV_ARG
- name: Upload LoComo results
if: always()
uses: actions/upload-artifact@v7
with:
name: locomo-results-${{ github.sha }}
path: hindsight-dev/benchmarks/locomo/results/
retention-days: 90
+3 -35
View File
@@ -31,10 +31,8 @@ jobs:
run: |
if [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/pyproject.toml" ]; then
echo "type=python" >> $GITHUB_OUTPUT
elif [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/package.json" ]; then
echo "type=typescript" >> $GITHUB_OUTPUT
else
echo "type=plugin" >> $GITHUB_OUTPUT
echo "type=typescript" >> $GITHUB_OUTPUT
fi
# ── Python integrations (litellm, pydantic-ai, crewai) ──────────────────
@@ -47,7 +45,7 @@ jobs:
- name: Set up Python
if: steps.type.outputs.type == 'python'
uses: actions/setup-python@v6
uses: actions/setup-python@v5
with:
python-version-file: ".python-version"
@@ -65,44 +63,14 @@ jobs:
# ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
# ── Plugin integrations (claude-code) — no package to publish ───────────
- name: Plugin release
if: steps.type.outputs.type == 'plugin'
run: |
echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
# ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
- name: Set up Node.js
if: steps.type.outputs.type == 'typescript'
uses: actions/setup-node@v6
with:
node-version: '22'
registry-url: 'https://registry.npmjs.org'
cache: 'npm'
cache-dependency-path: package-lock.json
- name: Check integration lockfile
if: steps.type.outputs.type == 'typescript'
run: ./scripts/check-integration-lockfiles.sh
# Some integrations depend on workspace packages (hindsight-client,
# hindsight-all, hindsight-agent-sdk) via file: refs. Install from root
# so npm resolves them, then build the workspace deps before the integration.
- name: Install root workspace dependencies
if: steps.type.outputs.type == 'typescript'
run: npm ci
- name: Build workspace deps (hindsight-client, hindsight-all, hindsight-agent-sdk)
if: steps.type.outputs.type == 'typescript'
run: |
npm run build --workspace=hindsight-clients/typescript
npm run build --workspace=hindsight-all-npm
npm run build --workspace=hindsight-tools/hindsight-agent-sdk
- name: Install integration dependencies
- name: Install dependencies
if: steps.type.outputs.type == 'typescript'
working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
run: npm ci
-65
View File
@@ -1,65 +0,0 @@
name: Release Tool
on:
push:
tags:
- 'tools/**'
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Extract tool info
id: info
run: |
# refs/tags/tools/self-driving-agents/v0.0.1 → tool=self-driving-agents, version=0.0.1
TAG="${GITHUB_REF#refs/tags/}"
TOOL=$(echo "$TAG" | cut -d'/' -f2)
VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
echo "tool=$TOOL" >> $GITHUB_OUTPUT
echo "version=$VERSION" >> $GITHUB_OUTPUT
echo "tag=$TAG" >> $GITHUB_OUTPUT
echo "Tool: $TOOL, Version: $VERSION"
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: '22'
registry-url: 'https://registry.npmjs.org'
cache: 'npm'
cache-dependency-path: package-lock.json
# Tools live under hindsight-tools/ and may depend on workspace packages
# (e.g. @vectorize-io/hindsight-client). Install from root so npm resolves
# workspace deps, then build any required workspace packages first.
- name: Install root workspace dependencies
run: npm ci
- name: Build hindsight-client (workspace dep)
run: npm run build --workspace=hindsight-clients/typescript
- name: Build hindsight-agent-sdk (workspace dep)
run: npm run build --workspace=hindsight-tools/hindsight-agent-sdk
- name: Build tool
run: npm run build --workspace=hindsight-tools/${{ steps.info.outputs.tool }}
- name: Publish to npm
working-directory: ./hindsight-tools/${{ steps.info.outputs.tool }}
run: |
set +e
OUTPUT=$(npm publish --access public 2>&1)
EXIT_CODE=$?
echo "$OUTPUT"
if [ $EXIT_CODE -ne 0 ]; then
if echo "$OUTPUT" | grep -q "cannot publish over"; then
echo "Package version already published, skipping..."
exit 0
fi
exit $EXIT_CODE
fi
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+3 -67
View File
@@ -150,55 +150,6 @@ jobs:
path: hindsight-clients/typescript/*.tgz
retention-days: 1
release-hindsight-all-npm:
runs-on: ubuntu-latest
environment: npm
steps:
- uses: actions/checkout@v6
- name: Set up Node.js
uses: actions/setup-node@v6
with:
node-version: '22'
registry-url: 'https://registry.npmjs.org'
cache: 'npm'
cache-dependency-path: package-lock.json
- name: Install dependencies
run: npm ci --workspace=hindsight-all-npm
- name: Build
run: npm run build --workspace=hindsight-all-npm
- name: Publish to npm
working-directory: ./hindsight-all-npm
run: |
set +e
OUTPUT=$(npm publish --access public 2>&1)
EXIT_CODE=$?
echo "$OUTPUT"
if [ $EXIT_CODE -ne 0 ]; then
if echo "$OUTPUT" | grep -q "cannot publish over"; then
echo "Package version already published, skipping..."
exit 0
fi
exit $EXIT_CODE
fi
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Pack for GitHub release
working-directory: ./hindsight-all-npm
run: npm pack
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
name: hindsight-all-npm
path: hindsight-all-npm/*.tgz
retention-days: 1
release-control-plane:
runs-on: ubuntu-latest
environment: npm
@@ -431,7 +382,7 @@ jobs:
- uses: actions/checkout@v6
- name: Install Helm
uses: azure/setup-helm@v5
uses: azure/setup-helm@v4
with:
version: 'latest'
@@ -456,7 +407,7 @@ jobs:
create-github-release:
runs-on: ubuntu-latest
needs: [release-python-packages, release-typescript-client, release-hindsight-all-npm, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
needs: [release-python-packages, release-typescript-client, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
permissions:
contents: write
@@ -485,24 +436,12 @@ jobs:
name: control-plane
path: ./artifacts/control-plane
- name: Download hindsight-embed npm wrapper
uses: actions/download-artifact@v8
with:
name: hindsight-all-npm
path: ./artifacts/hindsight-all-npm
- name: Download Rust CLI (Linux)
uses: actions/download-artifact@v8
with:
name: rust-cli-hindsight-linux-amd64
path: ./artifacts/rust-cli-linux
- name: Download Rust CLI (Linux ARM)
uses: actions/download-artifact@v8
with:
name: rust-cli-hindsight-linux-arm64
path: ./artifacts/rust-cli-linux-arm64
- name: Download Rust CLI (macOS Intel)
uses: actions/download-artifact@v8
with:
@@ -533,13 +472,10 @@ jobs:
cp artifacts/python-packages/hindsight-embed/dist/* release-assets/ || true
# TypeScript client
cp artifacts/typescript-client/*.tgz release-assets/ || true
# hindsight-embed npm wrapper
cp artifacts/hindsight-all-npm/*.tgz release-assets/ || true
# Control Plane
cp artifacts/control-plane/*.tgz release-assets/ || true
# Rust CLI binaries
cp artifacts/rust-cli-linux/hindsight-linux-amd64 release-assets/ || true
cp artifacts/rust-cli-linux-arm64/hindsight-linux-arm64 release-assets/ || true
cp artifacts/rust-cli-darwin-amd64/hindsight-darwin-amd64 release-assets/ || true
cp artifacts/rust-cli-darwin-arm64/hindsight-darwin-arm64 release-assets/ || true
# Helm chart
@@ -547,7 +483,7 @@ jobs:
ls -la release-assets/
- name: Create GitHub Release
uses: softprops/action-gh-release@v3
uses: softprops/action-gh-release@v2
with:
files: release-assets/*
generate_release_notes: true
File diff suppressed because it is too large Load Diff
+1 -2
View File
@@ -50,8 +50,7 @@ hindsight-dev/benchmarks/perf/results/
benchmarks/results/
hindsight-cli/target
hindsight-clients/rust/target
.claude/*
!.claude/skills/
.claude
whats-next.md
TASK.md
# Changelog is now tracked in hindsight-docs/src/pages/changelog.md
-7
View File
@@ -1,7 +0,0 @@
{
"semi": true,
"singleQuote": false,
"tabWidth": 2,
"trailingComma": "es5",
"printWidth": 100
}
+59 -81
View File
@@ -11,15 +11,9 @@ Hindsight is an agent memory system that provides long-term memory for AI agents
## Development Commands
### Local Development (API + UI)
```bash
# Start both API server and control plane UI
./scripts/dev/start.sh
```
### API Server (Python/FastAPI)
```bash
# Start API server only (loads .env automatically)
# Start API server (loads .env automatically)
./scripts/dev/start-api.sh
# Run all tests (parallelized with pytest-xdist)
@@ -68,9 +62,8 @@ cd hindsight-control-plane && npm run dev
./scripts/benchmarks/run-locomo.sh
# Performance benchmarks
./scripts/benchmarks/run-perf-test.sh # System perf (mock LLM + pg0)
./scripts/benchmarks/run-perf-test.sh --scale tiny # Quick smoke test
./scripts/benchmarks/run-consolidation.sh
./scripts/benchmarks/run-retain-perf.sh --document <path> # Requires API server running
# Results viewer
./scripts/benchmarks/start-visualizer.sh # View results at localhost:8001
@@ -80,16 +73,17 @@ cd hindsight-control-plane && npm run dev
### Monorepo Structure
- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)
- **hindsight/**: Embedded Python bundle (hindsight-all package)
- **hindsight-control-plane/**: Admin UI (Next.js, npm)
- **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)
- **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)
- **hindsight-docs/**: Docusaurus documentation site
- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)
- **hindsight-integrations/**: Framework integrations (LiteLLM, OpenAI)
- **hindsight-dev/**: Development tools and benchmarks
### Core Engine (hindsight-api-slim/hindsight_api/engine/)
- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
- `memory_engine.py`: Main orchestrator (~170KB) for retain/recall/reflect operations
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, Groq, MiniMax, Ollama, LM Studio
- `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
- `cross_encoder.py`: Reranking (local or TEI)
- `entity_resolver.py`: Entity extraction and normalization
@@ -102,13 +96,13 @@ cd hindsight-control-plane && npm run dev
**search/**: Multi-strategy retrieval
- `retrieval.py`: Main retrieval orchestrator
- `graph_retrieval.py`: Graph retrieval abstract base class
- `link_expansion_retrieval.py`: Link expansion graph retrieval
- `graph_retrieval.py`: Entity/relationship graph traversal
- `mpfp_retrieval.py`: Multi-Path Fact Propagation retrieval
- `fusion.py`: Reciprocal rank fusion for combining results
- `reranking.py`: Cross-encoder reranking
### API Layer (hindsight-api-slim/hindsight_api/api/)
- `http.py`: FastAPI HTTP routers for all REST endpoints
- `http.py`: FastAPI HTTP routers (~80KB) for all REST endpoints
- `mcp.py`: Model Context Protocol server implementation
Main operations:
@@ -123,17 +117,12 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
### Adding Database Migrations
Hindsight runs the same Alembic tree against PostgreSQL and Oracle 23ai. Each
migration file dispatches through `run_for_dialect`, which calls either
`_pg_upgrade` or `_oracle_upgrade` based on the live connection. A pytest lint
(`tests/test_migration_shape.py`) fails CI if a migration omits the dispatcher.
1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:
- File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)
- Use a unique hex revision ID (12 chars)
- Set `down_revision` to the previous migration's revision ID
2. **Migration template** (the `script.py.mako` template scaffolds this; fill in the bodies):
2. **Migration template**:
```python
"""Description of the migration
@@ -144,58 +133,25 @@ migration file dispatches through `run_for_dialect`, which calls either
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "f1a2b3c4d5e6"
down_revision: str | Sequence[str] | None = "<previous_revision_id>"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_schema_prefix() -> str:
"""Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _pg_schema_prefix()
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"CREATE INDEX ... ON {schema}table_name(...)")
def _pg_downgrade() -> None:
schema = _pg_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}index_name")
def _oracle_upgrade() -> None:
# Oracle 23ai equivalent. Use op.get_bind().exec_driver_sql for forms
# that Alembic core does not model (vector/text indexes, partitions).
op.execute("CREATE INDEX ... ON table_name(...)")
def _oracle_downgrade() -> None:
op.execute("DROP INDEX IF EXISTS index_name")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}index_name")
```
**Dialect-only migrations.** If a change genuinely doesn't apply to one
dialect (e.g. enabling `pg_trgm` is PG-only), omit the unused slot:
```python
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade) # oracle slot intentionally absent → no-op
```
Make the asymmetry deliberate. Don't leave an Oracle slot empty just because
you didn't think about it — copy-pasting a PG migration without the Oracle
half is exactly how schemas drift.
3. **Run migrations locally**:
```bash
# Set database URL and run migrations for the base schema plus all tenants
@@ -208,17 +164,11 @@ migration file dispatches through `run_for_dialect`, which calls either
## Key Conventions
### Code Quality
**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).
**Always run the lint script after making Python or TypeScript/Node changes:**
```bash
./scripts/hooks/lint.sh
```
**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Prettier for TypeScript).
### Memory Banks
- Each bank is an isolated memory store (like a "brain" for one user/agent)
@@ -250,20 +200,48 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
- Update the client type definition in `lib/api.ts`
- Update any UI components that need to use the new parameter
### Adding New Integrations
### Python Style
- Python 3.11+, type hints required
- Async throughout (asyncpg, async FastAPI)
- Pydantic models for request/response
- Ruff for linting (line-length 120)
- No Python files at project root - maintain clean directory structure
- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
### Type Safety with Pydantic Models
**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
- Use Pydantic `BaseModel` for all data structures passed between functions
- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
- Avoid `dict.get()` patterns - use typed model attributes instead
- Parse external data (JSON, API responses) into Pydantic models at the boundary
- This catches type errors at parse time, not deep in business logic
1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.
2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.
3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.
4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).
```python
# BAD - error-prone dict access
def process(data: dict) -> str:
return data.get("name", "") # No validation, silent failures
If any of these are missing, the integration is incomplete and must not be pushed or merged.
# GOOD - typed and validated
class UserData(BaseModel):
name: str
created_at: datetime
### Changelogs
@field_validator("created_at", mode="before")
@classmethod
def ensure_tz_aware(cls, v):
if isinstance(v, str):
v = datetime.fromisoformat(v.replace("Z", "+00:00"))
if v.tzinfo is None:
return v.replace(tzinfo=timezone.utc)
return v
Never add "Unreleased" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.
def process(data: UserData) -> str:
return data.name # Type-safe, validated at construction
```
### TypeScript Style
- Next.js App Router for control plane
- Tailwind CSS with shadcn/ui components
### Adding New API Configuration Flags
@@ -277,17 +255,17 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
- Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
- Add `DEFAULT_*` constant for the default value
- Add field to `HindsightConfig` dataclass with type annotation
- **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API
- **Mark as hierarchical or static** by adding to `_HIERARCHICAL_FIELDS` set (hierarchical) or leaving it out (static)
- Add initialization in `from_env()` method
```python
# Configurable field (can be overridden per-tenant/bank via API)
_CONFIGURABLE_FIELDS = {
# Hierarchical field (can be overridden per-bank)
_HIERARCHICAL_FIELDS = {
...,
"my_setting", # Add here for configurable
"my_setting", # Add here for hierarchical
}
# Static field - just don't add to _CONFIGURABLE_FIELDS
# Static field - just don't add to _HIERARCHICAL_FIELDS
```
2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
-3
View File
@@ -7,7 +7,6 @@
[![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![gitcgr](https://gitcgr.com/badge/vectorize-io/hindsight.svg)](https://gitcgr.com/vectorize-io/hindsight)
![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
<br/>
@@ -84,8 +83,6 @@ cd docker/docker-compose
docker compose up
```
> Oracle AI Database is also supported for enterprise deployments with full feature parity. See the [storage documentation](https://hindsight.vectorize.io/developer/storage) for details.
>API: http://localhost:8888
>UI: http://localhost:9999
+5 -97
View File
@@ -1,28 +1,6 @@
#!/bin/bash
set -e
# =============================================================================
# Embedded pg0 data integrity check (#675)
#
# When using embedded pg0, check if the data directory has existing PostgreSQL
# data before starting. If the directory exists but appears empty/corrupt
# (e.g., missing PG_VERSION file), log a warning. This helps diagnose data
# loss scenarios where a container restart caused the data directory to be
# wiped despite a volume mount being present.
# =============================================================================
PG0_DATA_DIR="${HOME}/.pg0"
if [ -d "$PG0_DATA_DIR" ]; then
# Look for actual PostgreSQL data directories (pg0 creates subdirs per instance)
if compgen -G "$PG0_DATA_DIR"/*/PG_VERSION > /dev/null 2>&1; then
echo "✅ Existing pg0 data directory detected at $PG0_DATA_DIR"
elif [ "$(ls -A "$PG0_DATA_DIR" 2>/dev/null)" ]; then
echo "⚠️ WARNING: pg0 data directory exists at $PG0_DATA_DIR but no PG_VERSION found."
echo " This may indicate data corruption or an incomplete previous shutdown."
echo " If you see all migrations running from scratch after this, your data may have been lost."
echo " See: https://github.com/vectorize-io/hindsight/issues/675"
fi
fi
# Service flags (default to true if not set)
ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
ENABLE_CP="${HINDSIGHT_ENABLE_CP:-true}"
@@ -93,63 +71,6 @@ if [ "${HINDSIGHT_WAIT_FOR_DEPS:-false}" = "true" ]; then
done
fi
# =============================================================================
# Graceful shutdown handler (#675)
#
# Docker sends SIGTERM on `docker stop`/`docker restart`. Without a trap, child
# processes (hindsight-api + pg0, control-plane) are killed abruptly. For the
# embedded pg0 database this can cause data loss when the data directory is on
# a Docker volume that gets remounted after restart.
#
# The trap forwards SIGTERM to all tracked child PIDs so that:
# - hindsight-api receives the signal and can run its shutdown hooks
# - pg0 gets a clean PostgreSQL shutdown (checkpoint + WAL flush)
# - The control-plane Node.js process exits cleanly
# =============================================================================
# Guard against concurrent cleanup (e.g., child crash + SIGTERM arriving together)
SHUTTING_DOWN=false
cleanup() {
if $SHUTTING_DOWN; then return; fi
SHUTTING_DOWN=true
echo ""
echo "🛑 Received shutdown signal, stopping services gracefully..."
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
kill -TERM "$pid" 2>/dev/null
fi
done
# Give processes time to shut down cleanly (pg0 needs to flush WAL).
# NOTE: Docker's default stop_grace_period is 10s. If you use the default,
# either set stop_grace_period: 30s in your compose file / docker stop -t 30,
# or Docker will SIGKILL the container before this timeout expires.
local timeout=30
for ((i=1; i<=timeout; i++)); do
local all_stopped=true
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
all_stopped=false
break
fi
done
if $all_stopped; then
echo "✅ All services stopped cleanly"
exit 0
fi
sleep 1
done
# Force kill if still running after timeout
echo "⚠️ Timeout reached, forcing shutdown..."
for pid in "${PIDS[@]}"; do
if kill -0 "$pid" 2>/dev/null; then
kill -9 "$pid" 2>/dev/null
fi
done
exit 1
}
trap cleanup SIGTERM SIGINT
# Track PIDs for wait
PIDS=()
@@ -217,21 +138,8 @@ if [ ${#PIDS[@]} -eq 0 ]; then
exit 1
fi
# Wait for any process to exit (use wait -n with trap-safe loop)
while true; do
# wait -n returns when any child exits; it also returns on signal delivery
# (the trap handler will run and exit, so this loop is just for robustness).
# `&& true` prevents `set -e` from killing the script when wait -n returns
# non-zero (child exited with error or no backgrounded children remain).
wait -n && true
# Check if any tracked PID has exited
for pid in "${PIDS[@]}"; do
if ! kill -0 "$pid" 2>/dev/null; then
wait "$pid" 2>/dev/null
exit_code=$?
echo "⚠️ Service (PID $pid) exited with code $exit_code"
# Trigger cleanup for remaining services
cleanup
fi
done
done
# Wait for any process to exit
wait -n
# Exit with status of first exited process
exit $?
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: hindsight
description: Hindsight helm chart
type: application
version: 0.5.6
appVersion: "0.5.6"
version: 0.4.19
appVersion: "0.4.19"
keywords:
- ai
- memory
@@ -95,27 +95,6 @@ spec:
{{- toYaml .Values.api.readinessProbe | nindent 10 }}
resources:
{{- toYaml .Values.api.resources | nindent 10 }}
{{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumeMounts }}
volumeMounts:
{{- if .Values.api.persistence.modelCache.enabled }}
- name: model-cache
mountPath: /home/hindsight/.cache
{{- end }}
{{- with .Values.api.extraVolumeMounts }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumes }}
volumes:
{{- if .Values.api.persistence.modelCache.enabled }}
- name: model-cache
persistentVolumeClaim:
claimName: {{ include "hindsight.fullname" . }}-api-model-cache
{{- end }}
{{- with .Values.api.extraVolumes }}
{{- toYaml . | nindent 6 }}
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
@@ -1,21 +0,0 @@
{{- if and .Values.api.enabled .Values.api.persistence.modelCache.enabled }}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: {{ include "hindsight.fullname" . }}-api-model-cache
labels:
{{- include "hindsight.api.labels" . | nindent 4 }}
{{- with .Values.api.persistence.modelCache.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
accessModes:
{{- toYaml .Values.api.persistence.modelCache.accessModes | nindent 4 }}
{{- if .Values.api.persistence.modelCache.storageClass }}
storageClassName: {{ .Values.api.persistence.modelCache.storageClass }}
{{- end }}
resources:
requests:
storage: {{ .Values.api.persistence.modelCache.size }}
{{- end }}
@@ -95,16 +95,6 @@ spec:
{{- toYaml .Values.worker.readinessProbe | nindent 10 }}
resources:
{{- toYaml .Values.worker.resources | nindent 10 }}
{{- if or .Values.worker.persistence.modelCache.enabled .Values.worker.extraVolumeMounts }}
volumeMounts:
{{- if .Values.worker.persistence.modelCache.enabled }}
- name: model-cache
mountPath: /home/hindsight/.cache
{{- end }}
{{- with .Values.worker.extraVolumeMounts }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
@@ -117,26 +107,4 @@ spec:
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.worker.extraVolumes }}
volumes:
{{- toYaml . | nindent 6 }}
{{- end }}
{{- if .Values.worker.persistence.modelCache.enabled }}
volumeClaimTemplates:
- metadata:
name: model-cache
{{- with .Values.worker.persistence.modelCache.annotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
accessModes:
{{- toYaml .Values.worker.persistence.modelCache.accessModes | nindent 8 }}
{{- if .Values.worker.persistence.modelCache.storageClass }}
storageClassName: {{ .Values.worker.persistence.modelCache.storageClass }}
{{- end }}
resources:
requests:
storage: {{ .Values.worker.persistence.modelCache.size }}
{{- end }}
{{- end }}
-53
View File
@@ -67,33 +67,6 @@ api:
# Pod affinity/anti-affinity (overrides global affinity for this component)
# affinity: {}
# Persistent volume for local model cache (reranker, embeddings)
# Models are downloaded to /home/hindsight/.cache on first use.
# Without persistence, models are re-downloaded on every pod restart.
persistence:
modelCache:
enabled: false
size: 5Gi
storageClass: ""
accessModes:
- ReadWriteOnce
annotations: {}
# Extra volume mounts for the api container
# e.g.
# extraVolumeMounts:
# - name: my-volume
# mountPath: /mnt/my-volume
extraVolumeMounts: []
# Extra volumes for the api pod
# e.g.
# extraVolumes:
# - name: my-volume
# configMap:
# name: my-configmap
extraVolumes: []
# Environment variables
env:
#HINDSIGHT_API_LLM_PROVIDER: "groq"
@@ -167,32 +140,6 @@ worker:
# Pod affinity/anti-affinity (overrides global affinity for this component)
# affinity: {}
# Persistent volume for local model cache (reranker, embeddings)
# Uses volumeClaimTemplates since worker is a StatefulSet.
persistence:
modelCache:
enabled: false
size: 5Gi
storageClass: ""
accessModes:
- ReadWriteOnce
annotations: {}
# Extra volume mounts for the worker container
# e.g.
# extraVolumeMounts:
# - name: my-volume
# mountPath: /mnt/my-volume
extraVolumeMounts: []
# Extra volumes for the worker pod
# e.g.
# extraVolumes:
# - name: my-volume
# configMap:
# name: my-configmap
extraVolumes: []
# Secret environment variables (inherited from api.secrets if not specified)
secrets: {}
-4
View File
@@ -1,4 +0,0 @@
node_modules
dist
*.tgz
.DS_Store
-80
View File
@@ -1,80 +0,0 @@
# @vectorize-io/hindsight-all
Node.js equivalent of the Python [`hindsight-all`](https://pypi.org/project/hindsight-all/) package — programmatic lifecycle manager for a local Hindsight daemon. Use this when you want to embed Hindsight in a Node application without hand-rolling subprocess management.
This package deliberately does **not** ship an HTTP client. Once the daemon is running, talk to it with [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client) against `server.getBaseUrl()`. The two packages compose — one owns the daemon process, the other owns the HTTP API surface.
## Requirements
- **Node.js >= 22** — uses global `fetch` and `AbortSignal.timeout`.
- **`uv` / `uvx`** on `PATH` — used to download and run the underlying `hindsight-embed` daemon on first use. Install via <https://docs.astral.sh/uv/>.
## Install
```bash
npm install @vectorize-io/hindsight-all @vectorize-io/hindsight-client
```
## Example
```ts
import { HindsightServer, consoleLogger } from "@vectorize-io/hindsight-all";
import { HindsightClient } from "@vectorize-io/hindsight-client";
const server = new HindsightServer({
profile: "my-app",
port: 9077,
env: {
HINDSIGHT_API_LLM_PROVIDER: "anthropic",
HINDSIGHT_API_LLM_API_KEY: process.env.ANTHROPIC_API_KEY,
HINDSIGHT_API_LLM_MODEL: "claude-sonnet-4-20250514",
HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT: "0",
},
logger: consoleLogger,
});
await server.start();
const client = new HindsightClient({ baseUrl: server.getBaseUrl() });
await client.retain("user-123", "User prefers dark mode and concise answers.", {
documentId: "pref-2026-04-01",
});
const recall = await client.recall("user-123", "what are the user preferences?");
console.log(recall.results);
await server.stop();
```
For a remote Hindsight API, skip `HindsightServer` entirely and just point `HindsightClient` at the remote URL.
## Open config — forward-compatible with new daemon flags
`HindsightServerOptions` is designed so every new environment variable or CLI flag in the underlying Hindsight daemon can be used without waiting for a wrapper release:
- **`env`** accepts an arbitrary `Record<string, string>`. Every entry is exported into the daemon process and written into the profile config via `--env KEY=VALUE`.
- **`extraProfileCreateArgs`** / **`extraDaemonStartArgs`** append raw args to the respective commands.
## Development against a local checkout
If you're hacking on the Python `hindsight-embed` package in the same monorepo, point the server at the local path — it'll use `uv run --directory <path>` instead of `uvx`:
```ts
new HindsightServer({
embedPackagePath: "/path/to/hindsight-embed",
// ...
});
```
## API surface
- `HindsightServer` — daemon lifecycle (`start`, `stop`, `checkHealth`, `getBaseUrl`, `getProfile`).
- `Logger` interface plus `silentLogger` (default) and `consoleLogger` helpers.
- `getEmbedCommand(opts)` — low-level helper that returns the `[cmd, ...args]` tuple used to invoke the underlying Python CLI.
For memory operations (retain, recall, reflect, bank management, stats) use [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client).
## License
MIT
-57
View File
@@ -1,57 +0,0 @@
{
"name": "@vectorize-io/hindsight-all",
"version": "0.5.6",
"description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"type": "module",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"keywords": [
"hindsight",
"hindsight-all",
"memory",
"ai",
"agent",
"long-term-memory",
"llm",
"embedded-server"
],
"author": "Vectorize <[email protected]>",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/vectorize-io/hindsight.git",
"directory": "hindsight-all-npm"
},
"files": [
"dist",
"README.md"
],
"scripts": {
"build": "tsup",
"dev": "tsup --watch",
"clean": "rm -rf dist",
"test": "vitest run src",
"test:watch": "vitest src",
"prepublishOnly": "npm run clean && npm run build"
},
"devDependencies": {
"@types/node": "^22.0.0",
"tsup": "^8.5.1",
"typescript": "^5.7.0",
"vitest": "^4.1.2"
},
"engines": {
"node": ">=22"
},
"overrides": {
"rollup": "^4.59.0",
"picomatch": ">=2.3.2 <3.0.0 || >=4.0.4",
"vite": ">=8.0.5"
}
}
-36
View File
@@ -1,36 +0,0 @@
import { describe, it, expect } from "vitest";
import { getEmbedCommand } from "./command.js";
describe("getEmbedCommand", () => {
it("defaults to uvx hindsight-embed@latest", () => {
expect(getEmbedCommand()).toEqual(["uvx", "hindsight-embed@latest"]);
});
it("honours an explicit version", () => {
expect(getEmbedCommand({ embedVersion: "0.5.0" })).toEqual(["uvx", "[email protected]"]);
});
it("treats an empty version as latest", () => {
expect(getEmbedCommand({ embedVersion: "" })).toEqual(["uvx", "hindsight-embed@latest"]);
});
it("uses uv run --directory when a local path is given", () => {
expect(getEmbedCommand({ embedPackagePath: "/abs/path" })).toEqual([
"uv",
"run",
"--directory",
"/abs/path",
"hindsight-embed",
]);
});
it("local path takes precedence over version", () => {
expect(getEmbedCommand({ embedPackagePath: "/abs/path", embedVersion: "0.5.0" })).toEqual([
"uv",
"run",
"--directory",
"/abs/path",
"hindsight-embed",
]);
});
});
-25
View File
@@ -1,25 +0,0 @@
/**
* Resolve the command that invokes the `hindsight-embed` Python CLI.
*
* - If `embedPackagePath` is set, runs the package from a local checkout via
* `uv run --directory <path> hindsight-embed`. Used for in-repo development.
* - Otherwise runs it via `uvx hindsight-embed@<version>` so no global install
* is required.
*
* Returns the argv as `[command, ...baseArgs]` suitable for `spawn()` /
* `execFile()` (never shell-interpolated).
*/
export interface EmbedCommandOptions {
/** Version spec passed to uvx (e.g. "latest", "0.5.0"). Default: "latest". */
embedVersion?: string;
/** Local checkout path. When set, overrides `embedVersion` and uses `uv run`. */
embedPackagePath?: string;
}
export function getEmbedCommand(opts: EmbedCommandOptions = {}): string[] {
if (opts.embedPackagePath) {
return ["uv", "run", "--directory", opts.embedPackagePath, "hindsight-embed"];
}
const version = opts.embedVersion && opts.embedVersion.length > 0 ? opts.embedVersion : "latest";
return ["uvx", `hindsight-embed@${version}`];
}
-7
View File
@@ -1,7 +0,0 @@
export { HindsightServer } from "./server.js";
export { getEmbedCommand } from "./command.js";
export { silentLogger, consoleLogger } from "./logger.js";
export type { Logger } from "./logger.js";
export type { EmbedCommandOptions } from "./command.js";
export type { HindsightServerOptions } from "./types.js";
-29
View File
@@ -1,29 +0,0 @@
/**
* Pluggable logger interface.
*
* This package does not own any logging infrastructure — consumers inject
* whatever they want (console, pino, openclaw's logger, a no-op). The default
* is silent so embedding this package never adds noise to an unrelated app.
*/
export interface Logger {
debug(msg: string): void;
info(msg: string): void;
warn(msg: string): void;
error(msg: string): void;
}
/** Logger that drops every call. Used when no logger is passed. */
export const silentLogger: Logger = {
debug: () => {},
info: () => {},
warn: () => {},
error: () => {},
};
/** Logger that writes to the standard console. Handy for CLIs and tests. */
export const consoleLogger: Logger = {
debug: (msg) => console.debug(msg),
info: (msg) => console.log(msg),
warn: (msg) => console.warn(msg),
error: (msg) => console.error(msg),
};
-35
View File
@@ -1,35 +0,0 @@
import { describe, it, expect } from "vitest";
import { HindsightServer } from "./server.js";
describe("HindsightServer construction", () => {
it("defaults base URL to http://127.0.0.1:8888", () => {
const server = new HindsightServer();
expect(server.getBaseUrl()).toBe("http://127.0.0.1:8888");
expect(server.getProfile()).toBe("default");
});
it("honours custom profile, port, and host", () => {
const server = new HindsightServer({ profile: "app", port: 9077, host: "0.0.0.0" });
expect(server.getProfile()).toBe("app");
expect(server.getBaseUrl()).toBe("http://0.0.0.0:9077");
});
it("accepts open env pass-through without complaining about unknown keys", () => {
const server = new HindsightServer({
env: {
HINDSIGHT_API_LLM_PROVIDER: "openai",
HINDSIGHT_API_LLM_MODEL: "gpt-4o-mini",
// A field that does not exist today — should still be accepted
HINDSIGHT_FUTURE_FLAG: "enabled",
},
});
expect(server).toBeInstanceOf(HindsightServer);
});
it("exposes checkHealth that returns false when no daemon is running", async () => {
// Random high port that nothing is listening on.
const server = new HindsightServer({ port: 1, readyTimeoutMs: 100 });
const healthy = await server.checkHealth();
expect(healthy).toBe(false);
});
});
-322
View File
@@ -1,322 +0,0 @@
import { spawn } from "child_process";
import { getEmbedCommand } from "./command.js";
import { silentLogger } from "./logger.js";
import type { Logger } from "./logger.js";
import type { HindsightServerOptions } from "./types.js";
const DEFAULT_PORT = 8888;
const DEFAULT_HOST = "127.0.0.1";
const DEFAULT_PROFILE = "default";
const DEFAULT_READY_TIMEOUT_MS = 30_000;
const DEFAULT_READY_POLL_INTERVAL_MS = 1_000;
/**
* Manages the lifecycle of a local Hindsight daemon from a Node.js process.
*
* On {@link start}, this class:
* 1. Resolves the `hindsight-embed` command (via `uvx` or a local `uv run`).
* 2. Runs `profile create <name> --merge --port <port> [--env K=V ...]`
* with every entry in {@link HindsightServerOptions.env} forwarded as
* an `--env` flag.
* 3. Runs `daemon --profile <name> start` and waits for the start command
* to exit.
* 4. Polls `http://host:port/health` until it returns `200` or the
* `readyTimeoutMs` budget is exhausted.
*
* On {@link stop}, it runs `daemon --profile <name> stop` and returns once
* the command exits (or after a short grace period).
*
* This is the Node.js equivalent of the Python `hindsight-all` package's
* `HindsightServer`: a thin programmatic lifecycle wrapper around the
* Hindsight daemon. It does NOT ship an HTTP client — once `start()`
* resolves, use `@vectorize-io/hindsight-client` against `getBaseUrl()` for
* retain / recall / reflect.
*
* The class is deliberately transparent about the daemon: new CLI flags or
* environment variables never require a code change here — callers can pass
* them via `env`, `extraProfileCreateArgs`, or `extraDaemonStartArgs`.
*/
export class HindsightServer {
private readonly profile: string;
private readonly port: number;
private readonly host: string;
private readonly baseUrl: string;
private readonly embedVersion: string | undefined;
private readonly embedPackagePath: string | undefined;
private readonly userEnv: Record<string, string | undefined>;
private readonly extraProfileCreateArgs: string[];
private readonly extraDaemonStartArgs: string[];
private readonly platformCpuWorkaround: boolean;
private readonly readyTimeoutMs: number;
private readonly readyPollIntervalMs: number;
private readonly logger: Logger;
constructor(opts: HindsightServerOptions = {}) {
this.profile = opts.profile ?? DEFAULT_PROFILE;
this.port = opts.port ?? DEFAULT_PORT;
this.host = opts.host ?? DEFAULT_HOST;
this.baseUrl = `http://${this.host}:${this.port}`;
this.embedVersion = opts.embedVersion;
this.embedPackagePath = opts.embedPackagePath;
this.userEnv = opts.env ?? {};
this.extraProfileCreateArgs = opts.extraProfileCreateArgs ?? [];
this.extraDaemonStartArgs = opts.extraDaemonStartArgs ?? [];
this.platformCpuWorkaround = opts.platformCpuWorkaround ?? process.platform === "darwin";
this.readyTimeoutMs = opts.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
this.readyPollIntervalMs = opts.readyPollIntervalMs ?? DEFAULT_READY_POLL_INTERVAL_MS;
this.logger = opts.logger ?? silentLogger;
}
/** The base URL the daemon listens on (`http://host:port`). */
getBaseUrl(): string {
return this.baseUrl;
}
/** The profile name this server operates on. */
getProfile(): string {
return this.profile;
}
/**
* Ensure the daemon is configured and running. Idempotent — the underlying
* `profile create --merge` and `daemon start` commands tolerate re-runs.
*/
async start(): Promise<void> {
this.logger.info(`[hindsight] starting daemon for profile "${this.profile}"`);
const env = this.buildEnv();
await this.configureProfile(env);
await this.startDaemon(env);
await this.waitForReady();
this.logger.info(`[hindsight] daemon ready at ${this.baseUrl}`);
}
/** Stop the daemon. Never throws — logs and resolves even on failure. */
async stop(): Promise<void> {
this.logger.info(`[hindsight] stopping daemon for profile "${this.profile}"`);
const [cmd, ...baseArgs] = getEmbedCommand({
embedVersion: this.embedVersion,
embedPackagePath: this.embedPackagePath,
});
const args = [...baseArgs, "daemon", "--profile", this.profile, "stop"];
const child = spawn(cmd, args, { stdio: "pipe" });
this.pipeOutput(child, "daemon.stop");
await new Promise<void>((resolve) => {
const timeout = setTimeout(() => {
this.logger.warn(`[hindsight] daemon stop timed out after 5s`);
resolve();
}, 5_000);
child.on("exit", () => {
clearTimeout(timeout);
this.logger.info(`[hindsight] daemon stopped`);
resolve();
});
child.on("error", (err) => {
clearTimeout(timeout);
this.logger.warn(`[hindsight] error stopping daemon: ${err.message}`);
resolve();
});
});
}
/** Probe `/health` once with a short timeout. */
async checkHealth(): Promise<boolean> {
try {
const res = await fetch(`${this.baseUrl}/health`, {
signal: AbortSignal.timeout(2_000),
});
return res.ok;
} catch {
return false;
}
}
// -------------------------------------------------------------------------
// Internal
// -------------------------------------------------------------------------
/**
* Merge the process env, the caller-supplied `env`, and (on macOS) the
* embeddings CPU workaround. Caller-supplied values always win over the
* workaround; undefined values are dropped.
*/
private buildEnv(): NodeJS.ProcessEnv {
const merged: NodeJS.ProcessEnv = { ...process.env };
if (this.platformCpuWorkaround && process.platform === "darwin") {
merged["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1";
merged["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1";
}
for (const [key, value] of Object.entries(this.userEnv)) {
if (value !== undefined) {
merged[key] = value;
}
}
return merged;
}
/**
* Run `profile create <name> --merge --port <port> [--env K=V ...]`.
* Every entry in the merged env that was passed via {@link userEnv} (or
* auto-applied by the CPU workaround) is forwarded as `--env`.
*/
private async configureProfile(env: NodeJS.ProcessEnv): Promise<void> {
this.logger.info(`[hindsight] configuring profile "${this.profile}"`);
const [cmd, ...baseArgs] = getEmbedCommand({
embedVersion: this.embedVersion,
embedPackagePath: this.embedPackagePath,
});
const createArgs = [
...baseArgs,
"profile",
"create",
this.profile,
"--merge",
"--port",
String(this.port),
];
// Forward every env var that the caller intended for the daemon as --env.
// We only forward keys the caller explicitly set (userEnv) plus the CPU
// workaround values — not the entire process.env, to avoid leaking random
// host state into profile config.
const envForProfile = this.collectProfileEnv(env);
for (const [key, value] of Object.entries(envForProfile)) {
createArgs.push("--env", `${key}=${value}`);
}
createArgs.push(...this.extraProfileCreateArgs);
await this.runCommand(cmd, createArgs, env, "profile.create");
}
/** Collect only the env vars that should be written into the profile file. */
private collectProfileEnv(env: NodeJS.ProcessEnv): Record<string, string> {
const out: Record<string, string> = {};
// 1. User-supplied env — always forwarded.
for (const [key, value] of Object.entries(this.userEnv)) {
if (value !== undefined) {
out[key] = value;
}
}
// 2. CPU workaround — only if auto-applied and not already overridden.
if (this.platformCpuWorkaround && process.platform === "darwin") {
const cpuKeys = [
"HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU",
"HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU",
];
for (const key of cpuKeys) {
if (!(key in out) && env[key] !== undefined) {
out[key] = env[key] as string;
}
}
}
return out;
}
private async startDaemon(env: NodeJS.ProcessEnv): Promise<void> {
const [cmd, ...baseArgs] = getEmbedCommand({
embedVersion: this.embedVersion,
embedPackagePath: this.embedPackagePath,
});
const args = [
...baseArgs,
"daemon",
"--profile",
this.profile,
"start",
...this.extraDaemonStartArgs,
];
await this.runCommand(cmd, args, env, "daemon.start");
}
/**
* Spawn `cmd` with `args`, pipe its output through the logger, and resolve
* once it exits with code 0. Rejects on non-zero exit or spawn error.
*/
private async runCommand(
cmd: string,
args: string[],
env: NodeJS.ProcessEnv,
label: string
): Promise<void> {
const child = spawn(cmd, args, { stdio: "pipe", env });
let output = "";
child.stdout?.on("data", (data: Buffer) => {
const text = data.toString();
output += text;
for (const line of text.trimEnd().split("\n")) {
if (line) this.logger.info(`[hindsight:${label}] ${line}`);
}
});
child.stderr?.on("data", (data: Buffer) => {
const text = data.toString();
output += text;
for (const line of text.trimEnd().split("\n")) {
if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
}
});
await new Promise<void>((resolve, reject) => {
child.on("exit", (code) => {
if (code === 0) {
resolve();
} else {
reject(new Error(`${label} failed with code ${code}: ${output.trim()}`));
}
});
child.on("error", (err) => {
reject(new Error(`${label} failed to spawn: ${err.message}`, { cause: err }));
});
});
}
/** Stream a spawned child's stdout/stderr through the logger without blocking. */
private pipeOutput(child: ReturnType<typeof spawn>, label: string): void {
child.stdout?.on("data", (data: Buffer) => {
for (const line of data.toString().trimEnd().split("\n")) {
if (line) this.logger.info(`[hindsight:${label}] ${line}`);
}
});
child.stderr?.on("data", (data: Buffer) => {
for (const line of data.toString().trimEnd().split("\n")) {
if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
}
});
}
/** Poll `/health` until it succeeds or `readyTimeoutMs` elapses. */
private async waitForReady(): Promise<void> {
const deadline = Date.now() + this.readyTimeoutMs;
let attempt = 0;
while (Date.now() < deadline) {
attempt++;
try {
const res = await fetch(`${this.baseUrl}/health`, {
signal: AbortSignal.timeout(this.readyPollIntervalMs),
});
if (res.ok) {
this.logger.debug(`[hindsight] health check passed (attempt ${attempt})`);
return;
}
} catch {
// expected while the daemon is still booting
}
await new Promise((resolve) => setTimeout(resolve, this.readyPollIntervalMs));
}
throw new Error(
`Hindsight daemon did not become ready within ${this.readyTimeoutMs}ms at ${this.baseUrl}`
);
}
}
-54
View File
@@ -1,54 +0,0 @@
import type { Logger } from "./logger.js";
/**
* Options for {@link HindsightServer}.
*
* The server is intentionally thin and pass-through: anything configurable
* on the daemon side (env vars or CLI flags) can be set here without needing
* a new dedicated option. Use {@link env} for `HINDSIGHT_*` / `OPENAI_API_KEY` /
* custom provider settings, and the two `extra*` arrays to append raw CLI
* args to `profile create` or `daemon start`.
*
* For talking to the daemon after `start()`, use `@vectorize-io/hindsight-client`
* against `server.getBaseUrl()`. This package does not ship its own HTTP
* client.
*/
export interface HindsightServerOptions {
/** Profile name used for `--profile <name>` on every sub-command. Default: `"default"`. */
profile?: string;
/** TCP port the daemon listens on. Default: `8888`. */
port?: number;
/** Hostname the daemon binds to (for health checks). Default: `127.0.0.1`. */
host?: string;
/** Version of the underlying `hindsight-embed` PyPI package to run via `uvx`. Default: `"latest"`. */
embedVersion?: string;
/** Local path to a `hindsight-embed` checkout — takes precedence over `embedVersion`. */
embedPackagePath?: string;
/**
* Environment variables passed to the daemon process AND written into the
* profile via repeated `--env KEY=VALUE` flags. This is the preferred way
* to surface any `HINDSIGHT_API_*` / `HINDSIGHT_EMBED_*` setting — adding a
* new daemon env var never requires a wrapper update.
*
* Values of `undefined` are dropped (so you can spread conditionally).
*/
env?: Record<string, string | undefined>;
/** Extra args appended verbatim to `hindsight-embed profile create <name> --merge ...`. */
extraProfileCreateArgs?: string[];
/** Extra args appended verbatim to `hindsight-embed daemon --profile <name> start ...`. */
extraDaemonStartArgs?: string[];
/**
* On macOS, automatically set
* `HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1` and
* `HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1` to avoid Metal/MPS crashes in
* daemon mode. Default: `true` on `darwin`, ignored elsewhere. Any value set
* explicitly in {@link env} wins over the auto-applied value.
*/
platformCpuWorkaround?: boolean;
/** Max time (ms) to wait for `/health` to return 200. Default: `30_000`. */
readyTimeoutMs?: number;
/** Polling interval (ms) while waiting for `/health`. Default: `1_000`. */
readyPollIntervalMs?: number;
/** Optional pluggable logger. Default: silent. */
logger?: Logger;
}
-18
View File
@@ -1,18 +0,0 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "ES2022",
"lib": ["ES2022"],
"moduleResolution": "node",
"declaration": true,
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist", "src/**/*.test.ts"]
}
-11
View File
@@ -1,11 +0,0 @@
import { defineConfig } from "tsup";
export default defineConfig({
entry: ["src/index.ts"],
format: ["esm"],
dts: true,
outDir: "dist",
clean: true,
sourcemap: true,
bundle: true,
});
-8
View File
@@ -1,8 +0,0 @@
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["src/**/*.test.ts"],
environment: "node",
},
});
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "hindsight-all-slim"
version = "0.5.6"
version = "0.4.19"
description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
readme = "README.md"
requires-python = ">=3.11"
+35 -103
View File
@@ -34,6 +34,7 @@ Using context manager:
"""
import logging
import os
import threading
from typing import Optional
@@ -71,11 +72,8 @@ class HindsightEmbedded:
llm_model: Model name to use
llm_base_url: Optional custom base URL for LLM API
database_url: Optional database URL override (default: profile-specific pg0)
idle_timeout: Seconds before daemon auto-exits when idle (default: 0, disabled)
idle_timeout: Seconds before daemon auto-exits when idle (default: 300)
log_level: Daemon log level (default: "info")
ui: Whether to start the control plane web UI alongside the daemon (default: False)
ui_port: Port for the UI. Defaults to daemon_port + 10000.
ui_hostname: Hostname to bind the UI to. Defaults to "0.0.0.0".
"""
def __init__(
@@ -86,11 +84,8 @@ class HindsightEmbedded:
llm_model: str = "openai/gpt-oss-120b",
llm_base_url: Optional[str] = None,
database_url: Optional[str] = None,
idle_timeout: int = 0,
idle_timeout: int = 300,
log_level: str = "info",
ui: bool = False,
ui_port: Optional[int] = None,
ui_hostname: str = "0.0.0.0",
):
"""
Initialize the embedded client (daemon starts on first use).
@@ -102,11 +97,8 @@ class HindsightEmbedded:
llm_model: Model name to use
llm_base_url: Optional custom base URL for LLM API
database_url: Optional database URL override
idle_timeout: Seconds before daemon auto-exits when idle (0 = disabled)
idle_timeout: Seconds before daemon auto-exits when idle
log_level: Daemon log level
ui: Whether to start the control plane web UI alongside the daemon
ui_port: Port for the UI (defaults to daemon_port + 10000)
ui_hostname: Hostname to bind the UI to (defaults to "0.0.0.0")
"""
self.profile = profile
@@ -125,10 +117,6 @@ class HindsightEmbedded:
if database_url:
self.config["HINDSIGHT_EMBED_API_DATABASE_URL"] = database_url
self._ui = ui
self._ui_port = ui_port
self._ui_hostname = ui_hostname
self._client: Optional[Hindsight] = None
self._lock = threading.Lock()
self._started = False
@@ -142,50 +130,23 @@ class HindsightEmbedded:
self._memories_api: Optional[MemoriesAPI] = None
def _ensure_started(self):
"""Ensure daemon is running (thread-safe), restarting if crashed."""
"""Ensure daemon is running (thread-safe)."""
if self._started and self._client is not None:
if self._manager.is_running(self.profile):
return
# Daemon crashed — reset state and fall through to restart
logger.warning(
"Daemon for profile '%s' is no longer responsive, restarting...",
self.profile,
)
try:
self._client.close()
except Exception:
logger.debug("Error closing stale client", exc_info=True)
self._client = None
self._started = False
return
with self._lock:
# Double-check after acquiring lock
if self._started and self._client is not None:
if self._manager.is_running(self.profile):
return
logger.warning(
"Daemon for profile '%s' is no longer responsive (lock path), restarting...",
self.profile,
)
try:
self._client.close()
except Exception:
logger.debug("Error closing stale client", exc_info=True)
self._client = None
self._started = False
return
if self._closed:
raise RuntimeError(
"Cannot use HindsightEmbedded after it has been closed"
)
raise RuntimeError("Cannot use HindsightEmbedded after it has been closed")
# Use embed manager interface for daemon management
logger.info(f"Ensuring daemon is running for profile '{self.profile}'...")
success = self._manager.ensure_running(self.config, self.profile)
if not success:
raise RuntimeError(
f"Failed to start daemon for profile '{self.profile}'"
)
raise RuntimeError(f"Failed to start daemon for profile '{self.profile}'")
# Get daemon URL and create client
daemon_url = self._manager.get_url(self.profile)
@@ -193,15 +154,6 @@ class HindsightEmbedded:
self._started = True
logger.info(f"Connected to daemon at {daemon_url}")
# Start UI if requested
if self._ui:
logger.info(f"Starting UI for profile '{self.profile}'...")
ui_started = self._manager.start_ui(
self.profile, self._ui_port, self._ui_hostname
)
if not ui_started:
logger.warning(f"Failed to start UI for profile '{self.profile}'")
def _cleanup(self, stop_daemon_on_close: bool = False):
"""
Cleanup client resources (idempotent).
@@ -213,47 +165,20 @@ class HindsightEmbedded:
if self._closed:
return
acquired = self._lock.acquire(timeout=5.0)
if not acquired:
# Lock is held by another thread (e.g. _ensure_started).
# Mark closed to prevent new operations but skip shared-state
# teardown — the daemon's idle timeout handles the rest.
logger.warning(
"Cleanup lock acquisition timed out for profile '%s'; "
"marking closed, daemon will idle-stop on its own",
self.profile,
)
self._closed = True
return
try:
with self._lock:
if self._closed:
return
if self._client is not None:
try:
self._client.close()
except Exception:
logger.debug(
"Error closing client for profile '%s'",
self.profile,
exc_info=True,
)
self._client.close()
self._client = None
# Stop UI if it was started
if self._ui and self._started:
logger.info(f"Stopping UI for profile '{self.profile}'...")
self._manager.stop_ui(self.profile, self._ui_port)
# Optionally stop daemon (daemon has idle timeout, so not required)
if stop_daemon_on_close and self._started:
logger.info(f"Stopping daemon for profile '{self.profile}'...")
self._manager.stop(self.profile)
self._closed = True
finally:
self._lock.release()
def close(self, stop_daemon: bool = False):
"""
@@ -276,10 +201,23 @@ class HindsightEmbedded:
This allows HindsightEmbedded to expose all HindsightClient methods
without manually wrapping each one.
"""
# Ensure server is started (and restart if crashed) before proxying
# Ensure server is started before proxying
self._ensure_started()
return getattr(self._client, name)
# Get the attribute from the underlying client
attr = getattr(self._client, name)
# If it's a callable, wrap it to ensure server is started
# (shouldn't be needed since _ensure_started already called, but defensive)
if callable(attr):
def wrapper(*args, **kwargs):
self._ensure_started()
return attr(*args, **kwargs)
return wrapper
return attr
def __enter__(self):
"""Context manager entry - ensures server is started."""
@@ -404,8 +342,11 @@ class HindsightEmbedded:
"""
Get the underlying Hindsight client for direct access.
Ensures daemon is started (and restarts it if it has crashed) before
returning the client.
WARNING: Using this property directly means daemon restarts won't be
handled automatically. Prefer using the API namespaces (banks, mental_models,
directives, memories) or direct method calls on HindsightEmbedded instead.
Ensures daemon is started before returning the client.
Returns:
Hindsight: The underlying client instance
@@ -416,8 +357,9 @@ class HindsightEmbedded:
embedded = HindsightEmbedded(profile="myapp", ...)
# Direct access (not recommended - daemon crashes won't be handled)
client = embedded.client
banks = client.list_banks()
banks = client.list_banks() # If daemon crashes, this will fail
```
"""
self._ensure_started()
@@ -431,15 +373,5 @@ class HindsightEmbedded:
@property
def is_running(self) -> bool:
"""Check if the client is initialized and the daemon is responsive."""
return (
self._started
and not self._closed
and self._client is not None
and self._manager.is_running(self.profile)
)
@property
def ui_url(self) -> str:
"""Get the UI URL for this profile."""
return self._manager.get_ui_url(self.profile)
"""Check if the client is initialized."""
return self._started and not self._closed and self._client is not None
View File
+1 -4
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "hindsight-all"
version = "0.5.6"
version = "0.4.19"
description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
readme = "README.md"
requires-python = ">=3.11"
@@ -20,9 +20,6 @@ hindsight-client = { workspace = true }
hindsight-embed = { workspace = true }
[project.optional-dependencies]
local-llm = [
"hindsight-api-slim[local-llm]>=0.4.17",
]
test = [
"pytest>=7.0.0",
"pytest-asyncio>=0.21.0",
@@ -1,56 +0,0 @@
"""
Unit test for _cleanup lock timeout behavior.
Verifies that _cleanup completes even when the lock is held by another thread,
instead of hanging indefinitely (fixes #952).
"""
import threading
import time
from unittest.mock import MagicMock, patch
import pytest
def test_cleanup_completes_when_lock_held():
"""
_cleanup should complete (best-effort) even when self._lock is held
by another thread, e.g. during a long _ensure_started call.
"""
with patch.dict("sys.modules", {
"hindsight_client": MagicMock(),
"hindsight_embed": MagicMock(),
"hindsight.api_namespaces": MagicMock(),
}):
from hindsight.embedded import HindsightEmbedded
client = HindsightEmbedded.__new__(HindsightEmbedded)
client.profile = "test"
client._lock = threading.Lock()
client._closed = False
client._client = None
client._started = False
client._ui = False
# Simulate another thread holding the lock
client._lock.acquire()
cleanup_done = threading.Event()
def run_cleanup():
client._cleanup()
cleanup_done.set()
t = threading.Thread(target=run_cleanup)
t.start()
# Cleanup should complete within the timeout (5s) + margin
assert cleanup_done.wait(timeout=8.0), (
"_cleanup hung instead of timing out on lock acquisition"
)
# Release the lock from the simulating thread
client._lock.release()
t.join(timeout=1.0)
assert client._closed, "Client should be marked as closed after cleanup"
+17 -121
View File
@@ -15,8 +15,6 @@ import os
import uuid
import pytest
import urllib.request
import json
from hindsight import HindsightEmbedded
@@ -25,20 +23,12 @@ from hindsight import HindsightEmbedded
def llm_config():
"""Get LLM configuration from environment (session-scoped)."""
# Try both naming conventions
provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv(
"HINDSIGHT_LLM_PROVIDER", "groq"
)
api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv(
"HINDSIGHT_LLM_API_KEY", ""
)
model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv(
"HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b"
)
provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv("HINDSIGHT_LLM_PROVIDER", "groq")
api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv("HINDSIGHT_LLM_API_KEY", "")
model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv("HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b")
if not api_key:
pytest.skip(
"LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY."
)
pytest.skip("LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY.")
return {
"llm_provider": provider,
@@ -88,9 +78,7 @@ def test_embedded_context_manager(llm_config):
# Recall memory
recall_results = client.recall(bank_id=bank_id, query="context")
assert isinstance(recall_results.results, list), (
"Recall should return results list"
)
assert isinstance(recall_results.results, list), "Recall should return results list"
# Server should be stopped after context exit
# Note: We can't check client.is_running here as client is out of scope
@@ -117,9 +105,7 @@ def test_embedded_complete_workflow(llm_config):
# Step 1: Create a memory bank
print(f"\n1. Creating memory bank: {bank_id}")
bank_response = client.create_bank(
bank_id=bank_id,
name="Test Assistant",
mission="Help with programming tasks",
bank_id=bank_id, name="Test Assistant", mission="Help with programming tasks"
)
assert bank_response.bank_id == bank_id
@@ -140,9 +126,7 @@ def test_embedded_complete_workflow(llm_config):
items=[
{"content": "User works with pandas and numpy."},
{"content": "User likes matplotlib for visualization."},
{
"content": "User is interested in machine learning with scikit-learn."
},
{"content": "User is interested in machine learning with scikit-learn."},
],
)
assert batch_response.success
@@ -150,9 +134,7 @@ def test_embedded_complete_workflow(llm_config):
# Step 4: Recall memories
print("\n4. Recalling memories...")
recall_response = client.recall(
bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000
)
recall_response = client.recall(bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000)
assert isinstance(recall_response.results, list)
assert len(recall_response.results) > 0
print(f" Found {len(recall_response.results)} relevant memories")
@@ -170,9 +152,7 @@ def test_embedded_complete_workflow(llm_config):
# Verify answer mentions relevant tools
answer_lower = reflect_response.text.lower()
assert any(
term in answer_lower for term in ["python", "pandas", "numpy", "data"]
)
assert any(term in answer_lower for term in ["python", "pandas", "numpy", "data"])
# Step 6: List memories
print("\n6. Listing memories...")
@@ -235,9 +215,7 @@ def test_embedded_method_proxying(llm_config):
assert bank.bank_id == bank_id
# Test mission setting
mission_response = client.set_mission(
bank_id=bank_id, mission="Test mission for proxying"
)
mission_response = client.set_mission(bank_id=bank_id, mission="Test mission for proxying")
assert mission_response.bank_id == bank_id
# Test retain
@@ -286,9 +264,7 @@ def test_embedded_multiple_banks(llm_config):
# Create second bank and store data
client.create_bank(bank_id=bank2_id, name="Bank 2")
client.retain(
bank_id=bank2_id, content="Bob uses JavaScript for web development"
)
client.retain(bank_id=bank2_id, content="Bob uses JavaScript for web development")
# Recall from both banks
results1 = client.recall(bank_id=bank1_id, query="programming language")
@@ -299,9 +275,9 @@ def test_embedded_multiple_banks(llm_config):
# Verify banks are isolated (each should only see their own content)
# This is a basic check - content isolation is tested more thoroughly in other tests
assert results1.results[0].text != results2.results[0].text or len(
results1.results
) != len(results2.results)
assert results1.results[0].text != results2.results[0].text or len(results1.results) != len(
results2.results
)
finally:
client.close()
@@ -320,14 +296,10 @@ def test_embedded_profile_isolation(llm_config):
try:
# Store data in profile1
client1.retain(
bank_id=bank_id, content="User likes TypeScript for frontend development"
)
client1.retain(bank_id=bank_id, content="User likes TypeScript for frontend development")
# Store different data in profile2
client2.retain(
bank_id=bank_id, content="User prefers Rust for systems programming"
)
client2.retain(bank_id=bank_id, content="User prefers Rust for systems programming")
# Each profile should only see its own data
results1 = client1.recall(bank_id=bank_id, query="programming preference")
@@ -362,81 +334,5 @@ def test_embedded_error_after_close(llm_config):
assert not client.is_running
# Trying to use it after close should raise an error
with pytest.raises(
RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"
):
with pytest.raises(RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"):
client.retain(bank_id=bank_id, content="This should fail")
def test_embedded_ui_flag(llm_config):
"""
Test that ui=True starts the control plane UI alongside the daemon,
and that the UI's health endpoint reports a connected dataplane.
"""
profile = f"test_ui_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", ui=True, **llm_config)
try:
# First use triggers daemon + UI startup
result = client.retain(bank_id=bank_id, content="UI integration test content")
assert result.success, "Retain should succeed"
assert client.is_running, "Daemon should be running"
# Verify UI is reachable and reports connected dataplane
ui_url = client.ui_url
assert ui_url, "ui_url should be set"
health_url = f"{ui_url}/api/health"
with urllib.request.urlopen(health_url, timeout=10) as resp:
health = json.loads(resp.read().decode())
assert health["status"] == "ok", (
f"UI health status should be 'ok', got: {health['status']}"
)
assert health["dataplane"]["status"] == "connected", (
f"Dataplane should be connected, got: {health['dataplane']}"
)
finally:
client.close()
def test_embedded_daemon_crash_recovery(llm_config):
"""
Test that HindsightEmbedded recovers when the daemon crashes.
Simulates a crash by stopping the daemon, then verifies
that the next operation transparently restarts it.
"""
profile = f"test_crash_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
try:
# Start daemon and store a memory
result = client.retain(bank_id=bank_id, content="Before crash")
assert result.success, "Initial retain should succeed"
assert client.is_running, "Daemon should be running"
original_url = client.url
# Simulate daemon crash by stopping it
client._manager.stop(client.profile)
assert not client._manager.is_running(client.profile), (
"Daemon should be stopped after simulated crash"
)
# Next operation should transparently restart the daemon
result2 = client.retain(bank_id=bank_id, content="After crash recovery")
assert result2.success, "Retain after crash recovery should succeed"
assert client.is_running, "Daemon should be running again after recovery"
# Verify recall still works
recall_result = client.recall(bank_id=bank_id, query="crash")
assert isinstance(recall_result.results, list), "Recall should return results"
finally:
client.close()
+1 -1
View File
@@ -46,4 +46,4 @@ __all__ = [
"RemoteTEICrossEncoder",
"LLMConfig",
]
__version__ = "0.5.6"
__version__ = "0.4.19"
+9 -140
View File
@@ -1,7 +1,5 @@
"""PostgreSQL-only admin utilities (backup, restore, migration, worker management).
Not supported on Oracle backends. Uses asyncpg.connect() directly, binary COPY,
TRUNCATE CASCADE, and REFRESH MATERIALIZED VIEW — all inherently PG-specific.
"""
Hindsight Admin CLI - backup and restore operations.
"""
import asyncio
@@ -17,10 +15,15 @@ import asyncpg
import typer
from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
from ..engine.schema import fq_table_explicit as _fq_table
from ..extensions import TenantExtension, load_extension
from ..pg0 import parse_pg0_url, resolve_database_url
def _fq_table(table: str, schema: str) -> str:
"""Get fully-qualified table name with schema prefix."""
return f"{schema}.{table}"
# Setup logging
logging.basicConfig(
level=logging.INFO,
@@ -246,7 +249,7 @@ async def _run_migration(
schemas = list(dict.fromkeys(schemas))
for schema in schemas:
run_migrations(resolved_url, schema=schema, migration_database_url=config.migration_database_url)
run_migrations(resolved_url, schema=schema)
if embedding_dimension is not None:
for schema in schemas:
@@ -372,140 +375,6 @@ def decommission_worker(
typer.echo(f"No tasks found for worker '{worker_id}'")
async def _decommission_all_workers(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
"""Release all processing tasks from all workers, setting them back to pending status."""
is_pg0, instance_name, _ = parse_pg0_url(db_url)
if is_pg0:
typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
resolved_url = await resolve_database_url(db_url)
conn = await asyncpg.connect(resolved_url)
try:
table = _fq_table("async_operations", schema)
rows = await conn.fetch(
f"""
UPDATE {table}
SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
WHERE status = 'processing'
RETURNING operation_id, worker_id, operation_type
""",
)
return [dict(r) for r in rows]
finally:
await conn.close()
@app.command(name="decommission-workers")
def decommission_workers(
schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
):
"""Release all processing tasks from all workers (sets status back to pending).
Use this command to recover from situations where one or more workers have crashed
or been removed without graceful shutdown. All tasks currently in 'processing' status
will be released back to the queue regardless of which worker owns them.
"""
config = HindsightConfig.from_env()
if not config.database_url:
typer.echo("Error: Database URL not configured.", err=True)
typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
raise typer.Exit(1)
if not yes:
typer.confirm(
"This will release ALL processing tasks from ALL workers back to pending. Continue?",
abort=True,
)
typer.echo(f"Decommissioning all workers (schema: {schema})...")
released = asyncio.run(_decommission_all_workers(config.database_url, schema))
if released:
# Group by worker_id for summary
by_worker: dict[str, int] = {}
for row in released:
wid = row["worker_id"] or "unknown"
by_worker[wid] = by_worker.get(wid, 0) + 1
typer.echo(f"Released {len(released)} task(s):")
for wid, count in by_worker.items():
typer.echo(f" {wid}: {count} task(s)")
else:
typer.echo("No processing tasks found")
async def _worker_status(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
"""Get all processing tasks grouped by worker with their last update time."""
is_pg0, instance_name, _ = parse_pg0_url(db_url)
if is_pg0:
typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
resolved_url = await resolve_database_url(db_url)
conn = await asyncpg.connect(resolved_url)
try:
table = _fq_table("async_operations", schema)
rows = await conn.fetch(
f"""
SELECT worker_id, operation_id, operation_type, bank_id,
claimed_at, updated_at,
now() - claimed_at AS running_for,
now() - updated_at AS last_update_ago
FROM {table}
WHERE status = 'processing'
ORDER BY worker_id, claimed_at
""",
)
return [dict(r) for r in rows]
finally:
await conn.close()
@app.command(name="worker-status")
def worker_status(
schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
):
"""Show all currently processing tasks grouped by worker.
Displays each worker's active tasks with operation type, bank, how long
the task has been running, and when it was last updated. Useful for
identifying dead workers with orphaned tasks.
"""
config = HindsightConfig.from_env()
if not config.database_url:
typer.echo("Error: Database URL not configured.", err=True)
typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
raise typer.Exit(1)
rows = asyncio.run(_worker_status(config.database_url, schema))
if not rows:
typer.echo("No processing tasks found")
return
# Group by worker_id
by_worker: dict[str, list[dict[str, Any]]] = {}
for row in rows:
wid = row["worker_id"] or "unknown"
by_worker.setdefault(wid, []).append(row)
typer.echo(f"Processing tasks across {len(by_worker)} worker(s):\n")
for wid, tasks in by_worker.items():
typer.echo(f"Worker: {wid} ({len(tasks)} task(s))")
for task in tasks:
op_id = str(task["operation_id"])[:8]
running_for = task["running_for"]
last_update = task["last_update_ago"]
typer.echo(
f" {op_id} {task['operation_type']:<20s} bank={task['bank_id']}"
f" running={running_for} last_update={last_update} ago"
)
typer.echo("")
def main():
app()
@@ -1,42 +0,0 @@
"""Dialect dispatcher for Alembic migrations.
Each migration file declares a ``_pg_upgrade``/``_oracle_upgrade`` (and matching
downgrades) function and routes ``upgrade()``/``downgrade()`` through
``run_for_dialect``. The helper inspects the live connection's dialect name and
runs the matching function — or no-ops if the migration doesn't apply to the
current backend.
Use ``None`` (or omit the kwarg) when a migration intentionally has no effect
on a dialect; the helper treats it as a no-op.
"""
from __future__ import annotations
from collections.abc import Callable
from alembic import op
DialectFn = Callable[[], None]
_SUPPORTED = ("postgresql", "oracle")
def run_for_dialect(
*,
pg: DialectFn | None = None,
oracle: DialectFn | None = None,
) -> None:
"""Dispatch to the function matching the current bind's dialect.
Args:
pg: Function to run when the active bind is PostgreSQL.
oracle: Function to run when the active bind is Oracle.
Unrecognized dialects raise; an explicit ``None`` for the active dialect
is a no-op (the migration deliberately does nothing here).
"""
name = op.get_bind().dialect.name
if name not in _SUPPORTED:
raise RuntimeError(f"Unsupported dialect for migration dispatch: {name!r}. Expected one of {_SUPPORTED}.")
fn = {"postgresql": pg, "oracle": oracle}[name]
if fn is not None:
fn()
+76 -101
View File
@@ -1,37 +1,28 @@
"""
Alembic environment for Hindsight.
Supports two dialects:
* PostgreSQL (sync psycopg2 driver) — default; uses ``search_path`` for
multi-tenant schema isolation and forces read-write transactions to work
around Supabase's read-only-by-default sessions.
* Oracle 23ai (``oracledb`` driver) — uses ``CURRENT_SCHEMA`` for tenant
isolation; no equivalent of ``search_path`` or read-only session quirks.
Each migration file dispatches its DDL through ``alembic._dialect.run_for_dialect``
so a single revision tree serves both backends.
Alembic environment configuration for SQLAlchemy with pgvector.
Uses synchronous psycopg2 driver for migrations to avoid pgbouncer issues.
"""
import logging
import os
from pathlib import Path
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
from alembic import context
from dotenv import load_dotenv
from sqlalchemy import Connection, engine_from_config, pool
from sqlalchemy.engine import Engine
from sqlalchemy import engine_from_config, pool
from hindsight_api.db_url import is_oracle_url, to_libpq_url
# Import your models here
from hindsight_api.models import Base
def load_env() -> None:
"""Load environment variables from .env (skipped if already configured)."""
# Load environment variables based on HINDSIGHT_API_DATABASE_URL env var or default to local
def load_env():
"""Load environment variables from .env"""
# Check if HINDSIGHT_API_DATABASE_URL is already set (e.g., by CI/CD)
if os.getenv("HINDSIGHT_API_DATABASE_URL"):
return
# Look for .env file in the parent directory (root of the workspace)
root_dir = Path(__file__).parent.parent.parent
env_file = root_dir / ".env"
@@ -41,45 +32,30 @@ def load_env() -> None:
load_env()
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Note: We don't call fileConfig() here to avoid overriding the application's logging configuration.
# Alembic will use the existing logging configuration from the application.
# add your model's MetaData object here
# for 'autogenerate' support
target_metadata = Base.metadata
def _normalize_oracle_url(url: str) -> str:
"""Coerce an Oracle URL into the SQLAlchemy form the oracledb dialect expects.
Two issues to handle:
1. Force the ``oracle+oracledb`` driver — bare ``oracle://`` defaults to
cx_Oracle.
2. Map a path-style service to ``?service_name=...``. SQLAlchemy's oracledb
dialect treats the URL path as a *SID* (legacy), but Oracle Free /
Autonomous DB only register a service name. Without this rewrite we get
``DPY-6003: SID "FREEPDB1" is not registered`` even though the listener
is happy to accept the same name as a service.
"""
parts = urlsplit(url)
if not parts.scheme.startswith("oracle"):
return url
new_scheme = "oracle+oracledb" if parts.scheme == "oracle" else parts.scheme
service = parts.path.lstrip("/")
new_query = parts.query
new_path = parts.path
# Promote /SERVICE to ?service_name=SERVICE unless the caller already
# supplied an explicit ?sid= or ?service_name=.
if service and "service_name=" not in new_query and "sid=" not in new_query:
params = [(k, v) for k, v in parse_qsl(new_query, keep_blank_values=True)]
params.append(("service_name", service))
new_query = urlencode(params)
new_path = ""
return urlunsplit((new_scheme, parts.netloc, new_path, new_query, parts.fragment))
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def get_database_url() -> str:
"""Resolve the migration URL from Alembic config or env, normalizing per-dialect."""
"""
Get and process the database URL from config or environment.
Returns the URL with the correct driver (psycopg2) for migrations.
"""
# Get database URL from config (set programmatically) or environment
database_url = config.get_main_option("sqlalchemy.url")
if not database_url:
database_url = os.getenv("HINDSIGHT_API_DATABASE_URL")
@@ -89,18 +65,30 @@ def get_database_url() -> str:
"Set HINDSIGHT_API_DATABASE_URL environment variable or pass database_url to run_migrations()."
)
if is_oracle_url(database_url):
database_url = _normalize_oracle_url(database_url)
else:
# PG: convert SQLAlchemy-style asyncpg URLs and ?ssl= params to libpq form
# for the sync engine used during migrations.
database_url = to_libpq_url(database_url)
# For migrations, use psycopg2 (sync driver) to avoid pgbouncer prepared statement issues
if database_url.startswith("postgresql+asyncpg://"):
database_url = database_url.replace("postgresql+asyncpg://", "postgresql://", 1)
elif database_url.startswith("postgres+asyncpg://"):
database_url = database_url.replace("postgres+asyncpg://", "postgresql://", 1)
# Update config with processed URL for engine_from_config to use
config.set_main_option("sqlalchemy.url", database_url)
return database_url
def run_migrations_offline() -> None:
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
logging.info("running offline")
database_url = get_database_url()
@@ -115,40 +103,14 @@ def run_migrations_offline() -> None:
context.run_migrations()
def _configure_pg_session(engine: Engine, connection: Connection, target_schema: str | None) -> None:
"""PG-only: ensure the session is RW (Supabase) and bind ``search_path``."""
def run_migrations_online() -> None:
"""Run migrations in 'online' mode with synchronous engine."""
from sqlalchemy import event, text
@event.listens_for(engine, "connect")
def set_read_write_mode(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE")
if target_schema:
cursor.execute(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"')
cursor.execute(f'SET search_path TO "{target_schema}", public')
cursor.close()
get_database_url() # Process and set the database URL in config
connection.execute(text("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE"))
if target_schema:
connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"'))
connection.execute(text(f'SET search_path TO "{target_schema}", public'))
connection.commit()
def _configure_oracle_session(connection: Connection, target_schema: str | None) -> None:
"""Oracle: switch the session's default schema; tolerate DDL contention."""
from sqlalchemy import text
# Wait up to 30s for DDL locks instead of failing immediately (ORA-00054).
connection.execute(text("ALTER SESSION SET DDL_LOCK_TIMEOUT = 30"))
if target_schema:
connection.execute(text(f'ALTER SESSION SET CURRENT_SCHEMA = "{target_schema}"'))
def run_migrations_online() -> None:
database_url = get_database_url()
# Check if we're targeting a specific schema (for multi-tenant isolation)
target_schema = config.get_main_option("target_schema")
is_oracle = is_oracle_url(database_url)
connectable = engine_from_config(
config.get_section(config.config_ini_section, {}),
@@ -156,19 +118,37 @@ def run_migrations_online() -> None:
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
if is_oracle:
_configure_oracle_session(connection, target_schema)
else:
_configure_pg_session(connectable, connection, target_schema)
# Add event listener to ensure connection is in read-write mode
# This is needed for Supabase which may start connections in read-only mode
@event.listens_for(connectable, "connect")
def set_read_write_mode(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE")
# If targeting a specific schema, set search_path
# Include public in search_path for access to shared extensions (pgvector)
if target_schema:
cursor.execute(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"')
cursor.execute(f'SET search_path TO "{target_schema}", public')
cursor.close()
with connectable.connect() as connection:
# Also explicitly set read-write mode on this connection
connection.execute(text("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE"))
# If targeting a specific schema, set search_path
# Include public in search_path for access to shared extensions (pgvector)
if target_schema:
connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"'))
connection.execute(text(f'SET search_path TO "{target_schema}", public'))
connection.commit() # Commit the SET command
# Configure context with version_table_schema if using a specific schema
context_opts = {
"connection": connection,
"target_metadata": target_metadata,
}
if target_schema and not is_oracle:
# Oracle has no equivalent of PG's per-schema version table; the
# ``alembic_version`` table lives in CURRENT_SCHEMA implicitly.
if target_schema:
context_opts["version_table_schema"] = target_schema
context.configure(**context_opts)
@@ -176,12 +156,7 @@ def run_migrations_online() -> None:
with context.begin_transaction():
context.run_migrations()
# Always commit. PG needs it for the explicit RW-mode SET to persist;
# Oracle needs it because each DDL auto-commits but the trailing
# ``UPDATE alembic_version`` is plain DML that would otherwise stay in
# an open transaction and roll back when the connection closes —
# producing the "schema is created but the version row is one revision
# behind" failure mode.
# Explicit commit to ensure changes are persisted (especially for Supabase)
connection.commit()
@@ -11,8 +11,6 @@ from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = ${repr(up_revision)}
down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
@@ -20,27 +18,11 @@ branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
def _pg_upgrade() -> None:
"""PostgreSQL upgrade. Set to ``None`` below if this migration is Oracle-only."""
def upgrade() -> None:
"""Upgrade schema."""
${upgrades if upgrades else "pass"}
def _pg_downgrade() -> None:
${downgrades if downgrades else "pass"}
def _oracle_upgrade() -> None:
"""Oracle upgrade. Set to ``None`` below if this migration is Postgres-only."""
pass
def _oracle_downgrade() -> None:
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
"""Downgrade schema."""
${downgrades if downgrades else "pass"}
@@ -1,55 +0,0 @@
"""Recreate entities trigram index on LOWER(canonical_name) for case-insensitive matching
The previous GIN trigram index on canonical_name was case-sensitive, causing
"Alice" and "alice" to have different trigram sets. This recreates it on
LOWER(canonical_name) so the % operator matches case-insensitively.
Revision ID: 2eee35aa3cfc
Revises: d6e7f8a9b0c1
Create Date: 2026-03-31
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "2eee35aa3cfc"
down_revision: str | Sequence[str] | None = "d6e7f8a9b0c1"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# Drop the old case-sensitive trigram index
op.execute("DROP INDEX IF EXISTS entities_canonical_name_trgm_idx")
# Create case-insensitive trigram index on LOWER(canonical_name)
op.execute(
f"CREATE INDEX IF NOT EXISTS entities_canonical_name_lower_trgm_idx "
f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops)"
)
def _pg_downgrade() -> None:
op.execute("DROP INDEX IF EXISTS entities_canonical_name_lower_trgm_idx")
schema = _get_schema_prefix()
# Restore original case-sensitive index
op.execute(
f"CREATE INDEX IF NOT EXISTS entities_canonical_name_trgm_idx "
f"ON {schema}entities USING GIN (canonical_name gin_trgm_ops)"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -15,8 +15,6 @@ from pgvector.sqlalchemy import Vector
from sqlalchemy import text
from sqlalchemy.dialects import postgresql
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "5a366d414dce"
down_revision: str | Sequence[str] | None = None
@@ -114,7 +112,7 @@ def _detect_text_search_extension() -> str:
)
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Upgrade schema - create all tables from scratch."""
# Note: pgvector extension is installed globally BEFORE migrations run
@@ -465,7 +463,7 @@ def _pg_upgrade() -> None:
op.create_index("idx_unit_entities_entity", "unit_entities", ["entity_id"])
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Downgrade schema - drop all tables."""
# Drop tables in reverse dependency order
@@ -525,11 +523,3 @@ def _pg_downgrade() -> None:
# Drop extensions (optional - comment out if you want to keep them)
# op.execute('DROP EXTENSION IF EXISTS vector')
# op.execute('DROP EXTENSION IF EXISTS "uuid-ossp"')
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,50 +0,0 @@
"""Merge divergent migration heads for v0.5.3
v0.5.3 shipped with two migration heads that were never unified:
* ``c4x5y6z7a8b9`` — delta-refresh chain
(``add_last_refreshed_source_query`` ->
``add_structured_content_to_mental_models`` ->
``backsweep_orphan_observations_v2``)
* ``h3i4j5k6l7m8`` — per-bank vector indexes / audit log chain
(the ``merge_heads_and_add_unit_entities_index`` subtree)
Both fork from ``z1u2v3w4x5y6``. Upgrades from v0.5.2 still succeed — the
walker applies the three c4x5 revisions and leaves the database stamped at
both heads — but the result is a split DAG: ``alembic upgrade head``
(singular) is ambiguous, and any future migration has to pick one head as
its parent, orphaning the other.
This revision linearises the DAG into a single head. It has no schema
effect.
Revision ID: 8c6fa6f7230b
Revises: c4x5y6z7a8b9, h3i4j5k6l7m8
Create Date: 2026-04-18
"""
from collections.abc import Sequence
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "8c6fa6f7230b"
down_revision: str | Sequence[str] | None = ("c4x5y6z7a8b9", "h3i4j5k6l7m8")
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
pass
def _pg_downgrade() -> None:
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,128 +0,0 @@
"""Make memory_links.from_unit_id and memory_links.to_unit_id FKs deferrable.
Revision ID: 9f8e7d6c5b4a
Revises: o1a2b3c4d5e6
Create Date: 2026-05-03
Background
----------
Concurrent retain (which INSERTs into ``memory_links``) and any code path
that DELETEs a row whose deletion cascades into ``memory_links`` (e.g.
delta-retain superseding chunks, which CASCADEs chunks → memory_units →
memory_links) can deadlock under sustained single-tenant write load.
The deadlock cycle:
* Tx A: ``DELETE FROM chunks WHERE chunk_id = ANY(...)``
→ CASCADE acquires row locks on memory_units, then on memory_links rows
where ``to_unit_id`` matches the deleted units.
* Tx B: ``INSERT INTO memory_links (...)`` referencing one of the same
memory_units rows.
→ The immediate FK check takes ``FOR KEY SHARE`` on those memory_units
rows.
The two transactions take row locks on the same memory_units rows in
opposite orders depending on which side started first. PostgreSQL detects
the cycle and aborts one transaction; the loser is killed mid-batch, the
winner continues. Workers then retry, but under sustained write load the
pattern repeats.
Fix
---
Make both ``memory_links → memory_units`` FKs (``from_unit_id`` and
``to_unit_id``) ``DEFERRABLE INITIALLY DEFERRED``. This pushes the FK
check from INSERT time to COMMIT time:
* INSERT no longer takes ``FOR KEY SHARE`` on the memory_units row → no
contention with the cascading DELETE's row lock.
* At COMMIT the engine validates referential integrity in one shot. If a
cascade-DELETE has since removed the referenced unit, the INSERT
transaction commits OR fails with a clean FK violation (sqlstate
23503) instead of a deadlock (sqlstate 40P01).
The ``WHERE EXISTS`` filter already in ``_bulk_insert_links`` continues to
filter out the typical "stale unit_id" case at INSERT time; the deferred
FK is only the backstop for the narrow race window between the EXISTS
probe and COMMIT. ``ON DELETE CASCADE`` semantics are unchanged — only
the *timing* of the constraint check moves.
The ``entity_id`` FK on ``memory_links`` is not changed; entities are not
involved in the observed deadlock cycle and leaving the constraint
immediate keeps the error message specific when an entity row is missing.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "9f8e7d6c5b4a"
down_revision: str | Sequence[str] | None = "o1a2b3c4d5e6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
# The two FK constraints installed by the initial schema migration
# (5a366d414dce_initial_schema), mapped to the column they constrain.
# They reference memory_units(id) with ON DELETE CASCADE — that
# semantics is preserved; only the deferral attribute changes.
_FK_COLUMNS: dict[str, str] = {
"fk_memory_links_from_unit_id_memory_units": "from_unit_id",
"fk_memory_links_to_unit_id_memory_units": "to_unit_id",
}
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# PostgreSQL doesn't allow altering the deferrability of an existing
# constraint with ALTER CONSTRAINT — the constraint must be dropped
# and recreated. DROP IF EXISTS makes the migration safe to re-run
# on schemas where the constraint was already recreated.
for fk_name, column in _FK_COLUMNS.items():
op.execute(f"ALTER TABLE {schema}memory_links DROP CONSTRAINT IF EXISTS {fk_name}")
op.execute(
f"""
ALTER TABLE {schema}memory_links
ADD CONSTRAINT {fk_name}
FOREIGN KEY ({column})
REFERENCES {schema}memory_units (id)
ON DELETE CASCADE
DEFERRABLE INITIALLY DEFERRED
"""
)
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
# Revert to the default (NOT DEFERRABLE) form so a downgrade actually
# restores the prior schema state, even though that re-introduces the
# deadlock window.
for fk_name, column in _FK_COLUMNS.items():
op.execute(f"ALTER TABLE {schema}memory_links DROP CONSTRAINT IF EXISTS {fk_name}")
op.execute(
f"""
ALTER TABLE {schema}memory_links
ADD CONSTRAINT {fk_name}
FOREIGN KEY ({column})
REFERENCES {schema}memory_units (id)
ON DELETE CASCADE
"""
)
def upgrade() -> None:
# PG-only: Oracle's deferrable-FK semantics differ and the deadlock
# cycle was only observed on PostgreSQL. Oracle slot intentionally
# absent → no-op there.
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -15,8 +15,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a1b2c3d4e5f6"
down_revision: str | Sequence[str] | None = "y0t1u2v3w4x5"
branch_labels: str | Sequence[str] | None = None
@@ -29,7 +27,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Create file_storage table for BYTEA storage."""
schema = _get_schema_prefix()
@@ -54,7 +52,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove file_storage table and related columns."""
schema = _get_schema_prefix()
@@ -70,11 +68,3 @@ def _pg_downgrade() -> None:
# Drop file_storage table
op.execute(f"DROP TABLE IF EXISTS {schema}file_storage")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -18,8 +18,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a2b3c4d5e6f7"
down_revision: str | Sequence[str] | None = "aa2b3c4d5e6f"
branch_labels: str | Sequence[str] | None = None
@@ -35,7 +33,7 @@ def _detect_text_search_extension() -> str:
return os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
table = f"{schema}memory_units"
text_search_ext = _detect_text_search_extension()
@@ -67,7 +65,7 @@ def _pg_upgrade() -> None:
# pg_textsearch: no change — index operates on the base `text` column only
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
table = f"{schema}memory_units"
text_search_ext = _detect_text_search_extension()
@@ -88,11 +86,3 @@ def _pg_downgrade() -> None:
""")
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS text_signals")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -24,8 +24,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a2b3c4d5e6f8"
down_revision: str | Sequence[str] | None = "f7g8h9i0j1k2"
branch_labels: str | Sequence[str] | None = None
@@ -37,7 +35,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
@@ -50,15 +48,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,48 +0,0 @@
"""Add last_refreshed_source_query column to mental_models
Revision ID: a2v3w4x5y6z7
Revises: z1u2v3w4x5y6
Create Date: 2026-04-15
Tracks the source_query that was used during the most recent refresh.
Used by delta-mode refresh to detect when the query has changed: if it has,
delta mode falls back to a full regeneration because the surgical-edit
assumption (same topic, new facts) no longer holds.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a2v3w4x5y6z7"
down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"""
ALTER TABLE {schema}mental_models
ADD COLUMN IF NOT EXISTS last_refreshed_source_query TEXT
""")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS last_refreshed_source_query")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -13,8 +13,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a3b4c5d6e7f8"
down_revision: str | Sequence[str] | None = "g7h8i9j0k1l2"
branch_labels: str | Sequence[str] | None = None
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(
@@ -47,16 +45,8 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_consolidation_failed")
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidation_failed_at")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,152 +0,0 @@
"""Fix per-bank vector indexes to match configured extension
Revision ID: a4b5c6d7e8f9
Revises: 2eee35aa3cfc
Create Date: 2026-04-01
Migration d5e6f7a8b9c0 hardcoded HNSW when creating per-bank partial vector
indexes, ignoring HINDSIGHT_API_VECTOR_EXTENSION. Banks that existed when that
migration ran got HNSW indexes even when pgvectorscale (DiskANN) or vchord
was configured.
This migration detects the mismatch and recreates the affected indexes with
the correct type. Skipped entirely when the configured extension is pgvector
(the default), since those indexes are already correct.
"""
import os
from collections.abc import Sequence
from alembic import context, op
from sqlalchemy import text
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "a4b5c6d7e8f9"
down_revision: str | Sequence[str] | None = "2eee35aa3cfc"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
_FACT_TYPES: dict[str, str] = {
"world": "worl",
"experience": "expr",
"observation": "obsv",
}
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _target_index_type() -> str | None:
"""Return the target index type, or None if pgvector (no fix needed)."""
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
if ext == "pgvectorscale":
return "diskann"
elif ext == "vchord":
return "vchordrq"
return None
def _vector_index_using_clause() -> str:
"""Return the USING clause based on the configured vector extension."""
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
if ext == "pgvectorscale":
return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
elif ext == "vchord":
return "USING vchordrq (embedding vector_l2_ops)"
else:
return "USING hnsw (embedding vector_cosine_ops)"
def _pg_upgrade() -> None:
target = _target_index_type()
if target is None:
# pgvector — indexes are already HNSW, nothing to fix
return
bind = op.get_bind()
schema_name = context.config.get_main_option("target_schema")
schema = _get_schema_prefix()
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
using_clause = _vector_index_using_clause()
pg_schema = schema_name or "public"
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
for row in rows:
bank_id = row[0]
internal_id = str(row[1]).replace("-", "")[:16]
escaped_bank_id = bank_id.replace("'", "''")
for ft, ft_short in _FACT_TYPES.items():
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
# Check if this index exists and what type it is
idx_info = bind.execute(
text("SELECT indexdef FROM pg_indexes WHERE schemaname = :schema AND indexname = :idx"),
{"schema": pg_schema, "idx": idx_name},
).fetchone()
if idx_info is None:
# Index doesn't exist — create it with the correct type
bind.execute(
text(
f"CREATE INDEX IF NOT EXISTS {idx_name} "
f"ON {table_ref} {using_clause} "
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
)
)
continue
indexdef = idx_info[0].lower()
if target in indexdef:
# Already the correct type
continue
# Wrong type — drop and recreate
bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
bind.execute(
text(
f"CREATE INDEX IF NOT EXISTS {idx_name} "
f"ON {table_ref} {using_clause} "
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
)
)
def _pg_downgrade() -> None:
# Downgrade recreates indexes as HNSW (the original hardcoded behavior)
target = _target_index_type()
if target is None:
return
bind = op.get_bind()
schema_name = context.config.get_main_option("target_schema")
schema = _get_schema_prefix()
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
for row in rows:
bank_id = row[0]
internal_id = str(row[1]).replace("-", "")[:16]
escaped_bank_id = bank_id.replace("'", "''")
for ft, ft_short in _FACT_TYPES.items():
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
bind.execute(
text(
f"CREATE INDEX IF NOT EXISTS {idx_name} "
f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
)
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "aa2b3c4d5e6f"
down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
branch_labels: str | Sequence[str] | None = None
@@ -26,21 +24,13 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date DROP NOT NULL")
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
# Backfill NULLs with now() before restoring the NOT NULL constraint
op.execute(f"UPDATE {schema}memory_units SET event_date = now() WHERE event_date IS NULL")
op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date SET NOT NULL")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,42 +0,0 @@
"""add content_hash to chunks table for delta retain
Revision ID: b3c4d5e6f7a8
Revises: a3b4c5d6e7f8
Create Date: 2026-03-25
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "b3c4d5e6f7a8"
down_revision: str | Sequence[str] | None = "a3b4c5d6e7f8"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# Add content_hash column to chunks table for delta comparison
op.execute(f"ALTER TABLE {schema}chunks ADD COLUMN IF NOT EXISTS content_hash TEXT")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}chunks DROP COLUMN IF EXISTS content_hash")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -22,8 +22,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "b3c4d5e6f7g8"
down_revision: str | Sequence[str] | None = "c1a2b3d4e5f6"
branch_labels: str | Sequence[str] | None = None
@@ -35,7 +33,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# Partial index on occurred_start (covers "occurred_start BETWEEN $4 AND $5")
op.execute("COMMIT")
@@ -60,7 +58,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_mentioned_at")
@@ -68,11 +66,3 @@ def _pg_downgrade() -> None:
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_end")
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_start")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,54 +0,0 @@
"""Add structured_content JSONB column to mental_models
Revision ID: b3w4x5y6z7a8
Revises: a2v3w4x5y6z7
Create Date: 2026-04-16
Stores the structured representation of a mental model document (sections,
blocks). The plain ``content`` column remains the rendered markdown shown to
users. ``structured_content`` is the source of truth for delta-mode refreshes:
each refresh applies a list of typed operations to the structured doc, then
re-renders to markdown — so unchanged sections come through byte-identical
without an LLM round-trip.
Nullable: existing markdown-only mental models continue to work in full mode;
the column is populated lazily the first time a model is refreshed in delta
mode.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "b3w4x5y6z7a8"
down_revision: str | Sequence[str] | None = "a2v3w4x5y6z7"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"""
ALTER TABLE {schema}mental_models
ADD COLUMN IF NOT EXISTS structured_content JSONB
""")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS structured_content")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -14,8 +14,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "b4c5d6e7f8a9"
down_revision: str | Sequence[str] | None = "a2b3c4d5e6f7"
branch_labels: str | Sequence[str] | None = None
@@ -27,18 +25,10 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS observation_scopes JSONB")
def _pg_downgrade() -> None:
pass # intentionally no-op — safe to leave the column in place
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
pass # intentionally no-op — safe to leave the column in place
@@ -1,91 +0,0 @@
"""Backfill entity_cooccurrences.last_cooccurred from memory_units event time
Revision ID: b5d4e3f2a1c9
Revises: o1a2b3c4d5e6
Create Date: 2026-04-24
The writer path in `entity_resolver.link_units_to_entities_batch` historically
stamped `entity_cooccurrences.last_cooccurred` with `datetime.now(UTC)` at
flush time, ignoring the source memory unit's event date. For normal online
retains that's fine (now ≈ event time), but for any corpus that was
backfilled in a single session — migrating from another memory system, for
example — every co-occurrence collapsed to the import moment, which hid the
underlying knowledge timeline from the dashboard's entity graph recency heat
and from any downstream consumer of the column.
The writer is fixed in the same change set to propagate the unit's event_date;
this migration repairs historical rows by reading the true event time off
`unit_entities × memory_units` (falling back to `created_at` when
`mentioned_at` / `occurred_start` are NULL, so rows never regress).
Oracle slot is intentionally absent: the Oracle baseline (`o1a2b3c4d5e6`)
landed days before this fix, so any Oracle deployment runs the corrected
writer against an effectively empty `entity_cooccurrences` — there is no
historical residue on Oracle to repair. PG-only matches the asymmetry of
the data, not negligence.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "b5d4e3f2a1c9"
down_revision: str | Sequence[str] | None = "o1a2b3c4d5e6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# Recompute last_cooccurred from the true event time per entity pair.
# COALESCE picks the first non-null of mentioned_at / occurred_start /
# created_at so banks without event-time metadata still see a sane value
# (equivalent to the pre-fix behaviour) instead of NULL.
#
# The self-join on `unit_entities` is O(k²) per memory_unit in the number
# of distinct entities mentioned (k). For typical units k is small (single
# digits), but a bank with units containing hundreds of entities and tens
# of millions of co-occurrence rows may want to run this off-hours — the
# whole UPDATE is one statement, so it locks every targeted ec row for
# the duration. The migration is one-time; subsequent online writes
# already carry event time via the writer fix.
op.execute(
f"""
UPDATE {schema}entity_cooccurrences ec
SET last_cooccurred = sub.event_time
FROM (
SELECT
LEAST(ue1.entity_id, ue2.entity_id) AS e1,
GREATEST(ue1.entity_id, ue2.entity_id) AS e2,
MAX(COALESCE(mu.mentioned_at, mu.occurred_start, mu.created_at)) AS event_time
FROM {schema}memory_units mu
JOIN {schema}unit_entities ue1 ON ue1.unit_id = mu.id
JOIN {schema}unit_entities ue2 ON ue2.unit_id = mu.id AND ue1.entity_id <> ue2.entity_id
GROUP BY 1, 2
) sub
WHERE ec.entity_id_1 = sub.e1 AND ec.entity_id_2 = sub.e2
"""
)
def _pg_downgrade() -> None:
# No-op: the previous column value was `now()` at the time of write and
# isn't recoverable. Rolling back the code is sufficient — new writes will
# revert to the old behaviour for subsequent retains.
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade) # oracle slot intentionally absent — see header
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "b7c4d8e9f1a2"
down_revision: str | Sequence[str] | None = "5a366d414dce"
@@ -21,7 +19,7 @@ branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add chunks table and link memory_units to chunks."""
# Create chunks table with single text PK (bank_id_document_id_chunk_index)
@@ -58,7 +56,7 @@ def _pg_upgrade() -> None:
op.create_index("idx_memory_units_chunk_id", "memory_units", ["chunk_id"])
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove chunks table and chunk_id from memory_units."""
# Drop index and foreign key from memory_units
@@ -70,11 +68,3 @@ def _pg_downgrade() -> None:
op.drop_index("idx_chunks_bank_id", table_name="chunks")
op.drop_index("idx_chunks_document_id", table_name="chunks")
op.drop_table("chunks")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -11,11 +11,8 @@ block; see migrations.py for how this is handled safely.
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "c1a2b3d4e5f6"
down_revision: str | Sequence[str] | None = "b4c5d6e7f8a9"
branch_labels: str | Sequence[str] | None = None
@@ -27,22 +24,10 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
# pg_trgm ships with most PostgreSQL installations as a contrib module.
def upgrade() -> None:
# pg_trgm ships with every standard PostgreSQL installation as a contrib module.
# It enables fast similarity lookups via GIN indexes, used for entity name matching.
# On managed services (e.g. Azure Flexible Server), the extension may not be
# available or may require manual enablement. We gracefully skip the index
# creation if the extension cannot be loaded — the entity resolver will
# auto-detect and fall back to the "full" lookup strategy at runtime. See #626.
conn = op.get_bind()
try:
conn.execute(sa.text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
except Exception:
# Extension not available (managed Postgres, insufficient privileges, etc.)
# Roll back the failed statement and skip index creation.
conn.execute(sa.text("ROLLBACK"))
conn.execute(sa.text("BEGIN"))
return
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
schema = _get_schema_prefix()
# GIN index on canonical_name enables sub-millisecond trigram similarity queries
@@ -54,16 +39,8 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}entities_canonical_name_trgm_idx")
# Note: not dropping pg_trgm extension as other indexes may depend on it
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,71 +0,0 @@
"""Add audit_log table for feature usage tracking.
Merge migration that combines the two existing heads (a3b4c5d6e7f8 + c8e5f2a3b4d1).
Stores raw request/response as JSONB for expandability without future migrations.
The metadata JSONB column allows adding arbitrary fields in the future.
Revision ID: c2d3e4f5g6h7
Revises: a3b4c5d6e7f8, c8e5f2a3b4d1
Create Date: 2026-03-26
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "c2d3e4f5g6h7"
down_revision: str | Sequence[str] | None = ("a3b4c5d6e7f8", "c8e5f2a3b4d1")
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
op.execute(
f"""
CREATE TABLE IF NOT EXISTS {schema}audit_log (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
action TEXT NOT NULL,
transport TEXT NOT NULL,
bank_id TEXT,
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
ended_at TIMESTAMPTZ,
request JSONB,
response JSONB,
metadata JSONB DEFAULT '{{}}'::jsonb
)
"""
)
op.execute(
f"CREATE INDEX IF NOT EXISTS idx_audit_log_action_started ON {schema}audit_log (action, started_at DESC)"
)
op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_bank_started ON {schema}audit_log (bank_id, started_at DESC)")
op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_started ON {schema}audit_log (started_at DESC)")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_started")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_bank_started")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_action_started")
op.execute(f"DROP TABLE IF EXISTS {schema}audit_log")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -9,8 +9,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "c3d4e5f6g7h8"
down_revision: str | Sequence[str] | None = ("a2b3c4d5e6f7", "a2b3c4d5e6f8")
branch_labels: str | Sequence[str] | None = None
@@ -22,19 +20,11 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,76 +0,0 @@
"""backsweep_orphan_observations_v2
Re-run of Pass 2 from migration ``g7h8i9j0k1l2_backsweep_orphan_observations``
to sweep observations that became orphaned between then and now.
Why we need it again:
``fact_storage.handle_document_tracking`` (the retain/upsert path) deleted
the existing document via the FK cascade — which removes the source
``memory_units`` — but never invalidated the observations derived from
them. Only the explicit ``MemoryEngine.delete_document`` API called
``_delete_stale_observations_for_memories``. Every document re-ingest
therefore left orphan observations whose ``source_memory_ids`` arrays
pointed at IDs that no longer existed in ``memory_units``.
``handle_document_tracking`` now calls the same cleanup helper before the
cascade, so no new orphans will accumulate going forward. This migration
cleans up the historical residue.
Identical to Pass 2 of g7h8i9j0k1l2. Pass 1 (memory_units whose bank is
gone) is intentionally not re-run; that scenario has no fresh source.
Revision ID: c4x5y6z7a8b9
Revises: b3w4x5y6z7a8
Create Date: 2026-04-16
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "c4x5y6z7a8b9"
down_revision: str | Sequence[str] | None = "b3w4x5y6z7a8"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
mu = f"{schema}memory_units"
# Delete observations whose every source_memory_id refers to a now-deleted
# memory_unit (or the array is empty). Observations with at least one
# surviving source are left alone — the consolidation engine will refresh
# their text on the next pass.
op.execute(
f"""
DELETE FROM {mu} orphan
WHERE orphan.fact_type = 'observation'
AND NOT EXISTS (
SELECT 1
FROM {mu} src
WHERE src.id = ANY(orphan.source_memory_ids)
AND src.bank_id = orphan.bank_id
)
"""
)
def _pg_downgrade() -> None:
# Deleted rows cannot be restored.
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,58 +0,0 @@
"""Add bank_id column to memory_links for direct filtering
The stats endpoint JOINs memory_links to memory_units just to filter by
bank_id. With millions of links this takes 18+ seconds. Adding bank_id
directly to memory_links lets Postgres push the filter down before the JOIN.
Revision ID: c5d6e7f8a9b0
Revises: b3c4d5e6f7a8
Create Date: 2026-03-26
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "c5d6e7f8a9b0"
down_revision: str | Sequence[str] | None = "b3c4d5e6f7a8"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# 1. Add nullable column
op.execute(f"ALTER TABLE {schema}memory_links ADD COLUMN IF NOT EXISTS bank_id TEXT")
# 2. Backfill from memory_units
op.execute(f"""
UPDATE {schema}memory_links ml
SET bank_id = mu.bank_id
FROM {schema}memory_units mu
WHERE ml.from_unit_id = mu.id
AND ml.bank_id IS NULL
""")
# 3. Set NOT NULL
op.execute(f"ALTER TABLE {schema}memory_links ALTER COLUMN bank_id SET NOT NULL")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}memory_links DROP COLUMN IF EXISTS bank_id")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "c8e5f2a3b4d1"
down_revision: str | Sequence[str] | None = "b7c4d8e9f1a2"
@@ -21,7 +19,7 @@ branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add retain_params JSONB column to documents table."""
# Add retain_params column to store parameters passed during retain
@@ -31,7 +29,7 @@ def _pg_upgrade() -> None:
op.create_index("idx_documents_retain_params", "documents", ["retain_params"], postgresql_using="gin")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove retain_params column from documents table."""
# Drop index
@@ -39,11 +37,3 @@ def _pg_downgrade() -> None:
# Drop column
op.drop_column("documents", "retain_params")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -34,8 +34,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "d2e3f4a5b6c7"
down_revision: str | Sequence[str] | None = "b3c4d5e6f7g8"
branch_labels: str | Sequence[str] | None = None
@@ -47,7 +45,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
@@ -77,17 +75,9 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_entity_covering")
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_to_type_weight")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -18,8 +18,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "d4e5f6g7h8i9"
down_revision: str | Sequence[str] | None = "d5e6f7a8b9c0"
branch_labels: str | Sequence[str] | None = None
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# DROP + CREATE CONCURRENTLY must run outside a transaction block.
op.execute("COMMIT")
@@ -44,7 +42,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute("COMMIT")
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
@@ -53,11 +51,3 @@ def _pg_downgrade() -> None:
f"ON {schema}memory_units USING GIN (source_memory_ids) "
f"WHERE source_memory_ids IS NOT NULL"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,4 +1,4 @@
"""Add internal_id to banks and per-(bank, fact_type) partial vector indexes
"""Add internal_id to banks and per-(bank, fact_type) partial HNSW indexes
Revision ID: d5e6f7a8b9c0
Revises: a3b4c5d6e7f8
@@ -6,33 +6,36 @@ Create Date: 2026-03-11
This migration:
1. Adds internal_id UUID column to banks (stable identifier for index naming)
2. Drops the global vector index (competes with per-bank partial indexes)
3. Creates per-(bank_id, fact_type) partial vector indexes for all existing banks
using the configured vector extension (HNSW for pgvector, DiskANN for
pgvectorscale, vchordrq for vchord).
(new banks get indexes created at bank-creation time via bank_utils.create_bank_vector_indexes)
2. Drops the global HNSW index (competes with per-bank partial indexes)
3. Creates per-(bank_id, fact_type) partial HNSW indexes for all existing banks
(new banks get indexes created at bank-creation time via bank_utils.create_bank_hnsw_indexes)
Why per-(bank, fact_type) indexes:
- fact_type-only partial indexes are never chosen by the planner when bank_id is in the WHERE
clause, because the idx_memory_units_bank_id B-tree index always wins at planning time.
- Per-(bank, fact_type) partial indexes have both predicates matching → planner selects them.
- The global vector index competes for larger partitions (world, observation) and must be dropped.
- The global HNSW index competes for larger partitions (world, observation) and must be dropped.
For large deployments, create indexes CONCURRENTLY before running this migration:
SELECT internal_id, bank_id FROM banks;
-- for each bank and each fact_type in (world, experience, observation):
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_mu_emb_{ft}_{uid16}
ON memory_units USING hnsw (embedding vector_cosine_ops)
WHERE fact_type = '{ft}' AND bank_id = '{bank_id}';
DROP INDEX CONCURRENTLY IF EXISTS idx_memory_units_embedding;
"""
import os
from collections.abc import Sequence
from alembic import context, op
from sqlalchemy import text
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "d5e6f7a8b9c0"
down_revision: str | Sequence[str] | None = "c3d4e5f6g7h8"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
_FACT_TYPES: dict[str, str] = {
_HNSW_FACT_TYPES: dict[str, str] = {
"world": "worl",
"experience": "expr",
"observation": "obsv",
@@ -44,18 +47,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _vector_index_using_clause() -> str:
"""Return the USING clause based on the configured vector extension."""
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
if ext == "pgvectorscale":
return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
elif ext == "vchord":
return "USING vchordrq (embedding vector_l2_ops)"
else:
return "USING hnsw (embedding vector_cosine_ops)"
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# 1. Add internal_id column to banks
@@ -64,41 +56,39 @@ def _pg_upgrade() -> None:
)
op.execute(f"ALTER TABLE {schema}banks ADD CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)")
# 2. Drop any fact_type-only partial indexes that may exist from prior migrations
# 2. Drop any fact_type-only partial HNSW indexes that may exist from prior migrations
# (bank_id B-tree always wins over them when bank_id is in the WHERE clause)
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_world")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_observation")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_experience")
# 4. Drop global vector index (competes with per-bank partial indexes)
# 4. Drop global HNSW index (competes with per-bank partial indexes)
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_embedding")
# 5. Create per-(bank, fact_type) partial vector indexes for all existing banks
# using the configured extension (HNSW / DiskANN / vchordrq)
# 5. Create per-(bank, fact_type) partial HNSW indexes for all existing banks
bind = op.get_bind()
schema_name = context.config.get_main_option("target_schema")
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
using_clause = _vector_index_using_clause()
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
for row in rows:
bank_id = row[0]
internal_id = str(row[1]).replace("-", "")[:16]
escaped_bank_id = bank_id.replace("'", "''")
for ft, ft_short in _FACT_TYPES.items():
for ft, ft_short in _HNSW_FACT_TYPES.items():
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
# Index name is schema-unqualified (indexes live in the schema of their table)
bind.execute(
text(
f"CREATE INDEX IF NOT EXISTS {idx_name} "
f"ON {table_ref} {using_clause} "
f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
)
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
# Drop per-bank HNSW indexes (iterate existing banks)
@@ -139,11 +129,3 @@ def _pg_downgrade() -> None:
# Drop internal_id column
op.execute(f"ALTER TABLE {schema}banks DROP CONSTRAINT IF EXISTS banks_internal_id_unique")
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS internal_id")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,67 +0,0 @@
"""Backfill mental_models.subtype for databases that ran h3c4d5e6f7g8 before the fix
Migration h3c4d5e6f7g8 used CREATE TABLE IF NOT EXISTS to create the
mental_models table with a subtype column. But on databases where the table
already existed (from the reflections -> mental_models rename chain), the
CREATE was a no-op and subtype was never added. A fix was later added to
h3c4d5e6f7g8 (Step 4b), but databases that had already run the migration
never re-execute it. This migration adds the missing columns idempotently.
Revision ID: d5y6z7a8b9c0
Revises: 8c6fa6f7230b
Create Date: 2026-04-18
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "d5y6z7a8b9c0"
down_revision: str | Sequence[str] | None = "8c6fa6f7230b"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# Add columns that h3c4d5e6f7g8 intended to create but missed when
# the table already existed from the reflections rename chain.
for col_ddl in [
"subtype VARCHAR(32) NOT NULL DEFAULT 'structural'",
"description TEXT NOT NULL DEFAULT ''",
"entity_id UUID",
"observations JSONB DEFAULT '{\"observations\": []}'::jsonb",
"links VARCHAR[]",
"last_updated TIMESTAMP WITH TIME ZONE",
]:
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS {col_ddl}")
# Ensure the CHECK constraint exists
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
op.execute(f"""
ALTER TABLE {schema}mental_models
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
""")
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
def _pg_downgrade() -> None:
# No-op: these columns are part of the intended schema
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,49 +0,0 @@
"""Drop unused metadata column from documents table
Revision ID: d6e7f8a9b0c1
Revises: c2d3e4f5g6h7, c5d6e7f8a9b0
Create Date: 2026-03-30
The metadata column on documents was always stored as an empty dict {}.
Actual document metadata is stored inside retain_params.metadata.
This migration was originally shipped in v0.4.22, then its file was deleted
in v0.5.0 (and its revision ID accidentally reused by 2eee35aa3cfc).
Restoring the file so that databases stamped at this revision can upgrade
cleanly to v0.5.x+.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "d6e7f8a9b0c1"
down_revision: str | Sequence[str] | None = ("c2d3e4f5g6h7", "c5d6e7f8a9b0")
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS metadata")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS metadata jsonb DEFAULT '{{}}'")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -8,8 +8,6 @@ Create Date: 2024-12-04 15:00:00.000000
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision = "d9f6a3b4c5e2"
down_revision = "c8e5f2a3b4d1"
@@ -23,7 +21,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade():
def upgrade():
schema = _get_schema_prefix()
# Drop old check constraint FIRST (before updating data)
@@ -40,7 +38,7 @@ def _pg_upgrade():
)
def _pg_downgrade():
def downgrade():
schema = _get_schema_prefix()
# Drop new check constraint FIRST
@@ -53,11 +51,3 @@ def _pg_downgrade():
op.create_check_constraint(
"memory_units_fact_type_check", "memory_units", "fact_type IN ('world', 'bank', 'opinion', 'observation')"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -14,8 +14,6 @@ from collections.abc import Sequence
import sqlalchemy as sa
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "e0a1b2c3d4e5"
down_revision: str | Sequence[str] | None = "rename_personality"
@@ -35,7 +33,7 @@ def _get_target_schema() -> str:
return schema if schema else "public"
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Convert Big Five disposition to 3-trait disposition."""
conn = op.get_bind()
schema = _get_schema_prefix()
@@ -77,7 +75,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Convert back to Big Five disposition."""
conn = op.get_bind()
schema = _get_schema_prefix()
@@ -111,11 +109,3 @@ def _pg_downgrade() -> None:
ALTER COLUMN disposition SET DEFAULT '{{"openness": 0.5, "conscientiousness": 0.5, "extraversion": 0.5, "agreeableness": 0.5, "neuroticism": 0.5, "bias_strength": 0.5}}'::jsonb
""")
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "e4f5a6b7c8d9"
down_revision: str | Sequence[str] | None = "d2e3f4a5b6c7"
branch_labels: str | Sequence[str] | None = None
@@ -25,7 +23,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(
@@ -56,17 +54,9 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_status_retry")
op.execute(f"ALTER TABLE {schema}async_operations DROP COLUMN IF EXISTS next_retry_at")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_webhooks_bank_id")
op.execute(f"DROP TABLE IF EXISTS {schema}webhooks")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -13,8 +13,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "e5f6g7h8i9j0"
down_revision: str | Sequence[str] | None = "d4e5f6g7h8i9"
branch_labels: str | Sequence[str] | None = None
@@ -26,7 +24,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# Remove orphaned async_operations rows whose bank no longer exists
@@ -69,15 +67,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS fk_async_operations_bank_id")
op.execute(f"ALTER TABLE {schema}webhooks DROP CONSTRAINT IF EXISTS fk_webhooks_bank_id")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -5,15 +5,13 @@ Revises: e0a1b2c3d4e5
Create Date: 2025-01-12
Add composite index on memory_links (from_unit_id, link_type, weight DESC)
to optimize graph traversal queries that need top-k edges per type.
to optimize MPFP graph traversal queries that need top-k edges per type.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "f1a2b3c4d5e6"
down_revision: str | Sequence[str] | None = "e0a1b2c3d4e5"
@@ -27,8 +25,8 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
"""Add composite index for efficient graph retrieval edge loading."""
def upgrade() -> None:
"""Add composite index for efficient MPFP edge loading."""
schema = _get_schema_prefix()
# Create composite index for efficient top-k per (from_node, link_type) queries
# This enables LATERAL joins to use index-only scans with early termination
@@ -40,15 +38,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove the composite index."""
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_links_from_type_weight")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -10,8 +10,6 @@ from collections.abc import Sequence
from alembic import op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "f6g7h8i9j0k1"
down_revision: str | Sequence[str] | None = "e5f6g7h8i9j0"
@@ -19,49 +17,22 @@ branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Change memory_units.chunk_id FK from SET NULL to CASCADE.
When a document is deleted the CASCADE reaches chunks first; with SET NULL
the memory_units rows survived with chunk_id = NULL, leaving ghost records.
Switching to CASCADE ensures they are removed together with their chunk.
"""
from alembic import context
schema = context.config.get_main_option("target_schema")
schema_prefix = f'"{schema}".' if schema else ""
# Use raw SQL with IF EXISTS so this is safe on schemas where the FK was
# already dropped or never existed under this name.
op.execute(f"ALTER TABLE {schema_prefix}memory_units DROP CONSTRAINT IF EXISTS memory_units_chunk_fkey")
# Use a DO block so the ADD is also idempotent: if the FK already exists (e.g.
# the schema was provisioned after the base migration already added it) the
# duplicate_object exception is swallowed rather than failing the migration.
op.execute(
f"""
DO $$ BEGIN
ALTER TABLE {schema_prefix}memory_units
ADD CONSTRAINT memory_units_chunk_fkey
FOREIGN KEY (chunk_id)
REFERENCES {schema_prefix}chunks (chunk_id)
ON DELETE CASCADE;
EXCEPTION
WHEN duplicate_object THEN NULL;
END $$;
"""
op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
op.create_foreign_key(
"memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="CASCADE"
)
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Revert to SET NULL behaviour."""
op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
op.create_foreign_key(
"memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="SET NULL"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "f7g8h9i0j1k2"
down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
branch_labels: str | Sequence[str] | None = None
@@ -25,19 +23,11 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}webhooks ADD COLUMN IF NOT EXISTS http_config JSONB NOT NULL DEFAULT '{{}}'")
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}webhooks DROP COLUMN IF EXISTS http_config")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "g2a3b4c5d6e7"
down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add tags column to memory_units and documents tables."""
schema = _get_schema_prefix()
@@ -41,18 +39,10 @@ def _pg_upgrade() -> None:
op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS tags VARCHAR[] NOT NULL DEFAULT '{{}}'")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove tags columns and index."""
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_tags")
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS tags")
op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS tags")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,93 +0,0 @@
"""remove_opinion_fact_type
Revision ID: g2h3i4j5k6l7
Revises: f1a2b3c4d5e6
Create Date: 2026-04-02
Remove the deprecated 'opinion' fact type: drop opinion-specific indexes,
update CHECK constraints, delete any remaining opinion rows, and drop the
confidence_score column (was only used for opinions, always NULL otherwise).
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "g2h3i4j5k6l7"
down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# 1. Delete any remaining opinion rows
op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
# 2. Drop opinion-specific indexes
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_confidence")
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_date")
# 3. Drop confidence_score constraints and column (only used for opinions, always NULL otherwise)
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS confidence_score_fact_type_check")
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_confidence_score_check")
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS confidence_score")
# 4. Replace fact_type CHECK constraint
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
op.execute(
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
f"CHECK (fact_type IN ('world', 'experience', 'observation'))"
)
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
# Restore confidence_score column
op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS confidence_score float")
op.execute(
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_confidence_score_check "
f"CHECK (confidence_score IS NULL OR (confidence_score >= 0.0 AND confidence_score <= 1.0))"
)
op.execute(
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT confidence_score_fact_type_check "
f"CHECK ((fact_type = 'opinion' AND confidence_score IS NOT NULL) OR "
f"(fact_type = 'observation') OR "
f"(fact_type NOT IN ('opinion', 'observation') AND confidence_score IS NULL))"
)
# Restore original fact_type CHECK constraint (with opinion)
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
op.execute(
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
f"CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))"
)
# Recreate opinion indexes
op.execute(
f"CREATE INDEX idx_memory_units_opinion_confidence ON {schema}memory_units "
f"(bank_id, confidence_score DESC) WHERE fact_type = 'opinion'"
)
op.execute(
f"CREATE INDEX idx_memory_units_opinion_date ON {schema}memory_units "
f"(bank_id, event_date DESC) WHERE fact_type = 'opinion'"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -22,8 +22,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "g7h8i9j0k1l2"
down_revision: str | Sequence[str] | None = "f6g7h8i9j0k1"
branch_labels: str | Sequence[str] | None = None
@@ -35,7 +33,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
mu = f"{schema}memory_units"
banks = f"{schema}banks"
@@ -68,14 +66,6 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
# Deleted rows cannot be restored.
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -17,8 +17,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "h3c4d5e6f7g8"
down_revision: str | Sequence[str] | None = "g2a3b4c5d6e7"
@@ -32,7 +30,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Apply mental models v4 changes."""
schema = _get_schema_prefix()
@@ -87,26 +85,6 @@ def _pg_upgrade() -> None:
)
""")
# Step 4b: If the table already existed (from reflections rename chain),
# it won't have the v4 columns. Add them idempotently so the migration
# works regardless of whether CREATE TABLE above was a no-op.
for col_ddl in [
"subtype VARCHAR(32) NOT NULL DEFAULT 'directive'",
"description TEXT NOT NULL DEFAULT ''",
"entity_id UUID",
"observations JSONB DEFAULT '{\"observations\": []}'::jsonb",
"links VARCHAR[]",
"last_updated TIMESTAMP WITH TIME ZONE",
]:
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS {col_ddl}")
# Ensure the subtype CHECK constraint exists (may not if table was renamed)
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
op.execute(f"""
ALTER TABLE {schema}mental_models
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
""")
# Step 5: Create indexes for efficient queries (if not exist)
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_bank_id ON {schema}mental_models(bank_id)")
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
@@ -115,7 +93,7 @@ def _pg_upgrade() -> None:
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_tags ON {schema}mental_models USING GIN(tags)")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Revert mental models v4 changes."""
schema = _get_schema_prefix()
@@ -132,11 +110,3 @@ def _pg_downgrade() -> None:
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission")
# Note: Cannot restore deleted observations - they are lost on downgrade
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,52 +0,0 @@
"""Merge 3 migration heads and add unit_entities composite index
Revision ID: h3i4j5k6l7m8
Revises: a4b5c6d7e8f9, g2h3i4j5k6l7
Create Date: 2026-04-07
Merges three unmerged migration heads into one, and adds a composite index
(entity_id, unit_id) on unit_entities for index-only scans in the LATERAL
entity expansion query.
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "h3i4j5k6l7m8"
down_revision: str | Sequence[str] | None = ("a4b5c6d7e8f9", "g2h3i4j5k6l7")
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
# Composite index enables index-only scans for entity_id -> unit_id lookups
op.execute(
f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity_unit ON {schema}unit_entities (entity_id, unit_id)"
)
# Drop the now-redundant single-column index
op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity")
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity_unit")
# Restore the single-column index
op.execute(f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity ON {schema}unit_entities (entity_id)")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -13,8 +13,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "i4d5e6f7g8h9"
down_revision: str | Sequence[str] | None = "h3c4d5e6f7g8"
@@ -28,7 +26,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Delete opinion memory_units."""
schema = _get_schema_prefix()
@@ -37,15 +35,7 @@ def _pg_upgrade() -> None:
op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Cannot restore deleted opinions."""
# Note: Cannot restore deleted opinions - they are lost on downgrade
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,49 +0,0 @@
"""Add 'cancelled' to async_operations status check constraint
Revision ID: i4j5k6l7m8n9
Revises: d5y6z7a8b9c0
Create Date: 2026-04-23
"""
from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "i4j5k6l7m8n9"
down_revision: str | Sequence[str] | None = "d5y6z7a8b9c0"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
op.execute(
f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
f"CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled'))"
)
def _pg_downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
op.execute(
f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
f"CHECK (status IN ('pending', 'processing', 'completed', 'failed'))"
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -15,8 +15,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "j5e6f7g8h9i0"
down_revision: str | Sequence[str] | None = "i4d5e6f7g8h9"
@@ -30,7 +28,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Create mental_model_versions table and add version tracking."""
schema = _get_schema_prefix()
@@ -83,7 +81,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove mental_model_versions table and version column."""
schema = _get_schema_prefix()
@@ -95,11 +93,3 @@ def _pg_downgrade() -> None:
# Remove version column from mental_models
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS version")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "k6f7g8h9i0j1"
down_revision: str | Sequence[str] | None = "j5e6f7g8h9i0"
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add 'directive' to mental_models subtype constraint."""
schema = _get_schema_prefix()
@@ -42,7 +40,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove 'directive' from mental_models subtype constraint."""
schema = _get_schema_prefix()
@@ -58,11 +56,3 @@ def _pg_downgrade() -> None:
ADD CONSTRAINT ck_mental_models_subtype
CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
""")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,38 +0,0 @@
"""No-op: observation_sources table is Oracle-only
Originally created the observation_sources junction table for all backends,
but PG uses native array ops on the source_memory_ids column (faster at scale).
Oracle creates this table in the o1a2b3c4d5e6 baseline migration instead.
Kept as a no-op to preserve the Alembic revision chain.
Revision ID: k6l7m8n9o0p1
Revises: i4j5k6l7m8n9
Create Date: 2026-04-24
"""
from collections.abc import Sequence
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "k6l7m8n9o0p1"
down_revision: str | Sequence[str] | None = "i4j5k6l7m8n9"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
# PG uses source_memory_ids[] array on memory_units — no junction table.
pass
def _pg_downgrade() -> None:
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -17,8 +17,6 @@ import sqlalchemy as sa
from alembic import context, op
from sqlalchemy.dialects import postgresql
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "l7g8h9i0j1k2"
down_revision: str | Sequence[str] | None = "k6f7g8h9i0j1"
@@ -32,7 +30,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add worker columns to async_operations."""
schema = _get_schema_prefix()
@@ -80,7 +78,7 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove worker columns from async_operations."""
schema = _get_schema_prefix()
@@ -109,11 +107,3 @@ def _pg_downgrade() -> None:
"worker_id",
schema=context.config.get_main_option("target_schema") or None,
)
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,31 +0,0 @@
"""Merge divergent heads from deferrable FK and cooccurrence backfill
Revision ID: m3rg3h3ad5f6
Revises: 9f8e7d6c5b4a, b5d4e3f2a1c9
Create Date: 2026-05-04
"""
from collections.abc import Sequence
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "m3rg3h3ad5f6"
down_revision: tuple[str, ...] = ("9f8e7d6c5b4a", "b5d4e3f2a1c9")
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _pg_upgrade() -> None:
pass
def _pg_downgrade() -> None:
pass
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "m8h9i0j1k2l3"
down_revision: str | Sequence[str] | None = "l7g8h9i0j1k2"
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Change mental_models.id from VARCHAR(64) to TEXT."""
schema = _get_schema_prefix()
@@ -35,17 +33,9 @@ def _pg_upgrade() -> None:
op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN id TYPE TEXT")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Revert mental_models.id from TEXT to VARCHAR(64)."""
schema = _get_schema_prefix()
# Note: This may fail if any id values exceed 64 characters
op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN id TYPE VARCHAR(64)")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -16,8 +16,6 @@ from collections.abc import Sequence
from alembic import context, op
from sqlalchemy import text
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "n9i0j1k2l3m4"
down_revision: str | Sequence[str] | None = "m8h9i0j1k2l3"
@@ -121,7 +119,7 @@ def _detect_text_search_extension() -> str:
)
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Create learnings and pinned_reflections tables."""
schema = _get_schema_prefix()
@@ -306,7 +304,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Drop learnings and pinned_reflections tables."""
schema = _get_schema_prefix()
@@ -317,11 +315,3 @@ def _pg_downgrade() -> None:
# Remove columns from banks
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS last_consolidated_at")
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission_changed_at")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -16,8 +16,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "o0j1k2l3m4n5"
down_revision: str | Sequence[str] | None = "n9i0j1k2l3m4"
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Migrate data and clean up old mental models."""
schema = _get_schema_prefix()
@@ -82,17 +80,15 @@ def _pg_upgrade() -> None:
# 4. Drop the mental_model_versions table (no longer used)
op.execute(f"DROP TABLE IF EXISTS {schema}mental_model_versions CASCADE")
# 5. Drop old constraints and add new one that allows current subtypes.
# 'pinned' is still used by the code for user-created mental models;
# 'directive' is used for system directives.
# 5. Drop old constraints and add new one that only allows 'directive'
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
op.execute(f"""
ALTER TABLE {schema}mental_models
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('directive', 'pinned'))
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype = 'directive')
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Reverse the migration (data migration is one-way, so this just removes constraints)."""
schema = _get_schema_prefix()
@@ -115,11 +111,3 @@ def _pg_downgrade() -> None:
)
# Note: Data migration cannot be reversed - pinned_reflections and learnings data remains
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -1,445 +0,0 @@
"""oracle_baseline
Brings a fresh Oracle 23ai database up to the current schema in a single step.
PostgreSQL is a no-op here — the prior 59 revisions already build the PG schema
incrementally; this revision just closes the loop so both dialects share a
single head from this point on.
After this migration ships, *every* new revision must fill both the ``_pg_*``
and ``_oracle_*`` slots (or explicitly leave one ``None``); a CI check enforces
that.
Tables mirror the PostgreSQL schema but use Oracle-native types:
- UUID -> RAW(16) DEFAULT SYS_GUID()
- TEXT / large VARCHAR -> CLOB
- JSONB -> CLOB with IS JSON CHECK
- BOOLEAN -> NUMBER(1)
- FLOAT -> BINARY_DOUBLE
- VARCHAR[] -> CLOB (JSON array stored as string)
- BYTEA -> BLOB
- vector(384) -> VECTOR(384, FLOAT32) (Oracle 23ai native)
Revision ID: o1a2b3c4d5e6
Revises: k6l7m8n9o0p1
Create Date: 2026-04-29
"""
from collections.abc import Sequence
from alembic import op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "o1a2b3c4d5e6"
down_revision: str | Sequence[str] | None = "k6l7m8n9o0p1"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
# ---------------------------------------------------------------------------
# Tables — created in dependency order
# ---------------------------------------------------------------------------
_TABLES: tuple[str, ...] = (
"""
CREATE TABLE IF NOT EXISTS banks (
bank_id VARCHAR2(256) NOT NULL,
internal_id RAW(16) DEFAULT SYS_GUID() NOT NULL,
name VARCHAR2(512),
disposition CLOB DEFAULT '{"skepticism":3,"literalism":3,"empathy":3}' NOT NULL
CONSTRAINT banks_disposition_json CHECK (disposition IS JSON),
mission CLOB,
personality CLOB DEFAULT '{}' NOT NULL
CONSTRAINT banks_personality_json CHECK (personality IS JSON),
config CLOB DEFAULT '{}' NOT NULL
CONSTRAINT banks_config_json CHECK (config IS JSON),
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_banks PRIMARY KEY (bank_id),
CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)
)
""",
"""
CREATE TABLE IF NOT EXISTS documents (
id VARCHAR2(512) NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
original_text CLOB,
content_hash VARCHAR2(128),
metadata CLOB DEFAULT '{}' NOT NULL
CONSTRAINT docs_metadata_json CHECK (metadata IS JSON),
retain_params CLOB CONSTRAINT docs_retain_params_json CHECK (retain_params IS JSON OR retain_params IS NULL),
file_storage_key VARCHAR2(512),
file_original_name VARCHAR2(512),
file_content_type VARCHAR2(256),
tags CLOB DEFAULT '[]' NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_documents PRIMARY KEY (id, bank_id),
CONSTRAINT fk_documents_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
)
""",
"""
CREATE TABLE IF NOT EXISTS chunks (
chunk_id VARCHAR2(512) NOT NULL,
document_id VARCHAR2(512) NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
chunk_index NUMBER(10) NOT NULL,
chunk_text CLOB NOT NULL,
content_hash VARCHAR2(128),
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_chunks PRIMARY KEY (chunk_id),
CONSTRAINT fk_chunks_document FOREIGN KEY (document_id, bank_id)
REFERENCES documents(id, bank_id) ON DELETE CASCADE
)
""",
# memory_units uses automatic list partitioning on bank_id at create time —
# no post-create ALTER required (we used to do that for legacy installs).
"""
CREATE TABLE IF NOT EXISTS memory_units (
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
document_id VARCHAR2(512),
chunk_id VARCHAR2(512),
text CLOB NOT NULL,
embedding VECTOR(384, FLOAT32),
context CLOB,
event_date TIMESTAMP WITH TIME ZONE NOT NULL,
occurred_start TIMESTAMP WITH TIME ZONE,
occurred_end TIMESTAMP WITH TIME ZONE,
mentioned_at TIMESTAMP WITH TIME ZONE,
fact_type VARCHAR2(64) DEFAULT 'world' NOT NULL,
confidence_score BINARY_DOUBLE,
access_count NUMBER(10) DEFAULT 0 NOT NULL,
consolidated_at TIMESTAMP WITH TIME ZONE,
observation_scopes CLOB CONSTRAINT mu_obs_scopes_json CHECK (observation_scopes IS JSON OR observation_scopes IS NULL),
tags CLOB DEFAULT '[]' NOT NULL,
metadata CLOB DEFAULT '{}' NOT NULL
CONSTRAINT mu_metadata_json CHECK (metadata IS JSON),
proof_count NUMBER(10) DEFAULT 1,
source_memory_ids CLOB,
history CLOB DEFAULT '[]'
CONSTRAINT mu_history_json CHECK (history IS JSON OR history IS NULL),
text_signals CLOB,
consolidation_failed_at TIMESTAMP WITH TIME ZONE,
search_vector CLOB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_memory_units PRIMARY KEY (id),
CONSTRAINT fk_mu_document FOREIGN KEY (document_id, bank_id)
REFERENCES documents(id, bank_id) ON DELETE CASCADE,
CONSTRAINT fk_mu_chunk FOREIGN KEY (chunk_id)
REFERENCES chunks(chunk_id) ON DELETE SET NULL,
CONSTRAINT chk_mu_fact_type CHECK (fact_type IN ('world', 'experience', 'observation')),
CONSTRAINT chk_mu_confidence CHECK (
confidence_score IS NULL
OR (confidence_score >= 0.0 AND confidence_score <= 1.0)
)
)
PARTITION BY LIST (bank_id) AUTOMATIC
(PARTITION p_default VALUES ('__default__'))
""",
"""
CREATE TABLE IF NOT EXISTS entities (
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
canonical_name VARCHAR2(512) NOT NULL,
metadata CLOB DEFAULT '{}' NOT NULL
CONSTRAINT ent_metadata_json CHECK (metadata IS JSON),
first_seen TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
last_seen TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
mention_count NUMBER(10) DEFAULT 1 NOT NULL,
CONSTRAINT pk_entities PRIMARY KEY (id)
)
""",
"""
CREATE TABLE IF NOT EXISTS unit_entities (
unit_id RAW(16) NOT NULL,
entity_id RAW(16) NOT NULL,
CONSTRAINT pk_unit_entities PRIMARY KEY (unit_id, entity_id),
CONSTRAINT fk_ue_unit FOREIGN KEY (unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
CONSTRAINT fk_ue_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE CASCADE
)
""",
"""
CREATE TABLE IF NOT EXISTS entity_cooccurrences (
entity_id_1 RAW(16) NOT NULL,
entity_id_2 RAW(16) NOT NULL,
cooccurrence_count NUMBER(10) DEFAULT 1 NOT NULL,
last_cooccurred TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_entity_cooccurrences PRIMARY KEY (entity_id_1, entity_id_2),
CONSTRAINT fk_ec_entity1 FOREIGN KEY (entity_id_1) REFERENCES entities(id) ON DELETE CASCADE,
CONSTRAINT fk_ec_entity2 FOREIGN KEY (entity_id_2) REFERENCES entities(id) ON DELETE CASCADE
)
""",
"""
CREATE TABLE IF NOT EXISTS memory_links (
from_unit_id RAW(16) NOT NULL,
to_unit_id RAW(16) NOT NULL,
link_type VARCHAR2(64) NOT NULL,
entity_id RAW(16),
bank_id VARCHAR2(256),
weight BINARY_DOUBLE DEFAULT 1.0 NOT NULL,
source_memory_ids CLOB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT fk_ml_from FOREIGN KEY (from_unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
CONSTRAINT fk_ml_to FOREIGN KEY (to_unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
CONSTRAINT fk_ml_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE CASCADE,
CONSTRAINT chk_ml_link_type CHECK (
link_type IN ('temporal', 'semantic', 'entity', 'causes', 'caused_by', 'enables', 'prevents')
),
CONSTRAINT chk_ml_weight CHECK (weight >= 0.0 AND weight <= 1.0)
)
""",
"""
CREATE TABLE IF NOT EXISTS mental_models (
id VARCHAR2(256) NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
subtype VARCHAR2(32) NOT NULL,
name VARCHAR2(256) NOT NULL,
description CLOB NOT NULL,
source_query CLOB,
content CLOB,
embedding VECTOR(384, FLOAT32),
entity_id RAW(16),
observations CLOB DEFAULT '{"observations":[]}' NOT NULL
CONSTRAINT mm_obs_json CHECK (observations IS JSON),
links CLOB,
tags CLOB DEFAULT '[]' NOT NULL,
max_tokens NUMBER(10) DEFAULT 2048 NOT NULL,
"trigger" CLOB DEFAULT '{"refresh_after_consolidation":false}' NOT NULL
CONSTRAINT mm_trigger_json CHECK ("trigger" IS JSON),
structured_content CLOB CONSTRAINT mm_sc_json CHECK (structured_content IS JSON OR structured_content IS NULL),
last_refreshed_source_query CLOB,
reflect_response CLOB CONSTRAINT mm_reflect_resp_json CHECK (reflect_response IS JSON OR reflect_response IS NULL),
history CLOB DEFAULT '[]' NOT NULL
CONSTRAINT mm_history_json CHECK (history IS JSON),
last_refreshed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
last_updated TIMESTAMP WITH TIME ZONE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_mental_models PRIMARY KEY (id, bank_id),
CONSTRAINT fk_mm_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE,
CONSTRAINT fk_mm_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE SET NULL,
CONSTRAINT chk_mm_subtype CHECK (subtype IN ('directive', 'pinned'))
)
""",
"""
CREATE TABLE IF NOT EXISTS directives (
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
name VARCHAR2(256) NOT NULL,
content CLOB NOT NULL,
priority NUMBER(10) DEFAULT 0 NOT NULL,
is_active NUMBER(1) DEFAULT 1 NOT NULL,
tags CLOB DEFAULT '[]' NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_directives PRIMARY KEY (id),
CONSTRAINT fk_dir_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
)
""",
"""
CREATE TABLE IF NOT EXISTS async_operations (
operation_id RAW(16) DEFAULT SYS_GUID() NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
operation_type VARCHAR2(128) NOT NULL,
status VARCHAR2(32) DEFAULT 'pending' NOT NULL,
worker_id VARCHAR2(256),
claimed_at TIMESTAMP WITH TIME ZONE,
retry_count NUMBER(10) DEFAULT 0 NOT NULL,
next_retry_at TIMESTAMP WITH TIME ZONE,
task_payload CLOB CONSTRAINT ao_payload_json CHECK (task_payload IS JSON OR task_payload IS NULL),
result_metadata CLOB DEFAULT '{}' NOT NULL
CONSTRAINT ao_result_json CHECK (result_metadata IS JSON),
error_message CLOB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
completed_at TIMESTAMP WITH TIME ZONE,
CONSTRAINT pk_async_operations PRIMARY KEY (operation_id),
CONSTRAINT fk_ao_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE,
CONSTRAINT chk_ao_status CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled'))
)
""",
"""
CREATE TABLE IF NOT EXISTS webhooks (
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
bank_id VARCHAR2(256) NOT NULL,
url VARCHAR2(2048) NOT NULL,
secret VARCHAR2(512),
event_types CLOB DEFAULT '[]' NOT NULL,
http_config CLOB DEFAULT '{}' NOT NULL
CONSTRAINT wh_http_config_json CHECK (http_config IS JSON),
enabled NUMBER(1) DEFAULT 1 NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
CONSTRAINT pk_webhooks PRIMARY KEY (id),
CONSTRAINT fk_wh_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
)
""",
"""
CREATE TABLE IF NOT EXISTS file_storage (
storage_key VARCHAR2(512) NOT NULL,
data BLOB NOT NULL,
CONSTRAINT pk_file_storage PRIMARY KEY (storage_key)
)
""",
"""
CREATE TABLE IF NOT EXISTS audit_log (
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
action VARCHAR2(128) NOT NULL,
transport VARCHAR2(64) NOT NULL,
bank_id VARCHAR2(256),
started_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
ended_at TIMESTAMP WITH TIME ZONE,
request CLOB CONSTRAINT al_request_json CHECK (request IS JSON OR request IS NULL),
response CLOB CONSTRAINT al_response_json CHECK (response IS JSON OR response IS NULL),
metadata CLOB DEFAULT '{}' NOT NULL
CONSTRAINT al_metadata_json CHECK (metadata IS JSON),
CONSTRAINT pk_audit_log PRIMARY KEY (id)
)
""",
"""
CREATE TABLE IF NOT EXISTS observation_sources (
observation_id RAW(16) NOT NULL,
source_id RAW(16) NOT NULL,
CONSTRAINT pk_observation_sources PRIMARY KEY (observation_id, source_id),
CONSTRAINT fk_obs_src_observation FOREIGN KEY (observation_id)
REFERENCES memory_units(id) ON DELETE CASCADE
)
""",
)
# ---------------------------------------------------------------------------
# B-tree indexes
# ---------------------------------------------------------------------------
_INDEXES: tuple[str, ...] = (
# documents
"CREATE INDEX idx_docs_bank_id ON documents(bank_id)",
"CREATE INDEX idx_docs_content_hash ON documents(content_hash)",
# chunks
"CREATE INDEX idx_chunks_document_id ON chunks(document_id)",
"CREATE INDEX idx_chunks_bank_id ON chunks(bank_id)",
# memory_units
"CREATE INDEX idx_mu_bank_id ON memory_units(bank_id)",
"CREATE INDEX idx_mu_document_id ON memory_units(document_id)",
"CREATE INDEX idx_mu_chunk_id ON memory_units(chunk_id)",
"CREATE INDEX idx_mu_event_date ON memory_units(event_date DESC)",
"CREATE INDEX idx_mu_bank_date ON memory_units(bank_id, event_date DESC)",
"CREATE INDEX idx_mu_access_count ON memory_units(access_count DESC)",
"CREATE INDEX idx_mu_fact_type ON memory_units(fact_type)",
"CREATE INDEX idx_mu_bank_fact_type ON memory_units(bank_id, fact_type)",
"CREATE INDEX idx_mu_bank_type_date ON memory_units(bank_id, fact_type, event_date DESC)",
# entities
"CREATE INDEX idx_ent_bank_id ON entities(bank_id)",
"CREATE INDEX idx_ent_canonical_name ON entities(canonical_name)",
"CREATE INDEX idx_ent_bank_name ON entities(bank_id, canonical_name)",
"CREATE UNIQUE INDEX idx_ent_bank_lower_name ON entities(bank_id, LOWER(canonical_name))",
# unit_entities
"CREATE INDEX idx_ue_unit ON unit_entities(unit_id)",
"CREATE INDEX idx_ue_entity ON unit_entities(entity_id)",
# entity_cooccurrences
"CREATE INDEX idx_ec_entity1 ON entity_cooccurrences(entity_id_1)",
"CREATE INDEX idx_ec_entity2 ON entity_cooccurrences(entity_id_2)",
"CREATE INDEX idx_ec_count ON entity_cooccurrences(cooccurrence_count DESC)",
# memory_links — function-based unique index uses NVL with the nil UUID raw
# to handle nullable entity_id (matches PG idx_memory_links_unique).
"CREATE UNIQUE INDEX idx_memory_links_unique ON memory_links("
"from_unit_id, to_unit_id, link_type, "
"NVL(entity_id, HEXTORAW('00000000000000000000000000000000')))",
"CREATE INDEX idx_ml_from_unit ON memory_links(from_unit_id)",
"CREATE INDEX idx_ml_to_unit ON memory_links(to_unit_id)",
"CREATE INDEX idx_ml_entity ON memory_links(entity_id)",
"CREATE INDEX idx_ml_link_type ON memory_links(link_type)",
"CREATE INDEX idx_ml_bank_id ON memory_links(bank_id)",
# directives
"CREATE INDEX idx_dir_bank_id ON directives(bank_id)",
"CREATE INDEX idx_dir_bank_active ON directives(bank_id, is_active)",
# mental_models
"CREATE INDEX idx_mm_bank_id ON mental_models(bank_id)",
"CREATE INDEX idx_mm_subtype ON mental_models(bank_id, subtype)",
"CREATE INDEX idx_mm_entity_id ON mental_models(entity_id)",
# async_operations
"CREATE INDEX idx_ao_bank_id ON async_operations(bank_id)",
"CREATE INDEX idx_ao_status ON async_operations(status)",
"CREATE INDEX idx_ao_bank_status ON async_operations(bank_id, status)",
"CREATE INDEX idx_ao_status_retry ON async_operations(status, next_retry_at)",
# webhooks
"CREATE INDEX idx_wh_bank_id ON webhooks(bank_id)",
# audit_log
"CREATE INDEX idx_al_action_started ON audit_log(action, started_at DESC)",
"CREATE INDEX idx_al_bank_started ON audit_log(bank_id, started_at DESC)",
"CREATE INDEX idx_al_started ON audit_log(started_at DESC)",
# observation_sources
"CREATE INDEX idx_obs_sources_source_id ON observation_sources(source_id, observation_id)",
)
_VECTOR_INDEX = (
"CREATE VECTOR INDEX idx_mu_embedding_hnsw ON memory_units(embedding) "
"ORGANIZATION NEIGHBOR PARTITIONS "
"DISTANCE COSINE "
"WITH TARGET ACCURACY 95"
)
# Oracle Text (CTXSYS.CONTEXT) — ``SYNC (ON COMMIT)`` makes it auto-update
# without a maintenance job. Doubled single quotes for the embedded literal.
_TEXT_INDEX = (
"BEGIN "
"EXECUTE IMMEDIATE '"
"CREATE INDEX idx_mu_content_text ON memory_units(text) "
"INDEXTYPE IS CTXSYS.CONTEXT "
"PARAMETERS (''SYNC (ON COMMIT)'')"
"'; "
"EXCEPTION WHEN OTHERS THEN "
"IF SQLCODE = -955 THEN NULL; ELSE RAISE; END IF; "
"END;"
)
def _execute_ignoring_955(sql: str) -> None:
"""Run a CREATE statement and swallow ORA-00955 (object already exists).
Wraps the statement in PL/SQL so the exception handler runs server-side —
no round-trip cost for the common case.
"""
block = (
"BEGIN "
"EXECUTE IMMEDIATE :stmt; "
"EXCEPTION WHEN OTHERS THEN "
"IF SQLCODE = -955 THEN NULL; ELSE RAISE; END IF; "
"END;"
)
op.get_bind().exec_driver_sql(block, {"stmt": sql.strip()})
def _oracle_upgrade() -> None:
bind = op.get_bind()
# Tolerate concurrent DDL instead of failing immediately (ORA-00054).
bind.exec_driver_sql("ALTER SESSION SET DDL_LOCK_TIMEOUT = 30")
for ddl in _TABLES:
_execute_ignoring_955(ddl)
for idx in _INDEXES:
_execute_ignoring_955(idx)
# Hindsight on Oracle requires 23ai with VECTOR support (ASSM tablespace)
# and the CTXSYS package for full-text. Both index creations must succeed
# — the migration fails hard if either feature is unavailable, by design.
# We only swallow ORA-00955 (object already exists) so reruns are safe.
_execute_ignoring_955(_VECTOR_INDEX)
bind.exec_driver_sql(_TEXT_INDEX)
def _oracle_downgrade() -> None:
# Baseline downgrades aren't supported — dropping every table here would
# destroy customer data. Use point-in-time recovery instead.
raise NotImplementedError("Cannot downgrade past the Oracle baseline.")
def upgrade() -> None:
run_for_dialect(oracle=_oracle_upgrade)
def downgrade() -> None:
run_for_dialect(oracle=_oracle_downgrade)
@@ -21,8 +21,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "p1k2l3m4n5o6"
down_revision: str | Sequence[str] | None = "o0j1k2l3m4n5"
@@ -36,7 +34,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Implement new knowledge architecture."""
schema = _get_schema_prefix()
@@ -128,7 +126,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Reverse the migration."""
schema = _get_schema_prefix()
@@ -194,11 +192,3 @@ def _pg_downgrade() -> None:
""")
# Note: mental_models table recreation is complex and would need separate handling
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "q2l3m4n5o6p7"
down_revision: str | Sequence[str] | None = "p1k2l3m4n5o6"
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add 'mental_model' to the fact_type check constraint."""
schema = _get_schema_prefix()
@@ -40,7 +38,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove 'mental_model' from the fact_type check constraint."""
schema = _get_schema_prefix()
@@ -50,11 +48,3 @@ def _pg_downgrade() -> None:
ADD CONSTRAINT memory_units_fact_type_check
CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))
""")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -14,8 +14,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "r3m4n5o6p7q8"
down_revision: str | Sequence[str] | None = "q2l3m4n5o6p7"
branch_labels: str | Sequence[str] | None = None
@@ -28,7 +26,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Add reflect_response JSONB column to reflections."""
schema = _get_schema_prefix()
@@ -39,7 +37,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Remove reflect_response column from reflections."""
schema = _get_schema_prefix()
@@ -47,11 +45,3 @@ def _pg_downgrade() -> None:
ALTER TABLE {schema}reflections
DROP COLUMN IF EXISTS reflect_response
""")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -12,8 +12,6 @@ import sqlalchemy as sa
from alembic import context, op
from sqlalchemy.dialects import postgresql
from hindsight_api.alembic._dialect import run_for_dialect
# revision identifiers, used by Alembic.
revision: str = "rename_personality"
down_revision: str | Sequence[str] | None = "d9f6a3b4c5e2"
@@ -27,7 +25,7 @@ def _get_target_schema() -> str:
return schema if schema else "public"
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Rename personality column to disposition in banks table (if it exists)."""
conn = op.get_bind()
target_schema = _get_target_schema()
@@ -71,7 +69,7 @@ def _pg_upgrade() -> None:
# else: disposition already exists, nothing to do
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Revert disposition column back to personality."""
conn = op.get_bind()
target_schema = _get_target_schema()
@@ -85,11 +83,3 @@ def _pg_downgrade() -> None:
)
if result.fetchone():
op.alter_column("banks", "disposition", new_column_name="personality")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -13,8 +13,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "s4n5o6p7q8r9"
down_revision: str | Sequence[str] | None = "r3m4n5o6p7q8"
branch_labels: str | Sequence[str] | None = None
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
schema = _get_schema_prefix()
# Add consolidated_at column to memory_units
@@ -48,16 +46,8 @@ def _pg_upgrade() -> None:
)
def _pg_downgrade() -> None:
def downgrade() -> None:
schema = _get_schema_prefix()
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_unconsolidated")
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidated_at")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)
@@ -17,8 +17,6 @@ from collections.abc import Sequence
from alembic import context, op
from hindsight_api.alembic._dialect import run_for_dialect
revision: str = "t5o6p7q8r9s0"
down_revision: str | Sequence[str] | None = "s4n5o6p7q8r9"
branch_labels: str | Sequence[str] | None = None
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _pg_upgrade() -> None:
def upgrade() -> None:
"""Rename mental_model -> observation and reflections -> mental_models."""
schema = _get_schema_prefix()
@@ -88,7 +86,7 @@ def _pg_upgrade() -> None:
""")
def _pg_downgrade() -> None:
def downgrade() -> None:
"""Reverse: observation -> mental_model and mental_models -> reflections."""
schema = _get_schema_prefix()
@@ -134,11 +132,3 @@ def _pg_downgrade() -> None:
ON {schema}memory_units(bank_id, fact_type)
WHERE fact_type = 'mental_model'
""")
def upgrade() -> None:
run_for_dialect(pg=_pg_upgrade)
def downgrade() -> None:
run_for_dialect(pg=_pg_downgrade)

Some files were not shown because too many files have changed in this diff Show More