Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4ed5d169a7 | ||
|
|
3629b350af | ||
|
|
7788927966 | ||
|
|
5005669497 | ||
|
|
f404f31ae2 | ||
|
|
7e191ecd9f | ||
|
|
1232c79652 | ||
|
|
b6dedd5e4b | ||
|
|
11d7a9a5a5 | ||
|
|
e5186129fb | ||
|
|
f828ae46d9 | ||
|
|
00280e65bf | ||
|
|
226b0b5fba |
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
|
||||
"name": "hindsight",
|
||||
"description": "Official Hindsight integrations for Claude Code",
|
||||
"owner": {
|
||||
"name": "vectorize-io"
|
||||
},
|
||||
"plugins": [
|
||||
{
|
||||
"name": "hindsight-memory",
|
||||
"description": "Automatic long-term memory for Claude Code via Hindsight",
|
||||
"source": "./hindsight-integrations/claude-code"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,205 +0,0 @@
|
||||
---
|
||||
name: code-review
|
||||
description: Review changed code against project standards. Checks for missing tests, dead code, type safety, lint issues, and coding conventions. Run after completing any implementation work.
|
||||
user_invocable: true
|
||||
---
|
||||
|
||||
# Code Review
|
||||
|
||||
Review all changed code against the project's quality standards and coding conventions.
|
||||
|
||||
## Code Standards
|
||||
|
||||
Read and internalize these standards before writing code. The review steps below verify compliance.
|
||||
|
||||
### Python Style
|
||||
- Python 3.11+, type hints required
|
||||
- Async throughout (asyncpg, async FastAPI)
|
||||
- Pydantic models for request/response
|
||||
- Ruff for linting (line-length 120)
|
||||
- No Python files at project root - maintain clean directory structure
|
||||
- **Never use multi-item tuple return values** — not even for internal/private functions. Always use a dataclass or Pydantic model. No exceptions, no "it's just two values" shortcuts. If a function returns more than one value, define a named type for it.
|
||||
|
||||
### Type Safety with Pydantic Models
|
||||
**NEVER use raw `dict` types for structured data** — this applies to all code, including internal helpers and private functions. If the dict has known keys, it must be a dataclass or Pydantic model:
|
||||
- Use Pydantic `BaseModel` for all data structures passed between functions
|
||||
- Use `@dataclass` for lightweight internal data containers when Pydantic validation isn't needed
|
||||
- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
|
||||
- Avoid `dict.get()` patterns - use typed model attributes instead
|
||||
- Parse external data (JSON, API responses) into Pydantic models at the boundary
|
||||
- This catches type errors at parse time, not deep in business logic
|
||||
- The only acceptable `dict` usage is for truly dynamic/unknown keys (e.g., arbitrary metadata, JSON blobs with no fixed schema)
|
||||
|
||||
```python
|
||||
# BAD - error-prone dict access
|
||||
def process(data: dict) -> str:
|
||||
return data.get("name", "") # No validation, silent failures
|
||||
|
||||
# GOOD - typed and validated
|
||||
class UserData(BaseModel):
|
||||
name: str
|
||||
created_at: datetime
|
||||
|
||||
@field_validator("created_at", mode="before")
|
||||
@classmethod
|
||||
def ensure_tz_aware(cls, v):
|
||||
if isinstance(v, str):
|
||||
v = datetime.fromisoformat(v.replace("Z", "+00:00"))
|
||||
if v.tzinfo is None:
|
||||
return v.replace(tzinfo=timezone.utc)
|
||||
return v
|
||||
|
||||
def process(data: UserData) -> str:
|
||||
return data.name # Type-safe, validated at construction
|
||||
```
|
||||
|
||||
### TypeScript Style
|
||||
- Next.js App Router for control plane
|
||||
- Tailwind CSS with shadcn/ui components
|
||||
|
||||
### Code Comments
|
||||
- **Always comment non-trivial technical decisions** with the reasoning behind the choice. If someone would ask "why is it done this way?", there should be a comment.
|
||||
- **Keep comments up to date with history** — when changing an approach, update the comment to explain what was tried before and why it was changed. Comments serve as a tracker of previous implementations that likely had problems.
|
||||
- Don't comment obvious code — only where the "why" isn't self-evident from the code itself.
|
||||
|
||||
```python
|
||||
# BAD - no context for future readers
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# GOOD - explains the non-obvious choice
|
||||
# Use return_exceptions=True to avoid cancelling sibling tasks on failure.
|
||||
# Previously we used TaskGroup but it cancelled all tasks when one failed,
|
||||
# causing partial writes that left orphaned entity links (see #412).
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
```
|
||||
|
||||
### Branch Hygiene
|
||||
- **Always start new feature branches from `origin/main`** — rebase to ensure a clean base.
|
||||
- **Only include commits relevant to the PR/branch/feature** — no unrelated changes. If the branch contains commits that don't belong, they must be removed before merging.
|
||||
|
||||
### General Principles
|
||||
- Don't add features, refactor code, or make "improvements" beyond what was asked
|
||||
- Don't add unnecessary error handling for impossible scenarios
|
||||
- Don't create helpers or abstractions for one-time operations
|
||||
- No backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
|
||||
- Three similar lines of code is better than a premature abstraction
|
||||
|
||||
## Review Steps
|
||||
|
||||
### 1. Check branch hygiene
|
||||
|
||||
- Run `git log --oneline main..HEAD` to list all commits on the branch.
|
||||
- Verify every commit is relevant to the feature/PR. Flag any unrelated commits.
|
||||
- Check the branch is based on a recent `origin/main` (no stale base).
|
||||
|
||||
### 2. Identify changed files
|
||||
|
||||
Run `git diff --name-only HEAD` (unstaged) and `git diff --cached --name-only` (staged) to get all changed files. If there are no local changes, diff against the base branch using `git diff main...HEAD --name-only` and `git diff main...HEAD` to review all commits on the current branch.
|
||||
|
||||
### 3. Run linters
|
||||
|
||||
```bash
|
||||
./scripts/hooks/lint.sh
|
||||
```
|
||||
|
||||
Report any failures. Do NOT fix them yourself — just report.
|
||||
|
||||
### 4. Check for dead code
|
||||
|
||||
For each changed Python file, check for:
|
||||
- Unused imports (Ruff should catch these, but verify)
|
||||
- Functions/methods/classes that were added but are never called from anywhere
|
||||
- Variables assigned but never read
|
||||
- Commented-out code blocks that should be removed
|
||||
|
||||
For each changed TypeScript file, check for:
|
||||
- Unused imports
|
||||
- Unused variables or functions
|
||||
- Commented-out code
|
||||
|
||||
### 5. Check type safety (Python)
|
||||
|
||||
For each changed Python file, check for violations:
|
||||
- **No raw `dict` for structured data** — must use Pydantic model or dataclass, even for internal/private functions (only exception: truly dynamic/unknown keys)
|
||||
- **No multi-item tuple returns** — must use dataclass or Pydantic model, even for internal/private functions (no exceptions)
|
||||
- **Missing type hints** on function parameters and return types
|
||||
- **Missing `@field_validator`** for datetime fields that should be timezone-aware
|
||||
|
||||
### 6. Check for missing tests
|
||||
|
||||
For each new or significantly changed function/endpoint/class:
|
||||
- Check if there is a corresponding test addition or update
|
||||
- New API endpoints MUST have integration tests
|
||||
- New utility functions MUST have unit tests
|
||||
- Bug fixes SHOULD have a regression test
|
||||
|
||||
Flag any new logic that lacks test coverage.
|
||||
|
||||
### 7. Check API consistency
|
||||
|
||||
If any files in `hindsight-api-slim/hindsight_api/api/` were changed:
|
||||
- Were the OpenAPI specs regenerated? (`./scripts/generate-openapi.sh`)
|
||||
- Were the client SDKs regenerated? (`./scripts/generate-clients.sh`)
|
||||
- Were the control plane proxy routes updated? (`hindsight-control-plane/src/app/api/`)
|
||||
|
||||
### 8. Check code comments
|
||||
|
||||
For each non-trivial change:
|
||||
- **New non-obvious logic** — is there a comment explaining the reasoning?
|
||||
- **Changed approach** — does the comment include what was done before and why it changed?
|
||||
- **Stale comments** — do existing comments near the changed code still accurately describe the behavior?
|
||||
|
||||
### 9. Check integration completeness
|
||||
|
||||
If any files in `hindsight-integrations/` were added or changed, verify:
|
||||
- **Tests exist** — the integration must have tests that simulate/exercise the external framework (not just pure unit tests of helpers). Check for a `tests/` directory with meaningful test files.
|
||||
- **CI job exists** — check `.github/workflows/test.yml` for a corresponding `test-<name>-integration` job. If missing, flag it.
|
||||
- **Release process** — check that the integration name is in the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh`. If missing, flag it.
|
||||
- **Code standards** — the integration code must follow all Python style rules (type hints, no raw dicts, no tuple returns, etc.).
|
||||
|
||||
### 10. Check MCP tool registration completeness
|
||||
|
||||
If any new MCP tools were added or existing tools renamed in `hindsight-api-slim/hindsight_api/mcp_tools.py`:
|
||||
- **`_ALL_TOOLS` set** in `mcp_tools.py` — must include the new tool name
|
||||
- **`tools_to_register` default set** in `register_mcp_tools()` in `mcp_tools.py` — must include the new tool name
|
||||
- **`_SINGLE_BANK_TOOLS` set** in `hindsight-api-slim/hindsight_api/api/mcp.py` — must include the new tool if it is bank-scoped (not a bank-management tool like `list_banks`/`create_bank`)
|
||||
- **`MCP_TOOL_GROUPS`** in `hindsight-control-plane/src/components/bank-config-view.tsx` — must include the new tool in the appropriate group for the UI tool selector
|
||||
- **Tool count assertions** in tests (e.g., `test_mcp_tools.py`) — must be updated to reflect the new count
|
||||
|
||||
### 11. Review against other coding standards
|
||||
|
||||
Check the diff for violations of the standards listed above:
|
||||
- Python files at project root (not allowed)
|
||||
- Missing async patterns (should be async throughout)
|
||||
- Pydantic models for request/response
|
||||
- Line length > 120 chars
|
||||
- New features/code beyond what was asked (over-engineering)
|
||||
- Unnecessary error handling for impossible scenarios
|
||||
- Premature abstractions or speculative helpers
|
||||
- Backwards-compatibility hacks (unused vars, re-exports, "removed" comments)
|
||||
|
||||
### 12. Report findings
|
||||
|
||||
Present a clear summary organized by severity:
|
||||
|
||||
**Must fix** — issues that will break CI or violate hard project rules:
|
||||
- Unrelated commits on the branch
|
||||
- Lint failures
|
||||
- Missing type hints on public functions
|
||||
- Raw dict usage for structured data (including internal code)
|
||||
- Multi-item tuple returns (including internal code)
|
||||
- Missing tests for new endpoints
|
||||
- New integration missing tests, CI job, or release-integration.sh entry
|
||||
|
||||
**Should fix** — issues that hurt code quality:
|
||||
- Dead code / unused imports missed by linter
|
||||
- Missing tests for non-trivial utility functions
|
||||
- Over-engineering beyond the task scope
|
||||
|
||||
**Note** — observations that may or may not need action:
|
||||
- API changes that might need client regeneration
|
||||
- Patterns that deviate from nearby code style
|
||||
|
||||
For each finding, include the file path, line number, and a brief explanation.
|
||||
|
||||
Do NOT auto-fix any issues. Report all findings and let the user decide what to address. If there are no findings, confirm the code looks good.
|
||||
+1
-7
@@ -2,7 +2,7 @@
|
||||
# Copy this file to .env and fill in your values
|
||||
|
||||
# LLM Configuration (Required)
|
||||
# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, volcano
|
||||
# Supported providers: openai, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax
|
||||
HINDSIGHT_API_LLM_PROVIDER=openai
|
||||
HINDSIGHT_API_LLM_API_KEY=your-api-key-here
|
||||
HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
||||
@@ -25,11 +25,6 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# HINDSIGHT_API_LLM_API_KEY=your-minimax-api-key
|
||||
# HINDSIGHT_API_LLM_MODEL=MiniMax-M2.7
|
||||
|
||||
# Example: DeepSeek configuration (https://api.deepseek.com)
|
||||
# HINDSIGHT_API_LLM_PROVIDER=deepseek
|
||||
# HINDSIGHT_API_LLM_API_KEY=your-deepseek-api-key
|
||||
# HINDSIGHT_API_LLM_MODEL=deepseek-v4-flash # or deepseek-v4-pro / deepseek-chat / deepseek-reasoner
|
||||
|
||||
# Example: LM Studio local configuration (Qwen 2.5 32B recommended)
|
||||
# HINDSIGHT_API_LLM_PROVIDER=lmstudio
|
||||
# HINDSIGHT_API_LLM_API_KEY=lmstudio
|
||||
@@ -49,7 +44,6 @@ HINDSIGHT_API_LOG_LEVEL=info
|
||||
|
||||
# Database (Optional - uses embedded pg0 by default)
|
||||
# HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
|
||||
# HINDSIGHT_API_MIGRATION_DATABASE_URL= # Direct PostgreSQL URL for migrations (bypasses PgBouncer). Falls back to DATABASE_URL.
|
||||
# HINDSIGHT_API_DATABASE_SCHEMA=public # PostgreSQL schema name (default: public)
|
||||
|
||||
# Vector Extension (Optional - uses pgvector by default)
|
||||
|
||||
@@ -34,7 +34,7 @@ jobs:
|
||||
env:
|
||||
UMAMI_URL: https://analytics.hindsight.vectorize.io
|
||||
UMAMI_WEBSITE_ID: ${{ secrets.UMAMI_WEBSITE_ID }}
|
||||
- uses: actions/upload-pages-artifact@v5
|
||||
- uses: actions/upload-pages-artifact@v4
|
||||
with:
|
||||
path: hindsight-docs/build
|
||||
deploy:
|
||||
@@ -44,5 +44,5 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
needs: build
|
||||
steps:
|
||||
- uses: actions/deploy-pages@v5
|
||||
- uses: actions/deploy-pages@v4
|
||||
id: deployment
|
||||
|
||||
@@ -1,197 +0,0 @@
|
||||
name: Performance Tests
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Run daily at 06:00 UTC
|
||||
- cron: "0 6 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
scale:
|
||||
description: "Test scale (perf-test)"
|
||||
type: choice
|
||||
options:
|
||||
- tiny
|
||||
- small
|
||||
- medium
|
||||
- large
|
||||
default: large
|
||||
suite:
|
||||
description: "Perf-test suite to run (blank = all)"
|
||||
type: choice
|
||||
options:
|
||||
- ""
|
||||
- retain
|
||||
- recall
|
||||
- recall-with-observations
|
||||
- consolidation
|
||||
default: ""
|
||||
locomo_max_conversations:
|
||||
description: "LoComo max conversations (0 = skip, blank = all)"
|
||||
type: number
|
||||
default: 0
|
||||
locomo_skip:
|
||||
description: "Skip LoComo job"
|
||||
type: boolean
|
||||
default: false
|
||||
ref:
|
||||
description: "Git ref to test (branch, tag, or SHA). Defaults to main."
|
||||
type: string
|
||||
default: ""
|
||||
|
||||
concurrency:
|
||||
group: perf-test
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
perf-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
prune-cache: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version-file: ".python-version"
|
||||
|
||||
- name: Cache HuggingFace models
|
||||
uses: actions/cache@v5
|
||||
with:
|
||||
path: ~/.cache/huggingface
|
||||
key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-huggingface-
|
||||
|
||||
- name: Pre-download models
|
||||
working-directory: ./hindsight-api-slim
|
||||
run: |
|
||||
uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
|
||||
from sentence_transformers import SentenceTransformer
|
||||
print('Downloading embedding model...')
|
||||
SentenceTransformer('BAAI/bge-small-en-v1.5')
|
||||
print('Model downloaded successfully')
|
||||
"
|
||||
|
||||
- name: Install hindsight-dev dependencies
|
||||
run: |
|
||||
cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
|
||||
|
||||
- name: "Suite: retain"
|
||||
if: inputs.suite == '' || inputs.suite == 'retain'
|
||||
run: |
|
||||
./scripts/benchmarks/run-perf-test.sh \
|
||||
--scale ${{ inputs.scale || 'large' }} \
|
||||
--suite retain \
|
||||
--output perf-results-retain.json
|
||||
|
||||
- name: "Suite: recall"
|
||||
if: inputs.suite == '' || inputs.suite == 'recall'
|
||||
run: |
|
||||
./scripts/benchmarks/run-perf-test.sh \
|
||||
--scale ${{ inputs.scale || 'large' }} \
|
||||
--suite recall \
|
||||
--output perf-results-recall.json
|
||||
|
||||
- name: "Suite: recall-with-observations"
|
||||
if: inputs.suite == '' || inputs.suite == 'recall-with-observations'
|
||||
run: |
|
||||
./scripts/benchmarks/run-perf-test.sh \
|
||||
--scale ${{ inputs.scale || 'large' }} \
|
||||
--suite recall-with-observations \
|
||||
--output perf-results-recall-with-observations.json
|
||||
|
||||
- name: "Suite: consolidation"
|
||||
if: inputs.suite == '' || inputs.suite == 'consolidation'
|
||||
run: |
|
||||
./scripts/benchmarks/run-perf-test.sh \
|
||||
--scale ${{ inputs.scale || 'large' }} \
|
||||
--suite consolidation \
|
||||
--output perf-results-consolidation.json
|
||||
|
||||
- name: Upload perf results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: perf-results-${{ github.sha }}
|
||||
path: hindsight-dev/perf-results-*.json
|
||||
retention-days: 90
|
||||
|
||||
locomo:
|
||||
if: inputs.locomo_skip != true
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
HINDSIGHT_API_LLM_PROVIDER: vertexai
|
||||
HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY: /tmp/gcp-credentials.json
|
||||
HINDSIGHT_API_LLM_MODEL: google/gemini-2.5-flash-lite
|
||||
HINDSIGHT_API_JUDGE_LLM_PROVIDER: vertexai
|
||||
HINDSIGHT_API_JUDGE_LLM_MODEL: google/gemini-2.5-flash-lite
|
||||
HINDSIGHT_API_ANSWER_LLM_PROVIDER: vertexai
|
||||
HINDSIGHT_API_ANSWER_LLM_MODEL: google/gemini-2.5-flash
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: ${{ inputs.ref || github.ref }}
|
||||
|
||||
- name: Setup GCP credentials
|
||||
run: |
|
||||
printf '%s' '${{ secrets.GCP_VERTEXAI_CREDENTIALS }}' > /tmp/gcp-credentials.json
|
||||
PROJECT_ID=$(jq -r '.project_id' /tmp/gcp-credentials.json)
|
||||
echo "HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=$PROJECT_ID" >> $GITHUB_ENV
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
prune-cache: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version-file: ".python-version"
|
||||
|
||||
- name: Cache HuggingFace models
|
||||
uses: actions/cache@v5
|
||||
with:
|
||||
path: ~/.cache/huggingface
|
||||
key: ${{ runner.os }}-huggingface-${{ hashFiles('hindsight-api-slim/pyproject.toml') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-huggingface-
|
||||
|
||||
- name: Pre-download models
|
||||
working-directory: ./hindsight-api-slim
|
||||
run: |
|
||||
uv run --frozen --all-extras --index-strategy unsafe-best-match python -c "
|
||||
from sentence_transformers import SentenceTransformer
|
||||
print('Downloading embedding model...')
|
||||
SentenceTransformer('BAAI/bge-small-en-v1.5')
|
||||
print('Model downloaded successfully')
|
||||
"
|
||||
|
||||
- name: Install hindsight-dev dependencies
|
||||
run: |
|
||||
cd hindsight-dev && uv sync --frozen --all-extras --index-strategy unsafe-best-match
|
||||
|
||||
- name: Run LoComo benchmark
|
||||
run: |
|
||||
MAX_CONV_ARG=""
|
||||
if [ "${{ inputs.locomo_max_conversations }}" != "0" ] && [ -n "${{ inputs.locomo_max_conversations }}" ]; then
|
||||
MAX_CONV_ARG="--max-conversations ${{ inputs.locomo_max_conversations }}"
|
||||
fi
|
||||
uv run python hindsight-dev/benchmarks/locomo/locomo_benchmark.py \
|
||||
--wait-consolidation \
|
||||
$MAX_CONV_ARG
|
||||
|
||||
- name: Upload LoComo results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: locomo-results-${{ github.sha }}
|
||||
path: hindsight-dev/benchmarks/locomo/results/
|
||||
retention-days: 90
|
||||
@@ -31,10 +31,8 @@ jobs:
|
||||
run: |
|
||||
if [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/pyproject.toml" ]; then
|
||||
echo "type=python" >> $GITHUB_OUTPUT
|
||||
elif [ -f "hindsight-integrations/${{ steps.info.outputs.integration }}/package.json" ]; then
|
||||
echo "type=typescript" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "type=plugin" >> $GITHUB_OUTPUT
|
||||
echo "type=typescript" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# ── Python integrations (litellm, pydantic-ai, crewai) ──────────────────
|
||||
@@ -47,7 +45,7 @@ jobs:
|
||||
|
||||
- name: Set up Python
|
||||
if: steps.type.outputs.type == 'python'
|
||||
uses: actions/setup-python@v6
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version-file: ".python-version"
|
||||
|
||||
@@ -65,44 +63,14 @@ jobs:
|
||||
|
||||
# ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
|
||||
|
||||
# ── Plugin integrations (claude-code) — no package to publish ───────────
|
||||
|
||||
- name: Plugin release
|
||||
if: steps.type.outputs.type == 'plugin'
|
||||
run: |
|
||||
echo "Plugin integration ${{ steps.info.outputs.integration }} v${{ steps.info.outputs.version }} — no package to publish."
|
||||
echo "Users install via: claude plugin marketplace add vectorize-io/hindsight --sparse hindsight-integrations"
|
||||
|
||||
# ── TypeScript integrations (ai-sdk, chat, openclaw) ────────────────────
|
||||
|
||||
- name: Set up Node.js
|
||||
if: steps.type.outputs.type == 'typescript'
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: '22'
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: package-lock.json
|
||||
|
||||
- name: Check integration lockfile
|
||||
if: steps.type.outputs.type == 'typescript'
|
||||
run: ./scripts/check-integration-lockfiles.sh
|
||||
|
||||
# Some integrations depend on workspace packages (hindsight-client,
|
||||
# hindsight-all, hindsight-agent-sdk) via file: refs. Install from root
|
||||
# so npm resolves them, then build the workspace deps before the integration.
|
||||
- name: Install root workspace dependencies
|
||||
if: steps.type.outputs.type == 'typescript'
|
||||
run: npm ci
|
||||
|
||||
- name: Build workspace deps (hindsight-client, hindsight-all, hindsight-agent-sdk)
|
||||
if: steps.type.outputs.type == 'typescript'
|
||||
run: |
|
||||
npm run build --workspace=hindsight-clients/typescript
|
||||
npm run build --workspace=hindsight-all-npm
|
||||
npm run build --workspace=hindsight-tools/hindsight-agent-sdk
|
||||
|
||||
- name: Install integration dependencies
|
||||
- name: Install dependencies
|
||||
if: steps.type.outputs.type == 'typescript'
|
||||
working-directory: ./hindsight-integrations/${{ steps.info.outputs.integration }}
|
||||
run: npm ci
|
||||
|
||||
@@ -1,65 +0,0 @@
|
||||
name: Release Tool
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'tools/**'
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Extract tool info
|
||||
id: info
|
||||
run: |
|
||||
# refs/tags/tools/self-driving-agents/v0.0.1 → tool=self-driving-agents, version=0.0.1
|
||||
TAG="${GITHUB_REF#refs/tags/}"
|
||||
TOOL=$(echo "$TAG" | cut -d'/' -f2)
|
||||
VERSION=$(echo "$TAG" | cut -d'/' -f3 | sed 's/^v//')
|
||||
echo "tool=$TOOL" >> $GITHUB_OUTPUT
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "tag=$TAG" >> $GITHUB_OUTPUT
|
||||
echo "Tool: $TOOL, Version: $VERSION"
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: '22'
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: package-lock.json
|
||||
|
||||
# Tools live under hindsight-tools/ and may depend on workspace packages
|
||||
# (e.g. @vectorize-io/hindsight-client). Install from root so npm resolves
|
||||
# workspace deps, then build any required workspace packages first.
|
||||
- name: Install root workspace dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Build hindsight-client (workspace dep)
|
||||
run: npm run build --workspace=hindsight-clients/typescript
|
||||
|
||||
- name: Build hindsight-agent-sdk (workspace dep)
|
||||
run: npm run build --workspace=hindsight-tools/hindsight-agent-sdk
|
||||
|
||||
- name: Build tool
|
||||
run: npm run build --workspace=hindsight-tools/${{ steps.info.outputs.tool }}
|
||||
|
||||
- name: Publish to npm
|
||||
working-directory: ./hindsight-tools/${{ steps.info.outputs.tool }}
|
||||
run: |
|
||||
set +e
|
||||
OUTPUT=$(npm publish --access public 2>&1)
|
||||
EXIT_CODE=$?
|
||||
echo "$OUTPUT"
|
||||
if [ $EXIT_CODE -ne 0 ]; then
|
||||
if echo "$OUTPUT" | grep -q "cannot publish over"; then
|
||||
echo "Package version already published, skipping..."
|
||||
exit 0
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
@@ -150,55 +150,6 @@ jobs:
|
||||
path: hindsight-clients/typescript/*.tgz
|
||||
retention-days: 1
|
||||
|
||||
release-hindsight-all-npm:
|
||||
runs-on: ubuntu-latest
|
||||
environment: npm
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: '22'
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci --workspace=hindsight-all-npm
|
||||
|
||||
- name: Build
|
||||
run: npm run build --workspace=hindsight-all-npm
|
||||
|
||||
- name: Publish to npm
|
||||
working-directory: ./hindsight-all-npm
|
||||
run: |
|
||||
set +e
|
||||
OUTPUT=$(npm publish --access public 2>&1)
|
||||
EXIT_CODE=$?
|
||||
echo "$OUTPUT"
|
||||
if [ $EXIT_CODE -ne 0 ]; then
|
||||
if echo "$OUTPUT" | grep -q "cannot publish over"; then
|
||||
echo "Package version already published, skipping..."
|
||||
exit 0
|
||||
fi
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
|
||||
- name: Pack for GitHub release
|
||||
working-directory: ./hindsight-all-npm
|
||||
run: npm pack
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: hindsight-all-npm
|
||||
path: hindsight-all-npm/*.tgz
|
||||
retention-days: 1
|
||||
|
||||
release-control-plane:
|
||||
runs-on: ubuntu-latest
|
||||
environment: npm
|
||||
@@ -431,7 +382,7 @@ jobs:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Install Helm
|
||||
uses: azure/setup-helm@v5
|
||||
uses: azure/setup-helm@v4
|
||||
with:
|
||||
version: 'latest'
|
||||
|
||||
@@ -456,7 +407,7 @@ jobs:
|
||||
|
||||
create-github-release:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [release-python-packages, release-typescript-client, release-hindsight-all-npm, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
|
||||
needs: [release-python-packages, release-typescript-client, release-control-plane, release-rust-cli, release-docker-images, release-helm-chart]
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
@@ -485,24 +436,12 @@ jobs:
|
||||
name: control-plane
|
||||
path: ./artifacts/control-plane
|
||||
|
||||
- name: Download hindsight-embed npm wrapper
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: hindsight-all-npm
|
||||
path: ./artifacts/hindsight-all-npm
|
||||
|
||||
- name: Download Rust CLI (Linux)
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: rust-cli-hindsight-linux-amd64
|
||||
path: ./artifacts/rust-cli-linux
|
||||
|
||||
- name: Download Rust CLI (Linux ARM)
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: rust-cli-hindsight-linux-arm64
|
||||
path: ./artifacts/rust-cli-linux-arm64
|
||||
|
||||
- name: Download Rust CLI (macOS Intel)
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
@@ -533,13 +472,10 @@ jobs:
|
||||
cp artifacts/python-packages/hindsight-embed/dist/* release-assets/ || true
|
||||
# TypeScript client
|
||||
cp artifacts/typescript-client/*.tgz release-assets/ || true
|
||||
# hindsight-embed npm wrapper
|
||||
cp artifacts/hindsight-all-npm/*.tgz release-assets/ || true
|
||||
# Control Plane
|
||||
cp artifacts/control-plane/*.tgz release-assets/ || true
|
||||
# Rust CLI binaries
|
||||
cp artifacts/rust-cli-linux/hindsight-linux-amd64 release-assets/ || true
|
||||
cp artifacts/rust-cli-linux-arm64/hindsight-linux-arm64 release-assets/ || true
|
||||
cp artifacts/rust-cli-darwin-amd64/hindsight-darwin-amd64 release-assets/ || true
|
||||
cp artifacts/rust-cli-darwin-arm64/hindsight-darwin-arm64 release-assets/ || true
|
||||
# Helm chart
|
||||
@@ -547,7 +483,7 @@ jobs:
|
||||
ls -la release-assets/
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@v3
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
files: release-assets/*
|
||||
generate_release_notes: true
|
||||
|
||||
+7
-1946
File diff suppressed because it is too large
Load Diff
+1
-2
@@ -50,8 +50,7 @@ hindsight-dev/benchmarks/perf/results/
|
||||
benchmarks/results/
|
||||
hindsight-cli/target
|
||||
hindsight-clients/rust/target
|
||||
.claude/*
|
||||
!.claude/skills/
|
||||
.claude
|
||||
whats-next.md
|
||||
TASK.md
|
||||
# Changelog is now tracked in hindsight-docs/src/pages/changelog.md
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"semi": true,
|
||||
"singleQuote": false,
|
||||
"tabWidth": 2,
|
||||
"trailingComma": "es5",
|
||||
"printWidth": 100
|
||||
}
|
||||
@@ -11,15 +11,9 @@ Hindsight is an agent memory system that provides long-term memory for AI agents
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Local Development (API + UI)
|
||||
```bash
|
||||
# Start both API server and control plane UI
|
||||
./scripts/dev/start.sh
|
||||
```
|
||||
|
||||
### API Server (Python/FastAPI)
|
||||
```bash
|
||||
# Start API server only (loads .env automatically)
|
||||
# Start API server (loads .env automatically)
|
||||
./scripts/dev/start-api.sh
|
||||
|
||||
# Run all tests (parallelized with pytest-xdist)
|
||||
@@ -68,9 +62,8 @@ cd hindsight-control-plane && npm run dev
|
||||
./scripts/benchmarks/run-locomo.sh
|
||||
|
||||
# Performance benchmarks
|
||||
./scripts/benchmarks/run-perf-test.sh # System perf (mock LLM + pg0)
|
||||
./scripts/benchmarks/run-perf-test.sh --scale tiny # Quick smoke test
|
||||
./scripts/benchmarks/run-consolidation.sh
|
||||
./scripts/benchmarks/run-retain-perf.sh --document <path> # Requires API server running
|
||||
|
||||
# Results viewer
|
||||
./scripts/benchmarks/start-visualizer.sh # View results at localhost:8001
|
||||
@@ -80,16 +73,17 @@ cd hindsight-control-plane && npm run dev
|
||||
|
||||
### Monorepo Structure
|
||||
- **hindsight-api-slim/**: Core FastAPI server with memory engine (Python, uv)
|
||||
- **hindsight/**: Embedded Python bundle (hindsight-all package)
|
||||
- **hindsight-control-plane/**: Admin UI (Next.js, npm)
|
||||
- **hindsight-cli/**: CLI tool (Rust, cargo, uses progenitor for API client)
|
||||
- **hindsight-clients/**: Generated SDK clients (Python, TypeScript, Rust)
|
||||
- **hindsight-docs/**: Docusaurus documentation site
|
||||
- **hindsight-integrations/**: Framework integrations (LiteLLM, CrewAI, LangGraph, Pydantic AI, AG2, Claude Code, etc.)
|
||||
- **hindsight-integrations/**: Framework integrations (LiteLLM, OpenAI)
|
||||
- **hindsight-dev/**: Development tools and benchmarks
|
||||
|
||||
### Core Engine (hindsight-api-slim/hindsight_api/engine/)
|
||||
- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
|
||||
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
|
||||
- `memory_engine.py`: Main orchestrator (~170KB) for retain/recall/reflect operations
|
||||
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, Groq, MiniMax, Ollama, LM Studio
|
||||
- `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
|
||||
- `cross_encoder.py`: Reranking (local or TEI)
|
||||
- `entity_resolver.py`: Entity extraction and normalization
|
||||
@@ -102,13 +96,13 @@ cd hindsight-control-plane && npm run dev
|
||||
|
||||
**search/**: Multi-strategy retrieval
|
||||
- `retrieval.py`: Main retrieval orchestrator
|
||||
- `graph_retrieval.py`: Graph retrieval abstract base class
|
||||
- `link_expansion_retrieval.py`: Link expansion graph retrieval
|
||||
- `graph_retrieval.py`: Entity/relationship graph traversal
|
||||
- `mpfp_retrieval.py`: Multi-Path Fact Propagation retrieval
|
||||
- `fusion.py`: Reciprocal rank fusion for combining results
|
||||
- `reranking.py`: Cross-encoder reranking
|
||||
|
||||
### API Layer (hindsight-api-slim/hindsight_api/api/)
|
||||
- `http.py`: FastAPI HTTP routers for all REST endpoints
|
||||
- `http.py`: FastAPI HTTP routers (~80KB) for all REST endpoints
|
||||
- `mcp.py`: Model Context Protocol server implementation
|
||||
|
||||
Main operations:
|
||||
@@ -123,17 +117,12 @@ Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
|
||||
|
||||
### Adding Database Migrations
|
||||
|
||||
Hindsight runs the same Alembic tree against PostgreSQL and Oracle 23ai. Each
|
||||
migration file dispatches through `run_for_dialect`, which calls either
|
||||
`_pg_upgrade` or `_oracle_upgrade` based on the live connection. A pytest lint
|
||||
(`tests/test_migration_shape.py`) fails CI if a migration omits the dispatcher.
|
||||
|
||||
1. **Create a new migration file** in `hindsight-api-slim/hindsight_api/alembic/versions/`:
|
||||
- File name format: `<revision_id>_<description>.py` (e.g., `f1a2b3c4d5e6_add_new_index.py`)
|
||||
- Use a unique hex revision ID (12 chars)
|
||||
- Set `down_revision` to the previous migration's revision ID
|
||||
|
||||
2. **Migration template** (the `script.py.mako` template scaffolds this; fill in the bodies):
|
||||
2. **Migration template**:
|
||||
```python
|
||||
"""Description of the migration
|
||||
|
||||
@@ -144,58 +133,25 @@ migration file dispatches through `run_for_dialect`, which calls either
|
||||
from collections.abc import Sequence
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "f1a2b3c4d5e6"
|
||||
down_revision: str | Sequence[str] | None = "<previous_revision_id>"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_schema_prefix() -> str:
|
||||
"""Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _pg_schema_prefix()
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"CREATE INDEX ... ON {schema}table_name(...)")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _pg_schema_prefix()
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}index_name")
|
||||
|
||||
|
||||
def _oracle_upgrade() -> None:
|
||||
# Oracle 23ai equivalent. Use op.get_bind().exec_driver_sql for forms
|
||||
# that Alembic core does not model (vector/text indexes, partitions).
|
||||
op.execute("CREATE INDEX ... ON table_name(...)")
|
||||
|
||||
|
||||
def _oracle_downgrade() -> None:
|
||||
op.execute("DROP INDEX IF EXISTS index_name")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}index_name")
|
||||
```
|
||||
|
||||
**Dialect-only migrations.** If a change genuinely doesn't apply to one
|
||||
dialect (e.g. enabling `pg_trgm` is PG-only), omit the unused slot:
|
||||
```python
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade) # oracle slot intentionally absent → no-op
|
||||
```
|
||||
Make the asymmetry deliberate. Don't leave an Oracle slot empty just because
|
||||
you didn't think about it — copy-pasting a PG migration without the Oracle
|
||||
half is exactly how schemas drift.
|
||||
|
||||
3. **Run migrations locally**:
|
||||
```bash
|
||||
# Set database URL and run migrations for the base schema plus all tenants
|
||||
@@ -208,17 +164,11 @@ migration file dispatches through `run_for_dialect`, which calls either
|
||||
## Key Conventions
|
||||
|
||||
### Code Quality
|
||||
|
||||
**Before writing code, read `.claude/skills/code-review/SKILL.md`** for the full coding standards (Python style, type safety, TypeScript style, general principles).
|
||||
|
||||
**Always run the lint script after making Python or TypeScript/Node changes:**
|
||||
```bash
|
||||
./scripts/hooks/lint.sh
|
||||
```
|
||||
|
||||
**After completing any implementation work, run `/code-review`** to verify your changes against project standards (missing tests, dead code, type safety, etc.). Fix any "must fix" issues before considering the task done.
|
||||
|
||||
**MANDATORY: Run `/code-review` before pushing code or creating a pull request.** Do not push or create a PR until all "must fix" issues are resolved.
|
||||
This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Prettier for TypeScript).
|
||||
|
||||
### Memory Banks
|
||||
- Each bank is an isolated memory store (like a "brain" for one user/agent)
|
||||
@@ -250,20 +200,48 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
|
||||
- Update the client type definition in `lib/api.ts`
|
||||
- Update any UI components that need to use the new parameter
|
||||
|
||||
### Adding New Integrations
|
||||
### Python Style
|
||||
- Python 3.11+, type hints required
|
||||
- Async throughout (asyncpg, async FastAPI)
|
||||
- Pydantic models for request/response
|
||||
- Ruff for linting (line-length 120)
|
||||
- No Python files at project root - maintain clean directory structure
|
||||
- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
|
||||
|
||||
Every new integration in `hindsight-integrations/` must satisfy all of the following before it can be merged:
|
||||
### Type Safety with Pydantic Models
|
||||
**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
|
||||
- Use Pydantic `BaseModel` for all data structures passed between functions
|
||||
- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
|
||||
- Avoid `dict.get()` patterns - use typed model attributes instead
|
||||
- Parse external data (JSON, API responses) into Pydantic models at the boundary
|
||||
- This catches type errors at parse time, not deep in business logic
|
||||
|
||||
1. **Tests are required** — tests must simulate or exercise the external system (mock the framework's interfaces and verify the integration actually calls Hindsight correctly). Pure unit tests of helper functions are not sufficient.
|
||||
2. **CI job** — add a test job in `.github/workflows/test.yml` following the existing pattern (e.g., `test-crewai-integration`). The job must build, install deps, and run `uv run pytest tests -v`. Also add the integration to `detect-changes` outputs so it only runs when its files change.
|
||||
3. **Release process** — add the integration name to the `VALID_INTEGRATIONS` array in `scripts/release-integration.sh` so it can be released via the standard release workflow.
|
||||
4. **Follow project code standards** — Python style, type safety, no raw dicts for structured data, no multi-item tuple returns (see `.claude/skills/code-review/SKILL.md`).
|
||||
```python
|
||||
# BAD - error-prone dict access
|
||||
def process(data: dict) -> str:
|
||||
return data.get("name", "") # No validation, silent failures
|
||||
|
||||
If any of these are missing, the integration is incomplete and must not be pushed or merged.
|
||||
# GOOD - typed and validated
|
||||
class UserData(BaseModel):
|
||||
name: str
|
||||
created_at: datetime
|
||||
|
||||
### Changelogs
|
||||
@field_validator("created_at", mode="before")
|
||||
@classmethod
|
||||
def ensure_tz_aware(cls, v):
|
||||
if isinstance(v, str):
|
||||
v = datetime.fromisoformat(v.replace("Z", "+00:00"))
|
||||
if v.tzinfo is None:
|
||||
return v.replace(tzinfo=timezone.utc)
|
||||
return v
|
||||
|
||||
Never add "Unreleased" entries to changelogs (e.g. `hindsight-docs/src/pages/changelog/**`). Changelog entries are written by the release script (`./scripts/release-integration.sh`) when a version is actually cut. If a bug fix or feature needs documenting before release, describe it in the PR/commit — the release tooling will surface it in the published changelog section.
|
||||
def process(data: UserData) -> str:
|
||||
return data.name # Type-safe, validated at construction
|
||||
```
|
||||
|
||||
### TypeScript Style
|
||||
- Next.js App Router for control plane
|
||||
- Tailwind CSS with shadcn/ui components
|
||||
|
||||
### Adding New API Configuration Flags
|
||||
|
||||
@@ -277,17 +255,17 @@ Fields must be categorized as either **hierarchical** (can be overridden per-ten
|
||||
- Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
|
||||
- Add `DEFAULT_*` constant for the default value
|
||||
- Add field to `HindsightConfig` dataclass with type annotation
|
||||
- **Mark as configurable** by adding to `_CONFIGURABLE_FIELDS` set if the field should be overridable per-tenant/bank via API
|
||||
- **Mark as hierarchical or static** by adding to `_HIERARCHICAL_FIELDS` set (hierarchical) or leaving it out (static)
|
||||
- Add initialization in `from_env()` method
|
||||
|
||||
```python
|
||||
# Configurable field (can be overridden per-tenant/bank via API)
|
||||
_CONFIGURABLE_FIELDS = {
|
||||
# Hierarchical field (can be overridden per-bank)
|
||||
_HIERARCHICAL_FIELDS = {
|
||||
...,
|
||||
"my_setting", # Add here for configurable
|
||||
"my_setting", # Add here for hierarchical
|
||||
}
|
||||
|
||||
# Static field - just don't add to _CONFIGURABLE_FIELDS
|
||||
# Static field - just don't add to _HIERARCHICAL_FIELDS
|
||||
```
|
||||
|
||||
2. **main.py** (`hindsight-api-slim/hindsight_api/main.py`):
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
[](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
|
||||
[](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://gitcgr.com/vectorize-io/hindsight)
|
||||

|
||||

|
||||
<br/>
|
||||
@@ -84,8 +83,6 @@ cd docker/docker-compose
|
||||
docker compose up
|
||||
```
|
||||
|
||||
> Oracle AI Database is also supported for enterprise deployments with full feature parity. See the [storage documentation](https://hindsight.vectorize.io/developer/storage) for details.
|
||||
|
||||
|
||||
>API: http://localhost:8888
|
||||
>UI: http://localhost:9999
|
||||
|
||||
@@ -1,28 +1,6 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# =============================================================================
|
||||
# Embedded pg0 data integrity check (#675)
|
||||
#
|
||||
# When using embedded pg0, check if the data directory has existing PostgreSQL
|
||||
# data before starting. If the directory exists but appears empty/corrupt
|
||||
# (e.g., missing PG_VERSION file), log a warning. This helps diagnose data
|
||||
# loss scenarios where a container restart caused the data directory to be
|
||||
# wiped despite a volume mount being present.
|
||||
# =============================================================================
|
||||
PG0_DATA_DIR="${HOME}/.pg0"
|
||||
if [ -d "$PG0_DATA_DIR" ]; then
|
||||
# Look for actual PostgreSQL data directories (pg0 creates subdirs per instance)
|
||||
if compgen -G "$PG0_DATA_DIR"/*/PG_VERSION > /dev/null 2>&1; then
|
||||
echo "✅ Existing pg0 data directory detected at $PG0_DATA_DIR"
|
||||
elif [ "$(ls -A "$PG0_DATA_DIR" 2>/dev/null)" ]; then
|
||||
echo "⚠️ WARNING: pg0 data directory exists at $PG0_DATA_DIR but no PG_VERSION found."
|
||||
echo " This may indicate data corruption or an incomplete previous shutdown."
|
||||
echo " If you see all migrations running from scratch after this, your data may have been lost."
|
||||
echo " See: https://github.com/vectorize-io/hindsight/issues/675"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Service flags (default to true if not set)
|
||||
ENABLE_API="${HINDSIGHT_ENABLE_API:-true}"
|
||||
ENABLE_CP="${HINDSIGHT_ENABLE_CP:-true}"
|
||||
@@ -93,63 +71,6 @@ if [ "${HINDSIGHT_WAIT_FOR_DEPS:-false}" = "true" ]; then
|
||||
done
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
# Graceful shutdown handler (#675)
|
||||
#
|
||||
# Docker sends SIGTERM on `docker stop`/`docker restart`. Without a trap, child
|
||||
# processes (hindsight-api + pg0, control-plane) are killed abruptly. For the
|
||||
# embedded pg0 database this can cause data loss when the data directory is on
|
||||
# a Docker volume that gets remounted after restart.
|
||||
#
|
||||
# The trap forwards SIGTERM to all tracked child PIDs so that:
|
||||
# - hindsight-api receives the signal and can run its shutdown hooks
|
||||
# - pg0 gets a clean PostgreSQL shutdown (checkpoint + WAL flush)
|
||||
# - The control-plane Node.js process exits cleanly
|
||||
# =============================================================================
|
||||
# Guard against concurrent cleanup (e.g., child crash + SIGTERM arriving together)
|
||||
SHUTTING_DOWN=false
|
||||
|
||||
cleanup() {
|
||||
if $SHUTTING_DOWN; then return; fi
|
||||
SHUTTING_DOWN=true
|
||||
|
||||
echo ""
|
||||
echo "🛑 Received shutdown signal, stopping services gracefully..."
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
kill -TERM "$pid" 2>/dev/null
|
||||
fi
|
||||
done
|
||||
# Give processes time to shut down cleanly (pg0 needs to flush WAL).
|
||||
# NOTE: Docker's default stop_grace_period is 10s. If you use the default,
|
||||
# either set stop_grace_period: 30s in your compose file / docker stop -t 30,
|
||||
# or Docker will SIGKILL the container before this timeout expires.
|
||||
local timeout=30
|
||||
for ((i=1; i<=timeout; i++)); do
|
||||
local all_stopped=true
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
all_stopped=false
|
||||
break
|
||||
fi
|
||||
done
|
||||
if $all_stopped; then
|
||||
echo "✅ All services stopped cleanly"
|
||||
exit 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
# Force kill if still running after timeout
|
||||
echo "⚠️ Timeout reached, forcing shutdown..."
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
kill -9 "$pid" 2>/dev/null
|
||||
fi
|
||||
done
|
||||
exit 1
|
||||
}
|
||||
trap cleanup SIGTERM SIGINT
|
||||
|
||||
# Track PIDs for wait
|
||||
PIDS=()
|
||||
|
||||
@@ -217,21 +138,8 @@ if [ ${#PIDS[@]} -eq 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Wait for any process to exit (use wait -n with trap-safe loop)
|
||||
while true; do
|
||||
# wait -n returns when any child exits; it also returns on signal delivery
|
||||
# (the trap handler will run and exit, so this loop is just for robustness).
|
||||
# `&& true` prevents `set -e` from killing the script when wait -n returns
|
||||
# non-zero (child exited with error or no backgrounded children remain).
|
||||
wait -n && true
|
||||
# Check if any tracked PID has exited
|
||||
for pid in "${PIDS[@]}"; do
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
wait "$pid" 2>/dev/null
|
||||
exit_code=$?
|
||||
echo "⚠️ Service (PID $pid) exited with code $exit_code"
|
||||
# Trigger cleanup for remaining services
|
||||
cleanup
|
||||
fi
|
||||
done
|
||||
done
|
||||
# Wait for any process to exit
|
||||
wait -n
|
||||
|
||||
# Exit with status of first exited process
|
||||
exit $?
|
||||
|
||||
@@ -2,8 +2,8 @@ apiVersion: v2
|
||||
name: hindsight
|
||||
description: Hindsight helm chart
|
||||
type: application
|
||||
version: 0.5.6
|
||||
appVersion: "0.5.6"
|
||||
version: 0.4.19
|
||||
appVersion: "0.4.19"
|
||||
keywords:
|
||||
- ai
|
||||
- memory
|
||||
|
||||
@@ -95,27 +95,6 @@ spec:
|
||||
{{- toYaml .Values.api.readinessProbe | nindent 10 }}
|
||||
resources:
|
||||
{{- toYaml .Values.api.resources | nindent 10 }}
|
||||
{{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumeMounts }}
|
||||
volumeMounts:
|
||||
{{- if .Values.api.persistence.modelCache.enabled }}
|
||||
- name: model-cache
|
||||
mountPath: /home/hindsight/.cache
|
||||
{{- end }}
|
||||
{{- with .Values.api.extraVolumeMounts }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- if or .Values.api.persistence.modelCache.enabled .Values.api.extraVolumes }}
|
||||
volumes:
|
||||
{{- if .Values.api.persistence.modelCache.enabled }}
|
||||
- name: model-cache
|
||||
persistentVolumeClaim:
|
||||
claimName: {{ include "hindsight.fullname" . }}-api-model-cache
|
||||
{{- end }}
|
||||
{{- with .Values.api.extraVolumes }}
|
||||
{{- toYaml . | nindent 6 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
{{- if and .Values.api.enabled .Values.api.persistence.modelCache.enabled }}
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: {{ include "hindsight.fullname" . }}-api-model-cache
|
||||
labels:
|
||||
{{- include "hindsight.api.labels" . | nindent 4 }}
|
||||
{{- with .Values.api.persistence.modelCache.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
accessModes:
|
||||
{{- toYaml .Values.api.persistence.modelCache.accessModes | nindent 4 }}
|
||||
{{- if .Values.api.persistence.modelCache.storageClass }}
|
||||
storageClassName: {{ .Values.api.persistence.modelCache.storageClass }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.api.persistence.modelCache.size }}
|
||||
{{- end }}
|
||||
@@ -95,16 +95,6 @@ spec:
|
||||
{{- toYaml .Values.worker.readinessProbe | nindent 10 }}
|
||||
resources:
|
||||
{{- toYaml .Values.worker.resources | nindent 10 }}
|
||||
{{- if or .Values.worker.persistence.modelCache.enabled .Values.worker.extraVolumeMounts }}
|
||||
volumeMounts:
|
||||
{{- if .Values.worker.persistence.modelCache.enabled }}
|
||||
- name: model-cache
|
||||
mountPath: /home/hindsight/.cache
|
||||
{{- end }}
|
||||
{{- with .Values.worker.extraVolumeMounts }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
@@ -117,26 +107,4 @@ spec:
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.worker.extraVolumes }}
|
||||
volumes:
|
||||
{{- toYaml . | nindent 6 }}
|
||||
{{- end }}
|
||||
{{- if .Values.worker.persistence.modelCache.enabled }}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: model-cache
|
||||
{{- with .Values.worker.persistence.modelCache.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
accessModes:
|
||||
{{- toYaml .Values.worker.persistence.modelCache.accessModes | nindent 8 }}
|
||||
{{- if .Values.worker.persistence.modelCache.storageClass }}
|
||||
storageClassName: {{ .Values.worker.persistence.modelCache.storageClass }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.worker.persistence.modelCache.size }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
@@ -67,33 +67,6 @@ api:
|
||||
# Pod affinity/anti-affinity (overrides global affinity for this component)
|
||||
# affinity: {}
|
||||
|
||||
# Persistent volume for local model cache (reranker, embeddings)
|
||||
# Models are downloaded to /home/hindsight/.cache on first use.
|
||||
# Without persistence, models are re-downloaded on every pod restart.
|
||||
persistence:
|
||||
modelCache:
|
||||
enabled: false
|
||||
size: 5Gi
|
||||
storageClass: ""
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
annotations: {}
|
||||
|
||||
# Extra volume mounts for the api container
|
||||
# e.g.
|
||||
# extraVolumeMounts:
|
||||
# - name: my-volume
|
||||
# mountPath: /mnt/my-volume
|
||||
extraVolumeMounts: []
|
||||
|
||||
# Extra volumes for the api pod
|
||||
# e.g.
|
||||
# extraVolumes:
|
||||
# - name: my-volume
|
||||
# configMap:
|
||||
# name: my-configmap
|
||||
extraVolumes: []
|
||||
|
||||
# Environment variables
|
||||
env:
|
||||
#HINDSIGHT_API_LLM_PROVIDER: "groq"
|
||||
@@ -167,32 +140,6 @@ worker:
|
||||
# Pod affinity/anti-affinity (overrides global affinity for this component)
|
||||
# affinity: {}
|
||||
|
||||
# Persistent volume for local model cache (reranker, embeddings)
|
||||
# Uses volumeClaimTemplates since worker is a StatefulSet.
|
||||
persistence:
|
||||
modelCache:
|
||||
enabled: false
|
||||
size: 5Gi
|
||||
storageClass: ""
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
annotations: {}
|
||||
|
||||
# Extra volume mounts for the worker container
|
||||
# e.g.
|
||||
# extraVolumeMounts:
|
||||
# - name: my-volume
|
||||
# mountPath: /mnt/my-volume
|
||||
extraVolumeMounts: []
|
||||
|
||||
# Extra volumes for the worker pod
|
||||
# e.g.
|
||||
# extraVolumes:
|
||||
# - name: my-volume
|
||||
# configMap:
|
||||
# name: my-configmap
|
||||
extraVolumes: []
|
||||
|
||||
# Secret environment variables (inherited from api.secrets if not specified)
|
||||
secrets: {}
|
||||
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
node_modules
|
||||
dist
|
||||
*.tgz
|
||||
.DS_Store
|
||||
@@ -1,80 +0,0 @@
|
||||
# @vectorize-io/hindsight-all
|
||||
|
||||
Node.js equivalent of the Python [`hindsight-all`](https://pypi.org/project/hindsight-all/) package — programmatic lifecycle manager for a local Hindsight daemon. Use this when you want to embed Hindsight in a Node application without hand-rolling subprocess management.
|
||||
|
||||
This package deliberately does **not** ship an HTTP client. Once the daemon is running, talk to it with [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client) against `server.getBaseUrl()`. The two packages compose — one owns the daemon process, the other owns the HTTP API surface.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Node.js >= 22** — uses global `fetch` and `AbortSignal.timeout`.
|
||||
- **`uv` / `uvx`** on `PATH` — used to download and run the underlying `hindsight-embed` daemon on first use. Install via <https://docs.astral.sh/uv/>.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install @vectorize-io/hindsight-all @vectorize-io/hindsight-client
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
```ts
|
||||
import { HindsightServer, consoleLogger } from "@vectorize-io/hindsight-all";
|
||||
import { HindsightClient } from "@vectorize-io/hindsight-client";
|
||||
|
||||
const server = new HindsightServer({
|
||||
profile: "my-app",
|
||||
port: 9077,
|
||||
env: {
|
||||
HINDSIGHT_API_LLM_PROVIDER: "anthropic",
|
||||
HINDSIGHT_API_LLM_API_KEY: process.env.ANTHROPIC_API_KEY,
|
||||
HINDSIGHT_API_LLM_MODEL: "claude-sonnet-4-20250514",
|
||||
HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT: "0",
|
||||
},
|
||||
logger: consoleLogger,
|
||||
});
|
||||
|
||||
await server.start();
|
||||
|
||||
const client = new HindsightClient({ baseUrl: server.getBaseUrl() });
|
||||
|
||||
await client.retain("user-123", "User prefers dark mode and concise answers.", {
|
||||
documentId: "pref-2026-04-01",
|
||||
});
|
||||
|
||||
const recall = await client.recall("user-123", "what are the user preferences?");
|
||||
console.log(recall.results);
|
||||
|
||||
await server.stop();
|
||||
```
|
||||
|
||||
For a remote Hindsight API, skip `HindsightServer` entirely and just point `HindsightClient` at the remote URL.
|
||||
|
||||
## Open config — forward-compatible with new daemon flags
|
||||
|
||||
`HindsightServerOptions` is designed so every new environment variable or CLI flag in the underlying Hindsight daemon can be used without waiting for a wrapper release:
|
||||
|
||||
- **`env`** accepts an arbitrary `Record<string, string>`. Every entry is exported into the daemon process and written into the profile config via `--env KEY=VALUE`.
|
||||
- **`extraProfileCreateArgs`** / **`extraDaemonStartArgs`** append raw args to the respective commands.
|
||||
|
||||
## Development against a local checkout
|
||||
|
||||
If you're hacking on the Python `hindsight-embed` package in the same monorepo, point the server at the local path — it'll use `uv run --directory <path>` instead of `uvx`:
|
||||
|
||||
```ts
|
||||
new HindsightServer({
|
||||
embedPackagePath: "/path/to/hindsight-embed",
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
## API surface
|
||||
|
||||
- `HindsightServer` — daemon lifecycle (`start`, `stop`, `checkHealth`, `getBaseUrl`, `getProfile`).
|
||||
- `Logger` interface plus `silentLogger` (default) and `consoleLogger` helpers.
|
||||
- `getEmbedCommand(opts)` — low-level helper that returns the `[cmd, ...args]` tuple used to invoke the underlying Python CLI.
|
||||
|
||||
For memory operations (retain, recall, reflect, bank management, stats) use [`@vectorize-io/hindsight-client`](https://www.npmjs.com/package/@vectorize-io/hindsight-client).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -1,57 +0,0 @@
|
||||
{
|
||||
"name": "@vectorize-io/hindsight-all",
|
||||
"version": "0.5.6",
|
||||
"description": "Node.js programmatic lifecycle manager for Hindsight — embeds a local hindsight daemon in a Node application. Pair with @vectorize-io/hindsight-client for memory operations.",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js"
|
||||
}
|
||||
},
|
||||
"keywords": [
|
||||
"hindsight",
|
||||
"hindsight-all",
|
||||
"memory",
|
||||
"ai",
|
||||
"agent",
|
||||
"long-term-memory",
|
||||
"llm",
|
||||
"embedded-server"
|
||||
],
|
||||
"author": "Vectorize <[email protected]>",
|
||||
"license": "MIT",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/vectorize-io/hindsight.git",
|
||||
"directory": "hindsight-all-npm"
|
||||
},
|
||||
"files": [
|
||||
"dist",
|
||||
"README.md"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "tsup",
|
||||
"dev": "tsup --watch",
|
||||
"clean": "rm -rf dist",
|
||||
"test": "vitest run src",
|
||||
"test:watch": "vitest src",
|
||||
"prepublishOnly": "npm run clean && npm run build"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.0.0",
|
||||
"tsup": "^8.5.1",
|
||||
"typescript": "^5.7.0",
|
||||
"vitest": "^4.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
},
|
||||
"overrides": {
|
||||
"rollup": "^4.59.0",
|
||||
"picomatch": ">=2.3.2 <3.0.0 || >=4.0.4",
|
||||
"vite": ">=8.0.5"
|
||||
}
|
||||
}
|
||||
@@ -1,36 +0,0 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { getEmbedCommand } from "./command.js";
|
||||
|
||||
describe("getEmbedCommand", () => {
|
||||
it("defaults to uvx hindsight-embed@latest", () => {
|
||||
expect(getEmbedCommand()).toEqual(["uvx", "hindsight-embed@latest"]);
|
||||
});
|
||||
|
||||
it("honours an explicit version", () => {
|
||||
expect(getEmbedCommand({ embedVersion: "0.5.0" })).toEqual(["uvx", "[email protected]"]);
|
||||
});
|
||||
|
||||
it("treats an empty version as latest", () => {
|
||||
expect(getEmbedCommand({ embedVersion: "" })).toEqual(["uvx", "hindsight-embed@latest"]);
|
||||
});
|
||||
|
||||
it("uses uv run --directory when a local path is given", () => {
|
||||
expect(getEmbedCommand({ embedPackagePath: "/abs/path" })).toEqual([
|
||||
"uv",
|
||||
"run",
|
||||
"--directory",
|
||||
"/abs/path",
|
||||
"hindsight-embed",
|
||||
]);
|
||||
});
|
||||
|
||||
it("local path takes precedence over version", () => {
|
||||
expect(getEmbedCommand({ embedPackagePath: "/abs/path", embedVersion: "0.5.0" })).toEqual([
|
||||
"uv",
|
||||
"run",
|
||||
"--directory",
|
||||
"/abs/path",
|
||||
"hindsight-embed",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -1,25 +0,0 @@
|
||||
/**
|
||||
* Resolve the command that invokes the `hindsight-embed` Python CLI.
|
||||
*
|
||||
* - If `embedPackagePath` is set, runs the package from a local checkout via
|
||||
* `uv run --directory <path> hindsight-embed`. Used for in-repo development.
|
||||
* - Otherwise runs it via `uvx hindsight-embed@<version>` so no global install
|
||||
* is required.
|
||||
*
|
||||
* Returns the argv as `[command, ...baseArgs]` suitable for `spawn()` /
|
||||
* `execFile()` (never shell-interpolated).
|
||||
*/
|
||||
export interface EmbedCommandOptions {
|
||||
/** Version spec passed to uvx (e.g. "latest", "0.5.0"). Default: "latest". */
|
||||
embedVersion?: string;
|
||||
/** Local checkout path. When set, overrides `embedVersion` and uses `uv run`. */
|
||||
embedPackagePath?: string;
|
||||
}
|
||||
|
||||
export function getEmbedCommand(opts: EmbedCommandOptions = {}): string[] {
|
||||
if (opts.embedPackagePath) {
|
||||
return ["uv", "run", "--directory", opts.embedPackagePath, "hindsight-embed"];
|
||||
}
|
||||
const version = opts.embedVersion && opts.embedVersion.length > 0 ? opts.embedVersion : "latest";
|
||||
return ["uvx", `hindsight-embed@${version}`];
|
||||
}
|
||||
@@ -1,7 +0,0 @@
|
||||
export { HindsightServer } from "./server.js";
|
||||
export { getEmbedCommand } from "./command.js";
|
||||
export { silentLogger, consoleLogger } from "./logger.js";
|
||||
|
||||
export type { Logger } from "./logger.js";
|
||||
export type { EmbedCommandOptions } from "./command.js";
|
||||
export type { HindsightServerOptions } from "./types.js";
|
||||
@@ -1,29 +0,0 @@
|
||||
/**
|
||||
* Pluggable logger interface.
|
||||
*
|
||||
* This package does not own any logging infrastructure — consumers inject
|
||||
* whatever they want (console, pino, openclaw's logger, a no-op). The default
|
||||
* is silent so embedding this package never adds noise to an unrelated app.
|
||||
*/
|
||||
export interface Logger {
|
||||
debug(msg: string): void;
|
||||
info(msg: string): void;
|
||||
warn(msg: string): void;
|
||||
error(msg: string): void;
|
||||
}
|
||||
|
||||
/** Logger that drops every call. Used when no logger is passed. */
|
||||
export const silentLogger: Logger = {
|
||||
debug: () => {},
|
||||
info: () => {},
|
||||
warn: () => {},
|
||||
error: () => {},
|
||||
};
|
||||
|
||||
/** Logger that writes to the standard console. Handy for CLIs and tests. */
|
||||
export const consoleLogger: Logger = {
|
||||
debug: (msg) => console.debug(msg),
|
||||
info: (msg) => console.log(msg),
|
||||
warn: (msg) => console.warn(msg),
|
||||
error: (msg) => console.error(msg),
|
||||
};
|
||||
@@ -1,35 +0,0 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { HindsightServer } from "./server.js";
|
||||
|
||||
describe("HindsightServer construction", () => {
|
||||
it("defaults base URL to http://127.0.0.1:8888", () => {
|
||||
const server = new HindsightServer();
|
||||
expect(server.getBaseUrl()).toBe("http://127.0.0.1:8888");
|
||||
expect(server.getProfile()).toBe("default");
|
||||
});
|
||||
|
||||
it("honours custom profile, port, and host", () => {
|
||||
const server = new HindsightServer({ profile: "app", port: 9077, host: "0.0.0.0" });
|
||||
expect(server.getProfile()).toBe("app");
|
||||
expect(server.getBaseUrl()).toBe("http://0.0.0.0:9077");
|
||||
});
|
||||
|
||||
it("accepts open env pass-through without complaining about unknown keys", () => {
|
||||
const server = new HindsightServer({
|
||||
env: {
|
||||
HINDSIGHT_API_LLM_PROVIDER: "openai",
|
||||
HINDSIGHT_API_LLM_MODEL: "gpt-4o-mini",
|
||||
// A field that does not exist today — should still be accepted
|
||||
HINDSIGHT_FUTURE_FLAG: "enabled",
|
||||
},
|
||||
});
|
||||
expect(server).toBeInstanceOf(HindsightServer);
|
||||
});
|
||||
|
||||
it("exposes checkHealth that returns false when no daemon is running", async () => {
|
||||
// Random high port that nothing is listening on.
|
||||
const server = new HindsightServer({ port: 1, readyTimeoutMs: 100 });
|
||||
const healthy = await server.checkHealth();
|
||||
expect(healthy).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,322 +0,0 @@
|
||||
import { spawn } from "child_process";
|
||||
import { getEmbedCommand } from "./command.js";
|
||||
import { silentLogger } from "./logger.js";
|
||||
import type { Logger } from "./logger.js";
|
||||
import type { HindsightServerOptions } from "./types.js";
|
||||
|
||||
const DEFAULT_PORT = 8888;
|
||||
const DEFAULT_HOST = "127.0.0.1";
|
||||
const DEFAULT_PROFILE = "default";
|
||||
const DEFAULT_READY_TIMEOUT_MS = 30_000;
|
||||
const DEFAULT_READY_POLL_INTERVAL_MS = 1_000;
|
||||
|
||||
/**
|
||||
* Manages the lifecycle of a local Hindsight daemon from a Node.js process.
|
||||
*
|
||||
* On {@link start}, this class:
|
||||
* 1. Resolves the `hindsight-embed` command (via `uvx` or a local `uv run`).
|
||||
* 2. Runs `profile create <name> --merge --port <port> [--env K=V ...]`
|
||||
* with every entry in {@link HindsightServerOptions.env} forwarded as
|
||||
* an `--env` flag.
|
||||
* 3. Runs `daemon --profile <name> start` and waits for the start command
|
||||
* to exit.
|
||||
* 4. Polls `http://host:port/health` until it returns `200` or the
|
||||
* `readyTimeoutMs` budget is exhausted.
|
||||
*
|
||||
* On {@link stop}, it runs `daemon --profile <name> stop` and returns once
|
||||
* the command exits (or after a short grace period).
|
||||
*
|
||||
* This is the Node.js equivalent of the Python `hindsight-all` package's
|
||||
* `HindsightServer`: a thin programmatic lifecycle wrapper around the
|
||||
* Hindsight daemon. It does NOT ship an HTTP client — once `start()`
|
||||
* resolves, use `@vectorize-io/hindsight-client` against `getBaseUrl()` for
|
||||
* retain / recall / reflect.
|
||||
*
|
||||
* The class is deliberately transparent about the daemon: new CLI flags or
|
||||
* environment variables never require a code change here — callers can pass
|
||||
* them via `env`, `extraProfileCreateArgs`, or `extraDaemonStartArgs`.
|
||||
*/
|
||||
export class HindsightServer {
|
||||
private readonly profile: string;
|
||||
private readonly port: number;
|
||||
private readonly host: string;
|
||||
private readonly baseUrl: string;
|
||||
private readonly embedVersion: string | undefined;
|
||||
private readonly embedPackagePath: string | undefined;
|
||||
private readonly userEnv: Record<string, string | undefined>;
|
||||
private readonly extraProfileCreateArgs: string[];
|
||||
private readonly extraDaemonStartArgs: string[];
|
||||
private readonly platformCpuWorkaround: boolean;
|
||||
private readonly readyTimeoutMs: number;
|
||||
private readonly readyPollIntervalMs: number;
|
||||
private readonly logger: Logger;
|
||||
|
||||
constructor(opts: HindsightServerOptions = {}) {
|
||||
this.profile = opts.profile ?? DEFAULT_PROFILE;
|
||||
this.port = opts.port ?? DEFAULT_PORT;
|
||||
this.host = opts.host ?? DEFAULT_HOST;
|
||||
this.baseUrl = `http://${this.host}:${this.port}`;
|
||||
this.embedVersion = opts.embedVersion;
|
||||
this.embedPackagePath = opts.embedPackagePath;
|
||||
this.userEnv = opts.env ?? {};
|
||||
this.extraProfileCreateArgs = opts.extraProfileCreateArgs ?? [];
|
||||
this.extraDaemonStartArgs = opts.extraDaemonStartArgs ?? [];
|
||||
this.platformCpuWorkaround = opts.platformCpuWorkaround ?? process.platform === "darwin";
|
||||
this.readyTimeoutMs = opts.readyTimeoutMs ?? DEFAULT_READY_TIMEOUT_MS;
|
||||
this.readyPollIntervalMs = opts.readyPollIntervalMs ?? DEFAULT_READY_POLL_INTERVAL_MS;
|
||||
this.logger = opts.logger ?? silentLogger;
|
||||
}
|
||||
|
||||
/** The base URL the daemon listens on (`http://host:port`). */
|
||||
getBaseUrl(): string {
|
||||
return this.baseUrl;
|
||||
}
|
||||
|
||||
/** The profile name this server operates on. */
|
||||
getProfile(): string {
|
||||
return this.profile;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure the daemon is configured and running. Idempotent — the underlying
|
||||
* `profile create --merge` and `daemon start` commands tolerate re-runs.
|
||||
*/
|
||||
async start(): Promise<void> {
|
||||
this.logger.info(`[hindsight] starting daemon for profile "${this.profile}"`);
|
||||
|
||||
const env = this.buildEnv();
|
||||
await this.configureProfile(env);
|
||||
await this.startDaemon(env);
|
||||
await this.waitForReady();
|
||||
|
||||
this.logger.info(`[hindsight] daemon ready at ${this.baseUrl}`);
|
||||
}
|
||||
|
||||
/** Stop the daemon. Never throws — logs and resolves even on failure. */
|
||||
async stop(): Promise<void> {
|
||||
this.logger.info(`[hindsight] stopping daemon for profile "${this.profile}"`);
|
||||
|
||||
const [cmd, ...baseArgs] = getEmbedCommand({
|
||||
embedVersion: this.embedVersion,
|
||||
embedPackagePath: this.embedPackagePath,
|
||||
});
|
||||
const args = [...baseArgs, "daemon", "--profile", this.profile, "stop"];
|
||||
|
||||
const child = spawn(cmd, args, { stdio: "pipe" });
|
||||
this.pipeOutput(child, "daemon.stop");
|
||||
|
||||
await new Promise<void>((resolve) => {
|
||||
const timeout = setTimeout(() => {
|
||||
this.logger.warn(`[hindsight] daemon stop timed out after 5s`);
|
||||
resolve();
|
||||
}, 5_000);
|
||||
child.on("exit", () => {
|
||||
clearTimeout(timeout);
|
||||
this.logger.info(`[hindsight] daemon stopped`);
|
||||
resolve();
|
||||
});
|
||||
child.on("error", (err) => {
|
||||
clearTimeout(timeout);
|
||||
this.logger.warn(`[hindsight] error stopping daemon: ${err.message}`);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** Probe `/health` once with a short timeout. */
|
||||
async checkHealth(): Promise<boolean> {
|
||||
try {
|
||||
const res = await fetch(`${this.baseUrl}/health`, {
|
||||
signal: AbortSignal.timeout(2_000),
|
||||
});
|
||||
return res.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Internal
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Merge the process env, the caller-supplied `env`, and (on macOS) the
|
||||
* embeddings CPU workaround. Caller-supplied values always win over the
|
||||
* workaround; undefined values are dropped.
|
||||
*/
|
||||
private buildEnv(): NodeJS.ProcessEnv {
|
||||
const merged: NodeJS.ProcessEnv = { ...process.env };
|
||||
|
||||
if (this.platformCpuWorkaround && process.platform === "darwin") {
|
||||
merged["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1";
|
||||
merged["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1";
|
||||
}
|
||||
|
||||
for (const [key, value] of Object.entries(this.userEnv)) {
|
||||
if (value !== undefined) {
|
||||
merged[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `profile create <name> --merge --port <port> [--env K=V ...]`.
|
||||
* Every entry in the merged env that was passed via {@link userEnv} (or
|
||||
* auto-applied by the CPU workaround) is forwarded as `--env`.
|
||||
*/
|
||||
private async configureProfile(env: NodeJS.ProcessEnv): Promise<void> {
|
||||
this.logger.info(`[hindsight] configuring profile "${this.profile}"`);
|
||||
|
||||
const [cmd, ...baseArgs] = getEmbedCommand({
|
||||
embedVersion: this.embedVersion,
|
||||
embedPackagePath: this.embedPackagePath,
|
||||
});
|
||||
const createArgs = [
|
||||
...baseArgs,
|
||||
"profile",
|
||||
"create",
|
||||
this.profile,
|
||||
"--merge",
|
||||
"--port",
|
||||
String(this.port),
|
||||
];
|
||||
|
||||
// Forward every env var that the caller intended for the daemon as --env.
|
||||
// We only forward keys the caller explicitly set (userEnv) plus the CPU
|
||||
// workaround values — not the entire process.env, to avoid leaking random
|
||||
// host state into profile config.
|
||||
const envForProfile = this.collectProfileEnv(env);
|
||||
for (const [key, value] of Object.entries(envForProfile)) {
|
||||
createArgs.push("--env", `${key}=${value}`);
|
||||
}
|
||||
|
||||
createArgs.push(...this.extraProfileCreateArgs);
|
||||
|
||||
await this.runCommand(cmd, createArgs, env, "profile.create");
|
||||
}
|
||||
|
||||
/** Collect only the env vars that should be written into the profile file. */
|
||||
private collectProfileEnv(env: NodeJS.ProcessEnv): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
|
||||
// 1. User-supplied env — always forwarded.
|
||||
for (const [key, value] of Object.entries(this.userEnv)) {
|
||||
if (value !== undefined) {
|
||||
out[key] = value;
|
||||
}
|
||||
}
|
||||
|
||||
// 2. CPU workaround — only if auto-applied and not already overridden.
|
||||
if (this.platformCpuWorkaround && process.platform === "darwin") {
|
||||
const cpuKeys = [
|
||||
"HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU",
|
||||
"HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU",
|
||||
];
|
||||
for (const key of cpuKeys) {
|
||||
if (!(key in out) && env[key] !== undefined) {
|
||||
out[key] = env[key] as string;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
private async startDaemon(env: NodeJS.ProcessEnv): Promise<void> {
|
||||
const [cmd, ...baseArgs] = getEmbedCommand({
|
||||
embedVersion: this.embedVersion,
|
||||
embedPackagePath: this.embedPackagePath,
|
||||
});
|
||||
const args = [
|
||||
...baseArgs,
|
||||
"daemon",
|
||||
"--profile",
|
||||
this.profile,
|
||||
"start",
|
||||
...this.extraDaemonStartArgs,
|
||||
];
|
||||
|
||||
await this.runCommand(cmd, args, env, "daemon.start");
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawn `cmd` with `args`, pipe its output through the logger, and resolve
|
||||
* once it exits with code 0. Rejects on non-zero exit or spawn error.
|
||||
*/
|
||||
private async runCommand(
|
||||
cmd: string,
|
||||
args: string[],
|
||||
env: NodeJS.ProcessEnv,
|
||||
label: string
|
||||
): Promise<void> {
|
||||
const child = spawn(cmd, args, { stdio: "pipe", env });
|
||||
let output = "";
|
||||
child.stdout?.on("data", (data: Buffer) => {
|
||||
const text = data.toString();
|
||||
output += text;
|
||||
for (const line of text.trimEnd().split("\n")) {
|
||||
if (line) this.logger.info(`[hindsight:${label}] ${line}`);
|
||||
}
|
||||
});
|
||||
child.stderr?.on("data", (data: Buffer) => {
|
||||
const text = data.toString();
|
||||
output += text;
|
||||
for (const line of text.trimEnd().split("\n")) {
|
||||
if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
|
||||
}
|
||||
});
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
child.on("exit", (code) => {
|
||||
if (code === 0) {
|
||||
resolve();
|
||||
} else {
|
||||
reject(new Error(`${label} failed with code ${code}: ${output.trim()}`));
|
||||
}
|
||||
});
|
||||
child.on("error", (err) => {
|
||||
reject(new Error(`${label} failed to spawn: ${err.message}`, { cause: err }));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/** Stream a spawned child's stdout/stderr through the logger without blocking. */
|
||||
private pipeOutput(child: ReturnType<typeof spawn>, label: string): void {
|
||||
child.stdout?.on("data", (data: Buffer) => {
|
||||
for (const line of data.toString().trimEnd().split("\n")) {
|
||||
if (line) this.logger.info(`[hindsight:${label}] ${line}`);
|
||||
}
|
||||
});
|
||||
child.stderr?.on("data", (data: Buffer) => {
|
||||
for (const line of data.toString().trimEnd().split("\n")) {
|
||||
if (line) this.logger.warn(`[hindsight:${label}] ${line}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/** Poll `/health` until it succeeds or `readyTimeoutMs` elapses. */
|
||||
private async waitForReady(): Promise<void> {
|
||||
const deadline = Date.now() + this.readyTimeoutMs;
|
||||
let attempt = 0;
|
||||
while (Date.now() < deadline) {
|
||||
attempt++;
|
||||
try {
|
||||
const res = await fetch(`${this.baseUrl}/health`, {
|
||||
signal: AbortSignal.timeout(this.readyPollIntervalMs),
|
||||
});
|
||||
if (res.ok) {
|
||||
this.logger.debug(`[hindsight] health check passed (attempt ${attempt})`);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// expected while the daemon is still booting
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, this.readyPollIntervalMs));
|
||||
}
|
||||
throw new Error(
|
||||
`Hindsight daemon did not become ready within ${this.readyTimeoutMs}ms at ${this.baseUrl}`
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,54 +0,0 @@
|
||||
import type { Logger } from "./logger.js";
|
||||
|
||||
/**
|
||||
* Options for {@link HindsightServer}.
|
||||
*
|
||||
* The server is intentionally thin and pass-through: anything configurable
|
||||
* on the daemon side (env vars or CLI flags) can be set here without needing
|
||||
* a new dedicated option. Use {@link env} for `HINDSIGHT_*` / `OPENAI_API_KEY` /
|
||||
* custom provider settings, and the two `extra*` arrays to append raw CLI
|
||||
* args to `profile create` or `daemon start`.
|
||||
*
|
||||
* For talking to the daemon after `start()`, use `@vectorize-io/hindsight-client`
|
||||
* against `server.getBaseUrl()`. This package does not ship its own HTTP
|
||||
* client.
|
||||
*/
|
||||
export interface HindsightServerOptions {
|
||||
/** Profile name used for `--profile <name>` on every sub-command. Default: `"default"`. */
|
||||
profile?: string;
|
||||
/** TCP port the daemon listens on. Default: `8888`. */
|
||||
port?: number;
|
||||
/** Hostname the daemon binds to (for health checks). Default: `127.0.0.1`. */
|
||||
host?: string;
|
||||
/** Version of the underlying `hindsight-embed` PyPI package to run via `uvx`. Default: `"latest"`. */
|
||||
embedVersion?: string;
|
||||
/** Local path to a `hindsight-embed` checkout — takes precedence over `embedVersion`. */
|
||||
embedPackagePath?: string;
|
||||
/**
|
||||
* Environment variables passed to the daemon process AND written into the
|
||||
* profile via repeated `--env KEY=VALUE` flags. This is the preferred way
|
||||
* to surface any `HINDSIGHT_API_*` / `HINDSIGHT_EMBED_*` setting — adding a
|
||||
* new daemon env var never requires a wrapper update.
|
||||
*
|
||||
* Values of `undefined` are dropped (so you can spread conditionally).
|
||||
*/
|
||||
env?: Record<string, string | undefined>;
|
||||
/** Extra args appended verbatim to `hindsight-embed profile create <name> --merge ...`. */
|
||||
extraProfileCreateArgs?: string[];
|
||||
/** Extra args appended verbatim to `hindsight-embed daemon --profile <name> start ...`. */
|
||||
extraDaemonStartArgs?: string[];
|
||||
/**
|
||||
* On macOS, automatically set
|
||||
* `HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1` and
|
||||
* `HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1` to avoid Metal/MPS crashes in
|
||||
* daemon mode. Default: `true` on `darwin`, ignored elsewhere. Any value set
|
||||
* explicitly in {@link env} wins over the auto-applied value.
|
||||
*/
|
||||
platformCpuWorkaround?: boolean;
|
||||
/** Max time (ms) to wait for `/health` to return 200. Default: `30_000`. */
|
||||
readyTimeoutMs?: number;
|
||||
/** Polling interval (ms) while waiting for `/health`. Default: `1_000`. */
|
||||
readyPollIntervalMs?: number;
|
||||
/** Optional pluggable logger. Default: silent. */
|
||||
logger?: Logger;
|
||||
}
|
||||
@@ -1,18 +0,0 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "ES2022",
|
||||
"lib": ["ES2022"],
|
||||
"moduleResolution": "node",
|
||||
"declaration": true,
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist", "src/**/*.test.ts"]
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
import { defineConfig } from "tsup";
|
||||
|
||||
export default defineConfig({
|
||||
entry: ["src/index.ts"],
|
||||
format: ["esm"],
|
||||
dts: true,
|
||||
outDir: "dist",
|
||||
clean: true,
|
||||
sourcemap: true,
|
||||
bundle: true,
|
||||
});
|
||||
@@ -1,8 +0,0 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
include: ["src/**/*.test.ts"],
|
||||
environment: "node",
|
||||
},
|
||||
});
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "hindsight-all-slim"
|
||||
version = "0.5.6"
|
||||
version = "0.4.19"
|
||||
description = "Hindsight: Agent Memory That Works Like Human Memory - Slim All-in-One Bundle"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
||||
@@ -34,6 +34,7 @@ Using context manager:
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
from typing import Optional
|
||||
|
||||
@@ -71,11 +72,8 @@ class HindsightEmbedded:
|
||||
llm_model: Model name to use
|
||||
llm_base_url: Optional custom base URL for LLM API
|
||||
database_url: Optional database URL override (default: profile-specific pg0)
|
||||
idle_timeout: Seconds before daemon auto-exits when idle (default: 0, disabled)
|
||||
idle_timeout: Seconds before daemon auto-exits when idle (default: 300)
|
||||
log_level: Daemon log level (default: "info")
|
||||
ui: Whether to start the control plane web UI alongside the daemon (default: False)
|
||||
ui_port: Port for the UI. Defaults to daemon_port + 10000.
|
||||
ui_hostname: Hostname to bind the UI to. Defaults to "0.0.0.0".
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
@@ -86,11 +84,8 @@ class HindsightEmbedded:
|
||||
llm_model: str = "openai/gpt-oss-120b",
|
||||
llm_base_url: Optional[str] = None,
|
||||
database_url: Optional[str] = None,
|
||||
idle_timeout: int = 0,
|
||||
idle_timeout: int = 300,
|
||||
log_level: str = "info",
|
||||
ui: bool = False,
|
||||
ui_port: Optional[int] = None,
|
||||
ui_hostname: str = "0.0.0.0",
|
||||
):
|
||||
"""
|
||||
Initialize the embedded client (daemon starts on first use).
|
||||
@@ -102,11 +97,8 @@ class HindsightEmbedded:
|
||||
llm_model: Model name to use
|
||||
llm_base_url: Optional custom base URL for LLM API
|
||||
database_url: Optional database URL override
|
||||
idle_timeout: Seconds before daemon auto-exits when idle (0 = disabled)
|
||||
idle_timeout: Seconds before daemon auto-exits when idle
|
||||
log_level: Daemon log level
|
||||
ui: Whether to start the control plane web UI alongside the daemon
|
||||
ui_port: Port for the UI (defaults to daemon_port + 10000)
|
||||
ui_hostname: Hostname to bind the UI to (defaults to "0.0.0.0")
|
||||
"""
|
||||
self.profile = profile
|
||||
|
||||
@@ -125,10 +117,6 @@ class HindsightEmbedded:
|
||||
if database_url:
|
||||
self.config["HINDSIGHT_EMBED_API_DATABASE_URL"] = database_url
|
||||
|
||||
self._ui = ui
|
||||
self._ui_port = ui_port
|
||||
self._ui_hostname = ui_hostname
|
||||
|
||||
self._client: Optional[Hindsight] = None
|
||||
self._lock = threading.Lock()
|
||||
self._started = False
|
||||
@@ -142,50 +130,23 @@ class HindsightEmbedded:
|
||||
self._memories_api: Optional[MemoriesAPI] = None
|
||||
|
||||
def _ensure_started(self):
|
||||
"""Ensure daemon is running (thread-safe), restarting if crashed."""
|
||||
"""Ensure daemon is running (thread-safe)."""
|
||||
if self._started and self._client is not None:
|
||||
if self._manager.is_running(self.profile):
|
||||
return
|
||||
# Daemon crashed — reset state and fall through to restart
|
||||
logger.warning(
|
||||
"Daemon for profile '%s' is no longer responsive, restarting...",
|
||||
self.profile,
|
||||
)
|
||||
try:
|
||||
self._client.close()
|
||||
except Exception:
|
||||
logger.debug("Error closing stale client", exc_info=True)
|
||||
self._client = None
|
||||
self._started = False
|
||||
return
|
||||
|
||||
with self._lock:
|
||||
# Double-check after acquiring lock
|
||||
if self._started and self._client is not None:
|
||||
if self._manager.is_running(self.profile):
|
||||
return
|
||||
logger.warning(
|
||||
"Daemon for profile '%s' is no longer responsive (lock path), restarting...",
|
||||
self.profile,
|
||||
)
|
||||
try:
|
||||
self._client.close()
|
||||
except Exception:
|
||||
logger.debug("Error closing stale client", exc_info=True)
|
||||
self._client = None
|
||||
self._started = False
|
||||
return
|
||||
|
||||
if self._closed:
|
||||
raise RuntimeError(
|
||||
"Cannot use HindsightEmbedded after it has been closed"
|
||||
)
|
||||
raise RuntimeError("Cannot use HindsightEmbedded after it has been closed")
|
||||
|
||||
# Use embed manager interface for daemon management
|
||||
logger.info(f"Ensuring daemon is running for profile '{self.profile}'...")
|
||||
success = self._manager.ensure_running(self.config, self.profile)
|
||||
if not success:
|
||||
raise RuntimeError(
|
||||
f"Failed to start daemon for profile '{self.profile}'"
|
||||
)
|
||||
raise RuntimeError(f"Failed to start daemon for profile '{self.profile}'")
|
||||
|
||||
# Get daemon URL and create client
|
||||
daemon_url = self._manager.get_url(self.profile)
|
||||
@@ -193,15 +154,6 @@ class HindsightEmbedded:
|
||||
self._started = True
|
||||
logger.info(f"Connected to daemon at {daemon_url}")
|
||||
|
||||
# Start UI if requested
|
||||
if self._ui:
|
||||
logger.info(f"Starting UI for profile '{self.profile}'...")
|
||||
ui_started = self._manager.start_ui(
|
||||
self.profile, self._ui_port, self._ui_hostname
|
||||
)
|
||||
if not ui_started:
|
||||
logger.warning(f"Failed to start UI for profile '{self.profile}'")
|
||||
|
||||
def _cleanup(self, stop_daemon_on_close: bool = False):
|
||||
"""
|
||||
Cleanup client resources (idempotent).
|
||||
@@ -213,47 +165,20 @@ class HindsightEmbedded:
|
||||
if self._closed:
|
||||
return
|
||||
|
||||
acquired = self._lock.acquire(timeout=5.0)
|
||||
if not acquired:
|
||||
# Lock is held by another thread (e.g. _ensure_started).
|
||||
# Mark closed to prevent new operations but skip shared-state
|
||||
# teardown — the daemon's idle timeout handles the rest.
|
||||
logger.warning(
|
||||
"Cleanup lock acquisition timed out for profile '%s'; "
|
||||
"marking closed, daemon will idle-stop on its own",
|
||||
self.profile,
|
||||
)
|
||||
self._closed = True
|
||||
return
|
||||
|
||||
try:
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
return
|
||||
|
||||
if self._client is not None:
|
||||
try:
|
||||
self._client.close()
|
||||
except Exception:
|
||||
logger.debug(
|
||||
"Error closing client for profile '%s'",
|
||||
self.profile,
|
||||
exc_info=True,
|
||||
)
|
||||
self._client.close()
|
||||
self._client = None
|
||||
|
||||
# Stop UI if it was started
|
||||
if self._ui and self._started:
|
||||
logger.info(f"Stopping UI for profile '{self.profile}'...")
|
||||
self._manager.stop_ui(self.profile, self._ui_port)
|
||||
|
||||
# Optionally stop daemon (daemon has idle timeout, so not required)
|
||||
if stop_daemon_on_close and self._started:
|
||||
logger.info(f"Stopping daemon for profile '{self.profile}'...")
|
||||
self._manager.stop(self.profile)
|
||||
|
||||
self._closed = True
|
||||
finally:
|
||||
self._lock.release()
|
||||
|
||||
def close(self, stop_daemon: bool = False):
|
||||
"""
|
||||
@@ -276,10 +201,23 @@ class HindsightEmbedded:
|
||||
This allows HindsightEmbedded to expose all HindsightClient methods
|
||||
without manually wrapping each one.
|
||||
"""
|
||||
# Ensure server is started (and restart if crashed) before proxying
|
||||
# Ensure server is started before proxying
|
||||
self._ensure_started()
|
||||
|
||||
return getattr(self._client, name)
|
||||
# Get the attribute from the underlying client
|
||||
attr = getattr(self._client, name)
|
||||
|
||||
# If it's a callable, wrap it to ensure server is started
|
||||
# (shouldn't be needed since _ensure_started already called, but defensive)
|
||||
if callable(attr):
|
||||
|
||||
def wrapper(*args, **kwargs):
|
||||
self._ensure_started()
|
||||
return attr(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
return attr
|
||||
|
||||
def __enter__(self):
|
||||
"""Context manager entry - ensures server is started."""
|
||||
@@ -404,8 +342,11 @@ class HindsightEmbedded:
|
||||
"""
|
||||
Get the underlying Hindsight client for direct access.
|
||||
|
||||
Ensures daemon is started (and restarts it if it has crashed) before
|
||||
returning the client.
|
||||
WARNING: Using this property directly means daemon restarts won't be
|
||||
handled automatically. Prefer using the API namespaces (banks, mental_models,
|
||||
directives, memories) or direct method calls on HindsightEmbedded instead.
|
||||
|
||||
Ensures daemon is started before returning the client.
|
||||
|
||||
Returns:
|
||||
Hindsight: The underlying client instance
|
||||
@@ -416,8 +357,9 @@ class HindsightEmbedded:
|
||||
|
||||
embedded = HindsightEmbedded(profile="myapp", ...)
|
||||
|
||||
# Direct access (not recommended - daemon crashes won't be handled)
|
||||
client = embedded.client
|
||||
banks = client.list_banks()
|
||||
banks = client.list_banks() # If daemon crashes, this will fail
|
||||
```
|
||||
"""
|
||||
self._ensure_started()
|
||||
@@ -431,15 +373,5 @@ class HindsightEmbedded:
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
"""Check if the client is initialized and the daemon is responsive."""
|
||||
return (
|
||||
self._started
|
||||
and not self._closed
|
||||
and self._client is not None
|
||||
and self._manager.is_running(self.profile)
|
||||
)
|
||||
|
||||
@property
|
||||
def ui_url(self) -> str:
|
||||
"""Get the UI URL for this profile."""
|
||||
return self._manager.get_ui_url(self.profile)
|
||||
"""Check if the client is initialized."""
|
||||
return self._started and not self._closed and self._client is not None
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "hindsight-all"
|
||||
version = "0.5.6"
|
||||
version = "0.4.19"
|
||||
description = "Hindsight: Agent Memory That Works Like Human Memory - All-in-One Bundle"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
@@ -20,9 +20,6 @@ hindsight-client = { workspace = true }
|
||||
hindsight-embed = { workspace = true }
|
||||
|
||||
[project.optional-dependencies]
|
||||
local-llm = [
|
||||
"hindsight-api-slim[local-llm]>=0.4.17",
|
||||
]
|
||||
test = [
|
||||
"pytest>=7.0.0",
|
||||
"pytest-asyncio>=0.21.0",
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
"""
|
||||
Unit test for _cleanup lock timeout behavior.
|
||||
|
||||
Verifies that _cleanup completes even when the lock is held by another thread,
|
||||
instead of hanging indefinitely (fixes #952).
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
def test_cleanup_completes_when_lock_held():
|
||||
"""
|
||||
_cleanup should complete (best-effort) even when self._lock is held
|
||||
by another thread, e.g. during a long _ensure_started call.
|
||||
"""
|
||||
with patch.dict("sys.modules", {
|
||||
"hindsight_client": MagicMock(),
|
||||
"hindsight_embed": MagicMock(),
|
||||
"hindsight.api_namespaces": MagicMock(),
|
||||
}):
|
||||
from hindsight.embedded import HindsightEmbedded
|
||||
|
||||
client = HindsightEmbedded.__new__(HindsightEmbedded)
|
||||
client.profile = "test"
|
||||
client._lock = threading.Lock()
|
||||
client._closed = False
|
||||
client._client = None
|
||||
client._started = False
|
||||
client._ui = False
|
||||
|
||||
# Simulate another thread holding the lock
|
||||
client._lock.acquire()
|
||||
|
||||
cleanup_done = threading.Event()
|
||||
|
||||
def run_cleanup():
|
||||
client._cleanup()
|
||||
cleanup_done.set()
|
||||
|
||||
t = threading.Thread(target=run_cleanup)
|
||||
t.start()
|
||||
|
||||
# Cleanup should complete within the timeout (5s) + margin
|
||||
assert cleanup_done.wait(timeout=8.0), (
|
||||
"_cleanup hung instead of timing out on lock acquisition"
|
||||
)
|
||||
|
||||
# Release the lock from the simulating thread
|
||||
client._lock.release()
|
||||
t.join(timeout=1.0)
|
||||
|
||||
assert client._closed, "Client should be marked as closed after cleanup"
|
||||
@@ -15,8 +15,6 @@ import os
|
||||
import uuid
|
||||
|
||||
import pytest
|
||||
import urllib.request
|
||||
import json
|
||||
|
||||
from hindsight import HindsightEmbedded
|
||||
|
||||
@@ -25,20 +23,12 @@ from hindsight import HindsightEmbedded
|
||||
def llm_config():
|
||||
"""Get LLM configuration from environment (session-scoped)."""
|
||||
# Try both naming conventions
|
||||
provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv(
|
||||
"HINDSIGHT_LLM_PROVIDER", "groq"
|
||||
)
|
||||
api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv(
|
||||
"HINDSIGHT_LLM_API_KEY", ""
|
||||
)
|
||||
model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv(
|
||||
"HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b"
|
||||
)
|
||||
provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv("HINDSIGHT_LLM_PROVIDER", "groq")
|
||||
api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv("HINDSIGHT_LLM_API_KEY", "")
|
||||
model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv("HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b")
|
||||
|
||||
if not api_key:
|
||||
pytest.skip(
|
||||
"LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY."
|
||||
)
|
||||
pytest.skip("LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY.")
|
||||
|
||||
return {
|
||||
"llm_provider": provider,
|
||||
@@ -88,9 +78,7 @@ def test_embedded_context_manager(llm_config):
|
||||
|
||||
# Recall memory
|
||||
recall_results = client.recall(bank_id=bank_id, query="context")
|
||||
assert isinstance(recall_results.results, list), (
|
||||
"Recall should return results list"
|
||||
)
|
||||
assert isinstance(recall_results.results, list), "Recall should return results list"
|
||||
|
||||
# Server should be stopped after context exit
|
||||
# Note: We can't check client.is_running here as client is out of scope
|
||||
@@ -117,9 +105,7 @@ def test_embedded_complete_workflow(llm_config):
|
||||
# Step 1: Create a memory bank
|
||||
print(f"\n1. Creating memory bank: {bank_id}")
|
||||
bank_response = client.create_bank(
|
||||
bank_id=bank_id,
|
||||
name="Test Assistant",
|
||||
mission="Help with programming tasks",
|
||||
bank_id=bank_id, name="Test Assistant", mission="Help with programming tasks"
|
||||
)
|
||||
assert bank_response.bank_id == bank_id
|
||||
|
||||
@@ -140,9 +126,7 @@ def test_embedded_complete_workflow(llm_config):
|
||||
items=[
|
||||
{"content": "User works with pandas and numpy."},
|
||||
{"content": "User likes matplotlib for visualization."},
|
||||
{
|
||||
"content": "User is interested in machine learning with scikit-learn."
|
||||
},
|
||||
{"content": "User is interested in machine learning with scikit-learn."},
|
||||
],
|
||||
)
|
||||
assert batch_response.success
|
||||
@@ -150,9 +134,7 @@ def test_embedded_complete_workflow(llm_config):
|
||||
|
||||
# Step 4: Recall memories
|
||||
print("\n4. Recalling memories...")
|
||||
recall_response = client.recall(
|
||||
bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000
|
||||
)
|
||||
recall_response = client.recall(bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000)
|
||||
assert isinstance(recall_response.results, list)
|
||||
assert len(recall_response.results) > 0
|
||||
print(f" Found {len(recall_response.results)} relevant memories")
|
||||
@@ -170,9 +152,7 @@ def test_embedded_complete_workflow(llm_config):
|
||||
|
||||
# Verify answer mentions relevant tools
|
||||
answer_lower = reflect_response.text.lower()
|
||||
assert any(
|
||||
term in answer_lower for term in ["python", "pandas", "numpy", "data"]
|
||||
)
|
||||
assert any(term in answer_lower for term in ["python", "pandas", "numpy", "data"])
|
||||
|
||||
# Step 6: List memories
|
||||
print("\n6. Listing memories...")
|
||||
@@ -235,9 +215,7 @@ def test_embedded_method_proxying(llm_config):
|
||||
assert bank.bank_id == bank_id
|
||||
|
||||
# Test mission setting
|
||||
mission_response = client.set_mission(
|
||||
bank_id=bank_id, mission="Test mission for proxying"
|
||||
)
|
||||
mission_response = client.set_mission(bank_id=bank_id, mission="Test mission for proxying")
|
||||
assert mission_response.bank_id == bank_id
|
||||
|
||||
# Test retain
|
||||
@@ -286,9 +264,7 @@ def test_embedded_multiple_banks(llm_config):
|
||||
|
||||
# Create second bank and store data
|
||||
client.create_bank(bank_id=bank2_id, name="Bank 2")
|
||||
client.retain(
|
||||
bank_id=bank2_id, content="Bob uses JavaScript for web development"
|
||||
)
|
||||
client.retain(bank_id=bank2_id, content="Bob uses JavaScript for web development")
|
||||
|
||||
# Recall from both banks
|
||||
results1 = client.recall(bank_id=bank1_id, query="programming language")
|
||||
@@ -299,9 +275,9 @@ def test_embedded_multiple_banks(llm_config):
|
||||
|
||||
# Verify banks are isolated (each should only see their own content)
|
||||
# This is a basic check - content isolation is tested more thoroughly in other tests
|
||||
assert results1.results[0].text != results2.results[0].text or len(
|
||||
results1.results
|
||||
) != len(results2.results)
|
||||
assert results1.results[0].text != results2.results[0].text or len(results1.results) != len(
|
||||
results2.results
|
||||
)
|
||||
|
||||
finally:
|
||||
client.close()
|
||||
@@ -320,14 +296,10 @@ def test_embedded_profile_isolation(llm_config):
|
||||
|
||||
try:
|
||||
# Store data in profile1
|
||||
client1.retain(
|
||||
bank_id=bank_id, content="User likes TypeScript for frontend development"
|
||||
)
|
||||
client1.retain(bank_id=bank_id, content="User likes TypeScript for frontend development")
|
||||
|
||||
# Store different data in profile2
|
||||
client2.retain(
|
||||
bank_id=bank_id, content="User prefers Rust for systems programming"
|
||||
)
|
||||
client2.retain(bank_id=bank_id, content="User prefers Rust for systems programming")
|
||||
|
||||
# Each profile should only see its own data
|
||||
results1 = client1.recall(bank_id=bank_id, query="programming preference")
|
||||
@@ -362,81 +334,5 @@ def test_embedded_error_after_close(llm_config):
|
||||
assert not client.is_running
|
||||
|
||||
# Trying to use it after close should raise an error
|
||||
with pytest.raises(
|
||||
RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"
|
||||
):
|
||||
with pytest.raises(RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"):
|
||||
client.retain(bank_id=bank_id, content="This should fail")
|
||||
|
||||
|
||||
def test_embedded_ui_flag(llm_config):
|
||||
"""
|
||||
Test that ui=True starts the control plane UI alongside the daemon,
|
||||
and that the UI's health endpoint reports a connected dataplane.
|
||||
"""
|
||||
profile = f"test_ui_{uuid.uuid4().hex[:8]}"
|
||||
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
client = HindsightEmbedded(profile=profile, log_level="info", ui=True, **llm_config)
|
||||
|
||||
try:
|
||||
# First use triggers daemon + UI startup
|
||||
result = client.retain(bank_id=bank_id, content="UI integration test content")
|
||||
assert result.success, "Retain should succeed"
|
||||
assert client.is_running, "Daemon should be running"
|
||||
|
||||
# Verify UI is reachable and reports connected dataplane
|
||||
ui_url = client.ui_url
|
||||
assert ui_url, "ui_url should be set"
|
||||
|
||||
health_url = f"{ui_url}/api/health"
|
||||
with urllib.request.urlopen(health_url, timeout=10) as resp:
|
||||
health = json.loads(resp.read().decode())
|
||||
|
||||
assert health["status"] == "ok", (
|
||||
f"UI health status should be 'ok', got: {health['status']}"
|
||||
)
|
||||
assert health["dataplane"]["status"] == "connected", (
|
||||
f"Dataplane should be connected, got: {health['dataplane']}"
|
||||
)
|
||||
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def test_embedded_daemon_crash_recovery(llm_config):
|
||||
"""
|
||||
Test that HindsightEmbedded recovers when the daemon crashes.
|
||||
|
||||
Simulates a crash by stopping the daemon, then verifies
|
||||
that the next operation transparently restarts it.
|
||||
"""
|
||||
profile = f"test_crash_{uuid.uuid4().hex[:8]}"
|
||||
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
|
||||
|
||||
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
|
||||
|
||||
try:
|
||||
# Start daemon and store a memory
|
||||
result = client.retain(bank_id=bank_id, content="Before crash")
|
||||
assert result.success, "Initial retain should succeed"
|
||||
assert client.is_running, "Daemon should be running"
|
||||
|
||||
original_url = client.url
|
||||
|
||||
# Simulate daemon crash by stopping it
|
||||
client._manager.stop(client.profile)
|
||||
assert not client._manager.is_running(client.profile), (
|
||||
"Daemon should be stopped after simulated crash"
|
||||
)
|
||||
|
||||
# Next operation should transparently restart the daemon
|
||||
result2 = client.retain(bank_id=bank_id, content="After crash recovery")
|
||||
assert result2.success, "Retain after crash recovery should succeed"
|
||||
assert client.is_running, "Daemon should be running again after recovery"
|
||||
|
||||
# Verify recall still works
|
||||
recall_result = client.recall(bank_id=bank_id, query="crash")
|
||||
assert isinstance(recall_result.results, list), "Recall should return results"
|
||||
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
@@ -46,4 +46,4 @@ __all__ = [
|
||||
"RemoteTEICrossEncoder",
|
||||
"LLMConfig",
|
||||
]
|
||||
__version__ = "0.5.6"
|
||||
__version__ = "0.4.19"
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
"""PostgreSQL-only admin utilities (backup, restore, migration, worker management).
|
||||
|
||||
Not supported on Oracle backends. Uses asyncpg.connect() directly, binary COPY,
|
||||
TRUNCATE CASCADE, and REFRESH MATERIALIZED VIEW — all inherently PG-specific.
|
||||
"""
|
||||
Hindsight Admin CLI - backup and restore operations.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -17,10 +15,15 @@ import asyncpg
|
||||
import typer
|
||||
|
||||
from ..config import DEFAULT_DATABASE_SCHEMA, HindsightConfig
|
||||
from ..engine.schema import fq_table_explicit as _fq_table
|
||||
from ..extensions import TenantExtension, load_extension
|
||||
from ..pg0 import parse_pg0_url, resolve_database_url
|
||||
|
||||
|
||||
def _fq_table(table: str, schema: str) -> str:
|
||||
"""Get fully-qualified table name with schema prefix."""
|
||||
return f"{schema}.{table}"
|
||||
|
||||
|
||||
# Setup logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
@@ -246,7 +249,7 @@ async def _run_migration(
|
||||
schemas = list(dict.fromkeys(schemas))
|
||||
|
||||
for schema in schemas:
|
||||
run_migrations(resolved_url, schema=schema, migration_database_url=config.migration_database_url)
|
||||
run_migrations(resolved_url, schema=schema)
|
||||
|
||||
if embedding_dimension is not None:
|
||||
for schema in schemas:
|
||||
@@ -372,140 +375,6 @@ def decommission_worker(
|
||||
typer.echo(f"No tasks found for worker '{worker_id}'")
|
||||
|
||||
|
||||
async def _decommission_all_workers(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
|
||||
"""Release all processing tasks from all workers, setting them back to pending status."""
|
||||
is_pg0, instance_name, _ = parse_pg0_url(db_url)
|
||||
if is_pg0:
|
||||
typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
|
||||
resolved_url = await resolve_database_url(db_url)
|
||||
|
||||
conn = await asyncpg.connect(resolved_url)
|
||||
try:
|
||||
table = _fq_table("async_operations", schema)
|
||||
rows = await conn.fetch(
|
||||
f"""
|
||||
UPDATE {table}
|
||||
SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
|
||||
WHERE status = 'processing'
|
||||
RETURNING operation_id, worker_id, operation_type
|
||||
""",
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@app.command(name="decommission-workers")
|
||||
def decommission_workers(
|
||||
schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
|
||||
yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
|
||||
):
|
||||
"""Release all processing tasks from all workers (sets status back to pending).
|
||||
|
||||
Use this command to recover from situations where one or more workers have crashed
|
||||
or been removed without graceful shutdown. All tasks currently in 'processing' status
|
||||
will be released back to the queue regardless of which worker owns them.
|
||||
"""
|
||||
config = HindsightConfig.from_env()
|
||||
|
||||
if not config.database_url:
|
||||
typer.echo("Error: Database URL not configured.", err=True)
|
||||
typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
if not yes:
|
||||
typer.confirm(
|
||||
"This will release ALL processing tasks from ALL workers back to pending. Continue?",
|
||||
abort=True,
|
||||
)
|
||||
|
||||
typer.echo(f"Decommissioning all workers (schema: {schema})...")
|
||||
|
||||
released = asyncio.run(_decommission_all_workers(config.database_url, schema))
|
||||
|
||||
if released:
|
||||
# Group by worker_id for summary
|
||||
by_worker: dict[str, int] = {}
|
||||
for row in released:
|
||||
wid = row["worker_id"] or "unknown"
|
||||
by_worker[wid] = by_worker.get(wid, 0) + 1
|
||||
|
||||
typer.echo(f"Released {len(released)} task(s):")
|
||||
for wid, count in by_worker.items():
|
||||
typer.echo(f" {wid}: {count} task(s)")
|
||||
else:
|
||||
typer.echo("No processing tasks found")
|
||||
|
||||
|
||||
async def _worker_status(db_url: str, schema: str = "public") -> list[dict[str, Any]]:
|
||||
"""Get all processing tasks grouped by worker with their last update time."""
|
||||
is_pg0, instance_name, _ = parse_pg0_url(db_url)
|
||||
if is_pg0:
|
||||
typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
|
||||
resolved_url = await resolve_database_url(db_url)
|
||||
|
||||
conn = await asyncpg.connect(resolved_url)
|
||||
try:
|
||||
table = _fq_table("async_operations", schema)
|
||||
rows = await conn.fetch(
|
||||
f"""
|
||||
SELECT worker_id, operation_id, operation_type, bank_id,
|
||||
claimed_at, updated_at,
|
||||
now() - claimed_at AS running_for,
|
||||
now() - updated_at AS last_update_ago
|
||||
FROM {table}
|
||||
WHERE status = 'processing'
|
||||
ORDER BY worker_id, claimed_at
|
||||
""",
|
||||
)
|
||||
return [dict(r) for r in rows]
|
||||
finally:
|
||||
await conn.close()
|
||||
|
||||
|
||||
@app.command(name="worker-status")
|
||||
def worker_status(
|
||||
schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
|
||||
):
|
||||
"""Show all currently processing tasks grouped by worker.
|
||||
|
||||
Displays each worker's active tasks with operation type, bank, how long
|
||||
the task has been running, and when it was last updated. Useful for
|
||||
identifying dead workers with orphaned tasks.
|
||||
"""
|
||||
config = HindsightConfig.from_env()
|
||||
|
||||
if not config.database_url:
|
||||
typer.echo("Error: Database URL not configured.", err=True)
|
||||
typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
rows = asyncio.run(_worker_status(config.database_url, schema))
|
||||
|
||||
if not rows:
|
||||
typer.echo("No processing tasks found")
|
||||
return
|
||||
|
||||
# Group by worker_id
|
||||
by_worker: dict[str, list[dict[str, Any]]] = {}
|
||||
for row in rows:
|
||||
wid = row["worker_id"] or "unknown"
|
||||
by_worker.setdefault(wid, []).append(row)
|
||||
|
||||
typer.echo(f"Processing tasks across {len(by_worker)} worker(s):\n")
|
||||
for wid, tasks in by_worker.items():
|
||||
typer.echo(f"Worker: {wid} ({len(tasks)} task(s))")
|
||||
for task in tasks:
|
||||
op_id = str(task["operation_id"])[:8]
|
||||
running_for = task["running_for"]
|
||||
last_update = task["last_update_ago"]
|
||||
typer.echo(
|
||||
f" {op_id} {task['operation_type']:<20s} bank={task['bank_id']}"
|
||||
f" running={running_for} last_update={last_update} ago"
|
||||
)
|
||||
typer.echo("")
|
||||
|
||||
|
||||
def main():
|
||||
app()
|
||||
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Dialect dispatcher for Alembic migrations.
|
||||
|
||||
Each migration file declares a ``_pg_upgrade``/``_oracle_upgrade`` (and matching
|
||||
downgrades) function and routes ``upgrade()``/``downgrade()`` through
|
||||
``run_for_dialect``. The helper inspects the live connection's dialect name and
|
||||
runs the matching function — or no-ops if the migration doesn't apply to the
|
||||
current backend.
|
||||
|
||||
Use ``None`` (or omit the kwarg) when a migration intentionally has no effect
|
||||
on a dialect; the helper treats it as a no-op.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from alembic import op
|
||||
|
||||
DialectFn = Callable[[], None]
|
||||
_SUPPORTED = ("postgresql", "oracle")
|
||||
|
||||
|
||||
def run_for_dialect(
|
||||
*,
|
||||
pg: DialectFn | None = None,
|
||||
oracle: DialectFn | None = None,
|
||||
) -> None:
|
||||
"""Dispatch to the function matching the current bind's dialect.
|
||||
|
||||
Args:
|
||||
pg: Function to run when the active bind is PostgreSQL.
|
||||
oracle: Function to run when the active bind is Oracle.
|
||||
|
||||
Unrecognized dialects raise; an explicit ``None`` for the active dialect
|
||||
is a no-op (the migration deliberately does nothing here).
|
||||
"""
|
||||
name = op.get_bind().dialect.name
|
||||
if name not in _SUPPORTED:
|
||||
raise RuntimeError(f"Unsupported dialect for migration dispatch: {name!r}. Expected one of {_SUPPORTED}.")
|
||||
fn = {"postgresql": pg, "oracle": oracle}[name]
|
||||
if fn is not None:
|
||||
fn()
|
||||
@@ -1,37 +1,28 @@
|
||||
"""
|
||||
Alembic environment for Hindsight.
|
||||
|
||||
Supports two dialects:
|
||||
|
||||
* PostgreSQL (sync psycopg2 driver) — default; uses ``search_path`` for
|
||||
multi-tenant schema isolation and forces read-write transactions to work
|
||||
around Supabase's read-only-by-default sessions.
|
||||
* Oracle 23ai (``oracledb`` driver) — uses ``CURRENT_SCHEMA`` for tenant
|
||||
isolation; no equivalent of ``search_path`` or read-only session quirks.
|
||||
|
||||
Each migration file dispatches its DDL through ``alembic._dialect.run_for_dialect``
|
||||
so a single revision tree serves both backends.
|
||||
Alembic environment configuration for SQLAlchemy with pgvector.
|
||||
Uses synchronous psycopg2 driver for migrations to avoid pgbouncer issues.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
|
||||
|
||||
from alembic import context
|
||||
from dotenv import load_dotenv
|
||||
from sqlalchemy import Connection, engine_from_config, pool
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
|
||||
from hindsight_api.db_url import is_oracle_url, to_libpq_url
|
||||
# Import your models here
|
||||
from hindsight_api.models import Base
|
||||
|
||||
|
||||
def load_env() -> None:
|
||||
"""Load environment variables from .env (skipped if already configured)."""
|
||||
# Load environment variables based on HINDSIGHT_API_DATABASE_URL env var or default to local
|
||||
def load_env():
|
||||
"""Load environment variables from .env"""
|
||||
# Check if HINDSIGHT_API_DATABASE_URL is already set (e.g., by CI/CD)
|
||||
if os.getenv("HINDSIGHT_API_DATABASE_URL"):
|
||||
return
|
||||
|
||||
# Look for .env file in the parent directory (root of the workspace)
|
||||
root_dir = Path(__file__).parent.parent.parent
|
||||
env_file = root_dir / ".env"
|
||||
|
||||
@@ -41,45 +32,30 @@ def load_env() -> None:
|
||||
|
||||
load_env()
|
||||
|
||||
# this is the Alembic Config object, which provides
|
||||
# access to the values within the .ini file in use.
|
||||
config = context.config
|
||||
|
||||
# Note: We don't call fileConfig() here to avoid overriding the application's logging configuration.
|
||||
# Alembic will use the existing logging configuration from the application.
|
||||
|
||||
# add your model's MetaData object here
|
||||
# for 'autogenerate' support
|
||||
target_metadata = Base.metadata
|
||||
|
||||
|
||||
def _normalize_oracle_url(url: str) -> str:
|
||||
"""Coerce an Oracle URL into the SQLAlchemy form the oracledb dialect expects.
|
||||
|
||||
Two issues to handle:
|
||||
|
||||
1. Force the ``oracle+oracledb`` driver — bare ``oracle://`` defaults to
|
||||
cx_Oracle.
|
||||
2. Map a path-style service to ``?service_name=...``. SQLAlchemy's oracledb
|
||||
dialect treats the URL path as a *SID* (legacy), but Oracle Free /
|
||||
Autonomous DB only register a service name. Without this rewrite we get
|
||||
``DPY-6003: SID "FREEPDB1" is not registered`` even though the listener
|
||||
is happy to accept the same name as a service.
|
||||
"""
|
||||
parts = urlsplit(url)
|
||||
if not parts.scheme.startswith("oracle"):
|
||||
return url
|
||||
|
||||
new_scheme = "oracle+oracledb" if parts.scheme == "oracle" else parts.scheme
|
||||
service = parts.path.lstrip("/")
|
||||
new_query = parts.query
|
||||
new_path = parts.path
|
||||
|
||||
# Promote /SERVICE to ?service_name=SERVICE unless the caller already
|
||||
# supplied an explicit ?sid= or ?service_name=.
|
||||
if service and "service_name=" not in new_query and "sid=" not in new_query:
|
||||
params = [(k, v) for k, v in parse_qsl(new_query, keep_blank_values=True)]
|
||||
params.append(("service_name", service))
|
||||
new_query = urlencode(params)
|
||||
new_path = ""
|
||||
|
||||
return urlunsplit((new_scheme, parts.netloc, new_path, new_query, parts.fragment))
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
# ... etc.
|
||||
|
||||
|
||||
def get_database_url() -> str:
|
||||
"""Resolve the migration URL from Alembic config or env, normalizing per-dialect."""
|
||||
"""
|
||||
Get and process the database URL from config or environment.
|
||||
|
||||
Returns the URL with the correct driver (psycopg2) for migrations.
|
||||
"""
|
||||
# Get database URL from config (set programmatically) or environment
|
||||
database_url = config.get_main_option("sqlalchemy.url")
|
||||
if not database_url:
|
||||
database_url = os.getenv("HINDSIGHT_API_DATABASE_URL")
|
||||
@@ -89,18 +65,30 @@ def get_database_url() -> str:
|
||||
"Set HINDSIGHT_API_DATABASE_URL environment variable or pass database_url to run_migrations()."
|
||||
)
|
||||
|
||||
if is_oracle_url(database_url):
|
||||
database_url = _normalize_oracle_url(database_url)
|
||||
else:
|
||||
# PG: convert SQLAlchemy-style asyncpg URLs and ?ssl= params to libpq form
|
||||
# for the sync engine used during migrations.
|
||||
database_url = to_libpq_url(database_url)
|
||||
# For migrations, use psycopg2 (sync driver) to avoid pgbouncer prepared statement issues
|
||||
if database_url.startswith("postgresql+asyncpg://"):
|
||||
database_url = database_url.replace("postgresql+asyncpg://", "postgresql://", 1)
|
||||
elif database_url.startswith("postgres+asyncpg://"):
|
||||
database_url = database_url.replace("postgres+asyncpg://", "postgresql://", 1)
|
||||
|
||||
# Update config with processed URL for engine_from_config to use
|
||||
config.set_main_option("sqlalchemy.url", database_url)
|
||||
|
||||
return database_url
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode.
|
||||
|
||||
This configures the context with just a URL
|
||||
and not an Engine, though an Engine is acceptable
|
||||
here as well. By skipping the Engine creation
|
||||
we don't even need a DBAPI to be available.
|
||||
|
||||
Calls to context.execute() here emit the given string to the
|
||||
script output.
|
||||
|
||||
"""
|
||||
logging.info("running offline")
|
||||
database_url = get_database_url()
|
||||
|
||||
@@ -115,40 +103,14 @@ def run_migrations_offline() -> None:
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def _configure_pg_session(engine: Engine, connection: Connection, target_schema: str | None) -> None:
|
||||
"""PG-only: ensure the session is RW (Supabase) and bind ``search_path``."""
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode with synchronous engine."""
|
||||
from sqlalchemy import event, text
|
||||
|
||||
@event.listens_for(engine, "connect")
|
||||
def set_read_write_mode(dbapi_connection, connection_record):
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE")
|
||||
if target_schema:
|
||||
cursor.execute(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"')
|
||||
cursor.execute(f'SET search_path TO "{target_schema}", public')
|
||||
cursor.close()
|
||||
get_database_url() # Process and set the database URL in config
|
||||
|
||||
connection.execute(text("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE"))
|
||||
if target_schema:
|
||||
connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"'))
|
||||
connection.execute(text(f'SET search_path TO "{target_schema}", public'))
|
||||
connection.commit()
|
||||
|
||||
|
||||
def _configure_oracle_session(connection: Connection, target_schema: str | None) -> None:
|
||||
"""Oracle: switch the session's default schema; tolerate DDL contention."""
|
||||
from sqlalchemy import text
|
||||
|
||||
# Wait up to 30s for DDL locks instead of failing immediately (ORA-00054).
|
||||
connection.execute(text("ALTER SESSION SET DDL_LOCK_TIMEOUT = 30"))
|
||||
if target_schema:
|
||||
connection.execute(text(f'ALTER SESSION SET CURRENT_SCHEMA = "{target_schema}"'))
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
database_url = get_database_url()
|
||||
# Check if we're targeting a specific schema (for multi-tenant isolation)
|
||||
target_schema = config.get_main_option("target_schema")
|
||||
is_oracle = is_oracle_url(database_url)
|
||||
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
@@ -156,19 +118,37 @@ def run_migrations_online() -> None:
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
if is_oracle:
|
||||
_configure_oracle_session(connection, target_schema)
|
||||
else:
|
||||
_configure_pg_session(connectable, connection, target_schema)
|
||||
# Add event listener to ensure connection is in read-write mode
|
||||
# This is needed for Supabase which may start connections in read-only mode
|
||||
@event.listens_for(connectable, "connect")
|
||||
def set_read_write_mode(dbapi_connection, connection_record):
|
||||
cursor = dbapi_connection.cursor()
|
||||
cursor.execute("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE")
|
||||
# If targeting a specific schema, set search_path
|
||||
# Include public in search_path for access to shared extensions (pgvector)
|
||||
if target_schema:
|
||||
cursor.execute(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"')
|
||||
cursor.execute(f'SET search_path TO "{target_schema}", public')
|
||||
cursor.close()
|
||||
|
||||
with connectable.connect() as connection:
|
||||
# Also explicitly set read-write mode on this connection
|
||||
connection.execute(text("SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE"))
|
||||
|
||||
# If targeting a specific schema, set search_path
|
||||
# Include public in search_path for access to shared extensions (pgvector)
|
||||
if target_schema:
|
||||
connection.execute(text(f'CREATE SCHEMA IF NOT EXISTS "{target_schema}"'))
|
||||
connection.execute(text(f'SET search_path TO "{target_schema}", public'))
|
||||
|
||||
connection.commit() # Commit the SET command
|
||||
|
||||
# Configure context with version_table_schema if using a specific schema
|
||||
context_opts = {
|
||||
"connection": connection,
|
||||
"target_metadata": target_metadata,
|
||||
}
|
||||
if target_schema and not is_oracle:
|
||||
# Oracle has no equivalent of PG's per-schema version table; the
|
||||
# ``alembic_version`` table lives in CURRENT_SCHEMA implicitly.
|
||||
if target_schema:
|
||||
context_opts["version_table_schema"] = target_schema
|
||||
|
||||
context.configure(**context_opts)
|
||||
@@ -176,12 +156,7 @@ def run_migrations_online() -> None:
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
# Always commit. PG needs it for the explicit RW-mode SET to persist;
|
||||
# Oracle needs it because each DDL auto-commits but the trailing
|
||||
# ``UPDATE alembic_version`` is plain DML that would otherwise stay in
|
||||
# an open transaction and roll back when the connection closes —
|
||||
# producing the "schema is created but the version row is one revision
|
||||
# behind" failure mode.
|
||||
# Explicit commit to ensure changes are persisted (especially for Supabase)
|
||||
connection.commit()
|
||||
|
||||
|
||||
|
||||
@@ -11,8 +11,6 @@ from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, Sequence[str], None] = ${repr(down_revision)}
|
||||
@@ -20,27 +18,11 @@ branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
"""PostgreSQL upgrade. Set to ``None`` below if this migration is Oracle-only."""
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema."""
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
|
||||
|
||||
def _oracle_upgrade() -> None:
|
||||
"""Oracle upgrade. Set to ``None`` below if this migration is Postgres-only."""
|
||||
pass
|
||||
|
||||
|
||||
def _oracle_downgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade, oracle=_oracle_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade, oracle=_oracle_downgrade)
|
||||
"""Downgrade schema."""
|
||||
${downgrades if downgrades else "pass"}
|
||||
|
||||
-55
@@ -1,55 +0,0 @@
|
||||
"""Recreate entities trigram index on LOWER(canonical_name) for case-insensitive matching
|
||||
|
||||
The previous GIN trigram index on canonical_name was case-sensitive, causing
|
||||
"Alice" and "alice" to have different trigram sets. This recreates it on
|
||||
LOWER(canonical_name) so the % operator matches case-insensitively.
|
||||
|
||||
Revision ID: 2eee35aa3cfc
|
||||
Revises: d6e7f8a9b0c1
|
||||
Create Date: 2026-03-31
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "2eee35aa3cfc"
|
||||
down_revision: str | Sequence[str] | None = "d6e7f8a9b0c1"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Drop the old case-sensitive trigram index
|
||||
op.execute("DROP INDEX IF EXISTS entities_canonical_name_trgm_idx")
|
||||
# Create case-insensitive trigram index on LOWER(canonical_name)
|
||||
op.execute(
|
||||
f"CREATE INDEX IF NOT EXISTS entities_canonical_name_lower_trgm_idx "
|
||||
f"ON {schema}entities USING GIN (LOWER(canonical_name) gin_trgm_ops)"
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
op.execute("DROP INDEX IF EXISTS entities_canonical_name_lower_trgm_idx")
|
||||
schema = _get_schema_prefix()
|
||||
# Restore original case-sensitive index
|
||||
op.execute(
|
||||
f"CREATE INDEX IF NOT EXISTS entities_canonical_name_trgm_idx "
|
||||
f"ON {schema}entities USING GIN (canonical_name gin_trgm_ops)"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
@@ -15,8 +15,6 @@ from pgvector.sqlalchemy import Vector
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "5a366d414dce"
|
||||
down_revision: str | Sequence[str] | None = None
|
||||
@@ -114,7 +112,7 @@ def _detect_text_search_extension() -> str:
|
||||
)
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema - create all tables from scratch."""
|
||||
|
||||
# Note: pgvector extension is installed globally BEFORE migrations run
|
||||
@@ -465,7 +463,7 @@ def _pg_upgrade() -> None:
|
||||
op.create_index("idx_unit_entities_entity", "unit_entities", ["entity_id"])
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Downgrade schema - drop all tables."""
|
||||
|
||||
# Drop tables in reverse dependency order
|
||||
@@ -525,11 +523,3 @@ def _pg_downgrade() -> None:
|
||||
# Drop extensions (optional - comment out if you want to keep them)
|
||||
# op.execute('DROP EXTENSION IF EXISTS vector')
|
||||
# op.execute('DROP EXTENSION IF EXISTS "uuid-ossp"')
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-50
@@ -1,50 +0,0 @@
|
||||
"""Merge divergent migration heads for v0.5.3
|
||||
|
||||
v0.5.3 shipped with two migration heads that were never unified:
|
||||
|
||||
* ``c4x5y6z7a8b9`` — delta-refresh chain
|
||||
(``add_last_refreshed_source_query`` ->
|
||||
``add_structured_content_to_mental_models`` ->
|
||||
``backsweep_orphan_observations_v2``)
|
||||
|
||||
* ``h3i4j5k6l7m8`` — per-bank vector indexes / audit log chain
|
||||
(the ``merge_heads_and_add_unit_entities_index`` subtree)
|
||||
|
||||
Both fork from ``z1u2v3w4x5y6``. Upgrades from v0.5.2 still succeed — the
|
||||
walker applies the three c4x5 revisions and leaves the database stamped at
|
||||
both heads — but the result is a split DAG: ``alembic upgrade head``
|
||||
(singular) is ambiguous, and any future migration has to pick one head as
|
||||
its parent, orphaning the other.
|
||||
|
||||
This revision linearises the DAG into a single head. It has no schema
|
||||
effect.
|
||||
|
||||
Revision ID: 8c6fa6f7230b
|
||||
Revises: c4x5y6z7a8b9, h3i4j5k6l7m8
|
||||
Create Date: 2026-04-18
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "8c6fa6f7230b"
|
||||
down_revision: str | Sequence[str] | None = ("c4x5y6z7a8b9", "h3i4j5k6l7m8")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
-128
@@ -1,128 +0,0 @@
|
||||
"""Make memory_links.from_unit_id and memory_links.to_unit_id FKs deferrable.
|
||||
|
||||
Revision ID: 9f8e7d6c5b4a
|
||||
Revises: o1a2b3c4d5e6
|
||||
Create Date: 2026-05-03
|
||||
|
||||
Background
|
||||
----------
|
||||
Concurrent retain (which INSERTs into ``memory_links``) and any code path
|
||||
that DELETEs a row whose deletion cascades into ``memory_links`` (e.g.
|
||||
delta-retain superseding chunks, which CASCADEs chunks → memory_units →
|
||||
memory_links) can deadlock under sustained single-tenant write load.
|
||||
|
||||
The deadlock cycle:
|
||||
|
||||
* Tx A: ``DELETE FROM chunks WHERE chunk_id = ANY(...)``
|
||||
→ CASCADE acquires row locks on memory_units, then on memory_links rows
|
||||
where ``to_unit_id`` matches the deleted units.
|
||||
* Tx B: ``INSERT INTO memory_links (...)`` referencing one of the same
|
||||
memory_units rows.
|
||||
→ The immediate FK check takes ``FOR KEY SHARE`` on those memory_units
|
||||
rows.
|
||||
|
||||
The two transactions take row locks on the same memory_units rows in
|
||||
opposite orders depending on which side started first. PostgreSQL detects
|
||||
the cycle and aborts one transaction; the loser is killed mid-batch, the
|
||||
winner continues. Workers then retry, but under sustained write load the
|
||||
pattern repeats.
|
||||
|
||||
Fix
|
||||
---
|
||||
Make both ``memory_links → memory_units`` FKs (``from_unit_id`` and
|
||||
``to_unit_id``) ``DEFERRABLE INITIALLY DEFERRED``. This pushes the FK
|
||||
check from INSERT time to COMMIT time:
|
||||
|
||||
* INSERT no longer takes ``FOR KEY SHARE`` on the memory_units row → no
|
||||
contention with the cascading DELETE's row lock.
|
||||
* At COMMIT the engine validates referential integrity in one shot. If a
|
||||
cascade-DELETE has since removed the referenced unit, the INSERT
|
||||
transaction commits OR fails with a clean FK violation (sqlstate
|
||||
23503) instead of a deadlock (sqlstate 40P01).
|
||||
|
||||
The ``WHERE EXISTS`` filter already in ``_bulk_insert_links`` continues to
|
||||
filter out the typical "stale unit_id" case at INSERT time; the deferred
|
||||
FK is only the backstop for the narrow race window between the EXISTS
|
||||
probe and COMMIT. ``ON DELETE CASCADE`` semantics are unchanged — only
|
||||
the *timing* of the constraint check moves.
|
||||
|
||||
The ``entity_id`` FK on ``memory_links`` is not changed; entities are not
|
||||
involved in the observed deadlock cycle and leaving the constraint
|
||||
immediate keeps the error message specific when an entity row is missing.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "9f8e7d6c5b4a"
|
||||
down_revision: str | Sequence[str] | None = "o1a2b3c4d5e6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Schema-qualifier for raw SQL on PG (multi-tenant search_path)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
# The two FK constraints installed by the initial schema migration
|
||||
# (5a366d414dce_initial_schema), mapped to the column they constrain.
|
||||
# They reference memory_units(id) with ON DELETE CASCADE — that
|
||||
# semantics is preserved; only the deferral attribute changes.
|
||||
_FK_COLUMNS: dict[str, str] = {
|
||||
"fk_memory_links_from_unit_id_memory_units": "from_unit_id",
|
||||
"fk_memory_links_to_unit_id_memory_units": "to_unit_id",
|
||||
}
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# PostgreSQL doesn't allow altering the deferrability of an existing
|
||||
# constraint with ALTER CONSTRAINT — the constraint must be dropped
|
||||
# and recreated. DROP IF EXISTS makes the migration safe to re-run
|
||||
# on schemas where the constraint was already recreated.
|
||||
for fk_name, column in _FK_COLUMNS.items():
|
||||
op.execute(f"ALTER TABLE {schema}memory_links DROP CONSTRAINT IF EXISTS {fk_name}")
|
||||
op.execute(
|
||||
f"""
|
||||
ALTER TABLE {schema}memory_links
|
||||
ADD CONSTRAINT {fk_name}
|
||||
FOREIGN KEY ({column})
|
||||
REFERENCES {schema}memory_units (id)
|
||||
ON DELETE CASCADE
|
||||
DEFERRABLE INITIALLY DEFERRED
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Revert to the default (NOT DEFERRABLE) form so a downgrade actually
|
||||
# restores the prior schema state, even though that re-introduces the
|
||||
# deadlock window.
|
||||
for fk_name, column in _FK_COLUMNS.items():
|
||||
op.execute(f"ALTER TABLE {schema}memory_links DROP CONSTRAINT IF EXISTS {fk_name}")
|
||||
op.execute(
|
||||
f"""
|
||||
ALTER TABLE {schema}memory_links
|
||||
ADD CONSTRAINT {fk_name}
|
||||
FOREIGN KEY ({column})
|
||||
REFERENCES {schema}memory_units (id)
|
||||
ON DELETE CASCADE
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# PG-only: Oracle's deferrable-FK semantics differ and the deadlock
|
||||
# cycle was only observed on PostgreSQL. Oracle slot intentionally
|
||||
# absent → no-op there.
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -15,8 +15,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a1b2c3d4e5f6"
|
||||
down_revision: str | Sequence[str] | None = "y0t1u2v3w4x5"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -29,7 +27,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Create file_storage table for BYTEA storage."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -54,7 +52,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove file_storage table and related columns."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -70,11 +68,3 @@ def _pg_downgrade() -> None:
|
||||
|
||||
# Drop file_storage table
|
||||
op.execute(f"DROP TABLE IF EXISTS {schema}file_storage")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -18,8 +18,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a2b3c4d5e6f7"
|
||||
down_revision: str | Sequence[str] | None = "aa2b3c4d5e6f"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -35,7 +33,7 @@ def _detect_text_search_extension() -> str:
|
||||
return os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
table = f"{schema}memory_units"
|
||||
text_search_ext = _detect_text_search_extension()
|
||||
@@ -67,7 +65,7 @@ def _pg_upgrade() -> None:
|
||||
# pg_textsearch: no change — index operates on the base `text` column only
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
table = f"{schema}memory_units"
|
||||
text_search_ext = _detect_text_search_extension()
|
||||
@@ -88,11 +86,3 @@ def _pg_downgrade() -> None:
|
||||
""")
|
||||
|
||||
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS text_signals")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -24,8 +24,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a2b3c4d5e6f8"
|
||||
down_revision: str | Sequence[str] | None = "f7g8h9i0j1k2"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -37,7 +35,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
|
||||
@@ -50,15 +48,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-48
@@ -1,48 +0,0 @@
|
||||
"""Add last_refreshed_source_query column to mental_models
|
||||
|
||||
Revision ID: a2v3w4x5y6z7
|
||||
Revises: z1u2v3w4x5y6
|
||||
Create Date: 2026-04-15
|
||||
|
||||
Tracks the source_query that was used during the most recent refresh.
|
||||
Used by delta-mode refresh to detect when the query has changed: if it has,
|
||||
delta mode falls back to a full regeneration because the surgical-edit
|
||||
assumption (same topic, new facts) no longer holds.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a2v3w4x5y6z7"
|
||||
down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}mental_models
|
||||
ADD COLUMN IF NOT EXISTS last_refreshed_source_query TEXT
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS last_refreshed_source_query")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -13,8 +13,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a3b4c5d6e7f8"
|
||||
down_revision: str | Sequence[str] | None = "g7h8i9j0k1l2"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(
|
||||
@@ -47,16 +45,8 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_consolidation_failed")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidation_failed_at")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-152
@@ -1,152 +0,0 @@
|
||||
"""Fix per-bank vector indexes to match configured extension
|
||||
|
||||
Revision ID: a4b5c6d7e8f9
|
||||
Revises: 2eee35aa3cfc
|
||||
Create Date: 2026-04-01
|
||||
|
||||
Migration d5e6f7a8b9c0 hardcoded HNSW when creating per-bank partial vector
|
||||
indexes, ignoring HINDSIGHT_API_VECTOR_EXTENSION. Banks that existed when that
|
||||
migration ran got HNSW indexes even when pgvectorscale (DiskANN) or vchord
|
||||
was configured.
|
||||
|
||||
This migration detects the mismatch and recreates the affected indexes with
|
||||
the correct type. Skipped entirely when the configured extension is pgvector
|
||||
(the default), since those indexes are already correct.
|
||||
"""
|
||||
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
from sqlalchemy import text
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "a4b5c6d7e8f9"
|
||||
down_revision: str | Sequence[str] | None = "2eee35aa3cfc"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
_FACT_TYPES: dict[str, str] = {
|
||||
"world": "worl",
|
||||
"experience": "expr",
|
||||
"observation": "obsv",
|
||||
}
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _target_index_type() -> str | None:
|
||||
"""Return the target index type, or None if pgvector (no fix needed)."""
|
||||
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
|
||||
if ext == "pgvectorscale":
|
||||
return "diskann"
|
||||
elif ext == "vchord":
|
||||
return "vchordrq"
|
||||
return None
|
||||
|
||||
|
||||
def _vector_index_using_clause() -> str:
|
||||
"""Return the USING clause based on the configured vector extension."""
|
||||
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
|
||||
if ext == "pgvectorscale":
|
||||
return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
|
||||
elif ext == "vchord":
|
||||
return "USING vchordrq (embedding vector_l2_ops)"
|
||||
else:
|
||||
return "USING hnsw (embedding vector_cosine_ops)"
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
target = _target_index_type()
|
||||
if target is None:
|
||||
# pgvector — indexes are already HNSW, nothing to fix
|
||||
return
|
||||
|
||||
bind = op.get_bind()
|
||||
schema_name = context.config.get_main_option("target_schema")
|
||||
schema = _get_schema_prefix()
|
||||
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
|
||||
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
|
||||
using_clause = _vector_index_using_clause()
|
||||
pg_schema = schema_name or "public"
|
||||
|
||||
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
|
||||
for row in rows:
|
||||
bank_id = row[0]
|
||||
internal_id = str(row[1]).replace("-", "")[:16]
|
||||
escaped_bank_id = bank_id.replace("'", "''")
|
||||
for ft, ft_short in _FACT_TYPES.items():
|
||||
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
|
||||
|
||||
# Check if this index exists and what type it is
|
||||
idx_info = bind.execute(
|
||||
text("SELECT indexdef FROM pg_indexes WHERE schemaname = :schema AND indexname = :idx"),
|
||||
{"schema": pg_schema, "idx": idx_name},
|
||||
).fetchone()
|
||||
|
||||
if idx_info is None:
|
||||
# Index doesn't exist — create it with the correct type
|
||||
bind.execute(
|
||||
text(
|
||||
f"CREATE INDEX IF NOT EXISTS {idx_name} "
|
||||
f"ON {table_ref} {using_clause} "
|
||||
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
indexdef = idx_info[0].lower()
|
||||
if target in indexdef:
|
||||
# Already the correct type
|
||||
continue
|
||||
|
||||
# Wrong type — drop and recreate
|
||||
bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
|
||||
bind.execute(
|
||||
text(
|
||||
f"CREATE INDEX IF NOT EXISTS {idx_name} "
|
||||
f"ON {table_ref} {using_clause} "
|
||||
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
# Downgrade recreates indexes as HNSW (the original hardcoded behavior)
|
||||
target = _target_index_type()
|
||||
if target is None:
|
||||
return
|
||||
|
||||
bind = op.get_bind()
|
||||
schema_name = context.config.get_main_option("target_schema")
|
||||
schema = _get_schema_prefix()
|
||||
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
|
||||
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
|
||||
|
||||
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
|
||||
for row in rows:
|
||||
bank_id = row[0]
|
||||
internal_id = str(row[1]).replace("-", "")[:16]
|
||||
escaped_bank_id = bank_id.replace("'", "''")
|
||||
for ft, ft_short in _FACT_TYPES.items():
|
||||
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
|
||||
bind.execute(text(f"DROP INDEX IF EXISTS {schema}{idx_name}"))
|
||||
bind.execute(
|
||||
text(
|
||||
f"CREATE INDEX IF NOT EXISTS {idx_name} "
|
||||
f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
|
||||
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "aa2b3c4d5e6f"
|
||||
down_revision: str | Sequence[str] | None = "z1u2v3w4x5y6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -26,21 +24,13 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date DROP NOT NULL")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Backfill NULLs with now() before restoring the NOT NULL constraint
|
||||
op.execute(f"UPDATE {schema}memory_units SET event_date = now() WHERE event_date IS NULL")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units ALTER COLUMN event_date SET NOT NULL")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-42
@@ -1,42 +0,0 @@
|
||||
"""add content_hash to chunks table for delta retain
|
||||
|
||||
Revision ID: b3c4d5e6f7a8
|
||||
Revises: a3b4c5d6e7f8
|
||||
Create Date: 2026-03-25
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "b3c4d5e6f7a8"
|
||||
down_revision: str | Sequence[str] | None = "a3b4c5d6e7f8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Add content_hash column to chunks table for delta comparison
|
||||
op.execute(f"ALTER TABLE {schema}chunks ADD COLUMN IF NOT EXISTS content_hash TEXT")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}chunks DROP COLUMN IF EXISTS content_hash")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -22,8 +22,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "b3c4d5e6f7g8"
|
||||
down_revision: str | Sequence[str] | None = "c1a2b3d4e5f6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -35,7 +33,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Partial index on occurred_start (covers "occurred_start BETWEEN $4 AND $5")
|
||||
op.execute("COMMIT")
|
||||
@@ -60,7 +58,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_mentioned_at")
|
||||
@@ -68,11 +66,3 @@ def _pg_downgrade() -> None:
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_end")
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_bank_occurred_start")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-54
@@ -1,54 +0,0 @@
|
||||
"""Add structured_content JSONB column to mental_models
|
||||
|
||||
Revision ID: b3w4x5y6z7a8
|
||||
Revises: a2v3w4x5y6z7
|
||||
Create Date: 2026-04-16
|
||||
|
||||
Stores the structured representation of a mental model document (sections,
|
||||
blocks). The plain ``content`` column remains the rendered markdown shown to
|
||||
users. ``structured_content`` is the source of truth for delta-mode refreshes:
|
||||
each refresh applies a list of typed operations to the structured doc, then
|
||||
re-renders to markdown — so unchanged sections come through byte-identical
|
||||
without an LLM round-trip.
|
||||
|
||||
Nullable: existing markdown-only mental models continue to work in full mode;
|
||||
the column is populated lazily the first time a model is refreshed in delta
|
||||
mode.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "b3w4x5y6z7a8"
|
||||
down_revision: str | Sequence[str] | None = "a2v3w4x5y6z7"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}mental_models
|
||||
ADD COLUMN IF NOT EXISTS structured_content JSONB
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS structured_content")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -14,8 +14,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "b4c5d6e7f8a9"
|
||||
down_revision: str | Sequence[str] | None = "a2b3c4d5e6f7"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -27,18 +25,10 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS observation_scopes JSONB")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
pass # intentionally no-op — safe to leave the column in place
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
pass # intentionally no-op — safe to leave the column in place
|
||||
|
||||
-91
@@ -1,91 +0,0 @@
|
||||
"""Backfill entity_cooccurrences.last_cooccurred from memory_units event time
|
||||
|
||||
Revision ID: b5d4e3f2a1c9
|
||||
Revises: o1a2b3c4d5e6
|
||||
Create Date: 2026-04-24
|
||||
|
||||
The writer path in `entity_resolver.link_units_to_entities_batch` historically
|
||||
stamped `entity_cooccurrences.last_cooccurred` with `datetime.now(UTC)` at
|
||||
flush time, ignoring the source memory unit's event date. For normal online
|
||||
retains that's fine (now ≈ event time), but for any corpus that was
|
||||
backfilled in a single session — migrating from another memory system, for
|
||||
example — every co-occurrence collapsed to the import moment, which hid the
|
||||
underlying knowledge timeline from the dashboard's entity graph recency heat
|
||||
and from any downstream consumer of the column.
|
||||
|
||||
The writer is fixed in the same change set to propagate the unit's event_date;
|
||||
this migration repairs historical rows by reading the true event time off
|
||||
`unit_entities × memory_units` (falling back to `created_at` when
|
||||
`mentioned_at` / `occurred_start` are NULL, so rows never regress).
|
||||
|
||||
Oracle slot is intentionally absent: the Oracle baseline (`o1a2b3c4d5e6`)
|
||||
landed days before this fix, so any Oracle deployment runs the corrected
|
||||
writer against an effectively empty `entity_cooccurrences` — there is no
|
||||
historical residue on Oracle to repair. PG-only matches the asymmetry of
|
||||
the data, not negligence.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "b5d4e3f2a1c9"
|
||||
down_revision: str | Sequence[str] | None = "o1a2b3c4d5e6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Recompute last_cooccurred from the true event time per entity pair.
|
||||
# COALESCE picks the first non-null of mentioned_at / occurred_start /
|
||||
# created_at so banks without event-time metadata still see a sane value
|
||||
# (equivalent to the pre-fix behaviour) instead of NULL.
|
||||
#
|
||||
# The self-join on `unit_entities` is O(k²) per memory_unit in the number
|
||||
# of distinct entities mentioned (k). For typical units k is small (single
|
||||
# digits), but a bank with units containing hundreds of entities and tens
|
||||
# of millions of co-occurrence rows may want to run this off-hours — the
|
||||
# whole UPDATE is one statement, so it locks every targeted ec row for
|
||||
# the duration. The migration is one-time; subsequent online writes
|
||||
# already carry event time via the writer fix.
|
||||
op.execute(
|
||||
f"""
|
||||
UPDATE {schema}entity_cooccurrences ec
|
||||
SET last_cooccurred = sub.event_time
|
||||
FROM (
|
||||
SELECT
|
||||
LEAST(ue1.entity_id, ue2.entity_id) AS e1,
|
||||
GREATEST(ue1.entity_id, ue2.entity_id) AS e2,
|
||||
MAX(COALESCE(mu.mentioned_at, mu.occurred_start, mu.created_at)) AS event_time
|
||||
FROM {schema}memory_units mu
|
||||
JOIN {schema}unit_entities ue1 ON ue1.unit_id = mu.id
|
||||
JOIN {schema}unit_entities ue2 ON ue2.unit_id = mu.id AND ue1.entity_id <> ue2.entity_id
|
||||
GROUP BY 1, 2
|
||||
) sub
|
||||
WHERE ec.entity_id_1 = sub.e1 AND ec.entity_id_2 = sub.e2
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
# No-op: the previous column value was `now()` at the time of write and
|
||||
# isn't recoverable. Rolling back the code is sufficient — new writes will
|
||||
# revert to the old behaviour for subsequent retains.
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade) # oracle slot intentionally absent — see header
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
@@ -12,8 +12,6 @@ import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "b7c4d8e9f1a2"
|
||||
down_revision: str | Sequence[str] | None = "5a366d414dce"
|
||||
@@ -21,7 +19,7 @@ branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add chunks table and link memory_units to chunks."""
|
||||
|
||||
# Create chunks table with single text PK (bank_id_document_id_chunk_index)
|
||||
@@ -58,7 +56,7 @@ def _pg_upgrade() -> None:
|
||||
op.create_index("idx_memory_units_chunk_id", "memory_units", ["chunk_id"])
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove chunks table and chunk_id from memory_units."""
|
||||
|
||||
# Drop index and foreign key from memory_units
|
||||
@@ -70,11 +68,3 @@ def _pg_downgrade() -> None:
|
||||
op.drop_index("idx_chunks_bank_id", table_name="chunks")
|
||||
op.drop_index("idx_chunks_document_id", table_name="chunks")
|
||||
op.drop_table("chunks")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+4
-27
@@ -11,11 +11,8 @@ block; see migrations.py for how this is handled safely.
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "c1a2b3d4e5f6"
|
||||
down_revision: str | Sequence[str] | None = "b4c5d6e7f8a9"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -27,22 +24,10 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
# pg_trgm ships with most PostgreSQL installations as a contrib module.
|
||||
def upgrade() -> None:
|
||||
# pg_trgm ships with every standard PostgreSQL installation as a contrib module.
|
||||
# It enables fast similarity lookups via GIN indexes, used for entity name matching.
|
||||
# On managed services (e.g. Azure Flexible Server), the extension may not be
|
||||
# available or may require manual enablement. We gracefully skip the index
|
||||
# creation if the extension cannot be loaded — the entity resolver will
|
||||
# auto-detect and fall back to the "full" lookup strategy at runtime. See #626.
|
||||
conn = op.get_bind()
|
||||
try:
|
||||
conn.execute(sa.text("CREATE EXTENSION IF NOT EXISTS pg_trgm"))
|
||||
except Exception:
|
||||
# Extension not available (managed Postgres, insufficient privileges, etc.)
|
||||
# Roll back the failed statement and skip index creation.
|
||||
conn.execute(sa.text("ROLLBACK"))
|
||||
conn.execute(sa.text("BEGIN"))
|
||||
return
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS pg_trgm")
|
||||
|
||||
schema = _get_schema_prefix()
|
||||
# GIN index on canonical_name enables sub-millisecond trigram similarity queries
|
||||
@@ -54,16 +39,8 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}entities_canonical_name_trgm_idx")
|
||||
# Note: not dropping pg_trgm extension as other indexes may depend on it
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
"""Add audit_log table for feature usage tracking.
|
||||
|
||||
Merge migration that combines the two existing heads (a3b4c5d6e7f8 + c8e5f2a3b4d1).
|
||||
|
||||
Stores raw request/response as JSONB for expandability without future migrations.
|
||||
The metadata JSONB column allows adding arbitrary fields in the future.
|
||||
|
||||
Revision ID: c2d3e4f5g6h7
|
||||
Revises: a3b4c5d6e7f8, c8e5f2a3b4d1
|
||||
Create Date: 2026-03-26
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "c2d3e4f5g6h7"
|
||||
down_revision: str | Sequence[str] | None = ("a3b4c5d6e7f8", "c8e5f2a3b4d1")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(
|
||||
f"""
|
||||
CREATE TABLE IF NOT EXISTS {schema}audit_log (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
action TEXT NOT NULL,
|
||||
transport TEXT NOT NULL,
|
||||
bank_id TEXT,
|
||||
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
ended_at TIMESTAMPTZ,
|
||||
request JSONB,
|
||||
response JSONB,
|
||||
metadata JSONB DEFAULT '{{}}'::jsonb
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
op.execute(
|
||||
f"CREATE INDEX IF NOT EXISTS idx_audit_log_action_started ON {schema}audit_log (action, started_at DESC)"
|
||||
)
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_bank_started ON {schema}audit_log (bank_id, started_at DESC)")
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_audit_log_started ON {schema}audit_log (started_at DESC)")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_started")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_bank_started")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_audit_log_action_started")
|
||||
op.execute(f"DROP TABLE IF EXISTS {schema}audit_log")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -9,8 +9,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "c3d4e5f6g7h8"
|
||||
down_revision: str | Sequence[str] | None = ("a2b3c4d5e6f7", "a2b3c4d5e6f8")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -22,19 +20,11 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS history JSONB DEFAULT '[]'::jsonb")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS history")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-76
@@ -1,76 +0,0 @@
|
||||
"""backsweep_orphan_observations_v2
|
||||
|
||||
Re-run of Pass 2 from migration ``g7h8i9j0k1l2_backsweep_orphan_observations``
|
||||
to sweep observations that became orphaned between then and now.
|
||||
|
||||
Why we need it again:
|
||||
``fact_storage.handle_document_tracking`` (the retain/upsert path) deleted
|
||||
the existing document via the FK cascade — which removes the source
|
||||
``memory_units`` — but never invalidated the observations derived from
|
||||
them. Only the explicit ``MemoryEngine.delete_document`` API called
|
||||
``_delete_stale_observations_for_memories``. Every document re-ingest
|
||||
therefore left orphan observations whose ``source_memory_ids`` arrays
|
||||
pointed at IDs that no longer existed in ``memory_units``.
|
||||
|
||||
``handle_document_tracking`` now calls the same cleanup helper before the
|
||||
cascade, so no new orphans will accumulate going forward. This migration
|
||||
cleans up the historical residue.
|
||||
|
||||
Identical to Pass 2 of g7h8i9j0k1l2. Pass 1 (memory_units whose bank is
|
||||
gone) is intentionally not re-run; that scenario has no fresh source.
|
||||
|
||||
Revision ID: c4x5y6z7a8b9
|
||||
Revises: b3w4x5y6z7a8
|
||||
Create Date: 2026-04-16
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "c4x5y6z7a8b9"
|
||||
down_revision: str | Sequence[str] | None = "b3w4x5y6z7a8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
mu = f"{schema}memory_units"
|
||||
|
||||
# Delete observations whose every source_memory_id refers to a now-deleted
|
||||
# memory_unit (or the array is empty). Observations with at least one
|
||||
# surviving source are left alone — the consolidation engine will refresh
|
||||
# their text on the next pass.
|
||||
op.execute(
|
||||
f"""
|
||||
DELETE FROM {mu} orphan
|
||||
WHERE orphan.fact_type = 'observation'
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM {mu} src
|
||||
WHERE src.id = ANY(orphan.source_memory_ids)
|
||||
AND src.bank_id = orphan.bank_id
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
# Deleted rows cannot be restored.
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
-58
@@ -1,58 +0,0 @@
|
||||
"""Add bank_id column to memory_links for direct filtering
|
||||
|
||||
The stats endpoint JOINs memory_links to memory_units just to filter by
|
||||
bank_id. With millions of links this takes 18+ seconds. Adding bank_id
|
||||
directly to memory_links lets Postgres push the filter down before the JOIN.
|
||||
|
||||
Revision ID: c5d6e7f8a9b0
|
||||
Revises: b3c4d5e6f7a8
|
||||
Create Date: 2026-03-26
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "c5d6e7f8a9b0"
|
||||
down_revision: str | Sequence[str] | None = "b3c4d5e6f7a8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# 1. Add nullable column
|
||||
op.execute(f"ALTER TABLE {schema}memory_links ADD COLUMN IF NOT EXISTS bank_id TEXT")
|
||||
|
||||
# 2. Backfill from memory_units
|
||||
op.execute(f"""
|
||||
UPDATE {schema}memory_links ml
|
||||
SET bank_id = mu.bank_id
|
||||
FROM {schema}memory_units mu
|
||||
WHERE ml.from_unit_id = mu.id
|
||||
AND ml.bank_id IS NULL
|
||||
""")
|
||||
|
||||
# 3. Set NOT NULL
|
||||
op.execute(f"ALTER TABLE {schema}memory_links ALTER COLUMN bank_id SET NOT NULL")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}memory_links DROP COLUMN IF EXISTS bank_id")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -12,8 +12,6 @@ import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "c8e5f2a3b4d1"
|
||||
down_revision: str | Sequence[str] | None = "b7c4d8e9f1a2"
|
||||
@@ -21,7 +19,7 @@ branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add retain_params JSONB column to documents table."""
|
||||
|
||||
# Add retain_params column to store parameters passed during retain
|
||||
@@ -31,7 +29,7 @@ def _pg_upgrade() -> None:
|
||||
op.create_index("idx_documents_retain_params", "documents", ["retain_params"], postgresql_using="gin")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove retain_params column from documents table."""
|
||||
|
||||
# Drop index
|
||||
@@ -39,11 +37,3 @@ def _pg_downgrade() -> None:
|
||||
|
||||
# Drop column
|
||||
op.drop_column("documents", "retain_params")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -34,8 +34,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "d2e3f4a5b6c7"
|
||||
down_revision: str | Sequence[str] | None = "b3c4d5e6f7g8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -47,7 +45,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# CREATE INDEX CONCURRENTLY cannot run inside a transaction block.
|
||||
@@ -77,17 +75,9 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_entity_covering")
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_links_to_type_weight")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -18,8 +18,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "d4e5f6g7h8i9"
|
||||
down_revision: str | Sequence[str] | None = "d5e6f7a8b9c0"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# DROP + CREATE CONCURRENTLY must run outside a transaction block.
|
||||
op.execute("COMMIT")
|
||||
@@ -44,7 +42,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute("COMMIT")
|
||||
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {schema}idx_memory_units_source_memory_ids")
|
||||
@@ -53,11 +51,3 @@ def _pg_downgrade() -> None:
|
||||
f"ON {schema}memory_units USING GIN (source_memory_ids) "
|
||||
f"WHERE source_memory_ids IS NOT NULL"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+21
-39
@@ -1,4 +1,4 @@
|
||||
"""Add internal_id to banks and per-(bank, fact_type) partial vector indexes
|
||||
"""Add internal_id to banks and per-(bank, fact_type) partial HNSW indexes
|
||||
|
||||
Revision ID: d5e6f7a8b9c0
|
||||
Revises: a3b4c5d6e7f8
|
||||
@@ -6,33 +6,36 @@ Create Date: 2026-03-11
|
||||
|
||||
This migration:
|
||||
1. Adds internal_id UUID column to banks (stable identifier for index naming)
|
||||
2. Drops the global vector index (competes with per-bank partial indexes)
|
||||
3. Creates per-(bank_id, fact_type) partial vector indexes for all existing banks
|
||||
using the configured vector extension (HNSW for pgvector, DiskANN for
|
||||
pgvectorscale, vchordrq for vchord).
|
||||
(new banks get indexes created at bank-creation time via bank_utils.create_bank_vector_indexes)
|
||||
2. Drops the global HNSW index (competes with per-bank partial indexes)
|
||||
3. Creates per-(bank_id, fact_type) partial HNSW indexes for all existing banks
|
||||
(new banks get indexes created at bank-creation time via bank_utils.create_bank_hnsw_indexes)
|
||||
|
||||
Why per-(bank, fact_type) indexes:
|
||||
- fact_type-only partial indexes are never chosen by the planner when bank_id is in the WHERE
|
||||
clause, because the idx_memory_units_bank_id B-tree index always wins at planning time.
|
||||
- Per-(bank, fact_type) partial indexes have both predicates matching → planner selects them.
|
||||
- The global vector index competes for larger partitions (world, observation) and must be dropped.
|
||||
- The global HNSW index competes for larger partitions (world, observation) and must be dropped.
|
||||
|
||||
For large deployments, create indexes CONCURRENTLY before running this migration:
|
||||
SELECT internal_id, bank_id FROM banks;
|
||||
-- for each bank and each fact_type in (world, experience, observation):
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_mu_emb_{ft}_{uid16}
|
||||
ON memory_units USING hnsw (embedding vector_cosine_ops)
|
||||
WHERE fact_type = '{ft}' AND bank_id = '{bank_id}';
|
||||
DROP INDEX CONCURRENTLY IF EXISTS idx_memory_units_embedding;
|
||||
"""
|
||||
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
from sqlalchemy import text
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "d5e6f7a8b9c0"
|
||||
down_revision: str | Sequence[str] | None = "c3d4e5f6g7h8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
_FACT_TYPES: dict[str, str] = {
|
||||
_HNSW_FACT_TYPES: dict[str, str] = {
|
||||
"world": "worl",
|
||||
"experience": "expr",
|
||||
"observation": "obsv",
|
||||
@@ -44,18 +47,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _vector_index_using_clause() -> str:
|
||||
"""Return the USING clause based on the configured vector extension."""
|
||||
ext = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
|
||||
if ext == "pgvectorscale":
|
||||
return "USING diskann (embedding vector_cosine_ops) WITH (num_neighbors = 50)"
|
||||
elif ext == "vchord":
|
||||
return "USING vchordrq (embedding vector_l2_ops)"
|
||||
else:
|
||||
return "USING hnsw (embedding vector_cosine_ops)"
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# 1. Add internal_id column to banks
|
||||
@@ -64,41 +56,39 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
op.execute(f"ALTER TABLE {schema}banks ADD CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)")
|
||||
|
||||
# 2. Drop any fact_type-only partial indexes that may exist from prior migrations
|
||||
# 2. Drop any fact_type-only partial HNSW indexes that may exist from prior migrations
|
||||
# (bank_id B-tree always wins over them when bank_id is in the WHERE clause)
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_world")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_observation")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_mu_emb_experience")
|
||||
|
||||
# 4. Drop global vector index (competes with per-bank partial indexes)
|
||||
# 4. Drop global HNSW index (competes with per-bank partial indexes)
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_embedding")
|
||||
|
||||
# 5. Create per-(bank, fact_type) partial vector indexes for all existing banks
|
||||
# using the configured extension (HNSW / DiskANN / vchordrq)
|
||||
# 5. Create per-(bank, fact_type) partial HNSW indexes for all existing banks
|
||||
bind = op.get_bind()
|
||||
schema_name = context.config.get_main_option("target_schema")
|
||||
table_ref = f'"{schema_name}".memory_units' if schema_name else "memory_units"
|
||||
banks_ref = f'"{schema_name}".banks' if schema_name else "banks"
|
||||
using_clause = _vector_index_using_clause()
|
||||
|
||||
rows = bind.execute(text(f"SELECT bank_id, internal_id FROM {banks_ref}")).fetchall() # noqa: S608
|
||||
for row in rows:
|
||||
bank_id = row[0]
|
||||
internal_id = str(row[1]).replace("-", "")[:16]
|
||||
escaped_bank_id = bank_id.replace("'", "''")
|
||||
for ft, ft_short in _FACT_TYPES.items():
|
||||
for ft, ft_short in _HNSW_FACT_TYPES.items():
|
||||
idx_name = f"idx_mu_emb_{ft_short}_{internal_id}"
|
||||
# Index name is schema-unqualified (indexes live in the schema of their table)
|
||||
bind.execute(
|
||||
text(
|
||||
f"CREATE INDEX IF NOT EXISTS {idx_name} "
|
||||
f"ON {table_ref} {using_clause} "
|
||||
f"ON {table_ref} USING hnsw (embedding vector_cosine_ops) "
|
||||
f"WHERE fact_type = '{ft}' AND bank_id = '{escaped_bank_id}'"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Drop per-bank HNSW indexes (iterate existing banks)
|
||||
@@ -139,11 +129,3 @@ def _pg_downgrade() -> None:
|
||||
# Drop internal_id column
|
||||
op.execute(f"ALTER TABLE {schema}banks DROP CONSTRAINT IF EXISTS banks_internal_id_unique")
|
||||
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS internal_id")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-67
@@ -1,67 +0,0 @@
|
||||
"""Backfill mental_models.subtype for databases that ran h3c4d5e6f7g8 before the fix
|
||||
|
||||
Migration h3c4d5e6f7g8 used CREATE TABLE IF NOT EXISTS to create the
|
||||
mental_models table with a subtype column. But on databases where the table
|
||||
already existed (from the reflections -> mental_models rename chain), the
|
||||
CREATE was a no-op and subtype was never added. A fix was later added to
|
||||
h3c4d5e6f7g8 (Step 4b), but databases that had already run the migration
|
||||
never re-execute it. This migration adds the missing columns idempotently.
|
||||
|
||||
Revision ID: d5y6z7a8b9c0
|
||||
Revises: 8c6fa6f7230b
|
||||
Create Date: 2026-04-18
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "d5y6z7a8b9c0"
|
||||
down_revision: str | Sequence[str] | None = "8c6fa6f7230b"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Add columns that h3c4d5e6f7g8 intended to create but missed when
|
||||
# the table already existed from the reflections rename chain.
|
||||
for col_ddl in [
|
||||
"subtype VARCHAR(32) NOT NULL DEFAULT 'structural'",
|
||||
"description TEXT NOT NULL DEFAULT ''",
|
||||
"entity_id UUID",
|
||||
"observations JSONB DEFAULT '{\"observations\": []}'::jsonb",
|
||||
"links VARCHAR[]",
|
||||
"last_updated TIMESTAMP WITH TIME ZONE",
|
||||
]:
|
||||
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS {col_ddl}")
|
||||
|
||||
# Ensure the CHECK constraint exists
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}mental_models
|
||||
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
|
||||
""")
|
||||
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
# No-op: these columns are part of the intended schema
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
-49
@@ -1,49 +0,0 @@
|
||||
"""Drop unused metadata column from documents table
|
||||
|
||||
Revision ID: d6e7f8a9b0c1
|
||||
Revises: c2d3e4f5g6h7, c5d6e7f8a9b0
|
||||
Create Date: 2026-03-30
|
||||
|
||||
The metadata column on documents was always stored as an empty dict {}.
|
||||
Actual document metadata is stored inside retain_params.metadata.
|
||||
|
||||
This migration was originally shipped in v0.4.22, then its file was deleted
|
||||
in v0.5.0 (and its revision ID accidentally reused by 2eee35aa3cfc).
|
||||
Restoring the file so that databases stamped at this revision can upgrade
|
||||
cleanly to v0.5.x+.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "d6e7f8a9b0c1"
|
||||
down_revision: str | Sequence[str] | None = ("c2d3e4f5g6h7", "c5d6e7f8a9b0")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS metadata")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS metadata jsonb DEFAULT '{{}}'")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -8,8 +8,6 @@ Create Date: 2024-12-04 15:00:00.000000
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "d9f6a3b4c5e2"
|
||||
down_revision = "c8e5f2a3b4d1"
|
||||
@@ -23,7 +21,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade():
|
||||
def upgrade():
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Drop old check constraint FIRST (before updating data)
|
||||
@@ -40,7 +38,7 @@ def _pg_upgrade():
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade():
|
||||
def downgrade():
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Drop new check constraint FIRST
|
||||
@@ -53,11 +51,3 @@ def _pg_downgrade():
|
||||
op.create_check_constraint(
|
||||
"memory_units_fact_type_check", "memory_units", "fact_type IN ('world', 'bank', 'opinion', 'observation')"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -14,8 +14,6 @@ from collections.abc import Sequence
|
||||
import sqlalchemy as sa
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "e0a1b2c3d4e5"
|
||||
down_revision: str | Sequence[str] | None = "rename_personality"
|
||||
@@ -35,7 +33,7 @@ def _get_target_schema() -> str:
|
||||
return schema if schema else "public"
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Convert Big Five disposition to 3-trait disposition."""
|
||||
conn = op.get_bind()
|
||||
schema = _get_schema_prefix()
|
||||
@@ -77,7 +75,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Convert back to Big Five disposition."""
|
||||
conn = op.get_bind()
|
||||
schema = _get_schema_prefix()
|
||||
@@ -111,11 +109,3 @@ def _pg_downgrade() -> None:
|
||||
ALTER COLUMN disposition SET DEFAULT '{{"openness": 0.5, "conscientiousness": 0.5, "extraversion": 0.5, "agreeableness": 0.5, "neuroticism": 0.5, "bias_strength": 0.5}}'::jsonb
|
||||
""")
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "e4f5a6b7c8d9"
|
||||
down_revision: str | Sequence[str] | None = "d2e3f4a5b6c7"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -25,7 +23,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(
|
||||
@@ -56,17 +54,9 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_status_retry")
|
||||
op.execute(f"ALTER TABLE {schema}async_operations DROP COLUMN IF EXISTS next_retry_at")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_webhooks_bank_id")
|
||||
op.execute(f"DROP TABLE IF EXISTS {schema}webhooks")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -13,8 +13,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "e5f6g7h8i9j0"
|
||||
down_revision: str | Sequence[str] | None = "d4e5f6g7h8i9"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -26,7 +24,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Remove orphaned async_operations rows whose bank no longer exists
|
||||
@@ -69,15 +67,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS fk_async_operations_bank_id")
|
||||
op.execute(f"ALTER TABLE {schema}webhooks DROP CONSTRAINT IF EXISTS fk_webhooks_bank_id")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+4
-14
@@ -5,15 +5,13 @@ Revises: e0a1b2c3d4e5
|
||||
Create Date: 2025-01-12
|
||||
|
||||
Add composite index on memory_links (from_unit_id, link_type, weight DESC)
|
||||
to optimize graph traversal queries that need top-k edges per type.
|
||||
to optimize MPFP graph traversal queries that need top-k edges per type.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "f1a2b3c4d5e6"
|
||||
down_revision: str | Sequence[str] | None = "e0a1b2c3d4e5"
|
||||
@@ -27,8 +25,8 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
"""Add composite index for efficient graph retrieval edge loading."""
|
||||
def upgrade() -> None:
|
||||
"""Add composite index for efficient MPFP edge loading."""
|
||||
schema = _get_schema_prefix()
|
||||
# Create composite index for efficient top-k per (from_node, link_type) queries
|
||||
# This enables LATERAL joins to use index-only scans with early termination
|
||||
@@ -40,15 +38,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove the composite index."""
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_links_from_type_weight")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+5
-34
@@ -10,8 +10,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "f6g7h8i9j0k1"
|
||||
down_revision: str | Sequence[str] | None = "e5f6g7h8i9j0"
|
||||
@@ -19,49 +17,22 @@ branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Change memory_units.chunk_id FK from SET NULL to CASCADE.
|
||||
|
||||
When a document is deleted the CASCADE reaches chunks first; with SET NULL
|
||||
the memory_units rows survived with chunk_id = NULL, leaving ghost records.
|
||||
Switching to CASCADE ensures they are removed together with their chunk.
|
||||
"""
|
||||
from alembic import context
|
||||
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
schema_prefix = f'"{schema}".' if schema else ""
|
||||
# Use raw SQL with IF EXISTS so this is safe on schemas where the FK was
|
||||
# already dropped or never existed under this name.
|
||||
op.execute(f"ALTER TABLE {schema_prefix}memory_units DROP CONSTRAINT IF EXISTS memory_units_chunk_fkey")
|
||||
# Use a DO block so the ADD is also idempotent: if the FK already exists (e.g.
|
||||
# the schema was provisioned after the base migration already added it) the
|
||||
# duplicate_object exception is swallowed rather than failing the migration.
|
||||
op.execute(
|
||||
f"""
|
||||
DO $$ BEGIN
|
||||
ALTER TABLE {schema_prefix}memory_units
|
||||
ADD CONSTRAINT memory_units_chunk_fkey
|
||||
FOREIGN KEY (chunk_id)
|
||||
REFERENCES {schema_prefix}chunks (chunk_id)
|
||||
ON DELETE CASCADE;
|
||||
EXCEPTION
|
||||
WHEN duplicate_object THEN NULL;
|
||||
END $$;
|
||||
"""
|
||||
op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
|
||||
op.create_foreign_key(
|
||||
"memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="CASCADE"
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Revert to SET NULL behaviour."""
|
||||
op.drop_constraint("memory_units_chunk_fkey", "memory_units", type_="foreignkey")
|
||||
op.create_foreign_key(
|
||||
"memory_units_chunk_fkey", "memory_units", "chunks", ["chunk_id"], ["chunk_id"], ondelete="SET NULL"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "f7g8h9i0j1k2"
|
||||
down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -25,19 +23,11 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}webhooks ADD COLUMN IF NOT EXISTS http_config JSONB NOT NULL DEFAULT '{{}}'")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}webhooks DROP COLUMN IF EXISTS http_config")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "g2a3b4c5d6e7"
|
||||
down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add tags column to memory_units and documents tables."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -41,18 +39,10 @@ def _pg_upgrade() -> None:
|
||||
op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS tags VARCHAR[] NOT NULL DEFAULT '{{}}'")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove tags columns and index."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_tags")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS tags")
|
||||
op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS tags")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-93
@@ -1,93 +0,0 @@
|
||||
"""remove_opinion_fact_type
|
||||
|
||||
Revision ID: g2h3i4j5k6l7
|
||||
Revises: f1a2b3c4d5e6
|
||||
Create Date: 2026-04-02
|
||||
|
||||
Remove the deprecated 'opinion' fact type: drop opinion-specific indexes,
|
||||
update CHECK constraints, delete any remaining opinion rows, and drop the
|
||||
confidence_score column (was only used for opinions, always NULL otherwise).
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "g2h3i4j5k6l7"
|
||||
down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# 1. Delete any remaining opinion rows
|
||||
op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
|
||||
|
||||
# 2. Drop opinion-specific indexes
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_confidence")
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_opinion_date")
|
||||
|
||||
# 3. Drop confidence_score constraints and column (only used for opinions, always NULL otherwise)
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS confidence_score_fact_type_check")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_confidence_score_check")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS confidence_score")
|
||||
|
||||
# 4. Replace fact_type CHECK constraint
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
|
||||
f"CHECK (fact_type IN ('world', 'experience', 'observation'))"
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Restore confidence_score column
|
||||
op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS confidence_score float")
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_confidence_score_check "
|
||||
f"CHECK (confidence_score IS NULL OR (confidence_score >= 0.0 AND confidence_score <= 1.0))"
|
||||
)
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT confidence_score_fact_type_check "
|
||||
f"CHECK ((fact_type = 'opinion' AND confidence_score IS NOT NULL) OR "
|
||||
f"(fact_type = 'observation') OR "
|
||||
f"(fact_type NOT IN ('opinion', 'observation') AND confidence_score IS NULL))"
|
||||
)
|
||||
|
||||
# Restore original fact_type CHECK constraint (with opinion)
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP CONSTRAINT IF EXISTS memory_units_fact_type_check")
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}memory_units ADD CONSTRAINT memory_units_fact_type_check "
|
||||
f"CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))"
|
||||
)
|
||||
|
||||
# Recreate opinion indexes
|
||||
op.execute(
|
||||
f"CREATE INDEX idx_memory_units_opinion_confidence ON {schema}memory_units "
|
||||
f"(bank_id, confidence_score DESC) WHERE fact_type = 'opinion'"
|
||||
)
|
||||
op.execute(
|
||||
f"CREATE INDEX idx_memory_units_opinion_date ON {schema}memory_units "
|
||||
f"(bank_id, event_date DESC) WHERE fact_type = 'opinion'"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -22,8 +22,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "g7h8i9j0k1l2"
|
||||
down_revision: str | Sequence[str] | None = "f6g7h8i9j0k1"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -35,7 +33,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
mu = f"{schema}memory_units"
|
||||
banks = f"{schema}banks"
|
||||
@@ -68,14 +66,6 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
# Deleted rows cannot be restored.
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
@@ -17,8 +17,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "h3c4d5e6f7g8"
|
||||
down_revision: str | Sequence[str] | None = "g2a3b4c5d6e7"
|
||||
@@ -32,7 +30,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Apply mental models v4 changes."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -87,26 +85,6 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
""")
|
||||
|
||||
# Step 4b: If the table already existed (from reflections rename chain),
|
||||
# it won't have the v4 columns. Add them idempotently so the migration
|
||||
# works regardless of whether CREATE TABLE above was a no-op.
|
||||
for col_ddl in [
|
||||
"subtype VARCHAR(32) NOT NULL DEFAULT 'directive'",
|
||||
"description TEXT NOT NULL DEFAULT ''",
|
||||
"entity_id UUID",
|
||||
"observations JSONB DEFAULT '{\"observations\": []}'::jsonb",
|
||||
"links VARCHAR[]",
|
||||
"last_updated TIMESTAMP WITH TIME ZONE",
|
||||
]:
|
||||
op.execute(f"ALTER TABLE {schema}mental_models ADD COLUMN IF NOT EXISTS {col_ddl}")
|
||||
|
||||
# Ensure the subtype CHECK constraint exists (may not if table was renamed)
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}mental_models
|
||||
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
|
||||
""")
|
||||
|
||||
# Step 5: Create indexes for efficient queries (if not exist)
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_bank_id ON {schema}mental_models(bank_id)")
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
|
||||
@@ -115,7 +93,7 @@ def _pg_upgrade() -> None:
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_tags ON {schema}mental_models USING GIN(tags)")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Revert mental models v4 changes."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -132,11 +110,3 @@ def _pg_downgrade() -> None:
|
||||
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission")
|
||||
|
||||
# Note: Cannot restore deleted observations - they are lost on downgrade
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-52
@@ -1,52 +0,0 @@
|
||||
"""Merge 3 migration heads and add unit_entities composite index
|
||||
|
||||
Revision ID: h3i4j5k6l7m8
|
||||
Revises: a4b5c6d7e8f9, g2h3i4j5k6l7
|
||||
Create Date: 2026-04-07
|
||||
|
||||
Merges three unmerged migration heads into one, and adds a composite index
|
||||
(entity_id, unit_id) on unit_entities for index-only scans in the LATERAL
|
||||
entity expansion query.
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "h3i4j5k6l7m8"
|
||||
down_revision: str | Sequence[str] | None = ("a4b5c6d7e8f9", "g2h3i4j5k6l7")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
# Composite index enables index-only scans for entity_id -> unit_id lookups
|
||||
op.execute(
|
||||
f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity_unit ON {schema}unit_entities (entity_id, unit_id)"
|
||||
)
|
||||
# Drop the now-redundant single-column index
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_unit_entities_entity_unit")
|
||||
# Restore the single-column index
|
||||
op.execute(f"CREATE INDEX IF NOT EXISTS idx_unit_entities_entity ON {schema}unit_entities (entity_id)")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
@@ -13,8 +13,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "i4d5e6f7g8h9"
|
||||
down_revision: str | Sequence[str] | None = "h3c4d5e6f7g8"
|
||||
@@ -28,7 +26,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Delete opinion memory_units."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -37,15 +35,7 @@ def _pg_upgrade() -> None:
|
||||
op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Cannot restore deleted opinions."""
|
||||
# Note: Cannot restore deleted opinions - they are lost on downgrade
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-49
@@ -1,49 +0,0 @@
|
||||
"""Add 'cancelled' to async_operations status check constraint
|
||||
|
||||
Revision ID: i4j5k6l7m8n9
|
||||
Revises: d5y6z7a8b9c0
|
||||
Create Date: 2026-04-23
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "i4j5k6l7m8n9"
|
||||
down_revision: str | Sequence[str] | None = "d5y6z7a8b9c0"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _get_schema_prefix() -> str:
|
||||
"""Get schema prefix for table names (required for multi-tenant support)."""
|
||||
schema = context.config.get_main_option("target_schema")
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
|
||||
f"CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled'))"
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
op.execute(f"ALTER TABLE {schema}async_operations DROP CONSTRAINT IF EXISTS async_operations_status_check")
|
||||
op.execute(
|
||||
f"ALTER TABLE {schema}async_operations ADD CONSTRAINT async_operations_status_check "
|
||||
f"CHECK (status IN ('pending', 'processing', 'completed', 'failed'))"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -15,8 +15,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "j5e6f7g8h9i0"
|
||||
down_revision: str | Sequence[str] | None = "i4d5e6f7g8h9"
|
||||
@@ -30,7 +28,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Create mental_model_versions table and add version tracking."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -83,7 +81,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove mental_model_versions table and version column."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -95,11 +93,3 @@ def _pg_downgrade() -> None:
|
||||
|
||||
# Remove version column from mental_models
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS version")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "k6f7g8h9i0j1"
|
||||
down_revision: str | Sequence[str] | None = "j5e6f7g8h9i0"
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add 'directive' to mental_models subtype constraint."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -42,7 +40,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove 'directive' from mental_models subtype constraint."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -58,11 +56,3 @@ def _pg_downgrade() -> None:
|
||||
ADD CONSTRAINT ck_mental_models_subtype
|
||||
CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
|
||||
""")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
"""No-op: observation_sources table is Oracle-only
|
||||
|
||||
Originally created the observation_sources junction table for all backends,
|
||||
but PG uses native array ops on the source_memory_ids column (faster at scale).
|
||||
Oracle creates this table in the o1a2b3c4d5e6 baseline migration instead.
|
||||
|
||||
Kept as a no-op to preserve the Alembic revision chain.
|
||||
|
||||
Revision ID: k6l7m8n9o0p1
|
||||
Revises: i4j5k6l7m8n9
|
||||
Create Date: 2026-04-24
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "k6l7m8n9o0p1"
|
||||
down_revision: str | Sequence[str] | None = "i4j5k6l7m8n9"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
# PG uses source_memory_ids[] array on memory_units — no junction table.
|
||||
pass
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -17,8 +17,6 @@ import sqlalchemy as sa
|
||||
from alembic import context, op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "l7g8h9i0j1k2"
|
||||
down_revision: str | Sequence[str] | None = "k6f7g8h9i0j1"
|
||||
@@ -32,7 +30,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add worker columns to async_operations."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -80,7 +78,7 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove worker columns from async_operations."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -109,11 +107,3 @@ def _pg_downgrade() -> None:
|
||||
"worker_id",
|
||||
schema=context.config.get_main_option("target_schema") or None,
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
-31
@@ -1,31 +0,0 @@
|
||||
"""Merge divergent heads from deferrable FK and cooccurrence backfill
|
||||
|
||||
Revision ID: m3rg3h3ad5f6
|
||||
Revises: 9f8e7d6c5b4a, b5d4e3f2a1c9
|
||||
Create Date: 2026-05-04
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "m3rg3h3ad5f6"
|
||||
down_revision: tuple[str, ...] = ("9f8e7d6c5b4a", "b5d4e3f2a1c9")
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
pass
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "m8h9i0j1k2l3"
|
||||
down_revision: str | Sequence[str] | None = "l7g8h9i0j1k2"
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Change mental_models.id from VARCHAR(64) to TEXT."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -35,17 +33,9 @@ def _pg_upgrade() -> None:
|
||||
op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN id TYPE TEXT")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Revert mental_models.id from TEXT to VARCHAR(64)."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Note: This may fail if any id values exceed 64 characters
|
||||
op.execute(f"ALTER TABLE {schema}mental_models ALTER COLUMN id TYPE VARCHAR(64)")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -16,8 +16,6 @@ from collections.abc import Sequence
|
||||
from alembic import context, op
|
||||
from sqlalchemy import text
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "n9i0j1k2l3m4"
|
||||
down_revision: str | Sequence[str] | None = "m8h9i0j1k2l3"
|
||||
@@ -121,7 +119,7 @@ def _detect_text_search_extension() -> str:
|
||||
)
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Create learnings and pinned_reflections tables."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -306,7 +304,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Drop learnings and pinned_reflections tables."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -317,11 +315,3 @@ def _pg_downgrade() -> None:
|
||||
# Remove columns from banks
|
||||
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS last_consolidated_at")
|
||||
op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission_changed_at")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+4
-16
@@ -16,8 +16,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "o0j1k2l3m4n5"
|
||||
down_revision: str | Sequence[str] | None = "n9i0j1k2l3m4"
|
||||
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Migrate data and clean up old mental models."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -82,17 +80,15 @@ def _pg_upgrade() -> None:
|
||||
# 4. Drop the mental_model_versions table (no longer used)
|
||||
op.execute(f"DROP TABLE IF EXISTS {schema}mental_model_versions CASCADE")
|
||||
|
||||
# 5. Drop old constraints and add new one that allows current subtypes.
|
||||
# 'pinned' is still used by the code for user-created mental models;
|
||||
# 'directive' is used for system directives.
|
||||
# 5. Drop old constraints and add new one that only allows 'directive'
|
||||
op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}mental_models
|
||||
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('directive', 'pinned'))
|
||||
ADD CONSTRAINT ck_mental_models_subtype CHECK (subtype = 'directive')
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Reverse the migration (data migration is one-way, so this just removes constraints)."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -115,11 +111,3 @@ def _pg_downgrade() -> None:
|
||||
)
|
||||
|
||||
# Note: Data migration cannot be reversed - pinned_reflections and learnings data remains
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
@@ -1,445 +0,0 @@
|
||||
"""oracle_baseline
|
||||
|
||||
Brings a fresh Oracle 23ai database up to the current schema in a single step.
|
||||
PostgreSQL is a no-op here — the prior 59 revisions already build the PG schema
|
||||
incrementally; this revision just closes the loop so both dialects share a
|
||||
single head from this point on.
|
||||
|
||||
After this migration ships, *every* new revision must fill both the ``_pg_*``
|
||||
and ``_oracle_*`` slots (or explicitly leave one ``None``); a CI check enforces
|
||||
that.
|
||||
|
||||
Tables mirror the PostgreSQL schema but use Oracle-native types:
|
||||
- UUID -> RAW(16) DEFAULT SYS_GUID()
|
||||
- TEXT / large VARCHAR -> CLOB
|
||||
- JSONB -> CLOB with IS JSON CHECK
|
||||
- BOOLEAN -> NUMBER(1)
|
||||
- FLOAT -> BINARY_DOUBLE
|
||||
- VARCHAR[] -> CLOB (JSON array stored as string)
|
||||
- BYTEA -> BLOB
|
||||
- vector(384) -> VECTOR(384, FLOAT32) (Oracle 23ai native)
|
||||
|
||||
Revision ID: o1a2b3c4d5e6
|
||||
Revises: k6l7m8n9o0p1
|
||||
Create Date: 2026-04-29
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "o1a2b3c4d5e6"
|
||||
down_revision: str | Sequence[str] | None = "k6l7m8n9o0p1"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tables — created in dependency order
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_TABLES: tuple[str, ...] = (
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS banks (
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
internal_id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
name VARCHAR2(512),
|
||||
disposition CLOB DEFAULT '{"skepticism":3,"literalism":3,"empathy":3}' NOT NULL
|
||||
CONSTRAINT banks_disposition_json CHECK (disposition IS JSON),
|
||||
mission CLOB,
|
||||
personality CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT banks_personality_json CHECK (personality IS JSON),
|
||||
config CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT banks_config_json CHECK (config IS JSON),
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_banks PRIMARY KEY (bank_id),
|
||||
CONSTRAINT banks_internal_id_unique UNIQUE (internal_id)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS documents (
|
||||
id VARCHAR2(512) NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
original_text CLOB,
|
||||
content_hash VARCHAR2(128),
|
||||
metadata CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT docs_metadata_json CHECK (metadata IS JSON),
|
||||
retain_params CLOB CONSTRAINT docs_retain_params_json CHECK (retain_params IS JSON OR retain_params IS NULL),
|
||||
file_storage_key VARCHAR2(512),
|
||||
file_original_name VARCHAR2(512),
|
||||
file_content_type VARCHAR2(256),
|
||||
tags CLOB DEFAULT '[]' NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_documents PRIMARY KEY (id, bank_id),
|
||||
CONSTRAINT fk_documents_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
chunk_id VARCHAR2(512) NOT NULL,
|
||||
document_id VARCHAR2(512) NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
chunk_index NUMBER(10) NOT NULL,
|
||||
chunk_text CLOB NOT NULL,
|
||||
content_hash VARCHAR2(128),
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_chunks PRIMARY KEY (chunk_id),
|
||||
CONSTRAINT fk_chunks_document FOREIGN KEY (document_id, bank_id)
|
||||
REFERENCES documents(id, bank_id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
# memory_units uses automatic list partitioning on bank_id at create time —
|
||||
# no post-create ALTER required (we used to do that for legacy installs).
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS memory_units (
|
||||
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
document_id VARCHAR2(512),
|
||||
chunk_id VARCHAR2(512),
|
||||
text CLOB NOT NULL,
|
||||
embedding VECTOR(384, FLOAT32),
|
||||
context CLOB,
|
||||
event_date TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||
occurred_start TIMESTAMP WITH TIME ZONE,
|
||||
occurred_end TIMESTAMP WITH TIME ZONE,
|
||||
mentioned_at TIMESTAMP WITH TIME ZONE,
|
||||
fact_type VARCHAR2(64) DEFAULT 'world' NOT NULL,
|
||||
confidence_score BINARY_DOUBLE,
|
||||
access_count NUMBER(10) DEFAULT 0 NOT NULL,
|
||||
consolidated_at TIMESTAMP WITH TIME ZONE,
|
||||
observation_scopes CLOB CONSTRAINT mu_obs_scopes_json CHECK (observation_scopes IS JSON OR observation_scopes IS NULL),
|
||||
tags CLOB DEFAULT '[]' NOT NULL,
|
||||
metadata CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT mu_metadata_json CHECK (metadata IS JSON),
|
||||
proof_count NUMBER(10) DEFAULT 1,
|
||||
source_memory_ids CLOB,
|
||||
history CLOB DEFAULT '[]'
|
||||
CONSTRAINT mu_history_json CHECK (history IS JSON OR history IS NULL),
|
||||
text_signals CLOB,
|
||||
consolidation_failed_at TIMESTAMP WITH TIME ZONE,
|
||||
search_vector CLOB,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_memory_units PRIMARY KEY (id),
|
||||
CONSTRAINT fk_mu_document FOREIGN KEY (document_id, bank_id)
|
||||
REFERENCES documents(id, bank_id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_mu_chunk FOREIGN KEY (chunk_id)
|
||||
REFERENCES chunks(chunk_id) ON DELETE SET NULL,
|
||||
CONSTRAINT chk_mu_fact_type CHECK (fact_type IN ('world', 'experience', 'observation')),
|
||||
CONSTRAINT chk_mu_confidence CHECK (
|
||||
confidence_score IS NULL
|
||||
OR (confidence_score >= 0.0 AND confidence_score <= 1.0)
|
||||
)
|
||||
)
|
||||
PARTITION BY LIST (bank_id) AUTOMATIC
|
||||
(PARTITION p_default VALUES ('__default__'))
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS entities (
|
||||
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
canonical_name VARCHAR2(512) NOT NULL,
|
||||
metadata CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT ent_metadata_json CHECK (metadata IS JSON),
|
||||
first_seen TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
last_seen TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
mention_count NUMBER(10) DEFAULT 1 NOT NULL,
|
||||
CONSTRAINT pk_entities PRIMARY KEY (id)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS unit_entities (
|
||||
unit_id RAW(16) NOT NULL,
|
||||
entity_id RAW(16) NOT NULL,
|
||||
CONSTRAINT pk_unit_entities PRIMARY KEY (unit_id, entity_id),
|
||||
CONSTRAINT fk_ue_unit FOREIGN KEY (unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_ue_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS entity_cooccurrences (
|
||||
entity_id_1 RAW(16) NOT NULL,
|
||||
entity_id_2 RAW(16) NOT NULL,
|
||||
cooccurrence_count NUMBER(10) DEFAULT 1 NOT NULL,
|
||||
last_cooccurred TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_entity_cooccurrences PRIMARY KEY (entity_id_1, entity_id_2),
|
||||
CONSTRAINT fk_ec_entity1 FOREIGN KEY (entity_id_1) REFERENCES entities(id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_ec_entity2 FOREIGN KEY (entity_id_2) REFERENCES entities(id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS memory_links (
|
||||
from_unit_id RAW(16) NOT NULL,
|
||||
to_unit_id RAW(16) NOT NULL,
|
||||
link_type VARCHAR2(64) NOT NULL,
|
||||
entity_id RAW(16),
|
||||
bank_id VARCHAR2(256),
|
||||
weight BINARY_DOUBLE DEFAULT 1.0 NOT NULL,
|
||||
source_memory_ids CLOB,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT fk_ml_from FOREIGN KEY (from_unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_ml_to FOREIGN KEY (to_unit_id) REFERENCES memory_units(id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_ml_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE CASCADE,
|
||||
CONSTRAINT chk_ml_link_type CHECK (
|
||||
link_type IN ('temporal', 'semantic', 'entity', 'causes', 'caused_by', 'enables', 'prevents')
|
||||
),
|
||||
CONSTRAINT chk_ml_weight CHECK (weight >= 0.0 AND weight <= 1.0)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS mental_models (
|
||||
id VARCHAR2(256) NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
subtype VARCHAR2(32) NOT NULL,
|
||||
name VARCHAR2(256) NOT NULL,
|
||||
description CLOB NOT NULL,
|
||||
source_query CLOB,
|
||||
content CLOB,
|
||||
embedding VECTOR(384, FLOAT32),
|
||||
entity_id RAW(16),
|
||||
observations CLOB DEFAULT '{"observations":[]}' NOT NULL
|
||||
CONSTRAINT mm_obs_json CHECK (observations IS JSON),
|
||||
links CLOB,
|
||||
tags CLOB DEFAULT '[]' NOT NULL,
|
||||
max_tokens NUMBER(10) DEFAULT 2048 NOT NULL,
|
||||
"trigger" CLOB DEFAULT '{"refresh_after_consolidation":false}' NOT NULL
|
||||
CONSTRAINT mm_trigger_json CHECK ("trigger" IS JSON),
|
||||
structured_content CLOB CONSTRAINT mm_sc_json CHECK (structured_content IS JSON OR structured_content IS NULL),
|
||||
last_refreshed_source_query CLOB,
|
||||
reflect_response CLOB CONSTRAINT mm_reflect_resp_json CHECK (reflect_response IS JSON OR reflect_response IS NULL),
|
||||
history CLOB DEFAULT '[]' NOT NULL
|
||||
CONSTRAINT mm_history_json CHECK (history IS JSON),
|
||||
last_refreshed_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
last_updated TIMESTAMP WITH TIME ZONE,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_mental_models PRIMARY KEY (id, bank_id),
|
||||
CONSTRAINT fk_mm_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE,
|
||||
CONSTRAINT fk_mm_entity FOREIGN KEY (entity_id) REFERENCES entities(id) ON DELETE SET NULL,
|
||||
CONSTRAINT chk_mm_subtype CHECK (subtype IN ('directive', 'pinned'))
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS directives (
|
||||
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
name VARCHAR2(256) NOT NULL,
|
||||
content CLOB NOT NULL,
|
||||
priority NUMBER(10) DEFAULT 0 NOT NULL,
|
||||
is_active NUMBER(1) DEFAULT 1 NOT NULL,
|
||||
tags CLOB DEFAULT '[]' NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_directives PRIMARY KEY (id),
|
||||
CONSTRAINT fk_dir_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS async_operations (
|
||||
operation_id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
operation_type VARCHAR2(128) NOT NULL,
|
||||
status VARCHAR2(32) DEFAULT 'pending' NOT NULL,
|
||||
worker_id VARCHAR2(256),
|
||||
claimed_at TIMESTAMP WITH TIME ZONE,
|
||||
retry_count NUMBER(10) DEFAULT 0 NOT NULL,
|
||||
next_retry_at TIMESTAMP WITH TIME ZONE,
|
||||
task_payload CLOB CONSTRAINT ao_payload_json CHECK (task_payload IS JSON OR task_payload IS NULL),
|
||||
result_metadata CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT ao_result_json CHECK (result_metadata IS JSON),
|
||||
error_message CLOB,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
completed_at TIMESTAMP WITH TIME ZONE,
|
||||
CONSTRAINT pk_async_operations PRIMARY KEY (operation_id),
|
||||
CONSTRAINT fk_ao_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE,
|
||||
CONSTRAINT chk_ao_status CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled'))
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS webhooks (
|
||||
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
bank_id VARCHAR2(256) NOT NULL,
|
||||
url VARCHAR2(2048) NOT NULL,
|
||||
secret VARCHAR2(512),
|
||||
event_types CLOB DEFAULT '[]' NOT NULL,
|
||||
http_config CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT wh_http_config_json CHECK (http_config IS JSON),
|
||||
enabled NUMBER(1) DEFAULT 1 NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
CONSTRAINT pk_webhooks PRIMARY KEY (id),
|
||||
CONSTRAINT fk_wh_bank FOREIGN KEY (bank_id) REFERENCES banks(bank_id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS file_storage (
|
||||
storage_key VARCHAR2(512) NOT NULL,
|
||||
data BLOB NOT NULL,
|
||||
CONSTRAINT pk_file_storage PRIMARY KEY (storage_key)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS audit_log (
|
||||
id RAW(16) DEFAULT SYS_GUID() NOT NULL,
|
||||
action VARCHAR2(128) NOT NULL,
|
||||
transport VARCHAR2(64) NOT NULL,
|
||||
bank_id VARCHAR2(256),
|
||||
started_at TIMESTAMP WITH TIME ZONE DEFAULT SYSTIMESTAMP NOT NULL,
|
||||
ended_at TIMESTAMP WITH TIME ZONE,
|
||||
request CLOB CONSTRAINT al_request_json CHECK (request IS JSON OR request IS NULL),
|
||||
response CLOB CONSTRAINT al_response_json CHECK (response IS JSON OR response IS NULL),
|
||||
metadata CLOB DEFAULT '{}' NOT NULL
|
||||
CONSTRAINT al_metadata_json CHECK (metadata IS JSON),
|
||||
CONSTRAINT pk_audit_log PRIMARY KEY (id)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS observation_sources (
|
||||
observation_id RAW(16) NOT NULL,
|
||||
source_id RAW(16) NOT NULL,
|
||||
CONSTRAINT pk_observation_sources PRIMARY KEY (observation_id, source_id),
|
||||
CONSTRAINT fk_obs_src_observation FOREIGN KEY (observation_id)
|
||||
REFERENCES memory_units(id) ON DELETE CASCADE
|
||||
)
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# B-tree indexes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_INDEXES: tuple[str, ...] = (
|
||||
# documents
|
||||
"CREATE INDEX idx_docs_bank_id ON documents(bank_id)",
|
||||
"CREATE INDEX idx_docs_content_hash ON documents(content_hash)",
|
||||
# chunks
|
||||
"CREATE INDEX idx_chunks_document_id ON chunks(document_id)",
|
||||
"CREATE INDEX idx_chunks_bank_id ON chunks(bank_id)",
|
||||
# memory_units
|
||||
"CREATE INDEX idx_mu_bank_id ON memory_units(bank_id)",
|
||||
"CREATE INDEX idx_mu_document_id ON memory_units(document_id)",
|
||||
"CREATE INDEX idx_mu_chunk_id ON memory_units(chunk_id)",
|
||||
"CREATE INDEX idx_mu_event_date ON memory_units(event_date DESC)",
|
||||
"CREATE INDEX idx_mu_bank_date ON memory_units(bank_id, event_date DESC)",
|
||||
"CREATE INDEX idx_mu_access_count ON memory_units(access_count DESC)",
|
||||
"CREATE INDEX idx_mu_fact_type ON memory_units(fact_type)",
|
||||
"CREATE INDEX idx_mu_bank_fact_type ON memory_units(bank_id, fact_type)",
|
||||
"CREATE INDEX idx_mu_bank_type_date ON memory_units(bank_id, fact_type, event_date DESC)",
|
||||
# entities
|
||||
"CREATE INDEX idx_ent_bank_id ON entities(bank_id)",
|
||||
"CREATE INDEX idx_ent_canonical_name ON entities(canonical_name)",
|
||||
"CREATE INDEX idx_ent_bank_name ON entities(bank_id, canonical_name)",
|
||||
"CREATE UNIQUE INDEX idx_ent_bank_lower_name ON entities(bank_id, LOWER(canonical_name))",
|
||||
# unit_entities
|
||||
"CREATE INDEX idx_ue_unit ON unit_entities(unit_id)",
|
||||
"CREATE INDEX idx_ue_entity ON unit_entities(entity_id)",
|
||||
# entity_cooccurrences
|
||||
"CREATE INDEX idx_ec_entity1 ON entity_cooccurrences(entity_id_1)",
|
||||
"CREATE INDEX idx_ec_entity2 ON entity_cooccurrences(entity_id_2)",
|
||||
"CREATE INDEX idx_ec_count ON entity_cooccurrences(cooccurrence_count DESC)",
|
||||
# memory_links — function-based unique index uses NVL with the nil UUID raw
|
||||
# to handle nullable entity_id (matches PG idx_memory_links_unique).
|
||||
"CREATE UNIQUE INDEX idx_memory_links_unique ON memory_links("
|
||||
"from_unit_id, to_unit_id, link_type, "
|
||||
"NVL(entity_id, HEXTORAW('00000000000000000000000000000000')))",
|
||||
"CREATE INDEX idx_ml_from_unit ON memory_links(from_unit_id)",
|
||||
"CREATE INDEX idx_ml_to_unit ON memory_links(to_unit_id)",
|
||||
"CREATE INDEX idx_ml_entity ON memory_links(entity_id)",
|
||||
"CREATE INDEX idx_ml_link_type ON memory_links(link_type)",
|
||||
"CREATE INDEX idx_ml_bank_id ON memory_links(bank_id)",
|
||||
# directives
|
||||
"CREATE INDEX idx_dir_bank_id ON directives(bank_id)",
|
||||
"CREATE INDEX idx_dir_bank_active ON directives(bank_id, is_active)",
|
||||
# mental_models
|
||||
"CREATE INDEX idx_mm_bank_id ON mental_models(bank_id)",
|
||||
"CREATE INDEX idx_mm_subtype ON mental_models(bank_id, subtype)",
|
||||
"CREATE INDEX idx_mm_entity_id ON mental_models(entity_id)",
|
||||
# async_operations
|
||||
"CREATE INDEX idx_ao_bank_id ON async_operations(bank_id)",
|
||||
"CREATE INDEX idx_ao_status ON async_operations(status)",
|
||||
"CREATE INDEX idx_ao_bank_status ON async_operations(bank_id, status)",
|
||||
"CREATE INDEX idx_ao_status_retry ON async_operations(status, next_retry_at)",
|
||||
# webhooks
|
||||
"CREATE INDEX idx_wh_bank_id ON webhooks(bank_id)",
|
||||
# audit_log
|
||||
"CREATE INDEX idx_al_action_started ON audit_log(action, started_at DESC)",
|
||||
"CREATE INDEX idx_al_bank_started ON audit_log(bank_id, started_at DESC)",
|
||||
"CREATE INDEX idx_al_started ON audit_log(started_at DESC)",
|
||||
# observation_sources
|
||||
"CREATE INDEX idx_obs_sources_source_id ON observation_sources(source_id, observation_id)",
|
||||
)
|
||||
|
||||
_VECTOR_INDEX = (
|
||||
"CREATE VECTOR INDEX idx_mu_embedding_hnsw ON memory_units(embedding) "
|
||||
"ORGANIZATION NEIGHBOR PARTITIONS "
|
||||
"DISTANCE COSINE "
|
||||
"WITH TARGET ACCURACY 95"
|
||||
)
|
||||
|
||||
# Oracle Text (CTXSYS.CONTEXT) — ``SYNC (ON COMMIT)`` makes it auto-update
|
||||
# without a maintenance job. Doubled single quotes for the embedded literal.
|
||||
_TEXT_INDEX = (
|
||||
"BEGIN "
|
||||
"EXECUTE IMMEDIATE '"
|
||||
"CREATE INDEX idx_mu_content_text ON memory_units(text) "
|
||||
"INDEXTYPE IS CTXSYS.CONTEXT "
|
||||
"PARAMETERS (''SYNC (ON COMMIT)'')"
|
||||
"'; "
|
||||
"EXCEPTION WHEN OTHERS THEN "
|
||||
"IF SQLCODE = -955 THEN NULL; ELSE RAISE; END IF; "
|
||||
"END;"
|
||||
)
|
||||
|
||||
|
||||
def _execute_ignoring_955(sql: str) -> None:
|
||||
"""Run a CREATE statement and swallow ORA-00955 (object already exists).
|
||||
|
||||
Wraps the statement in PL/SQL so the exception handler runs server-side —
|
||||
no round-trip cost for the common case.
|
||||
"""
|
||||
block = (
|
||||
"BEGIN "
|
||||
"EXECUTE IMMEDIATE :stmt; "
|
||||
"EXCEPTION WHEN OTHERS THEN "
|
||||
"IF SQLCODE = -955 THEN NULL; ELSE RAISE; END IF; "
|
||||
"END;"
|
||||
)
|
||||
op.get_bind().exec_driver_sql(block, {"stmt": sql.strip()})
|
||||
|
||||
|
||||
def _oracle_upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
# Tolerate concurrent DDL instead of failing immediately (ORA-00054).
|
||||
bind.exec_driver_sql("ALTER SESSION SET DDL_LOCK_TIMEOUT = 30")
|
||||
|
||||
for ddl in _TABLES:
|
||||
_execute_ignoring_955(ddl)
|
||||
|
||||
for idx in _INDEXES:
|
||||
_execute_ignoring_955(idx)
|
||||
|
||||
# Hindsight on Oracle requires 23ai with VECTOR support (ASSM tablespace)
|
||||
# and the CTXSYS package for full-text. Both index creations must succeed
|
||||
# — the migration fails hard if either feature is unavailable, by design.
|
||||
# We only swallow ORA-00955 (object already exists) so reruns are safe.
|
||||
_execute_ignoring_955(_VECTOR_INDEX)
|
||||
bind.exec_driver_sql(_TEXT_INDEX)
|
||||
|
||||
|
||||
def _oracle_downgrade() -> None:
|
||||
# Baseline downgrades aren't supported — dropping every table here would
|
||||
# destroy customer data. Use point-in-time recovery instead.
|
||||
raise NotImplementedError("Cannot downgrade past the Oracle baseline.")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(oracle=_oracle_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(oracle=_oracle_downgrade)
|
||||
+2
-12
@@ -21,8 +21,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "p1k2l3m4n5o6"
|
||||
down_revision: str | Sequence[str] | None = "o0j1k2l3m4n5"
|
||||
@@ -36,7 +34,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Implement new knowledge architecture."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -128,7 +126,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Reverse the migration."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -194,11 +192,3 @@ def _pg_downgrade() -> None:
|
||||
""")
|
||||
|
||||
# Note: mental_models table recreation is complex and would need separate handling
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -12,8 +12,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "q2l3m4n5o6p7"
|
||||
down_revision: str | Sequence[str] | None = "p1k2l3m4n5o6"
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add 'mental_model' to the fact_type check constraint."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -40,7 +38,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove 'mental_model' from the fact_type check constraint."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -50,11 +48,3 @@ def _pg_downgrade() -> None:
|
||||
ADD CONSTRAINT memory_units_fact_type_check
|
||||
CHECK (fact_type IN ('world', 'experience', 'opinion', 'observation'))
|
||||
""")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -14,8 +14,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "r3m4n5o6p7q8"
|
||||
down_revision: str | Sequence[str] | None = "q2l3m4n5o6p7"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -28,7 +26,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Add reflect_response JSONB column to reflections."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -39,7 +37,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Remove reflect_response column from reflections."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -47,11 +45,3 @@ def _pg_downgrade() -> None:
|
||||
ALTER TABLE {schema}reflections
|
||||
DROP COLUMN IF EXISTS reflect_response
|
||||
""")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -12,8 +12,6 @@ import sqlalchemy as sa
|
||||
from alembic import context, op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "rename_personality"
|
||||
down_revision: str | Sequence[str] | None = "d9f6a3b4c5e2"
|
||||
@@ -27,7 +25,7 @@ def _get_target_schema() -> str:
|
||||
return schema if schema else "public"
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Rename personality column to disposition in banks table (if it exists)."""
|
||||
conn = op.get_bind()
|
||||
target_schema = _get_target_schema()
|
||||
@@ -71,7 +69,7 @@ def _pg_upgrade() -> None:
|
||||
# else: disposition already exists, nothing to do
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Revert disposition column back to personality."""
|
||||
conn = op.get_bind()
|
||||
target_schema = _get_target_schema()
|
||||
@@ -85,11 +83,3 @@ def _pg_downgrade() -> None:
|
||||
)
|
||||
if result.fetchone():
|
||||
op.alter_column("banks", "disposition", new_column_name="personality")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -13,8 +13,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "s4n5o6p7q8r9"
|
||||
down_revision: str | Sequence[str] | None = "r3m4n5o6p7q8"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -27,7 +25,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Add consolidated_at column to memory_units
|
||||
@@ -48,16 +46,8 @@ def _pg_upgrade() -> None:
|
||||
)
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_unconsolidated")
|
||||
op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS consolidated_at")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
+2
-12
@@ -17,8 +17,6 @@ from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
|
||||
from hindsight_api.alembic._dialect import run_for_dialect
|
||||
|
||||
revision: str = "t5o6p7q8r9s0"
|
||||
down_revision: str | Sequence[str] | None = "s4n5o6p7q8r9"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
@@ -31,7 +29,7 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _pg_upgrade() -> None:
|
||||
def upgrade() -> None:
|
||||
"""Rename mental_model -> observation and reflections -> mental_models."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -88,7 +86,7 @@ def _pg_upgrade() -> None:
|
||||
""")
|
||||
|
||||
|
||||
def _pg_downgrade() -> None:
|
||||
def downgrade() -> None:
|
||||
"""Reverse: observation -> mental_model and mental_models -> reflections."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
@@ -134,11 +132,3 @@ def _pg_downgrade() -> None:
|
||||
ON {schema}memory_units(bank_id, fact_type)
|
||||
WHERE fact_type = 'mental_model'
|
||||
""")
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
run_for_dialect(pg=_pg_upgrade)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
run_for_dialect(pg=_pg_downgrade)
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user