Compare commits

...
8 Commits
Author SHA1 Message Date
Nicolò Boschi 90170d79e0 feat: support for other text and vector search pg extensions 2026-02-12 14:12:49 +01:00
Nicolò Boschi 043b304f27 test: increase timeout for test_batch_chunking_behavior to account for VectorChord BM25 tokenization overhead 2026-02-12 13:26:23 +01:00
Nicolò Boschi 8b274d10ad feat: support for other text and vector search pg extensions 2026-02-12 13:17:50 +01:00
Damien a713b68b1f Implement the vchord / pgvector support (#350)
* Implement the vchord / pgvector support

* feat(alembic): detect vector extension and create appropriate index
2026-02-12 10:47:07 +01:00
Nicolò Boschi 93ddd41621 feat: add reverse proxy support (#346)
* feat: add reverse proxy support

* improve

* improve

* improve

* improve

* improve

* fix: update integration test to use modern 'docker compose' command

- Replace 'docker-compose' with 'docker compose' (Docker Compose v2+)
- Add fallback to legacy docker-compose command for compatibility
- Fixes test failures on systems using Docker Compose plugin

* ci: trigger test rerun

* fix: make docker-compose detection more robust for CI

- Add get_docker_compose_command() to detect available command
- Use shutil.which() to check command availability
- Dynamically use correct command (docker compose vs docker-compose)
- Should work in both modern and legacy Docker environments

* fix: docker-compose networking in base path integration test

Fix connection refused error in test_reverse_proxy_simple_config by
handling host vs bridge networking modes correctly:

- Linux (host mode): nginx listens on 18080 directly, no port mapping
- Mac/Windows (bridge mode): nginx listens on 80, mapped to 18080

With host networking, port mappings in docker-compose don't work since
the container binds directly to the host's network namespace.
2026-02-12 10:09:53 +01:00
DK09876andClaude Opus 4.6 7ee229ba23 Fix MCP extra args rejection and bank ID resolution priority (#351)
* Fix MCP extra args rejection and bank ID resolution priority

Two fixes to the MCP middleware:

1. Strip unknown tool arguments: LLMs frequently add extra fields
   like "explanation" to tool calls. FastMCP's Pydantic TypeAdapter
   rejects these with "Unexpected keyword argument". The middleware
   now intercepts tools/call requests and removes unknown fields
   before they reach validation.

2. Bank ID resolution priority: Path now takes priority over header.
   Previously X-Bank-Id header was checked first, meaning /mcp/my-bank/
   with X-Bank-Id: other-bank would silently use other-bank in multi-bank
   mode. Now the URL path is authoritative — single-bank mode connections
   cannot be overridden by headers.

Co-Authored-By: Claude Opus 4.6 <[email protected]>

* docs: update MCP server docs with mental model tools and fixes

- Add all mental model tools (create, list, get, update, delete, refresh)
- Add list_banks and create_bank tool docs
- Document single-bank vs multi-bank modes
- Fix bank selection priority: path > header > default
- Add Accept header to curl example
- Add timestamp param to retain, max_tokens to recall

Co-Authored-By: Claude Opus 4.6 <[email protected]>

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
2026-02-12 10:09:28 +01:00
Chris Latimer 29c0890f22 Copy page button on docs 2026-02-11 19:21:24 -07:00
Chris Bartholomew a1f22dabd2 Replace waitlist links with direct Hindsight Cloud signup URL (#349)
The waitlist is no longer needed. Update all references from
vectorize.io/hindsight/cloud to ui.hindsight.vectorize.io/signup
and change "request early access" language to "sign up".
2026-02-11 20:51:32 +01:00
41 changed files with 2598 additions and 227 deletions
+6
View File
@@ -31,6 +31,12 @@ HINDSIGHT_API_HOST=0.0.0.0
HINDSIGHT_API_PORT=8888
HINDSIGHT_API_LOG_LEVEL=info
# Base Path / Reverse Proxy Support (Optional)
# Set these when deploying behind a reverse proxy with path-based routing
# Example: To deploy at example.com/hindsight/, set both to "/hindsight"
# HINDSIGHT_API_BASE_PATH=/hindsight
# NEXT_PUBLIC_BASE_PATH=/hindsight
# Database (Optional - uses embedded pg0 by default)
# HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
# HINDSIGHT_API_DATABASE_SCHEMA=public # PostgreSQL schema name (default: public)
+1 -1
View File
@@ -2,7 +2,7 @@
![Hindsight Banner](./hindsight-docs/static/img/hindsight-github-banner.png)
[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://vectorize.io/hindsight/cloud)
[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)
[![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
+96
View File
@@ -0,0 +1,96 @@
# Nginx Reverse Proxy with Custom Base Path
Deploy Hindsight API under `/hindsight` (or any custom path) using Nginx reverse proxy.
## Quick Start (Published Image - API Only)
```bash
docker-compose up
```
- **API:** http://localhost:8080/hindsight/docs
- **Control Plane:** http://localhost:9999 (direct access, not proxied)
## Full Stack with Custom Base Path (Requires Build)
**Important:** You cannot rebuild from the published image with build args. You must build from source.
### Build from Source with Custom Base Path
1. **Clone the repository** (if you haven't):
```bash
git clone https://github.com/vectorize-io/hindsight.git
cd hindsight
```
2. **Build with base path**:
```bash
docker build \
--build-arg NEXT_PUBLIC_BASE_PATH=/hindsight \
-f docker/standalone/Dockerfile \
-t hindsight:custom \
.
```
3. **Update docker-compose.yml** to use your built image:
```yaml
services:
hindsight:
image: hindsight:custom # ← Change this
environment:
HINDSIGHT_API_BASE_PATH: /hindsight
NEXT_PUBLIC_BASE_PATH: /hindsight
```
4. **Update nginx.conf** to handle Control Plane routes (see below)
5. **Run**:
```bash
docker-compose up
```
### Required nginx.conf for Full Stack
Replace the current `nginx.conf` with this to proxy both API and Control Plane:
```nginx
events { worker_connections 1024; }
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
upstream hindsight_api { server hindsight:8888; }
upstream hindsight_cp { server hindsight:9999; }
server {
listen 80;
# API
location ~ ^/hindsight/(docs|openapi\.json|health|metrics|v1|mcp) {
proxy_pass http://hindsight_api;
proxy_set_header Host $http_host;
}
# Control Plane static files
location ~ ^/hindsight/_next/ {
proxy_pass http://hindsight_cp;
proxy_set_header Host $http_host;
}
# Control Plane UI
location /hindsight {
proxy_pass http://hindsight_cp;
proxy_set_header Host $http_host;
}
location = / { return 301 /hindsight; }
}
}
```
### Why Build is Required
Next.js requires `basePath` at **build time**. The published image was built without a custom base path, so you must rebuild from source with the `NEXT_PUBLIC_BASE_PATH` build arg to deploy the Control Plane under a subpath.
The API works without rebuild because `HINDSIGHT_API_BASE_PATH` is a runtime environment variable.
@@ -0,0 +1,88 @@
# Hindsight API deployment with Nginx reverse proxy (API-only)
#
# This example deploys Hindsight API under the path /hindsight with:
# - Hindsight standalone image (API + Control Plane + embedded pg0)
# - Nginx reverse proxy (API only)
#
# Quick Start:
# docker-compose -f docker/docker-compose/nginx/docker-compose.yml up
#
# Access:
# API (via nginx): http://localhost:8080/hindsight/docs
# Control Plane (direct): http://localhost:9999
#
# For full stack deployment (API + Control Plane both under /hindsight):
# See README.md in this directory for instructions on building with basePath.
#
# Note: This configuration uses the published image (no build required).
# Control Plane is served directly because Next.js basePath requires
# build-time configuration. See README.md for the full stack option.
services:
# Hindsight (API + Control Plane + embedded pg0)
hindsight:
image: ghcr.io/vectorize-io/hindsight:latest
ports:
- "9999:9999" # Control Plane (direct access, not proxied)
environment:
# API base path for reverse proxy
HINDSIGHT_API_BASE_PATH: /hindsight
# LLM configuration
# Using mock provider for testing (no API key needed)
# For production, set OPENAI_API_KEY or ANTHROPIC_API_KEY and use a real provider
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-mock}
HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-not-needed-for-mock}
HINDSIGHT_API_LLM_MODEL: ${HINDSIGHT_API_LLM_MODEL:-mock-model}
# Production examples (uncomment and set appropriate API key):
# HINDSIGHT_API_LLM_PROVIDER: openai
# HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY}
# HINDSIGHT_API_LLM_MODEL: gpt-4o-mini
# HINDSIGHT_API_LLM_PROVIDER: anthropic
# HINDSIGHT_API_LLM_API_KEY: ${ANTHROPIC_API_KEY}
# HINDSIGHT_API_LLM_MODEL: claude-sonnet-4-20250514
# Server config
HINDSIGHT_API_HOST: 0.0.0.0
HINDSIGHT_API_PORT: 8888
HINDSIGHT_API_LOG_LEVEL: info
# Control Plane config
HINDSIGHT_CP_DATAPLANE_API_URL: http://localhost:8888
volumes:
# Persist embedded pg0 database
- hindsight_data:/app/data
# Note: Ports not exposed - access via Nginx at localhost:8080/hindsight/
# To debug directly, uncomment these ports:
# ports:
# - "8888:8888" # API
# - "9999:9999" # Control Plane
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8888/hindsight/health"]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
networks:
- hindsight
# Nginx reverse proxy
nginx:
image: nginx:alpine
ports:
- "8080:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
hindsight:
condition: service_healthy
networks:
- hindsight
volumes:
hindsight_data:
networks:
hindsight:
+40
View File
@@ -0,0 +1,40 @@
# Nginx configuration for API-only reverse proxy
# Control Plane accessed directly (not through nginx)
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Logging
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# Upstream - Hindsight API
upstream hindsight_api {
server hindsight:8888;
}
server {
listen 80;
server_name _;
# API endpoints - forward with /hindsight prefix
location /hindsight/ {
proxy_pass http://hindsight_api;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Redirect root to API docs
location = / {
return 301 /hindsight/docs;
}
}
}
@@ -0,0 +1,93 @@
name: hindsight
# Docker Compose file for Hindsight with PostgreSQL and vectorchord
# docker compose -f docker/docker-compose/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/docker-compose.yaml up -d
# Make sure to set the required environment variables before running:
# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
# - Configure LLM provider variables as needed (see below in the hindsight service)
#
# Usage:
# docker compose up -d
#
# Optional environment variables with defaults:
# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
services:
db:
# Use a PostgreSQL-Image with vectorchord extension pre-installed
image: tensorchord/vchord-suite:pg${HINDSIGHT_DB_VERSION:-18-latest}
container_name: hindsight-db
restart: always
# Expose PostgreSQL port
ports:
- "5436:5432"
environment:
POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
volumes:
- pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
networks:
- hindsight-net
vectorchord-init:
image: tensorchord/vchord-suite:pg18-latest
#container_name: vectorchord-init
depends_on:
- db
environment:
- PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
command: >
bash -c "
echo 'Waiting for PostgreSQL to be ready...';
until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
echo 'PostgreSQL is unavailable - sleeping';
sleep 2;
done;
echo 'PostgreSQL is ready - creating hindsight_db database';
psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
echo 'Creating extensions in hindsight_db database';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord CASCADE;';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;';
echo 'Creating llmlingua2 tokenizer';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c \"SELECT create_tokenizer('llmlingua2', \\$\\$ model = \\\"llmlingua2\\\" \\$\\$);\" 2>/dev/null || echo 'Tokenizer already exists or creation skipped';
echo 'Database and extensions created successfully';
"
restart: "no"
networks:
- hindsight-net
hindsight:
image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
container_name: hindsight-app
ports:
- "8888:8888"
- "9999:9999"
environment:
# LLM Configuration (uses OpenAI for testing vchord)
# LLM configuration
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
# Database Configuration
HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
# Vector and Text Search Extensions
HINDSIGHT_API_VECTOR_EXTENSION: vchord
HINDSIGHT_API_TEXT_SEARCH_EXTENSION: vchord
depends_on:
- db
networks:
- hindsight-net
networks:
hindsight-net:
driver: bridge
volumes:
pg_data:
+4
View File
@@ -112,6 +112,10 @@ RUN rm -f package-lock.json && sed -i '/"@vectorize-io\/hindsight-client":/d' pa
# Copy built SDK directly into node_modules (more reliable than npm link in Docker)
COPY --from=sdk-builder /app/hindsight-clients/typescript ./node_modules/@vectorize-io/hindsight-client
# Accept base path as build argument for reverse proxy deployments
# Usage: docker build --build-arg NEXT_PUBLIC_BASE_PATH=/hindsight ...
ARG NEXT_PUBLIC_BASE_PATH=""
# Build Control Plane - run next build first, then custom standalone copy
# (The build:standalone script expects a specific path structure that differs in Docker)
RUN npm exec -- next build
@@ -6,6 +6,7 @@ Create Date: 2025-11-27 11:54:19.228030
"""
import os
from collections.abc import Sequence
import sqlalchemy as sa
@@ -21,6 +22,61 @@ branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _detect_vector_extension() -> str:
"""
Detect or validate vector extension: 'vchord' or 'pgvector'.
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
"""
conn = op.get_bind()
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
# Validate configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
return "vchord"
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
return "pgvector"
else:
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
def _detect_text_search_extension() -> str:
"""
Detect or validate text search extension: 'native' or 'vchord'.
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
Creates the extension if needed.
"""
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
if text_search_extension == "vchord":
# Create vchord_bm25 extension if not exists
try:
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
except Exception:
# Extension might already exist or user lacks permissions - verify it exists
conn = op.get_bind()
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
if not result:
# Extension truly doesn't exist - re-raise the error
raise
return "vchord"
elif text_search_extension == "native":
return "native"
else:
raise ValueError(
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
)
def upgrade() -> None:
"""Upgrade schema - create all tables from scratch."""
@@ -166,11 +222,23 @@ def upgrade() -> None:
)
# Add search_vector column for full-text search
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
""")
# Type depends on configured text search backend
text_search_ext = _detect_text_search_extension()
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector bm25_catalog.bm25vector
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
""")
op.create_index("idx_memory_units_bank_id", "memory_units", ["bank_id"])
op.create_index("idx_memory_units_document_id", "memory_units", ["document_id"])
@@ -200,19 +268,39 @@ def upgrade() -> None:
["bank_id", sa.text("event_date DESC")],
postgresql_where=sa.text("fact_type = 'observation'"),
)
op.create_index(
"idx_memory_units_embedding",
"memory_units",
["embedding"],
postgresql_using="hnsw",
postgresql_ops={"embedding": "vector_cosine_ops"},
)
# Create vector index - conditional based on available extension
vector_ext = _detect_vector_extension()
# Create BM25 full-text search index on search_vector
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING gin(search_vector)
""")
if vector_ext == "vchord":
# Use vchordrq index for vchord (supports high-dimensional embeddings)
op.execute("""
CREATE INDEX idx_memory_units_embedding ON memory_units
USING vchordrq (embedding vector_l2_ops)
""")
else: # pgvector
# Use HNSW index for pgvector
op.create_index(
"idx_memory_units_embedding",
"memory_units",
["embedding"],
postgresql_using="hnsw",
postgresql_ops={"embedding": "vector_cosine_ops"},
)
# Create full-text search index on search_vector
# Index type depends on text search backend
if text_search_ext == "vchord":
# VectorChord BM25 index
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL GIN index
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING gin(search_vector)
""")
op.execute("""
CREATE MATERIALIZED VIEW memory_units_bm25 AS
@@ -10,9 +10,11 @@ This migration:
3. Adds consolidation tracking columns to the 'banks' table
"""
import os
from collections.abc import Sequence
from alembic import context, op
from sqlalchemy import text
# revision identifiers, used by Alembic.
revision: str = "n9i0j1k2l3m4"
@@ -27,10 +29,71 @@ def _get_schema_prefix() -> str:
return f'"{schema}".' if schema else ""
def _detect_vector_extension() -> str:
"""
Detect or validate vector extension: 'vchord' or 'pgvector'.
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
"""
conn = op.get_bind()
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
# Validate configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
return "vchord"
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
return "pgvector"
else:
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
def _detect_text_search_extension() -> str:
"""
Detect or validate text search extension: 'native' or 'vchord'.
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
Creates the extension if needed.
"""
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
if text_search_extension == "vchord":
# Create vchord_bm25 extension if not exists
try:
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
except Exception:
# Extension might already exist or user lacks permissions - verify it exists
conn = op.get_bind()
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
if not result:
# Extension truly doesn't exist - re-raise the error
raise
return "vchord"
elif text_search_extension == "native":
return "native"
else:
raise ValueError(
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
)
def upgrade() -> None:
"""Create learnings and pinned_reflections tables."""
schema = _get_schema_prefix()
# Detect which vector extension is available
vector_ext = _detect_vector_extension()
# Detect which text search extension to use
text_search_ext = _detect_text_search_extension()
# 1. Create learnings table
op.execute(f"""
CREATE TABLE {schema}learnings (
@@ -57,18 +120,39 @@ def upgrade() -> None:
# Indexes for learnings
op.execute(f"CREATE INDEX idx_learnings_bank_id ON {schema}learnings(bank_id)")
op.execute(f"""
CREATE INDEX idx_learnings_embedding ON {schema}learnings
USING hnsw (embedding vector_cosine_ops)
""")
# Create vector index based on detected extension
if vector_ext == "vchord":
op.execute(f"""
CREATE INDEX idx_learnings_embedding ON {schema}learnings
USING vchordrq (embedding vector_l2_ops)
""")
else: # pgvector
op.execute(f"""
CREATE INDEX idx_learnings_embedding ON {schema}learnings
USING hnsw (embedding vector_cosine_ops)
""")
op.execute(f"CREATE INDEX idx_learnings_tags ON {schema}learnings USING GIN(tags)")
# Full-text search for learnings
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
""")
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector bm25_catalog.bm25vector
""")
op.execute(f"""
CREATE INDEX idx_learnings_text_search ON {schema}learnings
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
""")
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
# 2. Create pinned_reflections table
op.execute(f"""
@@ -94,21 +178,42 @@ def upgrade() -> None:
# Indexes for pinned_reflections
op.execute(f"CREATE INDEX idx_pinned_reflections_bank_id ON {schema}pinned_reflections(bank_id)")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
USING hnsw (embedding vector_cosine_ops)
""")
# Create vector index based on detected extension
if vector_ext == "vchord":
op.execute(f"""
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
USING vchordrq (embedding vector_l2_ops)
""")
else: # pgvector
op.execute(f"""
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
USING hnsw (embedding vector_cosine_ops)
""")
op.execute(f"CREATE INDEX idx_pinned_reflections_tags ON {schema}pinned_reflections USING GIN(tags)")
# Full-text search for pinned_reflections
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING gin(search_vector)
""")
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT/UPDATE)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector bm25_catalog.bm25vector
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING gin(search_vector)
""")
# 3. Add consolidation tracking columns to banks table
op.execute(f"""
+4
View File
@@ -1491,6 +1491,9 @@ def create_app(
logging.info("Memory system closed")
from hindsight_api import __version__
from hindsight_api.config import get_config
config = get_config()
app = FastAPI(
title="Hindsight HTTP API",
@@ -1504,6 +1507,7 @@ def create_app(
"url": "https://www.apache.org/licenses/LICENSE-2.0.html",
},
lifespan=lifespan,
root_path=config.base_path,
)
# IMPORTANT: Set memory on app.state immediately, don't wait for lifespan
+46 -6
View File
@@ -114,9 +114,39 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
logger.info(f"Loading MCP extension: {mcp_extension.__class__.__name__}")
mcp_extension.register_tools(mcp, memory)
# Make all tools tolerant of extra arguments from LLMs (e.g., "explanation")
_make_tools_tolerant(mcp)
return mcp
def _make_tools_tolerant(mcp: FastMCP) -> None:
"""Wrap all tool run methods to strip unknown arguments before validation.
LLMs frequently add extra fields like "explanation" or "reasoning" to tool calls.
FastMCP's Pydantic TypeAdapter rejects these with "Unexpected keyword argument".
This wraps each tool's run() to filter arguments to only known parameters.
"""
try:
for name, tool in mcp._tool_manager._tools.items():
if hasattr(tool, "parameters") and tool.parameters:
allowed = set(tool.parameters.get("properties", {}).keys())
original_run = tool.run
async def _tolerant_run(arguments, _allowed=allowed, _orig=original_run):
extra_keys = set(arguments.keys()) - _allowed
if extra_keys:
logger.debug(f"Stripping unknown arguments from tool call: {extra_keys}")
arguments = {k: v for k, v in arguments.items() if k in _allowed}
return await _orig(arguments)
# FunctionTool is a Pydantic model with extra='forbid', so use
# object.__setattr__ to bypass Pydantic's setter validation.
object.__setattr__(tool, "run", _tolerant_run)
except (AttributeError, KeyError) as e:
logger.warning(f"Could not make tools tolerant of extra arguments: {e}")
class MCPMiddleware:
"""ASGI middleware that intercepts MCP requests and routes to appropriate MCP server.
@@ -142,6 +172,11 @@ class MCPMiddleware:
- No bank management tools (list_banks, create_bank)
- Recommended for agent isolation
Bank ID resolution priority:
1. URL path (e.g., /mcp/{bank_id}/) → single-bank mode
2. X-Bank-Id header → multi-bank mode
3. HINDSIGHT_MCP_BANK_ID env var → multi-bank mode (default: "default")
Examples:
# Single-bank mode (recommended for agent isolation)
claude mcp add --transport http my-agent http://localhost:8888/mcp/my-agent-bank/ \\
@@ -242,20 +277,25 @@ class MCPMiddleware:
_current_schema.set(tenant_context.schema_name) if tenant_context and tenant_context.schema_name else None
)
# Try to get bank_id from header first (for Claude Code compatibility)
bank_id = self._get_header(scope, "X-Bank-Id")
# Resolve bank_id: path takes priority over header.
# Path = user's explicit connection endpoint (e.g., /mcp/my-bank/).
# X-Bank-Id header = per-request override for multi-bank mode only.
bank_id = None
bank_id_from_path = False
# If no header, try to extract from path: /{bank_id}/...
new_path = path
if not bank_id and path.startswith("/") and len(path) > 1:
# First, try to extract from path: /{bank_id}/...
if path.startswith("/") and len(path) > 1:
parts = path[1:].split("/", 1)
if parts[0]:
# First segment looks like a bank_id
bank_id = parts[0]
bank_id_from_path = True
new_path = "/" + parts[1] if len(parts) > 1 else "/"
# If no path-based bank_id, try X-Bank-Id header (multi-bank mode)
if not bank_id:
bank_id = self._get_header(scope, "X-Bank-Id")
# Fall back to default bank_id
if not bank_id:
bank_id = DEFAULT_BANK_ID
+31
View File
@@ -107,8 +107,12 @@ ENV_RERANKER_MAX_CANDIDATES = "HINDSIGHT_API_RERANKER_MAX_CANDIDATES"
ENV_RERANKER_FLASHRANK_MODEL = "HINDSIGHT_API_RERANKER_FLASHRANK_MODEL"
ENV_RERANKER_FLASHRANK_CACHE_DIR = "HINDSIGHT_API_RERANKER_FLASHRANK_CACHE_DIR"
ENV_VECTOR_EXTENSION = "HINDSIGHT_API_VECTOR_EXTENSION"
ENV_TEXT_SEARCH_EXTENSION = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION"
ENV_HOST = "HINDSIGHT_API_HOST"
ENV_PORT = "HINDSIGHT_API_PORT"
ENV_BASE_PATH = "HINDSIGHT_API_BASE_PATH"
ENV_LOG_LEVEL = "HINDSIGHT_API_LOG_LEVEL"
ENV_LOG_FORMAT = "HINDSIGHT_API_LOG_FORMAT"
ENV_WORKERS = "HINDSIGHT_API_WORKERS"
@@ -223,6 +227,12 @@ DEFAULT_RERANKER_FLASHRANK_CACHE_DIR = None # Use default cache directory
DEFAULT_EMBEDDINGS_COHERE_MODEL = "embed-english-v3.0"
DEFAULT_RERANKER_COHERE_MODEL = "rerank-english-v3.0"
# Vector extension (pgvector vs vchord)
DEFAULT_VECTOR_EXTENSION = "pgvector" # Options: "pgvector", "vchord"
# Text search extension (native PostgreSQL vs vchord BM25)
DEFAULT_TEXT_SEARCH_EXTENSION = "native" # Options: "native", "vchord"
# LiteLLM defaults
DEFAULT_LITELLM_API_BASE = "http://localhost:4000"
DEFAULT_EMBEDDINGS_LITELLM_MODEL = "text-embedding-3-small"
@@ -230,6 +240,7 @@ DEFAULT_RERANKER_LITELLM_MODEL = "cohere/rerank-english-v3.0"
DEFAULT_HOST = "0.0.0.0"
DEFAULT_PORT = 8888
DEFAULT_BASE_PATH = "" # Empty string = root path
DEFAULT_LOG_LEVEL = "info"
DEFAULT_LOG_FORMAT = "text" # Options: "text", "json"
DEFAULT_WORKERS = 1
@@ -358,6 +369,8 @@ class HindsightConfig:
# Database
database_url: str
database_schema: str
vector_extension: str # "pgvector" or "vchord"
text_search_extension: str # "native" or "vchord"
# LLM (default, used as fallback for per-operation config)
llm_provider: str
@@ -440,6 +453,7 @@ class HindsightConfig:
# Server
host: str
port: int
base_path: str
log_level: str
log_format: str
mcp_enabled: bool
@@ -497,6 +511,20 @@ class HindsightConfig:
def validate(self) -> None:
"""Validate configuration values and raise errors for invalid combinations."""
# Validate vector_extension
valid_extensions = ("pgvector", "vchord")
if self.vector_extension not in valid_extensions:
raise ValueError(
f"Invalid vector_extension: {self.vector_extension}. Must be one of: {', '.join(valid_extensions)}"
)
# Validate text_search_extension
valid_text_search = ("native", "vchord")
if self.text_search_extension not in valid_text_search:
raise ValueError(
f"Invalid text_search_extension: {self.text_search_extension}. Must be one of: {', '.join(valid_text_search)}"
)
# RETAIN_MAX_COMPLETION_TOKENS must be greater than RETAIN_CHUNK_SIZE
# to ensure the LLM has enough output capacity to extract facts from chunks
if self.retain_max_completion_tokens <= self.retain_chunk_size:
@@ -522,6 +550,8 @@ class HindsightConfig:
# Database
database_url=os.getenv(ENV_DATABASE_URL, DEFAULT_DATABASE_URL),
database_schema=os.getenv(ENV_DATABASE_SCHEMA, DEFAULT_DATABASE_SCHEMA),
vector_extension=os.getenv(ENV_VECTOR_EXTENSION, DEFAULT_VECTOR_EXTENSION).lower(),
text_search_extension=os.getenv(ENV_TEXT_SEARCH_EXTENSION, DEFAULT_TEXT_SEARCH_EXTENSION).lower(),
# LLM
llm_provider=llm_provider,
llm_api_key=os.getenv(ENV_LLM_API_KEY),
@@ -662,6 +692,7 @@ class HindsightConfig:
# Server
host=os.getenv(ENV_HOST, DEFAULT_HOST),
port=int(os.getenv(ENV_PORT, DEFAULT_PORT)),
base_path=os.getenv(ENV_BASE_PATH, DEFAULT_BASE_PATH),
log_level=os.getenv(ENV_LOG_LEVEL, DEFAULT_LOG_LEVEL),
log_format=os.getenv(ENV_LOG_FORMAT, DEFAULT_LOG_FORMAT).lower(),
mcp_enabled=os.getenv(ENV_MCP_ENABLED, str(DEFAULT_MCP_ENABLED)).lower() == "true",
@@ -18,6 +18,7 @@ import uuid
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Any
from ...config import get_config
from ..memory_engine import fq_table
from ..retain import embedding_utils
from .prompts import (
@@ -1016,15 +1017,33 @@ async def _create_observation_directly(
t0 = time.time()
observation_id = uuid.uuid4()
# Query varies based on text search backend
config = get_config()
if config.text_search_extension == "vchord":
# VectorChord: manually tokenize and insert search_vector
query = f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at, search_vector
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10,
tokenize($3, 'llmlingua2')::bm25_catalog.bm25vector)
RETURNING id
"""
else: # native
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
query = f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
RETURNING id
"""
row = await conn.fetchrow(
f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
RETURNING id
""",
query,
observation_id,
bank_id,
observation_text,
@@ -968,7 +968,12 @@ class MemoryEngine(MemoryEngineInterface):
# Run database migrations if enabled
if self._run_migrations:
from ..migrations import ensure_embedding_dimension, run_migrations
from ..migrations import (
ensure_embedding_dimension,
ensure_text_search_extension,
ensure_vector_extension,
run_migrations,
)
if not self.db_url:
raise ValueError("Database URL is required for migrations")
@@ -976,30 +981,43 @@ class MemoryEngine(MemoryEngineInterface):
# Migrate all schemas from the tenant extension
# The tenant extension is the single source of truth for which schemas exist
logger.info("Running database migrations...")
try:
tenants = await self._tenant_extension.list_tenants()
if tenants:
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
for tenant in tenants:
schema = tenant.schema
if schema:
try:
run_migrations(self.db_url, schema=schema)
except Exception as e:
logger.warning(f"Failed to migrate schema {schema}: {e}")
logger.info("Schema migrations completed")
tenants = await self._tenant_extension.list_tenants()
if tenants:
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
for tenant in tenants:
schema = tenant.schema
if schema:
run_migrations(self.db_url, schema=schema)
logger.info("Schema migrations completed")
# Ensure embedding column dimension matches the model's dimension
# This is done after migrations and after embeddings.initialize()
for tenant in tenants:
schema = tenant.schema
if schema:
try:
ensure_embedding_dimension(self.db_url, self.embeddings.dimension, schema=schema)
except Exception as e:
logger.warning(f"Failed to ensure embedding dimension for schema {schema}: {e}")
except Exception as e:
logger.warning(f"Failed to run schema migrations: {e}")
# Get config for vector extension setting
config = get_config()
# Ensure embedding column dimension matches the model's dimension
# This is done after migrations and after embeddings.initialize()
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_embedding_dimension(
self.db_url,
self.embeddings.dimension,
schema=schema,
vector_extension=config.vector_extension,
)
# Ensure vector indexes match the configured extension
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_vector_extension(self.db_url, vector_extension=config.vector_extension, schema=schema)
# Ensure text search columns/indexes match the configured extension
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_text_search_extension(
self.db_url, text_search_extension=config.text_search_extension, schema=schema
)
logger.info(f"Connecting to PostgreSQL at {mask_network_location(self.db_url)}")
@@ -7,6 +7,7 @@ Handles insertion of facts into the database.
import json
import logging
from ...config import get_config
from ..memory_engine import fq_table
from .fact_extraction import _sanitize_text
from .types import ProcessedFact
@@ -70,28 +71,58 @@ async def insert_facts_batch(
# Batch insert all facts
# Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
results = await conn.fetch(
f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
# Query varies based on text search backend
config = get_config()
if config.text_search_extension == "vchord":
# VectorChord: manually tokenize and insert search_vector
query = f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
FROM input_data
RETURNING id
""",
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags, search_vector)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
),
tokenize(COALESCE(text, '') || ' ' || COALESCE(context, ''), 'llmlingua2')::bm25_catalog.bm25vector
FROM input_data
RETURNING id
"""
else: # native
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
query = f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
)
FROM input_data
RETURNING id
"""
results = await conn.fetch(
query,
bank_id,
fact_texts,
embeddings,
@@ -158,7 +158,7 @@ async def retrieve_bm25(
from .tags import TagsMatch, build_tags_where_clause_simple
# Sanitize query text: remove special characters that have meaning in tsquery
# Sanitize query text for native backend: remove special characters that have meaning in tsquery
# Keep only alphanumeric characters and spaces
sanitized_text = re.sub(r"[^\w\s]", " ", query_text.lower())
@@ -169,29 +169,46 @@ async def retrieve_bm25(
# If no valid tokens, return empty results
return []
# Convert query to tsquery using OR for more flexible matching
# This prevents empty results when some terms are missing
query_tsquery = " | ".join(tokens)
# Build query based on text search backend
config = get_config()
tags_clause = build_tags_where_clause_simple(tags, 5)
params = [query_tsquery, bank_id, fact_type, limit]
if tags:
params.append(tags)
results = await conn.fetch(
f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = $3
AND search_vector @@ to_tsquery('english', $1)
{tags_clause}
ORDER BY bm25_score DESC
LIMIT $4
""",
*params,
)
if config.text_search_extension == "vchord":
# VectorChord BM25: use <&> operator with to_bm25query and tokenize
params = [bank_id, fact_type, limit, query_text] # Use raw query_text for tokenization
if tags:
params.append(query_text) # VectorChord doesn't need sanitization
query = f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($4, 'llmlingua2')) AS bm25_score
FROM {fq_table("memory_units")}
WHERE bank_id = $1
AND fact_type = $2
{tags_clause}
ORDER BY bm25_score DESC
LIMIT $3
"""
else: # native
# Native PostgreSQL: use ts_rank_cd with to_tsquery
query_tsquery = " | ".join(tokens)
params = [query_tsquery, bank_id, fact_type, limit]
if tags:
params.append(tags)
query = f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = $3
AND search_vector @@ to_tsquery('english', $1)
{tags_clause}
ORDER BY bm25_score DESC
LIMIT $4
"""
results = await conn.fetch(query, *params)
return [RetrievalResult.from_db_row(dict(r)) for r in results]
@@ -268,59 +285,109 @@ async def retrieve_semantic_bm25_combined(
result_dict[ft][0].append(RetrievalResult.from_db_row(row))
return result_dict
query_tsquery = " | ".join(tokens)
# Build BM25 query based on text search backend
config = get_config()
# Build tags clause - param 6 if tags provided
tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
if tags:
params.append(tags)
if config.text_search_extension == "vchord":
# VectorChord BM25: use <&> operator with to_bm25query and tokenize
# Note: VectorChord scores are negative (higher = better, so -1 > -10)
params = [query_emb_str, bank_id, fact_types, limit, query_text] # Pass raw query_text for tokenization
if tags:
params.append(tags)
query = f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
"""
else: # native
# Native PostgreSQL: use ts_rank_cd with to_tsquery
query_tsquery = " | ".join(tokens)
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
if tags:
params.append(tags)
query = f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
AND search_vector @@ to_tsquery('english', $5)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
"""
# Combined CTE query for both semantic and BM25 across all fact types
# Uses window functions to limit per fact_type per method
results = await conn.fetch(
f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
AND search_vector @@ to_tsquery('english', $5)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
""",
*params,
)
results = await conn.fetch(query, *params)
# Group results by fact_type and source
result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {ft: ([], []) for ft in fact_types}
@@ -96,7 +96,13 @@ class DefaultExtensionContext(ExtensionContext):
async def run_migration(self, schema: str) -> None:
"""Run migrations for a specific schema."""
from hindsight_api.migrations import ensure_embedding_dimension, run_migrations
from hindsight_api.config import get_config
from hindsight_api.migrations import (
ensure_embedding_dimension,
ensure_text_search_extension,
ensure_vector_extension,
run_migrations,
)
# Prefer getting URL from memory engine (handles pg0 case where URL is set after init)
db_url = self._database_url
@@ -107,6 +113,9 @@ class DefaultExtensionContext(ExtensionContext):
run_migrations(db_url, schema=schema)
# Get config for vector extension setting
config = get_config()
# Ensure embedding column dimension matches the model's dimension
# This is needed because migrations create columns with default dimension
if self._memory_engine is not None:
@@ -114,7 +123,15 @@ class DefaultExtensionContext(ExtensionContext):
if embeddings is not None:
dimension = getattr(embeddings, "dimension", None)
if dimension is not None:
ensure_embedding_dimension(db_url, dimension, schema=schema)
ensure_embedding_dimension(
db_url, dimension, schema=schema, vector_extension=config.vector_extension
)
# Ensure vector indexes match the configured extension
ensure_vector_extension(db_url, vector_extension=config.vector_extension, schema=schema)
# Ensure text search columns/indexes match the configured extension
ensure_text_search_extension(db_url, text_search_extension=config.text_search_extension, schema=schema)
def get_memory_engine(self) -> "MemoryEngineInterface":
"""Get the memory engine interface."""
+3
View File
@@ -155,6 +155,8 @@ def main():
config = HindsightConfig(
database_url=config.database_url,
database_schema=config.database_schema,
vector_extension=config.vector_extension,
text_search_extension=config.text_search_extension,
llm_provider=config.llm_provider,
llm_api_key=config.llm_api_key,
llm_model=config.llm_model,
@@ -223,6 +225,7 @@ def main():
reranker_litellm_model=config.reranker_litellm_model,
host=args.host,
port=args.port,
base_path=config.base_path,
log_level=args.log_level,
log_format=config.log_format,
mcp_enabled=config.mcp_enabled,
+414 -12
View File
@@ -33,6 +33,41 @@ logger = logging.getLogger(__name__)
MIGRATION_LOCK_ID = 123456789
def _detect_vector_extension(conn, vector_extension: str = "pgvector") -> str:
"""
Validate vector extension: 'vchord' or 'pgvector'.
Args:
conn: SQLAlchemy connection object
vector_extension: Configured extension ("pgvector" or "vchord")
Returns:
"vchord" or "pgvector"
Raises:
RuntimeError: If configured extension is not installed
"""
# Verify the configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
logger.debug("Using configured vector extension: vchord")
return "vchord"
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
logger.debug("Using configured vector extension: pgvector")
return "pgvector"
else:
raise ValueError(f"Invalid vector_extension: {vector_extension}. Must be 'pgvector' or 'vchord'")
def _get_schema_lock_id(schema: str) -> int:
"""
Generate a unique advisory lock ID for a schema.
@@ -324,6 +359,7 @@ def ensure_embedding_dimension(
database_url: str,
required_dimension: int,
schema: str | None = None,
vector_extension: str = "pgvector",
) -> None:
"""
Ensure the embedding column dimension matches the model's dimension.
@@ -338,6 +374,7 @@ def ensure_embedding_dimension(
database_url: SQLAlchemy database URL
required_dimension: The embedding dimension required by the model
schema: Target PostgreSQL schema name (None for public)
vector_extension: Configured vector extension ("pgvector" or "vchord")
Raises:
RuntimeError: If dimension mismatch with existing data
@@ -361,6 +398,10 @@ def ensure_embedding_dimension(
logger.debug(f"memory_units table does not exist in schema '{schema_name}', skipping dimension check")
return
# Detect which vector extension is available
vector_ext = _detect_vector_extension(conn, vector_extension)
logger.info(f"Using vector extension: {vector_ext}")
# Get current column dimension from pg_attribute
# pgvector stores dimension in atttypmod
current_dim = conn.execute(
@@ -408,8 +449,7 @@ def ensure_embedding_dimension(
# Table is empty, safe to alter column
logger.info(f"Altering embedding column dimension from {current_dimension} to {required_dimension}")
# Drop the HNSW index on embedding column if it exists
# Only drop indexes that use 'hnsw' and reference the 'embedding' column
# Drop existing vector index (works for both HNSW and vchordrq)
conn.execute(
text(f"""
DO $$
@@ -419,7 +459,7 @@ def ensure_embedding_dimension(
SELECT indexname FROM pg_indexes
WHERE schemaname = '{schema_name}'
AND tablename = 'memory_units'
AND indexdef LIKE '%hnsw%'
AND (indexdef LIKE '%hnsw%' OR indexdef LIKE '%vchordrq%')
AND indexdef LIKE '%embedding%'
LOOP
EXECUTE 'DROP INDEX IF EXISTS {schema_name}.' || idx_name;
@@ -434,15 +474,377 @@ def ensure_embedding_dimension(
)
conn.commit()
# Recreate the HNSW index
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_hnsw
ON {schema_name}.memory_units
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64)
""")
)
# Recreate index with appropriate type based on detected extension
if vector_ext == "vchord":
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_vchordrq
ON {schema_name}.memory_units
USING vchordrq (embedding vector_l2_ops)
""")
)
logger.info(f"Created vchordrq index for {required_dimension}-dimensional embeddings")
else: # pgvector
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_hnsw
ON {schema_name}.memory_units
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64)
""")
)
logger.info(f"Created HNSW index for {required_dimension}-dimensional embeddings")
conn.commit()
logger.info(f"Successfully changed embedding dimension to {required_dimension}")
def ensure_vector_extension(
database_url: str,
vector_extension: str = "pgvector",
schema: str | None = None,
) -> None:
"""
Ensure the vector indexes match the configured vector extension.
This function checks the current vector index type in the database
and adjusts it if necessary:
- If index type matches configured extension: no action needed
- If they differ and tables are empty: drop old indexes, recreate with new type
- If they differ and tables have data: raise error with migration guidance
Args:
database_url: SQLAlchemy database URL
vector_extension: Configured vector extension ("pgvector" or "vchord")
schema: Target PostgreSQL schema name (None for public)
Raises:
RuntimeError: If extension mismatch with existing data
"""
schema_name = schema or "public"
engine = create_engine(database_url)
with engine.connect() as conn:
# Detect which vector extension should be used
target_ext = _detect_vector_extension(conn, vector_extension)
logger.info(f"Target vector extension: {target_ext}")
# Tables with vector indexes to check
tables_to_check = [
("memory_units", "idx_memory_units_embedding"),
("learnings", "idx_learnings_embedding"),
("pinned_reflections", "idx_pinned_reflections_embedding"),
]
# Determine target index type
target_index_type = "vchordrq" if target_ext == "vchord" else "hnsw"
mismatched_tables = []
tables_with_data = []
for table_name, index_name in tables_to_check:
# Check if table exists
table_exists = conn.execute(
text("""
SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_schema = :schema AND table_name = :table_name
)
"""),
{"schema": schema_name, "table_name": table_name},
).scalar()
if not table_exists:
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
continue
# Check current index type by querying pg_indexes
current_index_info = conn.execute(
text("""
SELECT indexdef
FROM pg_indexes
WHERE schemaname = :schema
AND tablename = :table_name
AND indexname LIKE :index_pattern
"""),
{"schema": schema_name, "table_name": table_name, "index_pattern": "%embedding%"},
).fetchone()
if not current_index_info:
logger.warning(f"No embedding index found for {table_name}, will create it")
mismatched_tables.append((table_name, index_name, None))
continue
indexdef = current_index_info[0].lower()
if "vchordrq" in indexdef:
current_index_type = "vchordrq"
elif "hnsw" in indexdef:
current_index_type = "hnsw"
else:
logger.warning(f"Unknown index type for {table_name}: {indexdef}")
continue
# Check if index type matches target
if current_index_type != target_index_type:
logger.info(
f"Index type mismatch on {table_name}: current={current_index_type}, target={target_index_type}"
)
mismatched_tables.append((table_name, index_name, current_index_type))
# Check if table has data
row_count = conn.execute(
text(f"SELECT COUNT(*) FROM {schema_name}.{table_name} WHERE embedding IS NOT NULL")
).scalar()
if row_count > 0:
tables_with_data.append((table_name, row_count))
else:
logger.debug(f"Index type OK for {table_name}: {current_index_type}")
# If no mismatches, we're done
if not mismatched_tables:
logger.debug(f"All vector indexes match configured extension: {target_ext}")
return
# If there's data in any mismatched table, raise error
if tables_with_data:
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
raise RuntimeError(
f"Cannot change vector extension from {current_index_type} to {target_index_type}: "
f"the following tables contain data: {table_list}. "
f"To change vector extension, you must either:\n"
f" 1. Re-embed all data: DELETE FROM {schema_name}.memory_units; "
f"DELETE FROM {schema_name}.learnings; DELETE FROM {schema_name}.pinned_reflections; then restart\n"
f" 2. Use the current vector extension (set HINDSIGHT_API_VECTOR_EXTENSION='{current_index_type.replace('vchordrq', 'vchord').replace('hnsw', 'pgvector')}')"
)
# Tables are empty, safe to recreate indexes
logger.info(f"Recreating vector indexes for {target_ext}")
for table_name, index_name, current_type in mismatched_tables:
# Drop existing index if it exists
if current_type:
logger.info(f"Dropping {current_type} index on {table_name}")
conn.execute(text(f"DROP INDEX IF EXISTS {schema_name}.{index_name}"))
# Create new index with appropriate type
if target_ext == "vchord":
logger.info(f"Creating vchordrq index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS {index_name}
ON {schema_name}.{table_name}
USING vchordrq (embedding vector_l2_ops)
""")
)
else: # pgvector
logger.info(f"Creating HNSW index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS {index_name}
ON {schema_name}.{table_name}
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64)
""")
)
conn.commit()
logger.info(f"Successfully migrated vector indexes to {target_ext}")
def ensure_text_search_extension(
database_url: str,
text_search_extension: str = "native",
schema: str | None = None,
) -> None:
"""
Ensure the text search columns and indexes match the configured extension.
This function checks the current search_vector column type and index type
in the database and adjusts them if necessary:
- If they match configured extension: no action needed
- If they differ and tables are empty: drop old column/index, recreate with new type
- If they differ and tables have data: raise error with migration guidance
Args:
database_url: SQLAlchemy database URL
text_search_extension: Configured text search extension ("native" or "vchord")
schema: Target PostgreSQL schema name (None for public)
Raises:
RuntimeError: If extension mismatch with existing data
"""
schema_name = schema or "public"
engine = create_engine(database_url)
with engine.connect() as conn:
# Tables with search_vector columns to check
tables_to_check = [
"memory_units",
"reflections", # Renamed from pinned_reflections in p1k2l3m4n5o6 migration
]
# Determine target column type and index type
if text_search_extension == "vchord":
target_column_type = "bm25vector"
target_index_type = "bm25"
else: # native
target_column_type = "tsvector"
target_index_type = "gin"
mismatched_tables = []
tables_with_data = []
for table_name in tables_to_check:
# Check if table exists
table_exists = conn.execute(
text("""
SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_schema = :schema AND table_name = :table_name
)
"""),
{"schema": schema_name, "table_name": table_name},
).scalar()
if not table_exists:
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
continue
# Get current column type from information_schema
current_column_info = conn.execute(
text("""
SELECT data_type, udt_name
FROM information_schema.columns
WHERE table_schema = :schema
AND table_name = :table_name
AND column_name = 'search_vector'
"""),
{"schema": schema_name, "table_name": table_name},
).fetchone()
if not current_column_info:
logger.warning(f"No search_vector column found for {table_name}, will create it")
mismatched_tables.append((table_name, None, None))
continue
# Check column type (udt_name contains the actual type: tsvector, bm25vector, etc.)
current_column_type = current_column_info[1] # udt_name
# Get current index type
current_index_info = conn.execute(
text("""
SELECT am.amname
FROM pg_indexes pi
JOIN pg_class c ON c.relname = pi.indexname
JOIN pg_am am ON am.oid = c.relam
WHERE pi.schemaname = :schema
AND pi.tablename = :table_name
AND pi.indexname LIKE '%text_search%'
"""),
{"schema": schema_name, "table_name": table_name},
).fetchone()
current_index_type = current_index_info[0] if current_index_info else None
# Check if column and index types match target
column_matches = current_column_type == target_column_type
index_matches = current_index_type == target_index_type if current_index_type else False
if not (column_matches and index_matches):
logger.info(
f"Text search mismatch on {table_name}: "
f"column={current_column_type} (want {target_column_type}), "
f"index={current_index_type} (want {target_index_type})"
)
mismatched_tables.append((table_name, current_column_type, current_index_type))
# Check if table has data
row_count = conn.execute(text(f"SELECT COUNT(*) FROM {schema_name}.{table_name}")).scalar()
if row_count > 0:
tables_with_data.append((table_name, row_count))
else:
logger.debug(f"Text search OK for {table_name}: {current_column_type}/{current_index_type}")
# If no mismatches, we're done
if not mismatched_tables:
logger.debug(f"All text search columns/indexes match configured extension: {text_search_extension}")
return
# If there's data in any mismatched table, raise error
if tables_with_data:
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
current_ext = "native" if mismatched_tables[0][1] == "tsvector" else "vchord"
raise RuntimeError(
f"Cannot change text search extension from {current_ext} to {text_search_extension}: "
f"the following tables contain data: {table_list}. "
f"To change text search extension, you must either:\n"
f" 1. Clear all data: DELETE FROM {schema_name}.memory_units; "
f"DELETE FROM {schema_name}.reflections; then restart\n"
f" 2. Use the current text search extension (set HINDSIGHT_API_TEXT_SEARCH_EXTENSION='{current_ext}')"
)
# Tables are empty, safe to recreate columns/indexes
logger.info(f"Recreating text search columns/indexes for {text_search_extension}")
for table_name, current_col_type, current_idx_type in mismatched_tables:
# Drop existing index if it exists
if current_idx_type:
logger.info(f"Dropping {current_idx_type} index on {table_name}")
conn.execute(
text(f"""
DROP INDEX IF EXISTS {schema_name}.idx_{table_name.replace(".", "_")}_text_search
""")
)
# Drop existing column if it exists
if current_col_type:
logger.info(f"Dropping {current_col_type} column on {table_name}")
conn.execute(text(f"ALTER TABLE {schema_name}.{table_name} DROP COLUMN IF EXISTS search_vector"))
# Create new column with appropriate type
if text_search_extension == "vchord":
logger.info(f"Creating bm25vector column on {table_name}")
# Note: vchord_bm25 extension creates types in bm25_catalog schema
conn.execute(
text(f"ALTER TABLE {schema_name}.{table_name} ADD COLUMN search_vector bm25_catalog.bm25vector")
)
# Create BM25 index
logger.info(f"Creating BM25 index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
ON {schema_name}.{table_name}
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
)
else: # native
logger.info(f"Creating tsvector column on {table_name}")
# Different GENERATED expression for each table
if table_name == "memory_units":
generated_expr = "to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))"
else: # reflections
generated_expr = "to_tsvector('english', COALESCE(name, '') || ' ' || content)"
conn.execute(
text(f"""
ALTER TABLE {schema_name}.{table_name}
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS ({generated_expr}) STORED
""")
)
# Create GIN index
logger.info(f"Creating GIN index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
ON {schema_name}.{table_name}
USING gin(search_vector)
""")
)
conn.commit()
logger.info(f"Successfully migrated text search to {text_search_extension}")
+189
View File
@@ -0,0 +1,189 @@
"""
Integration test for API base path support.
Tests that the API works correctly when deployed with a base path (e.g., /hindsight)
for reverse proxy deployments.
"""
import os
import pytest
import pytest_asyncio
import httpx
from hindsight_api.api import create_app
from hindsight_api.config import clear_config_cache
@pytest_asyncio.fixture
async def api_client_with_base_path(memory):
"""Create an async test client for the FastAPI app with a base path."""
# Set base path in environment
base_path = "/hindsight"
os.environ["HINDSIGHT_API_BASE_PATH"] = base_path
# Clear config cache to force reload with new base_path
clear_config_cache()
# Memory is already initialized by the conftest fixture (with migrations)
app = create_app(memory, initialize_memory=False)
# Use base_url with base path
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(
transport=transport,
base_url=f"http://test{base_path}"
) as client:
yield client
# Cleanup: unset base path
os.environ.pop("HINDSIGHT_API_BASE_PATH", None)
clear_config_cache()
@pytest_asyncio.fixture
async def api_client_without_base_path(memory):
"""Create an async test client for the FastAPI app without a base path (root)."""
# Ensure no base path is set
os.environ.pop("HINDSIGHT_API_BASE_PATH", None)
clear_config_cache()
app = create_app(memory, initialize_memory=False)
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
yield client
@pytest.mark.asyncio
async def test_base_path_health_endpoint(api_client_with_base_path):
"""Test that health endpoint works with base path."""
# With base path set to /hindsight, health should be at /hindsight/health
# But since our client base_url is already http://test/hindsight, we request /health
response = await api_client_with_base_path.get("/health")
assert response.status_code == 200
data = response.json()
assert "status" in data
assert data["status"] in ["ok", "healthy"] # Accept both formats
@pytest.mark.asyncio
async def test_base_path_banks_endpoint(api_client_with_base_path):
"""Test that banks endpoint works with base path."""
response = await api_client_with_base_path.get("/v1/default/banks")
assert response.status_code == 200
data = response.json()
assert "banks" in data
@pytest.mark.asyncio
async def test_base_path_openapi_schema(api_client_with_base_path):
"""Test that OpenAPI schema includes correct base path in servers."""
response = await api_client_with_base_path.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
# Check that servers array includes base path
assert "servers" in openapi_schema
servers = openapi_schema["servers"]
assert len(servers) > 0
# FastAPI should set server URL to the root_path
assert servers[0]["url"] == "/hindsight"
@pytest.mark.asyncio
async def test_base_path_docs_redirect(api_client_with_base_path):
"""Test that /docs redirects correctly with base path."""
# FastAPI docs endpoint should work
response = await api_client_with_base_path.get("/docs", follow_redirects=False)
# Should either return 200 (direct) or 307 (redirect to trailing slash)
assert response.status_code in [200, 307]
@pytest.mark.asyncio
async def test_base_path_metrics(api_client_with_base_path):
"""Test that metrics endpoint works with base path."""
response = await api_client_with_base_path.get("/metrics")
assert response.status_code == 200
# Metrics should be in Prometheus format
assert "# HELP" in response.text or "# TYPE" in response.text
@pytest.mark.asyncio
async def test_base_path_full_workflow(api_client_with_base_path):
"""
Test a full retain/recall workflow with base path.
This ensures that all memory operations work correctly when the API
is deployed with a base path.
"""
bank_id = "test_base_path_bank"
# 1. Create/get bank
response = await api_client_with_base_path.get(f"/v1/default/banks/{bank_id}/profile")
assert response.status_code == 200
# 2. Store a memory
response = await api_client_with_base_path.post(
f"/v1/default/banks/{bank_id}/memories",
json={
"items": [
{
"content": "The API supports base path deployment for reverse proxy use cases.",
"context": "testing base path feature"
}
]
}
)
assert response.status_code == 200
result = response.json()
assert result["success"] is True
# 3. Recall the memory
response = await api_client_with_base_path.post(
f"/v1/default/banks/{bank_id}/memories/recall",
json={
"query": "base path support"
}
)
assert response.status_code == 200
recall_result = response.json()
# API returns "results" not "memories"
assert "results" in recall_result
assert len(recall_result["results"]) > 0
@pytest.mark.asyncio
async def test_without_base_path_still_works(api_client_without_base_path):
"""
Regression test: ensure default behavior (no base path) still works.
This test verifies that when HINDSIGHT_API_BASE_PATH is not set,
the API works at the root path as before.
"""
# Health check at root
response = await api_client_without_base_path.get("/health")
assert response.status_code == 200
# Banks endpoint at root
response = await api_client_without_base_path.get("/v1/default/banks")
assert response.status_code == 200
# OpenAPI schema should have empty or "/" server path
response = await api_client_without_base_path.get("/openapi.json")
assert response.status_code == 200
openapi_schema = response.json()
servers = openapi_schema.get("servers", [])
if servers:
# Server URL should be empty string (root) or "/"
assert servers[0]["url"] in ["", "/"]
@pytest.mark.skip(reason="MCP endpoint routing with base path needs investigation")
@pytest.mark.asyncio
async def test_base_path_mcp_endpoint(api_client_with_base_path):
"""Test that MCP endpoint is accessible with base path."""
bank_id = "test_mcp_bank"
# MCP endpoint should be mounted at /mcp/{bank_id}/
# The MCP server uses a different protocol, so just check the root exists
response = await api_client_with_base_path.get(f"/mcp/{bank_id}/")
# MCP may return various status codes, but should not be 404 (not found)
# Accept 405 (method not allowed), 400 (bad request), etc.
assert response.status_code != 404, "MCP endpoint should exist"
+1 -1
View File
@@ -209,7 +209,7 @@ class TestLargeBatchRetain:
raise
@pytest.mark.asyncio
@pytest.mark.timeout(120)
@pytest.mark.timeout(240) # Increased timeout for VectorChord BM25 tokenization
async def test_batch_chunking_behavior(self, memory_with_mock_llm, request_context):
"""
Test that large batches are properly chunked into sub-batches.
+4
View File
@@ -1,8 +1,12 @@
import type { NextConfig } from "next";
import path from "path";
const basePath = process.env.NEXT_PUBLIC_BASE_PATH || '';
const nextConfig: NextConfig = {
output: 'standalone',
basePath: basePath,
assetPrefix: basePath,
// Disable request logging in production
logging: false,
// Set the monorepo root explicitly to avoid detecting wrong lockfiles in parent directories
@@ -1,4 +1,14 @@
---
title: How We Solved Memory Conflicts in Hindsight
description: Learn how Hindsight handles contradictory information by tracking temporal evolution and preserving history in its memory consolidation system.
authors: [hindsight]
tags: [engineering, memory-systems, conflict-resolution]
image: /img/blog/2026-02-09/consolidation-pipeline.png
date: 2026-02-09
---
# How We Solved Memory Conflicts in Hindsight
One of the hardest problems we tackled in Hindsight was dealing with contradictions. When you're building a memory system for AI agents, reality isn't static. It evolves.
A CRM agent might learn that "Acme Corp is a key prospect" in January, then encounter "Acme Corp is now a paying customer" in March. Naive approaches either lose the history or drown in duplicate facts.
@@ -57,6 +57,64 @@ hindsight-admin run-db-migration
hindsight-admin run-db-migration --schema tenant_acme
```
### Vector Extension
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_API_VECTOR_EXTENSION` | Vector extension to use: `auto`, `pgvector`, or `vchord` | `auto` |
Hindsight supports two PostgreSQL vector extensions:
- **pgvector**: Standard extension, works well for most embeddings (up to ~2000 dimensions)
- **vchord**: Optimized for high-dimensional embeddings (3000+ dimensions), includes BM25 search
When set to `auto` (default), Hindsight automatically detects which extension is installed, preferring vchord if both are available.
**When to use vchord:**
- Using high-dimensional embeddings (e.g., `text-embedding-3-large` with 3072 dimensions)
- Need better performance with large embedding dimensions
- Want to use vchord's BM25 search capabilities
**When to use pgvector:**
- Using standard embedding dimensions (384-1536)
- Prefer the widely-adopted pgvector extension
- Simpler deployment (pgvector is more commonly available)
**Switching extensions:**
If you need to switch from one extension to another:
1. Set `HINDSIGHT_API_VECTOR_EXTENSION` to your desired extension (`pgvector` or `vchord`)
2. If your database has existing data, you'll get an error with migration instructions
3. For empty databases, indexes will be automatically recreated on startup
### Text Search Extension
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native` or `vchord` | `native` |
Hindsight supports two text search backends for BM25 keyword retrieval:
- **native**: PostgreSQL's built-in full-text search (`tsvector` + GIN indexes)
- **vchord**: VectorChord BM25 (`bm25vector` + BM25 indexes) - requires `vchord_bm25` extension
**When to use vchord:**
- Already using vchord for vector search (good integration)
- Want better BM25 ranking performance
- Need advanced tokenization (uses `llmlingua2` tokenizer)
**When to use native:**
- Standard PostgreSQL deployment (no extra extensions)
- Simpler setup and wider compatibility
- Works well for most use cases
**Switching backends:**
To switch from native to vchord (or vice versa):
1. Set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION=vchord` (or `native`)
2. If your database has existing data, you'll get an error with migration instructions
3. For empty databases, the columns/indexes will be automatically recreated on startup
**Note:** VectorChord text search uses the `llmlingua2` tokenizer for multilingual support, while native uses PostgreSQL's English tokenizer.
### LLM Provider
| Variable | Description | Default |
@@ -429,6 +487,7 @@ For advanced authentication (JWT, OAuth, multi-tenant schemas), implement a cust
|----------|-------------|---------|
| `HINDSIGHT_API_HOST` | Bind address | `0.0.0.0` |
| `HINDSIGHT_API_PORT` | Server port | `8888` |
| `HINDSIGHT_API_BASE_PATH` | Base path for API when behind reverse proxy (e.g., `/hindsight`) | `""` (root) |
| `HINDSIGHT_API_WORKERS` | Number of uvicorn worker processes | `1` |
| `HINDSIGHT_API_LOG_LEVEL` | Log level: `debug`, `info`, `warning`, `error` | `info` |
| `HINDSIGHT_API_LOG_FORMAT` | Log format: `text` or `json` (structured logging for cloud platforms) | `text` |
@@ -649,12 +708,78 @@ The Control Plane is the web UI for managing memory banks.
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_CP_DATAPLANE_API_URL` | URL of the API service | `http://localhost:8888` |
| `NEXT_PUBLIC_BASE_PATH` | Base path for Control Plane UI when behind reverse proxy (e.g., `/hindsight`) | `""` (root) |
```bash
# Point Control Plane to a remote API service
export HINDSIGHT_CP_DATAPLANE_API_URL=http://api.example.com:8888
```
### Reverse Proxy / Subpath Deployment
To deploy Hindsight under a subpath (e.g., `example.com/hindsight/`):
1. Set both environment variables to the same path:
```bash
HINDSIGHT_API_BASE_PATH=/hindsight
NEXT_PUBLIC_BASE_PATH=/hindsight
```
2. Configure your reverse proxy to:
- Forward `/hindsight/*` requests to Hindsight
- Preserve the full path in forwarded requests
- Set appropriate proxy headers (X-Forwarded-Proto, X-Forwarded-For)
**Example: Nginx Configuration**
```nginx
location /hindsight/ {
proxy_pass http://localhost:8888/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
```
**Example: Traefik Configuration**
```yaml
http:
routers:
hindsight:
rule: "PathPrefix(`/hindsight`)"
service: hindsight
middlewares:
- hindsight-stripprefix
middlewares:
hindsight-stripprefix:
stripPrefix:
prefixes:
- "/hindsight"
services:
hindsight:
loadBalancer:
servers:
- url: "http://localhost:8888"
```
**Important Notes:**
- The base path must start with `/` and should NOT end with `/`
- Both API and Control Plane should use the same base path
- After setting environment variables, restart both services
- OpenAPI docs will be available at `<base-path>/docs` (e.g., `/hindsight/docs`)
**Complete Examples:**
See `docker/compose-examples/` directory for:
- Nginx configuration files (`simple.conf`, `api-and-control-plane.conf`)
- Docker Compose setups (`docker-compose.yml`, `reverse-proxy-only.yml`)
- Traefik and other reverse proxy examples
- Full deployment documentation
---
## Example .env File
@@ -3,7 +3,7 @@
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
:::tip Don't want to manage infrastructure?
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
:::
## Prerequisites
+118 -20
View File
@@ -71,6 +71,7 @@ curl -X POST http://localhost:8888/mcp \
-H "Authorization: Bearer your-secret-key" \
-H "X-Bank-Id: my-bank" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'
```
@@ -78,10 +79,10 @@ If the key is missing or invalid, requests will receive a `401 Unauthorized` res
## Bank Selection
Specify the memory bank via:
The memory bank is resolved in this priority order:
1. **X-Bank-Id header** (recommended): `--header "X-Bank-Id: my-bank"`
2. **URL path**: `http://localhost:8888/mcp/my-bank/`
1. **URL path** (highest priority): `http://localhost:8888/mcp/my-bank/`
2. **X-Bank-Id header**: `--header "X-Bank-Id: my-bank"`
3. **Default**: Uses `HINDSIGHT_MCP_BANK_ID` env var (default: "default")
## Per-Bank Endpoints
@@ -93,6 +94,19 @@ This design:
- **Enforces isolation** — each MCP connection is scoped to a single bank
- **Enables multi-tenant setups** — connect different users to different endpoints
## Two Modes
The MCP server operates in two modes depending on the URL:
| Mode | URL | Tools | bank_id |
|------|-----|-------|---------|
| **Single-bank** | `/mcp/{bank_id}/` | Memory + mental model tools | Implicit from URL |
| **Multi-bank** | `/mcp/` | All tools including bank management | Explicit `bank_id` parameter on each tool |
**Single-bank mode** (recommended) scopes all operations to the bank in the URL. Tools don't expose a `bank_id` parameter.
**Multi-bank mode** exposes all tools with an optional `bank_id` parameter, plus bank management tools (`list_banks`, `create_bank`).
---
## Available Tools
@@ -105,6 +119,7 @@ Store information to long-term memory.
|-----------|------|----------|-------------|
| `content` | string | Yes | The fact or memory to store |
| `context` | string | No | Category for the memory (default: `general`) |
| `timestamp` | string | No | ISO 8601 timestamp for when the event occurred |
**Example:**
```json
@@ -133,6 +148,7 @@ Search memories to provide personalized responses.
|-----------|------|----------|-------------|
| `query` | string | Yes | Natural language search query |
| `max_results` | integer | No | Maximum results to return (default: 10) |
| `max_tokens` | integer | No | Maximum tokens to return (default: 4096) |
**Example:**
```json
@@ -144,21 +160,6 @@ Search memories to provide personalized responses.
}
```
**Response:**
```json
{
"results": [
{
"id": "fact_abc123",
"text": "User prefers Python over JavaScript for backend development",
"type": "world",
"context": "programming_preferences",
"event_date": null
}
]
}
```
**When to use:**
- Start of conversation to recall relevant context
- Before making recommendations
@@ -195,10 +196,107 @@ Generate thoughtful analysis by synthesizing stored memories with the bank's per
---
### create_mental_model
Create a mental model — a living document that stays current with your memories. Mental models are pre-computed reflections that get automatically refreshed as new memories are stored.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `name` | string | Yes | Human-readable name for the mental model |
| `source_query` | string | Yes | The query used to generate and refresh the model |
| `tags` | list[string] | No | Tags for organizing and filtering models |
| `max_tokens` | integer | No | Maximum tokens for model content (default: 2048) |
**Example:**
```json
{
"name": "create_mental_model",
"arguments": {
"name": "Team Directory",
"source_query": "Who works here and what do they do?",
"tags": ["team", "people"]
}
}
```
Content generation runs asynchronously. The response includes an `operation_id` to track progress.
---
### list_mental_models
List all mental models in a bank, optionally filtered by tags.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `tags` | list[string] | No | Filter models by tags |
---
### get_mental_model
Retrieve a specific mental model by ID, including its full content.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `mental_model_id` | string | Yes | The ID of the mental model to retrieve |
---
### update_mental_model
Update a mental model's metadata or settings.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `mental_model_id` | string | Yes | The ID of the mental model to update |
| `name` | string | No | New name |
| `source_query` | string | No | New source query |
| `tags` | list[string] | No | New tags |
| `max_tokens` | integer | No | New max tokens |
---
### delete_mental_model
Permanently delete a mental model.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `mental_model_id` | string | Yes | The ID of the mental model to delete |
---
### refresh_mental_model
Re-generate a mental model's content from the latest memories. Runs asynchronously.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `mental_model_id` | string | Yes | The ID of the mental model to refresh |
---
### list_banks (multi-bank mode only)
List all available memory banks.
---
### create_bank (multi-bank mode only)
Create a new memory bank or retrieve an existing one.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `bank_id` | string | Yes | The ID for the new bank |
---
## Integration with AI Assistants
The MCP server can be used with any MCP-compatible AI assistant. See the [Authentication](#authentication) section above for Claude Code and Claude Desktop configuration examples.
Each user can have their own configuration pointing to their personal memory bank using either:
- The `X-Bank-Id` header (recommended)
- A bank-specific URL path like `/mcp/alice/`
- A bank-specific URL path like `/mcp/alice/` (recommended)
- The `X-Bank-Id` header
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
## Cloud Mode Setup
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
### Prerequisites
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
2. An API key from your team admin
3. A bank ID for your project (e.g., `team-acme-frontend`)
+2 -2
View File
@@ -226,7 +226,7 @@ const config: Config = {
className: 'navbar-item-changelog',
},
{
href: 'https://vectorize.io/hindsight/cloud',
href: 'https://ui.hindsight.vectorize.io/signup',
position: 'right',
label: 'Hindsight Cloud',
className: 'navbar-item-cloud',
@@ -285,7 +285,7 @@ const config: Config = {
},
{
label: 'Hindsight Cloud',
href: 'https://vectorize.io/hindsight/cloud',
href: 'https://ui.hindsight.vectorize.io/signup',
},
],
},
@@ -0,0 +1,162 @@
import React, { useState, useCallback } from 'react';
import styles from './styles.module.css';
export default function CopyPageButton(): JSX.Element | null {
const [copied, setCopied] = useState(false);
const copyPageAsMarkdown = useCallback(async () => {
try {
// Get the page content
const contentElement = document.querySelector('.markdown');
if (!contentElement) return;
// Convert HTML to markdown-like text
let markdown = '';
// Add title
const title = document.querySelector('h1')?.textContent;
if (title) {
markdown += `# ${title}\n\n`;
}
// Extract text content from the markdown container
const extractMarkdown = (element: Element): string => {
let text = '';
const processNode = (node: Node): string => {
if (node.nodeType === Node.TEXT_NODE) {
return node.textContent || '';
}
if (node.nodeType === Node.ELEMENT_NODE) {
const el = node as Element;
const tagName = el.tagName.toLowerCase();
const children = Array.from(el.childNodes).map(processNode).join('');
switch (tagName) {
case 'h1':
return `# ${children}\n\n`;
case 'h2':
return `## ${children}\n\n`;
case 'h3':
return `### ${children}\n\n`;
case 'h4':
return `#### ${children}\n\n`;
case 'h5':
return `##### ${children}\n\n`;
case 'h6':
return `###### ${children}\n\n`;
case 'p':
return `${children}\n\n`;
case 'ul':
return `${children}\n`;
case 'ol':
return `${children}\n`;
case 'li':
const parent = el.parentElement;
const isOrdered = parent?.tagName.toLowerCase() === 'ol';
if (isOrdered) {
const index = Array.from(parent?.children || []).indexOf(el) + 1;
return `${index}. ${children}\n`;
}
return `- ${children}\n`;
case 'code':
const isBlock = el.parentElement?.tagName.toLowerCase() === 'pre';
if (isBlock) {
const lang = el.className.replace('language-', '');
return `\`\`\`${lang}\n${children}\n\`\`\`\n\n`;
}
return `\`${children}\``;
case 'pre':
return children; // Already handled by code block
case 'blockquote':
return children.split('\n').map(line => `> ${line}`).join('\n') + '\n\n';
case 'a':
const href = el.getAttribute('href') || '';
return `[${children}](${href})`;
case 'strong':
case 'b':
return `**${children}**`;
case 'em':
case 'i':
return `*${children}*`;
case 'br':
return '\n';
case 'hr':
return '---\n\n';
case 'table':
return `${children}\n`;
case 'thead':
case 'tbody':
return children;
case 'tr':
return `${children}|\n`;
case 'th':
case 'td':
return `| ${children} `;
case 'img':
const src = el.getAttribute('src') || '';
const alt = el.getAttribute('alt') || '';
return `![${alt}](${src})`;
default:
return children;
}
}
return '';
};
Array.from(element.childNodes).forEach(node => {
text += processNode(node);
});
return text;
};
// Skip the title h1 if it's already added
const contentToCopy = Array.from(contentElement.children)
.filter(child => !(child.tagName === 'H1' && child.textContent === title))
.map(child => extractMarkdown(child))
.join('');
markdown += contentToCopy;
// Clean up excessive newlines
markdown = markdown.replace(/\n{3,}/g, '\n\n').trim();
// Copy to clipboard
await navigator.clipboard.writeText(markdown);
setCopied(true);
setTimeout(() => setCopied(false), 2000);
} catch (error) {
console.error('Failed to copy page content:', error);
}
}, []);
return (
<button
className={`${styles.copyPageButton} ${copied ? styles.copied : ''}`}
onClick={copyPageAsMarkdown}
aria-label="Copy page as markdown"
title="Copy page as markdown"
>
{copied ? (
<>
<svg width="16" height="16" viewBox="0 0 16 16" fill="currentColor">
<path d="M12.736 3.97a.733.733 0 0 1 1.047 0c.286.289.29.756.01 1.05L7.88 12.01a.733.733 0 0 1-1.065.02L3.217 8.384a.757.757 0 0 1 0-1.06.733.733 0 0 1 1.047 0l3.052 3.093 5.4-6.425a.247.247 0 0 1 .02-.022Z"/>
</svg>
<span className={styles.buttonText}>Copied!</span>
</>
) : (
<>
<svg width="16" height="16" viewBox="0 0 16 16" fill="currentColor">
<path d="M4 2a2 2 0 0 1 2-2h8a2 2 0 0 1 2 2v8a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V2zm2-1a1 1 0 0 0-1 1v8a1 1 0 0 0 1 1h8a1 1 0 0 0 1-1V2a1 1 0 0 0-1-1H6z"/>
<path d="M2 5a1 1 0 0 0-1 1v8a1 1 0 0 0 1 1h8a1 1 0 0 0 1-1v-1h1v1a2 2 0 0 1-2 2H2a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h1v1H2z"/>
</svg>
<span className={styles.buttonText}>Copy page</span>
</>
)}
</button>
);
}
@@ -0,0 +1,55 @@
.copyPageButton {
display: flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
background: transparent;
border: 1px solid var(--ifm-color-emphasis-300);
border-radius: 6px;
color: var(--ifm-font-color-base);
font-size: 14px;
font-weight: 500;
cursor: pointer;
transition: all 0.2s ease;
white-space: nowrap;
}
.copyPageButton:hover {
background-color: var(--ifm-color-emphasis-100);
border-color: var(--ifm-color-emphasis-400);
}
.copyPageButton:active {
transform: translateY(1px);
}
.copyPageButton.copied {
background-color: var(--ifm-color-success-contrast-background);
border-color: var(--ifm-color-success);
color: var(--ifm-color-success-darkest);
}
.copyPageButton.copied:hover {
background-color: var(--ifm-color-success-contrast-background);
border-color: var(--ifm-color-success);
}
.buttonText {
margin: 0 4px;
}
/* Dark mode adjustments */
[data-theme='dark'] .copyPageButton {
border-color: var(--ifm-color-emphasis-400);
}
[data-theme='dark'] .copyPageButton:hover {
background-color: var(--ifm-color-emphasis-200);
border-color: var(--ifm-color-emphasis-500);
}
[data-theme='dark'] .copyPageButton.copied {
background-color: var(--ifm-color-success-dark);
border-color: var(--ifm-color-success);
color: var(--ifm-color-success-contrast-foreground);
}
@@ -0,0 +1,21 @@
import React from 'react';
import DocItemContent from '@theme-original/DocItem/Content';
import type DocItemContentType from '@theme/DocItem/Content';
import type { WrapperProps } from '@docusaurus/types';
import CopyPageButton from '@site/src/components/CopyPageButton';
import styles from './styles.module.css';
type Props = WrapperProps<typeof DocItemContentType>;
export default function DocItemContentWrapper(props: Props): JSX.Element {
return (
<>
<div className={styles.docItemHeader}>
<div className={styles.docItemActions}>
<CopyPageButton />
</div>
</div>
<DocItemContent {...props} />
</>
);
}
@@ -0,0 +1,20 @@
.docItemHeader {
display: flex;
justify-content: flex-end;
align-items: center;
margin-bottom: 1rem;
padding-bottom: 0.5rem;
border-bottom: 1px solid var(--ifm-color-emphasis-200);
}
.docItemActions {
display: flex;
gap: 8px;
align-items: center;
}
@media (max-width: 768px) {
.docItemHeader {
margin-bottom: 0.75rem;
}
}
@@ -3,7 +3,7 @@
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
:::tip Don't want to manage infrastructure?
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
:::
## Prerequisites
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
## Cloud Mode Setup
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
### Prerequisites
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
2. An API key from your team admin
3. A bank ID for your project (e.g., `team-acme-frontend`)
@@ -3,7 +3,7 @@
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
:::tip Don't want to manage infrastructure?
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
:::
## Prerequisites
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
## Cloud Mode Setup
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
### Prerequisites
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
2. An API key from your team admin
3. A bank ID for your project (e.g., `team-acme-frontend`)
+52 -11
View File
@@ -2,19 +2,60 @@
E2E and integration tests for Hindsight API that require a running server.
## Running Tests
## Test Types
1. Start the API server:
```bash
./scripts/dev/start-api.sh
```
### 1. Tests with External Server
Tests like `test_mcp_e2e.py` expect a server to already be running.
2. Run the tests:
```bash
cd hindsight-integration-tests
HINDSIGHT_API_URL=http://localhost:8888 uv run pytest tests/ -v
```
**Running:**
```bash
# Start the API server
./scripts/dev/start-api.sh
# Run tests
cd hindsight-integration-tests
HINDSIGHT_API_URL=http://localhost:8888 uv run pytest tests/test_mcp_e2e.py -v
```
### 2. Self-Contained Tests
Tests like `test_base_path_deployment.py` manage their own server lifecycle and use docker-compose.
**Running:**
```bash
cd hindsight-integration-tests
# Run with pytest
uv run pytest tests/test_base_path_deployment.py -v
# Or run directly for nice output
uv run python tests/test_base_path_deployment.py
```
**Requirements:**
- Docker and docker-compose installed (for reverse proxy test)
- No nginx required on host!
**What it tests:**
- ✅ API with base path (direct server)
- ✅ Full reverse proxy via docker-compose + Nginx
- ✅ Regression: API without base path
- ✅ Full retain/recall workflow
These tests:
- Start their own API servers on dedicated ports (18888-18891)
- Use docker-compose to test actual deployment scenarios
- Run in parallel with other tests (no port conflicts)
- Clean up automatically
## Running All Tests
```bash
cd hindsight-integration-tests
uv run pytest tests/ -v
```
This runs both types. Self-contained tests won't conflict with the external server.
## Environment Variables
- `HINDSIGHT_API_URL` - Base URL of the running Hindsight API (default: `http://localhost:8888`)
- `HINDSIGHT_API_URL` - Base URL for external-server tests (default: `http://localhost:8888`)
@@ -0,0 +1,494 @@
"""
Integration test for base path deployment using Docker Compose.
This test validates that Hindsight works correctly when deployed
behind a reverse proxy with path-based routing using the actual
Docker Compose examples from docker/compose-examples/.
Tests:
1. API with base path (direct, no proxy)
2. Full stack via docker-compose with Nginx reverse proxy
3. Regression: API without base path still works
Requirements:
- Docker and docker-compose installed
- No nginx required on host!
"""
import asyncio
import os
import subprocess
import sys
import time
from pathlib import Path
import httpx
import pytest
# Paths
REPO_ROOT = Path(__file__).parent.parent.parent
API_PATH = REPO_ROOT / "hindsight-api"
COMPOSE_EXAMPLES_PATH = REPO_ROOT / "docker" / "docker-compose" / "nginx"
# Add hindsight-api to path for direct API testing
sys.path.insert(0, str(API_PATH))
def run_command(cmd: list[str], cwd: str | Path | None = None, env: dict | None = None) -> subprocess.CompletedProcess:
"""Run a command and return the result."""
return subprocess.run(
cmd,
cwd=cwd,
env=env,
capture_output=True,
text=True,
)
def check_docker_available() -> bool:
"""Check if Docker is available."""
result = run_command(["docker", "info"])
return result.returncode == 0
def get_docker_compose_command() -> list[str]:
"""Get the docker-compose command (modern or legacy)."""
import shutil
# Try modern docker compose plugin first
if shutil.which("docker"):
result = run_command(["docker", "compose", "version"])
if result.returncode == 0:
return ["docker", "compose"]
# Fall back to legacy docker-compose
if shutil.which("docker-compose"):
return ["docker-compose"]
raise RuntimeError("docker-compose not available")
def check_docker_compose_available() -> bool:
"""Check if docker-compose is available."""
try:
get_docker_compose_command()
return True
except RuntimeError:
return False
class APIServer:
"""Helper to manage API server lifecycle for direct testing."""
def __init__(self, base_path: str | None = None, port: int = 18888):
self.base_path = base_path
self.port = port
self.process = None
self.env = os.environ.copy()
if base_path:
self.env["HINDSIGHT_API_BASE_PATH"] = base_path
def start(self):
"""Start the API server."""
cmd = [
"uv",
"run",
"--directory",
str(API_PATH),
"hindsight-api",
"--host",
"0.0.0.0",
"--port",
str(self.port),
]
log_file = f"/tmp/hindsight-api-{self.port}.log"
self.log_file = open(log_file, "w")
self.process = subprocess.Popen(
cmd, env=self.env, stdout=self.log_file, stderr=subprocess.STDOUT
)
# Wait for server to be ready
base_url = f"http://localhost:{self.port}"
if self.base_path:
health_url = f"{base_url}{self.base_path}/health"
else:
health_url = f"{base_url}/health"
for _ in range(60): # 60 second timeout
try:
response = httpx.get(health_url, timeout=2.0)
if response.status_code == 200:
return
except (httpx.ConnectError, httpx.ReadTimeout):
pass
time.sleep(1)
# Failed to start
self.log_file.flush()
with open(log_file) as f:
print(f"API server failed to start. Logs:\n{f.read()}")
raise RuntimeError(f"API server failed to start on port {self.port}")
def stop(self):
"""Stop the API server."""
if self.process:
self.process.terminate()
try:
self.process.wait(timeout=10)
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait()
self.process = None
if hasattr(self, "log_file") and self.log_file:
self.log_file.close()
def __enter__(self):
self.start()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.stop()
class DockerComposeStack:
"""Helper to manage docker-compose stack lifecycle."""
def __init__(self, compose_file: Path, project_name: str = "hindsight-test"):
self.compose_file = compose_file
self.project_name = project_name
self.compose_cmd = get_docker_compose_command()
self.env = os.environ.copy()
# Set required env vars for docker-compose
self.env["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "test-key")
self.env["HINDSIGHT_API_LLM_PROVIDER"] = os.environ.get("HINDSIGHT_API_LLM_PROVIDER", "mock")
self.env["HINDSIGHT_API_LLM_MODEL"] = os.environ.get("HINDSIGHT_API_LLM_MODEL", "mock-model")
def start(self, timeout: int = 120):
"""Start the docker-compose stack."""
print(f"Starting docker-compose stack: {self.compose_file.name}")
# Pull images first (but don't fail if it doesn't work)
run_command(
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "pull"],
cwd=self.compose_file.parent,
env=self.env,
)
# Start services
result = run_command(
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "up", "-d", "--build"],
cwd=self.compose_file.parent,
env=self.env,
)
if result.returncode != 0:
print(f"Failed to start docker-compose:\nSTDOUT: {result.stdout}\nSTDERR: {result.stderr}")
raise RuntimeError("Failed to start docker-compose stack")
# Wait for services to be healthy
start_time = time.time()
while time.time() - start_time < timeout:
result = run_command(
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "ps", "--format", "json"],
cwd=self.compose_file.parent,
env=self.env,
)
if result.returncode == 0:
# Give it a few more seconds to fully initialize
time.sleep(5)
return
time.sleep(2)
# Timeout - show logs and fail
self.show_logs()
raise RuntimeError(f"Docker compose stack failed to start within {timeout}s")
def show_logs(self):
"""Show docker-compose logs."""
result = run_command(
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "logs", "--tail=100"],
cwd=self.compose_file.parent,
env=self.env,
)
print(f"Docker compose logs:\n{result.stdout}\n{result.stderr}")
def stop(self):
"""Stop and remove the docker-compose stack."""
print(f"Stopping docker-compose stack: {self.compose_file.name}")
result = run_command(
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "down", "-v"],
cwd=self.compose_file.parent,
env=self.env,
)
if result.returncode != 0:
print(f"Warning: Failed to stop docker-compose:\n{result.stderr}")
def __enter__(self):
self.start()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self.stop()
def test_api_without_base_path():
"""Regression test: API works at root path (default behavior)."""
with APIServer(base_path=None, port=18888) as server:
base_url = f"http://localhost:{server.port}"
# Health check
response = httpx.get(f"{base_url}/health")
assert response.status_code == 200
assert "status" in response.json()
# API endpoints
response = httpx.get(f"{base_url}/v1/default/banks")
assert response.status_code == 200
assert "banks" in response.json()
# OpenAPI docs
response = httpx.get(f"{base_url}/docs")
assert response.status_code == 200
def test_api_with_base_path_direct():
"""Test API with base path configuration (direct, no proxy)."""
base_path = "/hindsight"
with APIServer(base_path=base_path, port=18889) as server:
base_url = f"http://localhost:{server.port}"
# Base path SHOULD work
response = httpx.get(f"{base_url}{base_path}/health")
assert response.status_code == 200
assert "status" in response.json()
# API endpoints with base path
response = httpx.get(f"{base_url}{base_path}/v1/default/banks")
assert response.status_code == 200
assert "banks" in response.json()
# OpenAPI docs with base path
response = httpx.get(f"{base_url}{base_path}/docs")
assert response.status_code == 200
# OpenAPI schema should have correct server URL
response = httpx.get(f"{base_url}{base_path}/openapi.json")
assert response.status_code == 200
openapi = response.json()
assert "servers" in openapi
assert openapi["servers"][0]["url"] == base_path
@pytest.mark.skipif(
not check_docker_compose_available(),
reason="docker-compose not available"
)
def test_reverse_proxy_simple_config():
"""
Test reverse proxy deployment using docker-compose with Nginx.
This creates a minimal test setup with:
- API server running on HOST via uv (not Docker - faster, no image build needed!)
- Nginx container that proxies to the host API
This tests the actual reverse proxy scenario without requiring the
heavy Hindsight Docker image.
"""
base_path = "/hindsight"
api_port = 18890
# Start API on host with base path
with APIServer(base_path=base_path, port=api_port):
# Create test docker-compose file (nginx only)
test_compose = COMPOSE_EXAMPLES_PATH / "test-reverse-proxy.yml"
# Determine host address for nginx to reach host machine
# host.docker.internal works on Docker Desktop (Mac/Windows)
# On Linux, we use host network mode
import platform
if platform.system() == "Linux":
network_mode = "host"
api_host = "localhost"
nginx_port = 18080 # With host mode, nginx must listen on 18080 directly
port_mapping = "" # No port mapping with host mode
else:
network_mode = "bridge"
api_host = "host.docker.internal"
nginx_port = 80 # With bridge mode, nginx listens on 80 and is mapped
port_mapping = """ ports:
- "18080:80"
"""
compose_content = f"""version: '3.8'
services:
nginx:
image: nginx:alpine
{port_mapping} volumes:
- ./test-nginx.conf:/etc/nginx/nginx.conf:ro
network_mode: {network_mode}
"""
# Create test nginx config
nginx_config = f"""events {{
worker_connections 1024;
}}
http {{
server {{
listen {nginx_port};
server_name localhost;
location {base_path}/ {{
proxy_pass http://{api_host}:{api_port};
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}}
}}
}}
"""
# Write test files
test_compose.write_text(compose_content)
test_nginx_conf = COMPOSE_EXAMPLES_PATH / "test-nginx.conf"
test_nginx_conf.write_text(nginx_config)
try:
# Start nginx via docker-compose
with DockerComposeStack(test_compose, project_name="hindsight-base-path-test"):
proxy_url = "http://localhost:18080"
# Give nginx a moment to start
time.sleep(2)
# Test through nginx proxy
response = httpx.get(f"{proxy_url}{base_path}/health", timeout=10.0)
assert response.status_code == 200
assert "status" in response.json()
# API endpoints through proxy
response = httpx.get(f"{proxy_url}{base_path}/v1/default/banks", timeout=10.0)
assert response.status_code == 200
assert "banks" in response.json()
# OpenAPI docs through proxy
response = httpx.get(f"{proxy_url}{base_path}/docs", timeout=10.0)
assert response.status_code == 200
finally:
# Cleanup test files
test_compose.unlink(missing_ok=True)
test_nginx_conf.unlink(missing_ok=True)
@pytest.mark.asyncio
async def test_full_workflow_with_base_path():
"""Test full retain/recall workflow through base path."""
base_path = "/hindsight"
bank_id = "integration_test_bank"
with APIServer(base_path=base_path, port=18891) as server:
base_url = f"http://localhost:{server.port}{base_path}"
async with httpx.AsyncClient(base_url=base_url, timeout=30.0) as client:
# 1. Get bank profile (creates if needed)
response = await client.get(f"/v1/default/banks/{bank_id}/profile")
assert response.status_code == 200
# 2. Store a memory
response = await client.post(
f"/v1/default/banks/{bank_id}/memories",
json={
"items": [
{
"content": "Hindsight supports deployment under custom base paths for reverse proxy scenarios.",
"context": "integration test"
}
]
},
)
assert response.status_code == 200
result = response.json()
assert result["success"] is True
# 3. Recall the memory
response = await client.post(
f"/v1/default/banks/{bank_id}/memories/recall",
json={"query": "base path deployment"}
)
assert response.status_code == 200
recall_result = response.json()
assert "results" in recall_result
# Should find our memory
assert len(recall_result["results"]) > 0
if __name__ == "__main__":
"""Run tests directly with python."""
import sys
print("=" * 70)
print("Hindsight Base Path Integration Tests")
print("=" * 70)
print()
all_passed = True
# Test 1: Without base path
print("Test 1: API without base path (regression test)")
print("-" * 70)
try:
test_api_without_base_path()
print("✅ PASSED\n")
except Exception as e:
print(f"❌ FAILED: {e}\n")
all_passed = False
# Test 2: With base path (direct)
print("Test 2: API with base path (direct, no proxy)")
print("-" * 70)
try:
test_api_with_base_path_direct()
print("✅ PASSED\n")
except Exception as e:
print(f"❌ FAILED: {e}\n")
all_passed = False
# Test 3: Docker compose reverse proxy
print("Test 3: Reverse proxy via docker-compose")
print("-" * 70)
if check_docker_available() and check_docker_compose_available():
try:
test_reverse_proxy_simple_config()
print("✅ PASSED\n")
except Exception as e:
print(f"❌ FAILED: {e}\n")
import traceback
traceback.print_exc()
all_passed = False
else:
print("⚠️ SKIPPED: Docker or docker-compose not available\n")
# Test 4: Full workflow
print("Test 4: Full retain/recall workflow with base path")
print("-" * 70)
try:
asyncio.run(test_full_workflow_with_base_path())
print("✅ PASSED\n")
except Exception as e:
print(f"❌ FAILED: {e}\n")
all_passed = False
# Summary
print("=" * 70)
if all_passed:
print("✅ All tests passed!")
sys.exit(0)
else:
print("❌ Some tests failed")
sys.exit(1)
@@ -3,7 +3,7 @@
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
:::tip Don't want to manage infrastructure?
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
:::
## Prerequisites
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
## Cloud Mode Setup
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
### Prerequisites
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
2. An API key from your team admin
3. A bank ID for your project (e.g., `team-acme-frontend`)