Compare commits
8
Commits
python-deser
...
vchord
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
90170d79e0 | ||
|
|
043b304f27 | ||
|
|
8b274d10ad | ||
|
|
a713b68b1f | ||
|
|
93ddd41621 | ||
|
|
7ee229ba23 | ||
|
|
29c0890f22 | ||
|
|
a1f22dabd2 |
@@ -31,6 +31,12 @@ HINDSIGHT_API_HOST=0.0.0.0
|
||||
HINDSIGHT_API_PORT=8888
|
||||
HINDSIGHT_API_LOG_LEVEL=info
|
||||
|
||||
# Base Path / Reverse Proxy Support (Optional)
|
||||
# Set these when deploying behind a reverse proxy with path-based routing
|
||||
# Example: To deploy at example.com/hindsight/, set both to "/hindsight"
|
||||
# HINDSIGHT_API_BASE_PATH=/hindsight
|
||||
# NEXT_PUBLIC_BASE_PATH=/hindsight
|
||||
|
||||
# Database (Optional - uses embedded pg0 by default)
|
||||
# HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@host:5432/db
|
||||
# HINDSIGHT_API_DATABASE_SCHEMA=public # PostgreSQL schema name (default: public)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||

|
||||
|
||||
[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://vectorize.io/hindsight/cloud)
|
||||
[Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)
|
||||
|
||||
[](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
|
||||
[](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# Nginx Reverse Proxy with Custom Base Path
|
||||
|
||||
Deploy Hindsight API under `/hindsight` (or any custom path) using Nginx reverse proxy.
|
||||
|
||||
## Quick Start (Published Image - API Only)
|
||||
|
||||
```bash
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
- **API:** http://localhost:8080/hindsight/docs
|
||||
- **Control Plane:** http://localhost:9999 (direct access, not proxied)
|
||||
|
||||
## Full Stack with Custom Base Path (Requires Build)
|
||||
|
||||
**Important:** You cannot rebuild from the published image with build args. You must build from source.
|
||||
|
||||
### Build from Source with Custom Base Path
|
||||
|
||||
1. **Clone the repository** (if you haven't):
|
||||
```bash
|
||||
git clone https://github.com/vectorize-io/hindsight.git
|
||||
cd hindsight
|
||||
```
|
||||
|
||||
2. **Build with base path**:
|
||||
```bash
|
||||
docker build \
|
||||
--build-arg NEXT_PUBLIC_BASE_PATH=/hindsight \
|
||||
-f docker/standalone/Dockerfile \
|
||||
-t hindsight:custom \
|
||||
.
|
||||
```
|
||||
|
||||
3. **Update docker-compose.yml** to use your built image:
|
||||
```yaml
|
||||
services:
|
||||
hindsight:
|
||||
image: hindsight:custom # ← Change this
|
||||
environment:
|
||||
HINDSIGHT_API_BASE_PATH: /hindsight
|
||||
NEXT_PUBLIC_BASE_PATH: /hindsight
|
||||
```
|
||||
|
||||
4. **Update nginx.conf** to handle Control Plane routes (see below)
|
||||
|
||||
5. **Run**:
|
||||
```bash
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
### Required nginx.conf for Full Stack
|
||||
|
||||
Replace the current `nginx.conf` with this to proxy both API and Control Plane:
|
||||
|
||||
```nginx
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
upstream hindsight_api { server hindsight:8888; }
|
||||
upstream hindsight_cp { server hindsight:9999; }
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# API
|
||||
location ~ ^/hindsight/(docs|openapi\.json|health|metrics|v1|mcp) {
|
||||
proxy_pass http://hindsight_api;
|
||||
proxy_set_header Host $http_host;
|
||||
}
|
||||
|
||||
# Control Plane static files
|
||||
location ~ ^/hindsight/_next/ {
|
||||
proxy_pass http://hindsight_cp;
|
||||
proxy_set_header Host $http_host;
|
||||
}
|
||||
|
||||
# Control Plane UI
|
||||
location /hindsight {
|
||||
proxy_pass http://hindsight_cp;
|
||||
proxy_set_header Host $http_host;
|
||||
}
|
||||
|
||||
location = / { return 301 /hindsight; }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Why Build is Required
|
||||
|
||||
Next.js requires `basePath` at **build time**. The published image was built without a custom base path, so you must rebuild from source with the `NEXT_PUBLIC_BASE_PATH` build arg to deploy the Control Plane under a subpath.
|
||||
|
||||
The API works without rebuild because `HINDSIGHT_API_BASE_PATH` is a runtime environment variable.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Hindsight API deployment with Nginx reverse proxy (API-only)
|
||||
#
|
||||
# This example deploys Hindsight API under the path /hindsight with:
|
||||
# - Hindsight standalone image (API + Control Plane + embedded pg0)
|
||||
# - Nginx reverse proxy (API only)
|
||||
#
|
||||
# Quick Start:
|
||||
# docker-compose -f docker/docker-compose/nginx/docker-compose.yml up
|
||||
#
|
||||
# Access:
|
||||
# API (via nginx): http://localhost:8080/hindsight/docs
|
||||
# Control Plane (direct): http://localhost:9999
|
||||
#
|
||||
# For full stack deployment (API + Control Plane both under /hindsight):
|
||||
# See README.md in this directory for instructions on building with basePath.
|
||||
#
|
||||
# Note: This configuration uses the published image (no build required).
|
||||
# Control Plane is served directly because Next.js basePath requires
|
||||
# build-time configuration. See README.md for the full stack option.
|
||||
|
||||
services:
|
||||
# Hindsight (API + Control Plane + embedded pg0)
|
||||
hindsight:
|
||||
image: ghcr.io/vectorize-io/hindsight:latest
|
||||
ports:
|
||||
- "9999:9999" # Control Plane (direct access, not proxied)
|
||||
environment:
|
||||
# API base path for reverse proxy
|
||||
HINDSIGHT_API_BASE_PATH: /hindsight
|
||||
|
||||
# LLM configuration
|
||||
# Using mock provider for testing (no API key needed)
|
||||
# For production, set OPENAI_API_KEY or ANTHROPIC_API_KEY and use a real provider
|
||||
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-mock}
|
||||
HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-not-needed-for-mock}
|
||||
HINDSIGHT_API_LLM_MODEL: ${HINDSIGHT_API_LLM_MODEL:-mock-model}
|
||||
|
||||
# Production examples (uncomment and set appropriate API key):
|
||||
# HINDSIGHT_API_LLM_PROVIDER: openai
|
||||
# HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY}
|
||||
# HINDSIGHT_API_LLM_MODEL: gpt-4o-mini
|
||||
|
||||
# HINDSIGHT_API_LLM_PROVIDER: anthropic
|
||||
# HINDSIGHT_API_LLM_API_KEY: ${ANTHROPIC_API_KEY}
|
||||
# HINDSIGHT_API_LLM_MODEL: claude-sonnet-4-20250514
|
||||
|
||||
# Server config
|
||||
HINDSIGHT_API_HOST: 0.0.0.0
|
||||
HINDSIGHT_API_PORT: 8888
|
||||
HINDSIGHT_API_LOG_LEVEL: info
|
||||
|
||||
# Control Plane config
|
||||
HINDSIGHT_CP_DATAPLANE_API_URL: http://localhost:8888
|
||||
volumes:
|
||||
# Persist embedded pg0 database
|
||||
- hindsight_data:/app/data
|
||||
# Note: Ports not exposed - access via Nginx at localhost:8080/hindsight/
|
||||
# To debug directly, uncomment these ports:
|
||||
# ports:
|
||||
# - "8888:8888" # API
|
||||
# - "9999:9999" # Control Plane
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8888/hindsight/health"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
networks:
|
||||
- hindsight
|
||||
|
||||
# Nginx reverse proxy
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- "8080:80"
|
||||
volumes:
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
hindsight:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- hindsight
|
||||
|
||||
volumes:
|
||||
hindsight_data:
|
||||
|
||||
networks:
|
||||
hindsight:
|
||||
@@ -0,0 +1,40 @@
|
||||
# Nginx configuration for API-only reverse proxy
|
||||
# Control Plane accessed directly (not through nginx)
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
# Logging
|
||||
access_log /var/log/nginx/access.log;
|
||||
error_log /var/log/nginx/error.log;
|
||||
|
||||
# Upstream - Hindsight API
|
||||
upstream hindsight_api {
|
||||
server hindsight:8888;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
# API endpoints - forward with /hindsight prefix
|
||||
location /hindsight/ {
|
||||
proxy_pass http://hindsight_api;
|
||||
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
|
||||
# Redirect root to API docs
|
||||
location = / {
|
||||
return 301 /hindsight/docs;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
name: hindsight
|
||||
# Docker Compose file for Hindsight with PostgreSQL and vectorchord
|
||||
# docker compose -f docker/docker-compose/docker-compose.yaml down && sleep 2 && docker compose -f docker/docker-compose/docker-compose.yaml up -d
|
||||
# Make sure to set the required environment variables before running:
|
||||
# - HINDSIGHT_DB_PASSWORD: Password for the PostgreSQL user
|
||||
# - Configure LLM provider variables as needed (see below in the hindsight service)
|
||||
#
|
||||
# Usage:
|
||||
# docker compose up -d
|
||||
#
|
||||
# Optional environment variables with defaults:
|
||||
# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
|
||||
# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
|
||||
# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
|
||||
# - HINDSIGHT_DB_VERSION: PostgreSQL version (default: 18)
|
||||
|
||||
services:
|
||||
db:
|
||||
# Use a PostgreSQL-Image with vectorchord extension pre-installed
|
||||
image: tensorchord/vchord-suite:pg${HINDSIGHT_DB_VERSION:-18-latest}
|
||||
container_name: hindsight-db
|
||||
restart: always
|
||||
# Expose PostgreSQL port
|
||||
ports:
|
||||
- "5436:5432"
|
||||
environment:
|
||||
POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
|
||||
POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
|
||||
POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
|
||||
volumes:
|
||||
- pg_data:/var/lib/postgresql/${HINDSIGHT_DB_VERSION:-18}/docker
|
||||
networks:
|
||||
- hindsight-net
|
||||
|
||||
vectorchord-init:
|
||||
image: tensorchord/vchord-suite:pg18-latest
|
||||
#container_name: vectorchord-init
|
||||
depends_on:
|
||||
- db
|
||||
environment:
|
||||
- PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
|
||||
command: >
|
||||
bash -c "
|
||||
echo 'Waiting for PostgreSQL to be ready...';
|
||||
until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
|
||||
echo 'PostgreSQL is unavailable - sleeping';
|
||||
sleep 2;
|
||||
done;
|
||||
echo 'PostgreSQL is ready - creating hindsight_db database';
|
||||
psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
|
||||
echo 'Creating extensions in hindsight_db database';
|
||||
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord CASCADE;';
|
||||
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;';
|
||||
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;';
|
||||
echo 'Creating llmlingua2 tokenizer';
|
||||
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c \"SELECT create_tokenizer('llmlingua2', \\$\\$ model = \\\"llmlingua2\\\" \\$\\$);\" 2>/dev/null || echo 'Tokenizer already exists or creation skipped';
|
||||
echo 'Database and extensions created successfully';
|
||||
"
|
||||
restart: "no"
|
||||
networks:
|
||||
- hindsight-net
|
||||
|
||||
hindsight:
|
||||
image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
|
||||
container_name: hindsight-app
|
||||
ports:
|
||||
- "8888:8888"
|
||||
- "9999:9999"
|
||||
environment:
|
||||
# LLM Configuration (uses OpenAI for testing vchord)
|
||||
# LLM configuration
|
||||
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
|
||||
HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
|
||||
|
||||
# Database Configuration
|
||||
HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
|
||||
|
||||
# Vector and Text Search Extensions
|
||||
HINDSIGHT_API_VECTOR_EXTENSION: vchord
|
||||
HINDSIGHT_API_TEXT_SEARCH_EXTENSION: vchord
|
||||
|
||||
depends_on:
|
||||
- db
|
||||
networks:
|
||||
- hindsight-net
|
||||
|
||||
|
||||
networks:
|
||||
hindsight-net:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
pg_data:
|
||||
@@ -112,6 +112,10 @@ RUN rm -f package-lock.json && sed -i '/"@vectorize-io\/hindsight-client":/d' pa
|
||||
# Copy built SDK directly into node_modules (more reliable than npm link in Docker)
|
||||
COPY --from=sdk-builder /app/hindsight-clients/typescript ./node_modules/@vectorize-io/hindsight-client
|
||||
|
||||
# Accept base path as build argument for reverse proxy deployments
|
||||
# Usage: docker build --build-arg NEXT_PUBLIC_BASE_PATH=/hindsight ...
|
||||
ARG NEXT_PUBLIC_BASE_PATH=""
|
||||
|
||||
# Build Control Plane - run next build first, then custom standalone copy
|
||||
# (The build:standalone script expects a specific path structure that differs in Docker)
|
||||
RUN npm exec -- next build
|
||||
|
||||
@@ -6,6 +6,7 @@ Create Date: 2025-11-27 11:54:19.228030
|
||||
|
||||
"""
|
||||
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
|
||||
import sqlalchemy as sa
|
||||
@@ -21,6 +22,61 @@ branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def _detect_vector_extension() -> str:
|
||||
"""
|
||||
Detect or validate vector extension: 'vchord' or 'pgvector'.
|
||||
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
|
||||
"""
|
||||
conn = op.get_bind()
|
||||
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
|
||||
|
||||
# Validate configured extension is installed
|
||||
if vector_extension == "vchord":
|
||||
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
|
||||
if not vchord_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
|
||||
)
|
||||
return "vchord"
|
||||
elif vector_extension == "pgvector":
|
||||
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
|
||||
if not pgvector_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
|
||||
)
|
||||
return "pgvector"
|
||||
else:
|
||||
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
|
||||
|
||||
|
||||
def _detect_text_search_extension() -> str:
|
||||
"""
|
||||
Detect or validate text search extension: 'native' or 'vchord'.
|
||||
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
|
||||
Creates the extension if needed.
|
||||
"""
|
||||
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
|
||||
|
||||
if text_search_extension == "vchord":
|
||||
# Create vchord_bm25 extension if not exists
|
||||
try:
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
|
||||
except Exception:
|
||||
# Extension might already exist or user lacks permissions - verify it exists
|
||||
conn = op.get_bind()
|
||||
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
|
||||
if not result:
|
||||
# Extension truly doesn't exist - re-raise the error
|
||||
raise
|
||||
return "vchord"
|
||||
elif text_search_extension == "native":
|
||||
return "native"
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Upgrade schema - create all tables from scratch."""
|
||||
|
||||
@@ -166,11 +222,23 @@ def upgrade() -> None:
|
||||
)
|
||||
|
||||
# Add search_vector column for full-text search
|
||||
op.execute("""
|
||||
ALTER TABLE memory_units
|
||||
ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
|
||||
""")
|
||||
# Type depends on configured text search backend
|
||||
text_search_ext = _detect_text_search_extension()
|
||||
|
||||
if text_search_ext == "vchord":
|
||||
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
|
||||
# Note: vchord_bm25 extension creates types in bm25_catalog schema
|
||||
op.execute("""
|
||||
ALTER TABLE memory_units
|
||||
ADD COLUMN search_vector bm25_catalog.bm25vector
|
||||
""")
|
||||
else: # native
|
||||
# Native PostgreSQL: tsvector with automatic generation
|
||||
op.execute("""
|
||||
ALTER TABLE memory_units
|
||||
ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
|
||||
""")
|
||||
|
||||
op.create_index("idx_memory_units_bank_id", "memory_units", ["bank_id"])
|
||||
op.create_index("idx_memory_units_document_id", "memory_units", ["document_id"])
|
||||
@@ -200,19 +268,39 @@ def upgrade() -> None:
|
||||
["bank_id", sa.text("event_date DESC")],
|
||||
postgresql_where=sa.text("fact_type = 'observation'"),
|
||||
)
|
||||
op.create_index(
|
||||
"idx_memory_units_embedding",
|
||||
"memory_units",
|
||||
["embedding"],
|
||||
postgresql_using="hnsw",
|
||||
postgresql_ops={"embedding": "vector_cosine_ops"},
|
||||
)
|
||||
# Create vector index - conditional based on available extension
|
||||
vector_ext = _detect_vector_extension()
|
||||
|
||||
# Create BM25 full-text search index on search_vector
|
||||
op.execute("""
|
||||
CREATE INDEX idx_memory_units_text_search ON memory_units
|
||||
USING gin(search_vector)
|
||||
""")
|
||||
if vector_ext == "vchord":
|
||||
# Use vchordrq index for vchord (supports high-dimensional embeddings)
|
||||
op.execute("""
|
||||
CREATE INDEX idx_memory_units_embedding ON memory_units
|
||||
USING vchordrq (embedding vector_l2_ops)
|
||||
""")
|
||||
else: # pgvector
|
||||
# Use HNSW index for pgvector
|
||||
op.create_index(
|
||||
"idx_memory_units_embedding",
|
||||
"memory_units",
|
||||
["embedding"],
|
||||
postgresql_using="hnsw",
|
||||
postgresql_ops={"embedding": "vector_cosine_ops"},
|
||||
)
|
||||
|
||||
# Create full-text search index on search_vector
|
||||
# Index type depends on text search backend
|
||||
if text_search_ext == "vchord":
|
||||
# VectorChord BM25 index
|
||||
op.execute("""
|
||||
CREATE INDEX idx_memory_units_text_search ON memory_units
|
||||
USING bm25 (search_vector bm25_catalog.bm25_ops)
|
||||
""")
|
||||
else: # native
|
||||
# Native PostgreSQL GIN index
|
||||
op.execute("""
|
||||
CREATE INDEX idx_memory_units_text_search ON memory_units
|
||||
USING gin(search_vector)
|
||||
""")
|
||||
|
||||
op.execute("""
|
||||
CREATE MATERIALIZED VIEW memory_units_bm25 AS
|
||||
|
||||
+126
-21
@@ -10,9 +10,11 @@ This migration:
|
||||
3. Adds consolidation tracking columns to the 'banks' table
|
||||
"""
|
||||
|
||||
import os
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import context, op
|
||||
from sqlalchemy import text
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "n9i0j1k2l3m4"
|
||||
@@ -27,10 +29,71 @@ def _get_schema_prefix() -> str:
|
||||
return f'"{schema}".' if schema else ""
|
||||
|
||||
|
||||
def _detect_vector_extension() -> str:
|
||||
"""
|
||||
Detect or validate vector extension: 'vchord' or 'pgvector'.
|
||||
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
|
||||
"""
|
||||
conn = op.get_bind()
|
||||
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
|
||||
|
||||
# Validate configured extension is installed
|
||||
if vector_extension == "vchord":
|
||||
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
|
||||
if not vchord_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
|
||||
)
|
||||
return "vchord"
|
||||
elif vector_extension == "pgvector":
|
||||
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
|
||||
if not pgvector_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
|
||||
)
|
||||
return "pgvector"
|
||||
else:
|
||||
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
|
||||
|
||||
|
||||
def _detect_text_search_extension() -> str:
|
||||
"""
|
||||
Detect or validate text search extension: 'native' or 'vchord'.
|
||||
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
|
||||
Creates the extension if needed.
|
||||
"""
|
||||
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
|
||||
|
||||
if text_search_extension == "vchord":
|
||||
# Create vchord_bm25 extension if not exists
|
||||
try:
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
|
||||
except Exception:
|
||||
# Extension might already exist or user lacks permissions - verify it exists
|
||||
conn = op.get_bind()
|
||||
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
|
||||
if not result:
|
||||
# Extension truly doesn't exist - re-raise the error
|
||||
raise
|
||||
return "vchord"
|
||||
elif text_search_extension == "native":
|
||||
return "native"
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
|
||||
)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Create learnings and pinned_reflections tables."""
|
||||
schema = _get_schema_prefix()
|
||||
|
||||
# Detect which vector extension is available
|
||||
vector_ext = _detect_vector_extension()
|
||||
|
||||
# Detect which text search extension to use
|
||||
text_search_ext = _detect_text_search_extension()
|
||||
|
||||
# 1. Create learnings table
|
||||
op.execute(f"""
|
||||
CREATE TABLE {schema}learnings (
|
||||
@@ -57,18 +120,39 @@ def upgrade() -> None:
|
||||
|
||||
# Indexes for learnings
|
||||
op.execute(f"CREATE INDEX idx_learnings_bank_id ON {schema}learnings(bank_id)")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_learnings_embedding ON {schema}learnings
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
""")
|
||||
|
||||
# Create vector index based on detected extension
|
||||
if vector_ext == "vchord":
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_learnings_embedding ON {schema}learnings
|
||||
USING vchordrq (embedding vector_l2_ops)
|
||||
""")
|
||||
else: # pgvector
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_learnings_embedding ON {schema}learnings
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
""")
|
||||
|
||||
op.execute(f"CREATE INDEX idx_learnings_tags ON {schema}learnings USING GIN(tags)")
|
||||
|
||||
# Full-text search for learnings
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
|
||||
""")
|
||||
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
|
||||
if text_search_ext == "vchord":
|
||||
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
|
||||
# Note: vchord_bm25 extension creates types in bm25_catalog schema
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}learnings ADD COLUMN search_vector bm25_catalog.bm25vector
|
||||
""")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_learnings_text_search ON {schema}learnings
|
||||
USING bm25 (search_vector bm25_catalog.bm25_ops)
|
||||
""")
|
||||
else: # native
|
||||
# Native PostgreSQL: tsvector with automatic generation
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
|
||||
""")
|
||||
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
|
||||
|
||||
# 2. Create pinned_reflections table
|
||||
op.execute(f"""
|
||||
@@ -94,21 +178,42 @@ def upgrade() -> None:
|
||||
|
||||
# Indexes for pinned_reflections
|
||||
op.execute(f"CREATE INDEX idx_pinned_reflections_bank_id ON {schema}pinned_reflections(bank_id)")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
""")
|
||||
|
||||
# Create vector index based on detected extension
|
||||
if vector_ext == "vchord":
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
|
||||
USING vchordrq (embedding vector_l2_ops)
|
||||
""")
|
||||
else: # pgvector
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_embedding ON {schema}pinned_reflections
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
""")
|
||||
|
||||
op.execute(f"CREATE INDEX idx_pinned_reflections_tags ON {schema}pinned_reflections USING GIN(tags)")
|
||||
|
||||
# Full-text search for pinned_reflections
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
|
||||
""")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
|
||||
USING gin(search_vector)
|
||||
""")
|
||||
if text_search_ext == "vchord":
|
||||
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT/UPDATE)
|
||||
# Note: vchord_bm25 extension creates types in bm25_catalog schema
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector bm25_catalog.bm25vector
|
||||
""")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
|
||||
USING bm25 (search_vector bm25_catalog.bm25_ops)
|
||||
""")
|
||||
else: # native
|
||||
# Native PostgreSQL: tsvector with automatic generation
|
||||
op.execute(f"""
|
||||
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
|
||||
""")
|
||||
op.execute(f"""
|
||||
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
|
||||
USING gin(search_vector)
|
||||
""")
|
||||
|
||||
# 3. Add consolidation tracking columns to banks table
|
||||
op.execute(f"""
|
||||
|
||||
@@ -1491,6 +1491,9 @@ def create_app(
|
||||
logging.info("Memory system closed")
|
||||
|
||||
from hindsight_api import __version__
|
||||
from hindsight_api.config import get_config
|
||||
|
||||
config = get_config()
|
||||
|
||||
app = FastAPI(
|
||||
title="Hindsight HTTP API",
|
||||
@@ -1504,6 +1507,7 @@ def create_app(
|
||||
"url": "https://www.apache.org/licenses/LICENSE-2.0.html",
|
||||
},
|
||||
lifespan=lifespan,
|
||||
root_path=config.base_path,
|
||||
)
|
||||
|
||||
# IMPORTANT: Set memory on app.state immediately, don't wait for lifespan
|
||||
|
||||
@@ -114,9 +114,39 @@ def create_mcp_server(memory: MemoryEngine, multi_bank: bool = True) -> FastMCP:
|
||||
logger.info(f"Loading MCP extension: {mcp_extension.__class__.__name__}")
|
||||
mcp_extension.register_tools(mcp, memory)
|
||||
|
||||
# Make all tools tolerant of extra arguments from LLMs (e.g., "explanation")
|
||||
_make_tools_tolerant(mcp)
|
||||
|
||||
return mcp
|
||||
|
||||
|
||||
def _make_tools_tolerant(mcp: FastMCP) -> None:
|
||||
"""Wrap all tool run methods to strip unknown arguments before validation.
|
||||
|
||||
LLMs frequently add extra fields like "explanation" or "reasoning" to tool calls.
|
||||
FastMCP's Pydantic TypeAdapter rejects these with "Unexpected keyword argument".
|
||||
This wraps each tool's run() to filter arguments to only known parameters.
|
||||
"""
|
||||
try:
|
||||
for name, tool in mcp._tool_manager._tools.items():
|
||||
if hasattr(tool, "parameters") and tool.parameters:
|
||||
allowed = set(tool.parameters.get("properties", {}).keys())
|
||||
original_run = tool.run
|
||||
|
||||
async def _tolerant_run(arguments, _allowed=allowed, _orig=original_run):
|
||||
extra_keys = set(arguments.keys()) - _allowed
|
||||
if extra_keys:
|
||||
logger.debug(f"Stripping unknown arguments from tool call: {extra_keys}")
|
||||
arguments = {k: v for k, v in arguments.items() if k in _allowed}
|
||||
return await _orig(arguments)
|
||||
|
||||
# FunctionTool is a Pydantic model with extra='forbid', so use
|
||||
# object.__setattr__ to bypass Pydantic's setter validation.
|
||||
object.__setattr__(tool, "run", _tolerant_run)
|
||||
except (AttributeError, KeyError) as e:
|
||||
logger.warning(f"Could not make tools tolerant of extra arguments: {e}")
|
||||
|
||||
|
||||
class MCPMiddleware:
|
||||
"""ASGI middleware that intercepts MCP requests and routes to appropriate MCP server.
|
||||
|
||||
@@ -142,6 +172,11 @@ class MCPMiddleware:
|
||||
- No bank management tools (list_banks, create_bank)
|
||||
- Recommended for agent isolation
|
||||
|
||||
Bank ID resolution priority:
|
||||
1. URL path (e.g., /mcp/{bank_id}/) → single-bank mode
|
||||
2. X-Bank-Id header → multi-bank mode
|
||||
3. HINDSIGHT_MCP_BANK_ID env var → multi-bank mode (default: "default")
|
||||
|
||||
Examples:
|
||||
# Single-bank mode (recommended for agent isolation)
|
||||
claude mcp add --transport http my-agent http://localhost:8888/mcp/my-agent-bank/ \\
|
||||
@@ -242,20 +277,25 @@ class MCPMiddleware:
|
||||
_current_schema.set(tenant_context.schema_name) if tenant_context and tenant_context.schema_name else None
|
||||
)
|
||||
|
||||
# Try to get bank_id from header first (for Claude Code compatibility)
|
||||
bank_id = self._get_header(scope, "X-Bank-Id")
|
||||
# Resolve bank_id: path takes priority over header.
|
||||
# Path = user's explicit connection endpoint (e.g., /mcp/my-bank/).
|
||||
# X-Bank-Id header = per-request override for multi-bank mode only.
|
||||
bank_id = None
|
||||
bank_id_from_path = False
|
||||
|
||||
# If no header, try to extract from path: /{bank_id}/...
|
||||
new_path = path
|
||||
if not bank_id and path.startswith("/") and len(path) > 1:
|
||||
|
||||
# First, try to extract from path: /{bank_id}/...
|
||||
if path.startswith("/") and len(path) > 1:
|
||||
parts = path[1:].split("/", 1)
|
||||
if parts[0]:
|
||||
# First segment looks like a bank_id
|
||||
bank_id = parts[0]
|
||||
bank_id_from_path = True
|
||||
new_path = "/" + parts[1] if len(parts) > 1 else "/"
|
||||
|
||||
# If no path-based bank_id, try X-Bank-Id header (multi-bank mode)
|
||||
if not bank_id:
|
||||
bank_id = self._get_header(scope, "X-Bank-Id")
|
||||
|
||||
# Fall back to default bank_id
|
||||
if not bank_id:
|
||||
bank_id = DEFAULT_BANK_ID
|
||||
|
||||
@@ -107,8 +107,12 @@ ENV_RERANKER_MAX_CANDIDATES = "HINDSIGHT_API_RERANKER_MAX_CANDIDATES"
|
||||
ENV_RERANKER_FLASHRANK_MODEL = "HINDSIGHT_API_RERANKER_FLASHRANK_MODEL"
|
||||
ENV_RERANKER_FLASHRANK_CACHE_DIR = "HINDSIGHT_API_RERANKER_FLASHRANK_CACHE_DIR"
|
||||
|
||||
ENV_VECTOR_EXTENSION = "HINDSIGHT_API_VECTOR_EXTENSION"
|
||||
ENV_TEXT_SEARCH_EXTENSION = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION"
|
||||
|
||||
ENV_HOST = "HINDSIGHT_API_HOST"
|
||||
ENV_PORT = "HINDSIGHT_API_PORT"
|
||||
ENV_BASE_PATH = "HINDSIGHT_API_BASE_PATH"
|
||||
ENV_LOG_LEVEL = "HINDSIGHT_API_LOG_LEVEL"
|
||||
ENV_LOG_FORMAT = "HINDSIGHT_API_LOG_FORMAT"
|
||||
ENV_WORKERS = "HINDSIGHT_API_WORKERS"
|
||||
@@ -223,6 +227,12 @@ DEFAULT_RERANKER_FLASHRANK_CACHE_DIR = None # Use default cache directory
|
||||
DEFAULT_EMBEDDINGS_COHERE_MODEL = "embed-english-v3.0"
|
||||
DEFAULT_RERANKER_COHERE_MODEL = "rerank-english-v3.0"
|
||||
|
||||
# Vector extension (pgvector vs vchord)
|
||||
DEFAULT_VECTOR_EXTENSION = "pgvector" # Options: "pgvector", "vchord"
|
||||
|
||||
# Text search extension (native PostgreSQL vs vchord BM25)
|
||||
DEFAULT_TEXT_SEARCH_EXTENSION = "native" # Options: "native", "vchord"
|
||||
|
||||
# LiteLLM defaults
|
||||
DEFAULT_LITELLM_API_BASE = "http://localhost:4000"
|
||||
DEFAULT_EMBEDDINGS_LITELLM_MODEL = "text-embedding-3-small"
|
||||
@@ -230,6 +240,7 @@ DEFAULT_RERANKER_LITELLM_MODEL = "cohere/rerank-english-v3.0"
|
||||
|
||||
DEFAULT_HOST = "0.0.0.0"
|
||||
DEFAULT_PORT = 8888
|
||||
DEFAULT_BASE_PATH = "" # Empty string = root path
|
||||
DEFAULT_LOG_LEVEL = "info"
|
||||
DEFAULT_LOG_FORMAT = "text" # Options: "text", "json"
|
||||
DEFAULT_WORKERS = 1
|
||||
@@ -358,6 +369,8 @@ class HindsightConfig:
|
||||
# Database
|
||||
database_url: str
|
||||
database_schema: str
|
||||
vector_extension: str # "pgvector" or "vchord"
|
||||
text_search_extension: str # "native" or "vchord"
|
||||
|
||||
# LLM (default, used as fallback for per-operation config)
|
||||
llm_provider: str
|
||||
@@ -440,6 +453,7 @@ class HindsightConfig:
|
||||
# Server
|
||||
host: str
|
||||
port: int
|
||||
base_path: str
|
||||
log_level: str
|
||||
log_format: str
|
||||
mcp_enabled: bool
|
||||
@@ -497,6 +511,20 @@ class HindsightConfig:
|
||||
|
||||
def validate(self) -> None:
|
||||
"""Validate configuration values and raise errors for invalid combinations."""
|
||||
# Validate vector_extension
|
||||
valid_extensions = ("pgvector", "vchord")
|
||||
if self.vector_extension not in valid_extensions:
|
||||
raise ValueError(
|
||||
f"Invalid vector_extension: {self.vector_extension}. Must be one of: {', '.join(valid_extensions)}"
|
||||
)
|
||||
|
||||
# Validate text_search_extension
|
||||
valid_text_search = ("native", "vchord")
|
||||
if self.text_search_extension not in valid_text_search:
|
||||
raise ValueError(
|
||||
f"Invalid text_search_extension: {self.text_search_extension}. Must be one of: {', '.join(valid_text_search)}"
|
||||
)
|
||||
|
||||
# RETAIN_MAX_COMPLETION_TOKENS must be greater than RETAIN_CHUNK_SIZE
|
||||
# to ensure the LLM has enough output capacity to extract facts from chunks
|
||||
if self.retain_max_completion_tokens <= self.retain_chunk_size:
|
||||
@@ -522,6 +550,8 @@ class HindsightConfig:
|
||||
# Database
|
||||
database_url=os.getenv(ENV_DATABASE_URL, DEFAULT_DATABASE_URL),
|
||||
database_schema=os.getenv(ENV_DATABASE_SCHEMA, DEFAULT_DATABASE_SCHEMA),
|
||||
vector_extension=os.getenv(ENV_VECTOR_EXTENSION, DEFAULT_VECTOR_EXTENSION).lower(),
|
||||
text_search_extension=os.getenv(ENV_TEXT_SEARCH_EXTENSION, DEFAULT_TEXT_SEARCH_EXTENSION).lower(),
|
||||
# LLM
|
||||
llm_provider=llm_provider,
|
||||
llm_api_key=os.getenv(ENV_LLM_API_KEY),
|
||||
@@ -662,6 +692,7 @@ class HindsightConfig:
|
||||
# Server
|
||||
host=os.getenv(ENV_HOST, DEFAULT_HOST),
|
||||
port=int(os.getenv(ENV_PORT, DEFAULT_PORT)),
|
||||
base_path=os.getenv(ENV_BASE_PATH, DEFAULT_BASE_PATH),
|
||||
log_level=os.getenv(ENV_LOG_LEVEL, DEFAULT_LOG_LEVEL),
|
||||
log_format=os.getenv(ENV_LOG_FORMAT, DEFAULT_LOG_FORMAT).lower(),
|
||||
mcp_enabled=os.getenv(ENV_MCP_ENABLED, str(DEFAULT_MCP_ENABLED)).lower() == "true",
|
||||
|
||||
@@ -18,6 +18,7 @@ import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from ...config import get_config
|
||||
from ..memory_engine import fq_table
|
||||
from ..retain import embedding_utils
|
||||
from .prompts import (
|
||||
@@ -1016,15 +1017,33 @@ async def _create_observation_directly(
|
||||
|
||||
t0 = time.time()
|
||||
observation_id = uuid.uuid4()
|
||||
|
||||
# Query varies based on text search backend
|
||||
config = get_config()
|
||||
if config.text_search_extension == "vchord":
|
||||
# VectorChord: manually tokenize and insert search_vector
|
||||
query = f"""
|
||||
INSERT INTO {fq_table("memory_units")} (
|
||||
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
|
||||
tags, event_date, occurred_start, occurred_end, mentioned_at, search_vector
|
||||
)
|
||||
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10,
|
||||
tokenize($3, 'llmlingua2')::bm25_catalog.bm25vector)
|
||||
RETURNING id
|
||||
"""
|
||||
else: # native
|
||||
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
|
||||
query = f"""
|
||||
INSERT INTO {fq_table("memory_units")} (
|
||||
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
|
||||
tags, event_date, occurred_start, occurred_end, mentioned_at
|
||||
)
|
||||
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
|
||||
RETURNING id
|
||||
"""
|
||||
|
||||
row = await conn.fetchrow(
|
||||
f"""
|
||||
INSERT INTO {fq_table("memory_units")} (
|
||||
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
|
||||
tags, event_date, occurred_start, occurred_end, mentioned_at
|
||||
)
|
||||
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
|
||||
RETURNING id
|
||||
""",
|
||||
query,
|
||||
observation_id,
|
||||
bank_id,
|
||||
observation_text,
|
||||
|
||||
@@ -968,7 +968,12 @@ class MemoryEngine(MemoryEngineInterface):
|
||||
|
||||
# Run database migrations if enabled
|
||||
if self._run_migrations:
|
||||
from ..migrations import ensure_embedding_dimension, run_migrations
|
||||
from ..migrations import (
|
||||
ensure_embedding_dimension,
|
||||
ensure_text_search_extension,
|
||||
ensure_vector_extension,
|
||||
run_migrations,
|
||||
)
|
||||
|
||||
if not self.db_url:
|
||||
raise ValueError("Database URL is required for migrations")
|
||||
@@ -976,30 +981,43 @@ class MemoryEngine(MemoryEngineInterface):
|
||||
# Migrate all schemas from the tenant extension
|
||||
# The tenant extension is the single source of truth for which schemas exist
|
||||
logger.info("Running database migrations...")
|
||||
try:
|
||||
tenants = await self._tenant_extension.list_tenants()
|
||||
if tenants:
|
||||
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
try:
|
||||
run_migrations(self.db_url, schema=schema)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to migrate schema {schema}: {e}")
|
||||
logger.info("Schema migrations completed")
|
||||
tenants = await self._tenant_extension.list_tenants()
|
||||
if tenants:
|
||||
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
run_migrations(self.db_url, schema=schema)
|
||||
logger.info("Schema migrations completed")
|
||||
|
||||
# Ensure embedding column dimension matches the model's dimension
|
||||
# This is done after migrations and after embeddings.initialize()
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
try:
|
||||
ensure_embedding_dimension(self.db_url, self.embeddings.dimension, schema=schema)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure embedding dimension for schema {schema}: {e}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to run schema migrations: {e}")
|
||||
# Get config for vector extension setting
|
||||
config = get_config()
|
||||
|
||||
# Ensure embedding column dimension matches the model's dimension
|
||||
# This is done after migrations and after embeddings.initialize()
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
ensure_embedding_dimension(
|
||||
self.db_url,
|
||||
self.embeddings.dimension,
|
||||
schema=schema,
|
||||
vector_extension=config.vector_extension,
|
||||
)
|
||||
|
||||
# Ensure vector indexes match the configured extension
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
ensure_vector_extension(self.db_url, vector_extension=config.vector_extension, schema=schema)
|
||||
|
||||
# Ensure text search columns/indexes match the configured extension
|
||||
for tenant in tenants:
|
||||
schema = tenant.schema
|
||||
if schema:
|
||||
ensure_text_search_extension(
|
||||
self.db_url, text_search_extension=config.text_search_extension, schema=schema
|
||||
)
|
||||
|
||||
logger.info(f"Connecting to PostgreSQL at {mask_network_location(self.db_url)}")
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ Handles insertion of facts into the database.
|
||||
import json
|
||||
import logging
|
||||
|
||||
from ...config import get_config
|
||||
from ..memory_engine import fq_table
|
||||
from .fact_extraction import _sanitize_text
|
||||
from .types import ProcessedFact
|
||||
@@ -70,28 +71,58 @@ async def insert_facts_batch(
|
||||
|
||||
# Batch insert all facts
|
||||
# Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
|
||||
results = await conn.fetch(
|
||||
f"""
|
||||
WITH input_data AS (
|
||||
SELECT * FROM unnest(
|
||||
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
|
||||
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
|
||||
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
|
||||
)
|
||||
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
|
||||
SELECT
|
||||
$1,
|
||||
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id,
|
||||
COALESCE(
|
||||
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
|
||||
'{{}}'::varchar[]
|
||||
# Query varies based on text search backend
|
||||
config = get_config()
|
||||
if config.text_search_extension == "vchord":
|
||||
# VectorChord: manually tokenize and insert search_vector
|
||||
query = f"""
|
||||
WITH input_data AS (
|
||||
SELECT * FROM unnest(
|
||||
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
|
||||
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
|
||||
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
|
||||
)
|
||||
FROM input_data
|
||||
RETURNING id
|
||||
""",
|
||||
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags, search_vector)
|
||||
SELECT
|
||||
$1,
|
||||
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id,
|
||||
COALESCE(
|
||||
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
|
||||
'{{}}'::varchar[]
|
||||
),
|
||||
tokenize(COALESCE(text, '') || ' ' || COALESCE(context, ''), 'llmlingua2')::bm25_catalog.bm25vector
|
||||
FROM input_data
|
||||
RETURNING id
|
||||
"""
|
||||
else: # native
|
||||
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
|
||||
query = f"""
|
||||
WITH input_data AS (
|
||||
SELECT * FROM unnest(
|
||||
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
|
||||
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
|
||||
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
|
||||
)
|
||||
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
|
||||
SELECT
|
||||
$1,
|
||||
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
|
||||
context, fact_type, confidence_score, metadata, chunk_id, document_id,
|
||||
COALESCE(
|
||||
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
|
||||
'{{}}'::varchar[]
|
||||
)
|
||||
FROM input_data
|
||||
RETURNING id
|
||||
"""
|
||||
|
||||
results = await conn.fetch(
|
||||
query,
|
||||
bank_id,
|
||||
fact_texts,
|
||||
embeddings,
|
||||
|
||||
@@ -158,7 +158,7 @@ async def retrieve_bm25(
|
||||
|
||||
from .tags import TagsMatch, build_tags_where_clause_simple
|
||||
|
||||
# Sanitize query text: remove special characters that have meaning in tsquery
|
||||
# Sanitize query text for native backend: remove special characters that have meaning in tsquery
|
||||
# Keep only alphanumeric characters and spaces
|
||||
sanitized_text = re.sub(r"[^\w\s]", " ", query_text.lower())
|
||||
|
||||
@@ -169,29 +169,46 @@ async def retrieve_bm25(
|
||||
# If no valid tokens, return empty results
|
||||
return []
|
||||
|
||||
# Convert query to tsquery using OR for more flexible matching
|
||||
# This prevents empty results when some terms are missing
|
||||
query_tsquery = " | ".join(tokens)
|
||||
|
||||
# Build query based on text search backend
|
||||
config = get_config()
|
||||
tags_clause = build_tags_where_clause_simple(tags, 5)
|
||||
params = [query_tsquery, bank_id, fact_type, limit]
|
||||
if tags:
|
||||
params.append(tags)
|
||||
|
||||
results = await conn.fetch(
|
||||
f"""
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND fact_type = $3
|
||||
AND search_vector @@ to_tsquery('english', $1)
|
||||
{tags_clause}
|
||||
ORDER BY bm25_score DESC
|
||||
LIMIT $4
|
||||
""",
|
||||
*params,
|
||||
)
|
||||
if config.text_search_extension == "vchord":
|
||||
# VectorChord BM25: use <&> operator with to_bm25query and tokenize
|
||||
params = [bank_id, fact_type, limit, query_text] # Use raw query_text for tokenization
|
||||
if tags:
|
||||
params.append(query_text) # VectorChord doesn't need sanitization
|
||||
|
||||
query = f"""
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($4, 'llmlingua2')) AS bm25_score
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $1
|
||||
AND fact_type = $2
|
||||
{tags_clause}
|
||||
ORDER BY bm25_score DESC
|
||||
LIMIT $3
|
||||
"""
|
||||
else: # native
|
||||
# Native PostgreSQL: use ts_rank_cd with to_tsquery
|
||||
query_tsquery = " | ".join(tokens)
|
||||
params = [query_tsquery, bank_id, fact_type, limit]
|
||||
if tags:
|
||||
params.append(tags)
|
||||
|
||||
query = f"""
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND fact_type = $3
|
||||
AND search_vector @@ to_tsquery('english', $1)
|
||||
{tags_clause}
|
||||
ORDER BY bm25_score DESC
|
||||
LIMIT $4
|
||||
"""
|
||||
|
||||
results = await conn.fetch(query, *params)
|
||||
return [RetrievalResult.from_db_row(dict(r)) for r in results]
|
||||
|
||||
|
||||
@@ -268,59 +285,109 @@ async def retrieve_semantic_bm25_combined(
|
||||
result_dict[ft][0].append(RetrievalResult.from_db_row(row))
|
||||
return result_dict
|
||||
|
||||
query_tsquery = " | ".join(tokens)
|
||||
# Build BM25 query based on text search backend
|
||||
config = get_config()
|
||||
|
||||
# Build tags clause - param 6 if tags provided
|
||||
tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
|
||||
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
|
||||
if tags:
|
||||
params.append(tags)
|
||||
|
||||
if config.text_search_extension == "vchord":
|
||||
# VectorChord BM25: use <&> operator with to_bm25query and tokenize
|
||||
# Note: VectorChord scores are negative (higher = better, so -1 > -10)
|
||||
params = [query_emb_str, bank_id, fact_types, limit, query_text] # Pass raw query_text for tokenization
|
||||
if tags:
|
||||
params.append(tags)
|
||||
|
||||
query = f"""
|
||||
WITH semantic_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
1 - (embedding <=> $1::vector) AS similarity,
|
||||
NULL::float AS bm25_score,
|
||||
'semantic' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND embedding IS NOT NULL
|
||||
AND fact_type = ANY($3)
|
||||
AND (1 - (embedding <=> $1::vector)) >= 0.3
|
||||
{tags_clause}
|
||||
),
|
||||
bm25_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
NULL::float AS similarity,
|
||||
search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) AS bm25_score,
|
||||
'bm25' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) DESC) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND fact_type = ANY($3)
|
||||
{tags_clause}
|
||||
),
|
||||
semantic AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM semantic_ranked WHERE rn <= $4
|
||||
),
|
||||
bm25 AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM bm25_ranked WHERE rn <= $4
|
||||
)
|
||||
SELECT * FROM semantic
|
||||
UNION ALL
|
||||
SELECT * FROM bm25
|
||||
"""
|
||||
else: # native
|
||||
# Native PostgreSQL: use ts_rank_cd with to_tsquery
|
||||
query_tsquery = " | ".join(tokens)
|
||||
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
|
||||
if tags:
|
||||
params.append(tags)
|
||||
|
||||
query = f"""
|
||||
WITH semantic_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
1 - (embedding <=> $1::vector) AS similarity,
|
||||
NULL::float AS bm25_score,
|
||||
'semantic' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND embedding IS NOT NULL
|
||||
AND fact_type = ANY($3)
|
||||
AND (1 - (embedding <=> $1::vector)) >= 0.3
|
||||
{tags_clause}
|
||||
),
|
||||
bm25_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
NULL::float AS similarity,
|
||||
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
|
||||
'bm25' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND fact_type = ANY($3)
|
||||
AND search_vector @@ to_tsquery('english', $5)
|
||||
{tags_clause}
|
||||
),
|
||||
semantic AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM semantic_ranked WHERE rn <= $4
|
||||
),
|
||||
bm25 AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM bm25_ranked WHERE rn <= $4
|
||||
)
|
||||
SELECT * FROM semantic
|
||||
UNION ALL
|
||||
SELECT * FROM bm25
|
||||
"""
|
||||
|
||||
# Combined CTE query for both semantic and BM25 across all fact types
|
||||
# Uses window functions to limit per fact_type per method
|
||||
results = await conn.fetch(
|
||||
f"""
|
||||
WITH semantic_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
1 - (embedding <=> $1::vector) AS similarity,
|
||||
NULL::float AS bm25_score,
|
||||
'semantic' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND embedding IS NOT NULL
|
||||
AND fact_type = ANY($3)
|
||||
AND (1 - (embedding <=> $1::vector)) >= 0.3
|
||||
{tags_clause}
|
||||
),
|
||||
bm25_ranked AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
NULL::float AS similarity,
|
||||
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
|
||||
'bm25' AS source,
|
||||
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
|
||||
FROM {fq_table("memory_units")}
|
||||
WHERE bank_id = $2
|
||||
AND fact_type = ANY($3)
|
||||
AND search_vector @@ to_tsquery('english', $5)
|
||||
{tags_clause}
|
||||
),
|
||||
semantic AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM semantic_ranked WHERE rn <= $4
|
||||
),
|
||||
bm25 AS (
|
||||
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
|
||||
similarity, bm25_score, source
|
||||
FROM bm25_ranked WHERE rn <= $4
|
||||
)
|
||||
SELECT * FROM semantic
|
||||
UNION ALL
|
||||
SELECT * FROM bm25
|
||||
""",
|
||||
*params,
|
||||
)
|
||||
results = await conn.fetch(query, *params)
|
||||
|
||||
# Group results by fact_type and source
|
||||
result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {ft: ([], []) for ft in fact_types}
|
||||
|
||||
@@ -96,7 +96,13 @@ class DefaultExtensionContext(ExtensionContext):
|
||||
|
||||
async def run_migration(self, schema: str) -> None:
|
||||
"""Run migrations for a specific schema."""
|
||||
from hindsight_api.migrations import ensure_embedding_dimension, run_migrations
|
||||
from hindsight_api.config import get_config
|
||||
from hindsight_api.migrations import (
|
||||
ensure_embedding_dimension,
|
||||
ensure_text_search_extension,
|
||||
ensure_vector_extension,
|
||||
run_migrations,
|
||||
)
|
||||
|
||||
# Prefer getting URL from memory engine (handles pg0 case where URL is set after init)
|
||||
db_url = self._database_url
|
||||
@@ -107,6 +113,9 @@ class DefaultExtensionContext(ExtensionContext):
|
||||
|
||||
run_migrations(db_url, schema=schema)
|
||||
|
||||
# Get config for vector extension setting
|
||||
config = get_config()
|
||||
|
||||
# Ensure embedding column dimension matches the model's dimension
|
||||
# This is needed because migrations create columns with default dimension
|
||||
if self._memory_engine is not None:
|
||||
@@ -114,7 +123,15 @@ class DefaultExtensionContext(ExtensionContext):
|
||||
if embeddings is not None:
|
||||
dimension = getattr(embeddings, "dimension", None)
|
||||
if dimension is not None:
|
||||
ensure_embedding_dimension(db_url, dimension, schema=schema)
|
||||
ensure_embedding_dimension(
|
||||
db_url, dimension, schema=schema, vector_extension=config.vector_extension
|
||||
)
|
||||
|
||||
# Ensure vector indexes match the configured extension
|
||||
ensure_vector_extension(db_url, vector_extension=config.vector_extension, schema=schema)
|
||||
|
||||
# Ensure text search columns/indexes match the configured extension
|
||||
ensure_text_search_extension(db_url, text_search_extension=config.text_search_extension, schema=schema)
|
||||
|
||||
def get_memory_engine(self) -> "MemoryEngineInterface":
|
||||
"""Get the memory engine interface."""
|
||||
|
||||
@@ -155,6 +155,8 @@ def main():
|
||||
config = HindsightConfig(
|
||||
database_url=config.database_url,
|
||||
database_schema=config.database_schema,
|
||||
vector_extension=config.vector_extension,
|
||||
text_search_extension=config.text_search_extension,
|
||||
llm_provider=config.llm_provider,
|
||||
llm_api_key=config.llm_api_key,
|
||||
llm_model=config.llm_model,
|
||||
@@ -223,6 +225,7 @@ def main():
|
||||
reranker_litellm_model=config.reranker_litellm_model,
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
base_path=config.base_path,
|
||||
log_level=args.log_level,
|
||||
log_format=config.log_format,
|
||||
mcp_enabled=config.mcp_enabled,
|
||||
|
||||
@@ -33,6 +33,41 @@ logger = logging.getLogger(__name__)
|
||||
MIGRATION_LOCK_ID = 123456789
|
||||
|
||||
|
||||
def _detect_vector_extension(conn, vector_extension: str = "pgvector") -> str:
|
||||
"""
|
||||
Validate vector extension: 'vchord' or 'pgvector'.
|
||||
|
||||
Args:
|
||||
conn: SQLAlchemy connection object
|
||||
vector_extension: Configured extension ("pgvector" or "vchord")
|
||||
|
||||
Returns:
|
||||
"vchord" or "pgvector"
|
||||
|
||||
Raises:
|
||||
RuntimeError: If configured extension is not installed
|
||||
"""
|
||||
# Verify the configured extension is installed
|
||||
if vector_extension == "vchord":
|
||||
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
|
||||
if not vchord_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
|
||||
)
|
||||
logger.debug("Using configured vector extension: vchord")
|
||||
return "vchord"
|
||||
elif vector_extension == "pgvector":
|
||||
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
|
||||
if not pgvector_check:
|
||||
raise RuntimeError(
|
||||
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
|
||||
)
|
||||
logger.debug("Using configured vector extension: pgvector")
|
||||
return "pgvector"
|
||||
else:
|
||||
raise ValueError(f"Invalid vector_extension: {vector_extension}. Must be 'pgvector' or 'vchord'")
|
||||
|
||||
|
||||
def _get_schema_lock_id(schema: str) -> int:
|
||||
"""
|
||||
Generate a unique advisory lock ID for a schema.
|
||||
@@ -324,6 +359,7 @@ def ensure_embedding_dimension(
|
||||
database_url: str,
|
||||
required_dimension: int,
|
||||
schema: str | None = None,
|
||||
vector_extension: str = "pgvector",
|
||||
) -> None:
|
||||
"""
|
||||
Ensure the embedding column dimension matches the model's dimension.
|
||||
@@ -338,6 +374,7 @@ def ensure_embedding_dimension(
|
||||
database_url: SQLAlchemy database URL
|
||||
required_dimension: The embedding dimension required by the model
|
||||
schema: Target PostgreSQL schema name (None for public)
|
||||
vector_extension: Configured vector extension ("pgvector" or "vchord")
|
||||
|
||||
Raises:
|
||||
RuntimeError: If dimension mismatch with existing data
|
||||
@@ -361,6 +398,10 @@ def ensure_embedding_dimension(
|
||||
logger.debug(f"memory_units table does not exist in schema '{schema_name}', skipping dimension check")
|
||||
return
|
||||
|
||||
# Detect which vector extension is available
|
||||
vector_ext = _detect_vector_extension(conn, vector_extension)
|
||||
logger.info(f"Using vector extension: {vector_ext}")
|
||||
|
||||
# Get current column dimension from pg_attribute
|
||||
# pgvector stores dimension in atttypmod
|
||||
current_dim = conn.execute(
|
||||
@@ -408,8 +449,7 @@ def ensure_embedding_dimension(
|
||||
# Table is empty, safe to alter column
|
||||
logger.info(f"Altering embedding column dimension from {current_dimension} to {required_dimension}")
|
||||
|
||||
# Drop the HNSW index on embedding column if it exists
|
||||
# Only drop indexes that use 'hnsw' and reference the 'embedding' column
|
||||
# Drop existing vector index (works for both HNSW and vchordrq)
|
||||
conn.execute(
|
||||
text(f"""
|
||||
DO $$
|
||||
@@ -419,7 +459,7 @@ def ensure_embedding_dimension(
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE schemaname = '{schema_name}'
|
||||
AND tablename = 'memory_units'
|
||||
AND indexdef LIKE '%hnsw%'
|
||||
AND (indexdef LIKE '%hnsw%' OR indexdef LIKE '%vchordrq%')
|
||||
AND indexdef LIKE '%embedding%'
|
||||
LOOP
|
||||
EXECUTE 'DROP INDEX IF EXISTS {schema_name}.' || idx_name;
|
||||
@@ -434,15 +474,377 @@ def ensure_embedding_dimension(
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
# Recreate the HNSW index
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_hnsw
|
||||
ON {schema_name}.memory_units
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64)
|
||||
""")
|
||||
)
|
||||
# Recreate index with appropriate type based on detected extension
|
||||
if vector_ext == "vchord":
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_vchordrq
|
||||
ON {schema_name}.memory_units
|
||||
USING vchordrq (embedding vector_l2_ops)
|
||||
""")
|
||||
)
|
||||
logger.info(f"Created vchordrq index for {required_dimension}-dimensional embeddings")
|
||||
else: # pgvector
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX IF NOT EXISTS idx_memory_units_embedding_hnsw
|
||||
ON {schema_name}.memory_units
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64)
|
||||
""")
|
||||
)
|
||||
logger.info(f"Created HNSW index for {required_dimension}-dimensional embeddings")
|
||||
conn.commit()
|
||||
|
||||
logger.info(f"Successfully changed embedding dimension to {required_dimension}")
|
||||
|
||||
|
||||
def ensure_vector_extension(
|
||||
database_url: str,
|
||||
vector_extension: str = "pgvector",
|
||||
schema: str | None = None,
|
||||
) -> None:
|
||||
"""
|
||||
Ensure the vector indexes match the configured vector extension.
|
||||
|
||||
This function checks the current vector index type in the database
|
||||
and adjusts it if necessary:
|
||||
- If index type matches configured extension: no action needed
|
||||
- If they differ and tables are empty: drop old indexes, recreate with new type
|
||||
- If they differ and tables have data: raise error with migration guidance
|
||||
|
||||
Args:
|
||||
database_url: SQLAlchemy database URL
|
||||
vector_extension: Configured vector extension ("pgvector" or "vchord")
|
||||
schema: Target PostgreSQL schema name (None for public)
|
||||
|
||||
Raises:
|
||||
RuntimeError: If extension mismatch with existing data
|
||||
"""
|
||||
schema_name = schema or "public"
|
||||
|
||||
engine = create_engine(database_url)
|
||||
with engine.connect() as conn:
|
||||
# Detect which vector extension should be used
|
||||
target_ext = _detect_vector_extension(conn, vector_extension)
|
||||
logger.info(f"Target vector extension: {target_ext}")
|
||||
|
||||
# Tables with vector indexes to check
|
||||
tables_to_check = [
|
||||
("memory_units", "idx_memory_units_embedding"),
|
||||
("learnings", "idx_learnings_embedding"),
|
||||
("pinned_reflections", "idx_pinned_reflections_embedding"),
|
||||
]
|
||||
|
||||
# Determine target index type
|
||||
target_index_type = "vchordrq" if target_ext == "vchord" else "hnsw"
|
||||
|
||||
mismatched_tables = []
|
||||
tables_with_data = []
|
||||
|
||||
for table_name, index_name in tables_to_check:
|
||||
# Check if table exists
|
||||
table_exists = conn.execute(
|
||||
text("""
|
||||
SELECT EXISTS (
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = :schema AND table_name = :table_name
|
||||
)
|
||||
"""),
|
||||
{"schema": schema_name, "table_name": table_name},
|
||||
).scalar()
|
||||
|
||||
if not table_exists:
|
||||
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
|
||||
continue
|
||||
|
||||
# Check current index type by querying pg_indexes
|
||||
current_index_info = conn.execute(
|
||||
text("""
|
||||
SELECT indexdef
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = :schema
|
||||
AND tablename = :table_name
|
||||
AND indexname LIKE :index_pattern
|
||||
"""),
|
||||
{"schema": schema_name, "table_name": table_name, "index_pattern": "%embedding%"},
|
||||
).fetchone()
|
||||
|
||||
if not current_index_info:
|
||||
logger.warning(f"No embedding index found for {table_name}, will create it")
|
||||
mismatched_tables.append((table_name, index_name, None))
|
||||
continue
|
||||
|
||||
indexdef = current_index_info[0].lower()
|
||||
if "vchordrq" in indexdef:
|
||||
current_index_type = "vchordrq"
|
||||
elif "hnsw" in indexdef:
|
||||
current_index_type = "hnsw"
|
||||
else:
|
||||
logger.warning(f"Unknown index type for {table_name}: {indexdef}")
|
||||
continue
|
||||
|
||||
# Check if index type matches target
|
||||
if current_index_type != target_index_type:
|
||||
logger.info(
|
||||
f"Index type mismatch on {table_name}: current={current_index_type}, target={target_index_type}"
|
||||
)
|
||||
mismatched_tables.append((table_name, index_name, current_index_type))
|
||||
|
||||
# Check if table has data
|
||||
row_count = conn.execute(
|
||||
text(f"SELECT COUNT(*) FROM {schema_name}.{table_name} WHERE embedding IS NOT NULL")
|
||||
).scalar()
|
||||
|
||||
if row_count > 0:
|
||||
tables_with_data.append((table_name, row_count))
|
||||
else:
|
||||
logger.debug(f"Index type OK for {table_name}: {current_index_type}")
|
||||
|
||||
# If no mismatches, we're done
|
||||
if not mismatched_tables:
|
||||
logger.debug(f"All vector indexes match configured extension: {target_ext}")
|
||||
return
|
||||
|
||||
# If there's data in any mismatched table, raise error
|
||||
if tables_with_data:
|
||||
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
|
||||
raise RuntimeError(
|
||||
f"Cannot change vector extension from {current_index_type} to {target_index_type}: "
|
||||
f"the following tables contain data: {table_list}. "
|
||||
f"To change vector extension, you must either:\n"
|
||||
f" 1. Re-embed all data: DELETE FROM {schema_name}.memory_units; "
|
||||
f"DELETE FROM {schema_name}.learnings; DELETE FROM {schema_name}.pinned_reflections; then restart\n"
|
||||
f" 2. Use the current vector extension (set HINDSIGHT_API_VECTOR_EXTENSION='{current_index_type.replace('vchordrq', 'vchord').replace('hnsw', 'pgvector')}')"
|
||||
)
|
||||
|
||||
# Tables are empty, safe to recreate indexes
|
||||
logger.info(f"Recreating vector indexes for {target_ext}")
|
||||
|
||||
for table_name, index_name, current_type in mismatched_tables:
|
||||
# Drop existing index if it exists
|
||||
if current_type:
|
||||
logger.info(f"Dropping {current_type} index on {table_name}")
|
||||
conn.execute(text(f"DROP INDEX IF EXISTS {schema_name}.{index_name}"))
|
||||
|
||||
# Create new index with appropriate type
|
||||
if target_ext == "vchord":
|
||||
logger.info(f"Creating vchordrq index on {table_name}")
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX IF NOT EXISTS {index_name}
|
||||
ON {schema_name}.{table_name}
|
||||
USING vchordrq (embedding vector_l2_ops)
|
||||
""")
|
||||
)
|
||||
else: # pgvector
|
||||
logger.info(f"Creating HNSW index on {table_name}")
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX IF NOT EXISTS {index_name}
|
||||
ON {schema_name}.{table_name}
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64)
|
||||
""")
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
logger.info(f"Successfully migrated vector indexes to {target_ext}")
|
||||
|
||||
|
||||
def ensure_text_search_extension(
|
||||
database_url: str,
|
||||
text_search_extension: str = "native",
|
||||
schema: str | None = None,
|
||||
) -> None:
|
||||
"""
|
||||
Ensure the text search columns and indexes match the configured extension.
|
||||
|
||||
This function checks the current search_vector column type and index type
|
||||
in the database and adjusts them if necessary:
|
||||
- If they match configured extension: no action needed
|
||||
- If they differ and tables are empty: drop old column/index, recreate with new type
|
||||
- If they differ and tables have data: raise error with migration guidance
|
||||
|
||||
Args:
|
||||
database_url: SQLAlchemy database URL
|
||||
text_search_extension: Configured text search extension ("native" or "vchord")
|
||||
schema: Target PostgreSQL schema name (None for public)
|
||||
|
||||
Raises:
|
||||
RuntimeError: If extension mismatch with existing data
|
||||
"""
|
||||
schema_name = schema or "public"
|
||||
|
||||
engine = create_engine(database_url)
|
||||
with engine.connect() as conn:
|
||||
# Tables with search_vector columns to check
|
||||
tables_to_check = [
|
||||
"memory_units",
|
||||
"reflections", # Renamed from pinned_reflections in p1k2l3m4n5o6 migration
|
||||
]
|
||||
|
||||
# Determine target column type and index type
|
||||
if text_search_extension == "vchord":
|
||||
target_column_type = "bm25vector"
|
||||
target_index_type = "bm25"
|
||||
else: # native
|
||||
target_column_type = "tsvector"
|
||||
target_index_type = "gin"
|
||||
|
||||
mismatched_tables = []
|
||||
tables_with_data = []
|
||||
|
||||
for table_name in tables_to_check:
|
||||
# Check if table exists
|
||||
table_exists = conn.execute(
|
||||
text("""
|
||||
SELECT EXISTS (
|
||||
SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = :schema AND table_name = :table_name
|
||||
)
|
||||
"""),
|
||||
{"schema": schema_name, "table_name": table_name},
|
||||
).scalar()
|
||||
|
||||
if not table_exists:
|
||||
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
|
||||
continue
|
||||
|
||||
# Get current column type from information_schema
|
||||
current_column_info = conn.execute(
|
||||
text("""
|
||||
SELECT data_type, udt_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = :schema
|
||||
AND table_name = :table_name
|
||||
AND column_name = 'search_vector'
|
||||
"""),
|
||||
{"schema": schema_name, "table_name": table_name},
|
||||
).fetchone()
|
||||
|
||||
if not current_column_info:
|
||||
logger.warning(f"No search_vector column found for {table_name}, will create it")
|
||||
mismatched_tables.append((table_name, None, None))
|
||||
continue
|
||||
|
||||
# Check column type (udt_name contains the actual type: tsvector, bm25vector, etc.)
|
||||
current_column_type = current_column_info[1] # udt_name
|
||||
|
||||
# Get current index type
|
||||
current_index_info = conn.execute(
|
||||
text("""
|
||||
SELECT am.amname
|
||||
FROM pg_indexes pi
|
||||
JOIN pg_class c ON c.relname = pi.indexname
|
||||
JOIN pg_am am ON am.oid = c.relam
|
||||
WHERE pi.schemaname = :schema
|
||||
AND pi.tablename = :table_name
|
||||
AND pi.indexname LIKE '%text_search%'
|
||||
"""),
|
||||
{"schema": schema_name, "table_name": table_name},
|
||||
).fetchone()
|
||||
|
||||
current_index_type = current_index_info[0] if current_index_info else None
|
||||
|
||||
# Check if column and index types match target
|
||||
column_matches = current_column_type == target_column_type
|
||||
index_matches = current_index_type == target_index_type if current_index_type else False
|
||||
|
||||
if not (column_matches and index_matches):
|
||||
logger.info(
|
||||
f"Text search mismatch on {table_name}: "
|
||||
f"column={current_column_type} (want {target_column_type}), "
|
||||
f"index={current_index_type} (want {target_index_type})"
|
||||
)
|
||||
mismatched_tables.append((table_name, current_column_type, current_index_type))
|
||||
|
||||
# Check if table has data
|
||||
row_count = conn.execute(text(f"SELECT COUNT(*) FROM {schema_name}.{table_name}")).scalar()
|
||||
|
||||
if row_count > 0:
|
||||
tables_with_data.append((table_name, row_count))
|
||||
else:
|
||||
logger.debug(f"Text search OK for {table_name}: {current_column_type}/{current_index_type}")
|
||||
|
||||
# If no mismatches, we're done
|
||||
if not mismatched_tables:
|
||||
logger.debug(f"All text search columns/indexes match configured extension: {text_search_extension}")
|
||||
return
|
||||
|
||||
# If there's data in any mismatched table, raise error
|
||||
if tables_with_data:
|
||||
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
|
||||
current_ext = "native" if mismatched_tables[0][1] == "tsvector" else "vchord"
|
||||
raise RuntimeError(
|
||||
f"Cannot change text search extension from {current_ext} to {text_search_extension}: "
|
||||
f"the following tables contain data: {table_list}. "
|
||||
f"To change text search extension, you must either:\n"
|
||||
f" 1. Clear all data: DELETE FROM {schema_name}.memory_units; "
|
||||
f"DELETE FROM {schema_name}.reflections; then restart\n"
|
||||
f" 2. Use the current text search extension (set HINDSIGHT_API_TEXT_SEARCH_EXTENSION='{current_ext}')"
|
||||
)
|
||||
|
||||
# Tables are empty, safe to recreate columns/indexes
|
||||
logger.info(f"Recreating text search columns/indexes for {text_search_extension}")
|
||||
|
||||
for table_name, current_col_type, current_idx_type in mismatched_tables:
|
||||
# Drop existing index if it exists
|
||||
if current_idx_type:
|
||||
logger.info(f"Dropping {current_idx_type} index on {table_name}")
|
||||
conn.execute(
|
||||
text(f"""
|
||||
DROP INDEX IF EXISTS {schema_name}.idx_{table_name.replace(".", "_")}_text_search
|
||||
""")
|
||||
)
|
||||
|
||||
# Drop existing column if it exists
|
||||
if current_col_type:
|
||||
logger.info(f"Dropping {current_col_type} column on {table_name}")
|
||||
conn.execute(text(f"ALTER TABLE {schema_name}.{table_name} DROP COLUMN IF EXISTS search_vector"))
|
||||
|
||||
# Create new column with appropriate type
|
||||
if text_search_extension == "vchord":
|
||||
logger.info(f"Creating bm25vector column on {table_name}")
|
||||
# Note: vchord_bm25 extension creates types in bm25_catalog schema
|
||||
conn.execute(
|
||||
text(f"ALTER TABLE {schema_name}.{table_name} ADD COLUMN search_vector bm25_catalog.bm25vector")
|
||||
)
|
||||
|
||||
# Create BM25 index
|
||||
logger.info(f"Creating BM25 index on {table_name}")
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
|
||||
ON {schema_name}.{table_name}
|
||||
USING bm25 (search_vector bm25_catalog.bm25_ops)
|
||||
""")
|
||||
)
|
||||
else: # native
|
||||
logger.info(f"Creating tsvector column on {table_name}")
|
||||
# Different GENERATED expression for each table
|
||||
if table_name == "memory_units":
|
||||
generated_expr = "to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))"
|
||||
else: # reflections
|
||||
generated_expr = "to_tsvector('english', COALESCE(name, '') || ' ' || content)"
|
||||
|
||||
conn.execute(
|
||||
text(f"""
|
||||
ALTER TABLE {schema_name}.{table_name}
|
||||
ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS ({generated_expr}) STORED
|
||||
""")
|
||||
)
|
||||
|
||||
# Create GIN index
|
||||
logger.info(f"Creating GIN index on {table_name}")
|
||||
conn.execute(
|
||||
text(f"""
|
||||
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
|
||||
ON {schema_name}.{table_name}
|
||||
USING gin(search_vector)
|
||||
""")
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
logger.info(f"Successfully migrated text search to {text_search_extension}")
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
"""
|
||||
Integration test for API base path support.
|
||||
|
||||
Tests that the API works correctly when deployed with a base path (e.g., /hindsight)
|
||||
for reverse proxy deployments.
|
||||
"""
|
||||
import os
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
import httpx
|
||||
from hindsight_api.api import create_app
|
||||
from hindsight_api.config import clear_config_cache
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def api_client_with_base_path(memory):
|
||||
"""Create an async test client for the FastAPI app with a base path."""
|
||||
# Set base path in environment
|
||||
base_path = "/hindsight"
|
||||
os.environ["HINDSIGHT_API_BASE_PATH"] = base_path
|
||||
|
||||
# Clear config cache to force reload with new base_path
|
||||
clear_config_cache()
|
||||
|
||||
# Memory is already initialized by the conftest fixture (with migrations)
|
||||
app = create_app(memory, initialize_memory=False)
|
||||
|
||||
# Use base_url with base path
|
||||
transport = httpx.ASGITransport(app=app)
|
||||
async with httpx.AsyncClient(
|
||||
transport=transport,
|
||||
base_url=f"http://test{base_path}"
|
||||
) as client:
|
||||
yield client
|
||||
|
||||
# Cleanup: unset base path
|
||||
os.environ.pop("HINDSIGHT_API_BASE_PATH", None)
|
||||
clear_config_cache()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def api_client_without_base_path(memory):
|
||||
"""Create an async test client for the FastAPI app without a base path (root)."""
|
||||
# Ensure no base path is set
|
||||
os.environ.pop("HINDSIGHT_API_BASE_PATH", None)
|
||||
clear_config_cache()
|
||||
|
||||
app = create_app(memory, initialize_memory=False)
|
||||
transport = httpx.ASGITransport(app=app)
|
||||
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
yield client
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_health_endpoint(api_client_with_base_path):
|
||||
"""Test that health endpoint works with base path."""
|
||||
# With base path set to /hindsight, health should be at /hindsight/health
|
||||
# But since our client base_url is already http://test/hindsight, we request /health
|
||||
response = await api_client_with_base_path.get("/health")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "status" in data
|
||||
assert data["status"] in ["ok", "healthy"] # Accept both formats
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_banks_endpoint(api_client_with_base_path):
|
||||
"""Test that banks endpoint works with base path."""
|
||||
response = await api_client_with_base_path.get("/v1/default/banks")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert "banks" in data
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_openapi_schema(api_client_with_base_path):
|
||||
"""Test that OpenAPI schema includes correct base path in servers."""
|
||||
response = await api_client_with_base_path.get("/openapi.json")
|
||||
assert response.status_code == 200
|
||||
openapi_schema = response.json()
|
||||
|
||||
# Check that servers array includes base path
|
||||
assert "servers" in openapi_schema
|
||||
servers = openapi_schema["servers"]
|
||||
assert len(servers) > 0
|
||||
# FastAPI should set server URL to the root_path
|
||||
assert servers[0]["url"] == "/hindsight"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_docs_redirect(api_client_with_base_path):
|
||||
"""Test that /docs redirects correctly with base path."""
|
||||
# FastAPI docs endpoint should work
|
||||
response = await api_client_with_base_path.get("/docs", follow_redirects=False)
|
||||
# Should either return 200 (direct) or 307 (redirect to trailing slash)
|
||||
assert response.status_code in [200, 307]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_metrics(api_client_with_base_path):
|
||||
"""Test that metrics endpoint works with base path."""
|
||||
response = await api_client_with_base_path.get("/metrics")
|
||||
assert response.status_code == 200
|
||||
# Metrics should be in Prometheus format
|
||||
assert "# HELP" in response.text or "# TYPE" in response.text
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_full_workflow(api_client_with_base_path):
|
||||
"""
|
||||
Test a full retain/recall workflow with base path.
|
||||
|
||||
This ensures that all memory operations work correctly when the API
|
||||
is deployed with a base path.
|
||||
"""
|
||||
bank_id = "test_base_path_bank"
|
||||
|
||||
# 1. Create/get bank
|
||||
response = await api_client_with_base_path.get(f"/v1/default/banks/{bank_id}/profile")
|
||||
assert response.status_code == 200
|
||||
|
||||
# 2. Store a memory
|
||||
response = await api_client_with_base_path.post(
|
||||
f"/v1/default/banks/{bank_id}/memories",
|
||||
json={
|
||||
"items": [
|
||||
{
|
||||
"content": "The API supports base path deployment for reverse proxy use cases.",
|
||||
"context": "testing base path feature"
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
assert result["success"] is True
|
||||
|
||||
# 3. Recall the memory
|
||||
response = await api_client_with_base_path.post(
|
||||
f"/v1/default/banks/{bank_id}/memories/recall",
|
||||
json={
|
||||
"query": "base path support"
|
||||
}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
recall_result = response.json()
|
||||
# API returns "results" not "memories"
|
||||
assert "results" in recall_result
|
||||
assert len(recall_result["results"]) > 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_without_base_path_still_works(api_client_without_base_path):
|
||||
"""
|
||||
Regression test: ensure default behavior (no base path) still works.
|
||||
|
||||
This test verifies that when HINDSIGHT_API_BASE_PATH is not set,
|
||||
the API works at the root path as before.
|
||||
"""
|
||||
# Health check at root
|
||||
response = await api_client_without_base_path.get("/health")
|
||||
assert response.status_code == 200
|
||||
|
||||
# Banks endpoint at root
|
||||
response = await api_client_without_base_path.get("/v1/default/banks")
|
||||
assert response.status_code == 200
|
||||
|
||||
# OpenAPI schema should have empty or "/" server path
|
||||
response = await api_client_without_base_path.get("/openapi.json")
|
||||
assert response.status_code == 200
|
||||
openapi_schema = response.json()
|
||||
servers = openapi_schema.get("servers", [])
|
||||
if servers:
|
||||
# Server URL should be empty string (root) or "/"
|
||||
assert servers[0]["url"] in ["", "/"]
|
||||
|
||||
|
||||
@pytest.mark.skip(reason="MCP endpoint routing with base path needs investigation")
|
||||
@pytest.mark.asyncio
|
||||
async def test_base_path_mcp_endpoint(api_client_with_base_path):
|
||||
"""Test that MCP endpoint is accessible with base path."""
|
||||
bank_id = "test_mcp_bank"
|
||||
|
||||
# MCP endpoint should be mounted at /mcp/{bank_id}/
|
||||
# The MCP server uses a different protocol, so just check the root exists
|
||||
response = await api_client_with_base_path.get(f"/mcp/{bank_id}/")
|
||||
# MCP may return various status codes, but should not be 404 (not found)
|
||||
# Accept 405 (method not allowed), 400 (bad request), etc.
|
||||
assert response.status_code != 404, "MCP endpoint should exist"
|
||||
@@ -209,7 +209,7 @@ class TestLargeBatchRetain:
|
||||
raise
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.timeout(120)
|
||||
@pytest.mark.timeout(240) # Increased timeout for VectorChord BM25 tokenization
|
||||
async def test_batch_chunking_behavior(self, memory_with_mock_llm, request_context):
|
||||
"""
|
||||
Test that large batches are properly chunked into sub-batches.
|
||||
|
||||
@@ -1,8 +1,12 @@
|
||||
import type { NextConfig } from "next";
|
||||
import path from "path";
|
||||
|
||||
const basePath = process.env.NEXT_PUBLIC_BASE_PATH || '';
|
||||
|
||||
const nextConfig: NextConfig = {
|
||||
output: 'standalone',
|
||||
basePath: basePath,
|
||||
assetPrefix: basePath,
|
||||
// Disable request logging in production
|
||||
logging: false,
|
||||
// Set the monorepo root explicitly to avoid detecting wrong lockfiles in parent directories
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
---
|
||||
title: How We Solved Memory Conflicts in Hindsight
|
||||
description: Learn how Hindsight handles contradictory information by tracking temporal evolution and preserving history in its memory consolidation system.
|
||||
authors: [hindsight]
|
||||
tags: [engineering, memory-systems, conflict-resolution]
|
||||
image: /img/blog/2026-02-09/consolidation-pipeline.png
|
||||
date: 2026-02-09
|
||||
---
|
||||
|
||||
# How We Solved Memory Conflicts in Hindsight
|
||||
|
||||
One of the hardest problems we tackled in Hindsight was dealing with contradictions. When you're building a memory system for AI agents, reality isn't static. It evolves.
|
||||
|
||||
A CRM agent might learn that "Acme Corp is a key prospect" in January, then encounter "Acme Corp is now a paying customer" in March. Naive approaches either lose the history or drown in duplicate facts.
|
||||
|
||||
@@ -57,6 +57,64 @@ hindsight-admin run-db-migration
|
||||
hindsight-admin run-db-migration --schema tenant_acme
|
||||
```
|
||||
|
||||
### Vector Extension
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_VECTOR_EXTENSION` | Vector extension to use: `auto`, `pgvector`, or `vchord` | `auto` |
|
||||
|
||||
Hindsight supports two PostgreSQL vector extensions:
|
||||
- **pgvector**: Standard extension, works well for most embeddings (up to ~2000 dimensions)
|
||||
- **vchord**: Optimized for high-dimensional embeddings (3000+ dimensions), includes BM25 search
|
||||
|
||||
When set to `auto` (default), Hindsight automatically detects which extension is installed, preferring vchord if both are available.
|
||||
|
||||
**When to use vchord:**
|
||||
- Using high-dimensional embeddings (e.g., `text-embedding-3-large` with 3072 dimensions)
|
||||
- Need better performance with large embedding dimensions
|
||||
- Want to use vchord's BM25 search capabilities
|
||||
|
||||
**When to use pgvector:**
|
||||
- Using standard embedding dimensions (384-1536)
|
||||
- Prefer the widely-adopted pgvector extension
|
||||
- Simpler deployment (pgvector is more commonly available)
|
||||
|
||||
**Switching extensions:**
|
||||
|
||||
If you need to switch from one extension to another:
|
||||
1. Set `HINDSIGHT_API_VECTOR_EXTENSION` to your desired extension (`pgvector` or `vchord`)
|
||||
2. If your database has existing data, you'll get an error with migration instructions
|
||||
3. For empty databases, indexes will be automatically recreated on startup
|
||||
|
||||
### Text Search Extension
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native` or `vchord` | `native` |
|
||||
|
||||
Hindsight supports two text search backends for BM25 keyword retrieval:
|
||||
- **native**: PostgreSQL's built-in full-text search (`tsvector` + GIN indexes)
|
||||
- **vchord**: VectorChord BM25 (`bm25vector` + BM25 indexes) - requires `vchord_bm25` extension
|
||||
|
||||
**When to use vchord:**
|
||||
- Already using vchord for vector search (good integration)
|
||||
- Want better BM25 ranking performance
|
||||
- Need advanced tokenization (uses `llmlingua2` tokenizer)
|
||||
|
||||
**When to use native:**
|
||||
- Standard PostgreSQL deployment (no extra extensions)
|
||||
- Simpler setup and wider compatibility
|
||||
- Works well for most use cases
|
||||
|
||||
**Switching backends:**
|
||||
|
||||
To switch from native to vchord (or vice versa):
|
||||
1. Set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION=vchord` (or `native`)
|
||||
2. If your database has existing data, you'll get an error with migration instructions
|
||||
3. For empty databases, the columns/indexes will be automatically recreated on startup
|
||||
|
||||
**Note:** VectorChord text search uses the `llmlingua2` tokenizer for multilingual support, while native uses PostgreSQL's English tokenizer.
|
||||
|
||||
### LLM Provider
|
||||
|
||||
| Variable | Description | Default |
|
||||
@@ -429,6 +487,7 @@ For advanced authentication (JWT, OAuth, multi-tenant schemas), implement a cust
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_HOST` | Bind address | `0.0.0.0` |
|
||||
| `HINDSIGHT_API_PORT` | Server port | `8888` |
|
||||
| `HINDSIGHT_API_BASE_PATH` | Base path for API when behind reverse proxy (e.g., `/hindsight`) | `""` (root) |
|
||||
| `HINDSIGHT_API_WORKERS` | Number of uvicorn worker processes | `1` |
|
||||
| `HINDSIGHT_API_LOG_LEVEL` | Log level: `debug`, `info`, `warning`, `error` | `info` |
|
||||
| `HINDSIGHT_API_LOG_FORMAT` | Log format: `text` or `json` (structured logging for cloud platforms) | `text` |
|
||||
@@ -649,12 +708,78 @@ The Control Plane is the web UI for managing memory banks.
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_CP_DATAPLANE_API_URL` | URL of the API service | `http://localhost:8888` |
|
||||
| `NEXT_PUBLIC_BASE_PATH` | Base path for Control Plane UI when behind reverse proxy (e.g., `/hindsight`) | `""` (root) |
|
||||
|
||||
```bash
|
||||
# Point Control Plane to a remote API service
|
||||
export HINDSIGHT_CP_DATAPLANE_API_URL=http://api.example.com:8888
|
||||
```
|
||||
|
||||
### Reverse Proxy / Subpath Deployment
|
||||
|
||||
To deploy Hindsight under a subpath (e.g., `example.com/hindsight/`):
|
||||
|
||||
1. Set both environment variables to the same path:
|
||||
```bash
|
||||
HINDSIGHT_API_BASE_PATH=/hindsight
|
||||
NEXT_PUBLIC_BASE_PATH=/hindsight
|
||||
```
|
||||
|
||||
2. Configure your reverse proxy to:
|
||||
- Forward `/hindsight/*` requests to Hindsight
|
||||
- Preserve the full path in forwarded requests
|
||||
- Set appropriate proxy headers (X-Forwarded-Proto, X-Forwarded-For)
|
||||
|
||||
**Example: Nginx Configuration**
|
||||
|
||||
```nginx
|
||||
location /hindsight/ {
|
||||
proxy_pass http://localhost:8888/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
```
|
||||
|
||||
**Example: Traefik Configuration**
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
hindsight:
|
||||
rule: "PathPrefix(`/hindsight`)"
|
||||
service: hindsight
|
||||
middlewares:
|
||||
- hindsight-stripprefix
|
||||
|
||||
middlewares:
|
||||
hindsight-stripprefix:
|
||||
stripPrefix:
|
||||
prefixes:
|
||||
- "/hindsight"
|
||||
|
||||
services:
|
||||
hindsight:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://localhost:8888"
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
- The base path must start with `/` and should NOT end with `/`
|
||||
- Both API and Control Plane should use the same base path
|
||||
- After setting environment variables, restart both services
|
||||
- OpenAPI docs will be available at `<base-path>/docs` (e.g., `/hindsight/docs`)
|
||||
|
||||
**Complete Examples:**
|
||||
|
||||
See `docker/compose-examples/` directory for:
|
||||
- Nginx configuration files (`simple.conf`, `api-and-control-plane.conf`)
|
||||
- Docker Compose setups (`docker-compose.yml`, `reverse-proxy-only.yml`)
|
||||
- Traefik and other reverse proxy examples
|
||||
- Full deployment documentation
|
||||
|
||||
---
|
||||
|
||||
## Example .env File
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
|
||||
|
||||
:::tip Don't want to manage infrastructure?
|
||||
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
|
||||
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
|
||||
:::
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -71,6 +71,7 @@ curl -X POST http://localhost:8888/mcp \
|
||||
-H "Authorization: Bearer your-secret-key" \
|
||||
-H "X-Bank-Id: my-bank" \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Accept: application/json, text/event-stream" \
|
||||
-d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'
|
||||
```
|
||||
|
||||
@@ -78,10 +79,10 @@ If the key is missing or invalid, requests will receive a `401 Unauthorized` res
|
||||
|
||||
## Bank Selection
|
||||
|
||||
Specify the memory bank via:
|
||||
The memory bank is resolved in this priority order:
|
||||
|
||||
1. **X-Bank-Id header** (recommended): `--header "X-Bank-Id: my-bank"`
|
||||
2. **URL path**: `http://localhost:8888/mcp/my-bank/`
|
||||
1. **URL path** (highest priority): `http://localhost:8888/mcp/my-bank/`
|
||||
2. **X-Bank-Id header**: `--header "X-Bank-Id: my-bank"`
|
||||
3. **Default**: Uses `HINDSIGHT_MCP_BANK_ID` env var (default: "default")
|
||||
|
||||
## Per-Bank Endpoints
|
||||
@@ -93,6 +94,19 @@ This design:
|
||||
- **Enforces isolation** — each MCP connection is scoped to a single bank
|
||||
- **Enables multi-tenant setups** — connect different users to different endpoints
|
||||
|
||||
## Two Modes
|
||||
|
||||
The MCP server operates in two modes depending on the URL:
|
||||
|
||||
| Mode | URL | Tools | bank_id |
|
||||
|------|-----|-------|---------|
|
||||
| **Single-bank** | `/mcp/{bank_id}/` | Memory + mental model tools | Implicit from URL |
|
||||
| **Multi-bank** | `/mcp/` | All tools including bank management | Explicit `bank_id` parameter on each tool |
|
||||
|
||||
**Single-bank mode** (recommended) scopes all operations to the bank in the URL. Tools don't expose a `bank_id` parameter.
|
||||
|
||||
**Multi-bank mode** exposes all tools with an optional `bank_id` parameter, plus bank management tools (`list_banks`, `create_bank`).
|
||||
|
||||
---
|
||||
|
||||
## Available Tools
|
||||
@@ -105,6 +119,7 @@ Store information to long-term memory.
|
||||
|-----------|------|----------|-------------|
|
||||
| `content` | string | Yes | The fact or memory to store |
|
||||
| `context` | string | No | Category for the memory (default: `general`) |
|
||||
| `timestamp` | string | No | ISO 8601 timestamp for when the event occurred |
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
@@ -133,6 +148,7 @@ Search memories to provide personalized responses.
|
||||
|-----------|------|----------|-------------|
|
||||
| `query` | string | Yes | Natural language search query |
|
||||
| `max_results` | integer | No | Maximum results to return (default: 10) |
|
||||
| `max_tokens` | integer | No | Maximum tokens to return (default: 4096) |
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
@@ -144,21 +160,6 @@ Search memories to provide personalized responses.
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"id": "fact_abc123",
|
||||
"text": "User prefers Python over JavaScript for backend development",
|
||||
"type": "world",
|
||||
"context": "programming_preferences",
|
||||
"event_date": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Start of conversation to recall relevant context
|
||||
- Before making recommendations
|
||||
@@ -195,10 +196,107 @@ Generate thoughtful analysis by synthesizing stored memories with the bank's per
|
||||
|
||||
---
|
||||
|
||||
### create_mental_model
|
||||
|
||||
Create a mental model — a living document that stays current with your memories. Mental models are pre-computed reflections that get automatically refreshed as new memories are stored.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `name` | string | Yes | Human-readable name for the mental model |
|
||||
| `source_query` | string | Yes | The query used to generate and refresh the model |
|
||||
| `tags` | list[string] | No | Tags for organizing and filtering models |
|
||||
| `max_tokens` | integer | No | Maximum tokens for model content (default: 2048) |
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
{
|
||||
"name": "create_mental_model",
|
||||
"arguments": {
|
||||
"name": "Team Directory",
|
||||
"source_query": "Who works here and what do they do?",
|
||||
"tags": ["team", "people"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Content generation runs asynchronously. The response includes an `operation_id` to track progress.
|
||||
|
||||
---
|
||||
|
||||
### list_mental_models
|
||||
|
||||
List all mental models in a bank, optionally filtered by tags.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `tags` | list[string] | No | Filter models by tags |
|
||||
|
||||
---
|
||||
|
||||
### get_mental_model
|
||||
|
||||
Retrieve a specific mental model by ID, including its full content.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `mental_model_id` | string | Yes | The ID of the mental model to retrieve |
|
||||
|
||||
---
|
||||
|
||||
### update_mental_model
|
||||
|
||||
Update a mental model's metadata or settings.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `mental_model_id` | string | Yes | The ID of the mental model to update |
|
||||
| `name` | string | No | New name |
|
||||
| `source_query` | string | No | New source query |
|
||||
| `tags` | list[string] | No | New tags |
|
||||
| `max_tokens` | integer | No | New max tokens |
|
||||
|
||||
---
|
||||
|
||||
### delete_mental_model
|
||||
|
||||
Permanently delete a mental model.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `mental_model_id` | string | Yes | The ID of the mental model to delete |
|
||||
|
||||
---
|
||||
|
||||
### refresh_mental_model
|
||||
|
||||
Re-generate a mental model's content from the latest memories. Runs asynchronously.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `mental_model_id` | string | Yes | The ID of the mental model to refresh |
|
||||
|
||||
---
|
||||
|
||||
### list_banks (multi-bank mode only)
|
||||
|
||||
List all available memory banks.
|
||||
|
||||
---
|
||||
|
||||
### create_bank (multi-bank mode only)
|
||||
|
||||
Create a new memory bank or retrieve an existing one.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|------|----------|-------------|
|
||||
| `bank_id` | string | Yes | The ID for the new bank |
|
||||
|
||||
---
|
||||
|
||||
## Integration with AI Assistants
|
||||
|
||||
The MCP server can be used with any MCP-compatible AI assistant. See the [Authentication](#authentication) section above for Claude Code and Claude Desktop configuration examples.
|
||||
|
||||
Each user can have their own configuration pointing to their personal memory bank using either:
|
||||
- The `X-Bank-Id` header (recommended)
|
||||
- A bank-specific URL path like `/mcp/alice/`
|
||||
- A bank-specific URL path like `/mcp/alice/` (recommended)
|
||||
- The `X-Bank-Id` header
|
||||
|
||||
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
|
||||
|
||||
## Cloud Mode Setup
|
||||
|
||||
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
|
||||
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
|
||||
2. An API key from your team admin
|
||||
3. A bank ID for your project (e.g., `team-acme-frontend`)
|
||||
|
||||
|
||||
@@ -226,7 +226,7 @@ const config: Config = {
|
||||
className: 'navbar-item-changelog',
|
||||
},
|
||||
{
|
||||
href: 'https://vectorize.io/hindsight/cloud',
|
||||
href: 'https://ui.hindsight.vectorize.io/signup',
|
||||
position: 'right',
|
||||
label: 'Hindsight Cloud',
|
||||
className: 'navbar-item-cloud',
|
||||
@@ -285,7 +285,7 @@ const config: Config = {
|
||||
},
|
||||
{
|
||||
label: 'Hindsight Cloud',
|
||||
href: 'https://vectorize.io/hindsight/cloud',
|
||||
href: 'https://ui.hindsight.vectorize.io/signup',
|
||||
},
|
||||
],
|
||||
},
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
import React, { useState, useCallback } from 'react';
|
||||
import styles from './styles.module.css';
|
||||
|
||||
export default function CopyPageButton(): JSX.Element | null {
|
||||
const [copied, setCopied] = useState(false);
|
||||
|
||||
const copyPageAsMarkdown = useCallback(async () => {
|
||||
try {
|
||||
// Get the page content
|
||||
const contentElement = document.querySelector('.markdown');
|
||||
if (!contentElement) return;
|
||||
|
||||
// Convert HTML to markdown-like text
|
||||
let markdown = '';
|
||||
|
||||
// Add title
|
||||
const title = document.querySelector('h1')?.textContent;
|
||||
if (title) {
|
||||
markdown += `# ${title}\n\n`;
|
||||
}
|
||||
|
||||
// Extract text content from the markdown container
|
||||
const extractMarkdown = (element: Element): string => {
|
||||
let text = '';
|
||||
|
||||
const processNode = (node: Node): string => {
|
||||
if (node.nodeType === Node.TEXT_NODE) {
|
||||
return node.textContent || '';
|
||||
}
|
||||
|
||||
if (node.nodeType === Node.ELEMENT_NODE) {
|
||||
const el = node as Element;
|
||||
const tagName = el.tagName.toLowerCase();
|
||||
const children = Array.from(el.childNodes).map(processNode).join('');
|
||||
|
||||
switch (tagName) {
|
||||
case 'h1':
|
||||
return `# ${children}\n\n`;
|
||||
case 'h2':
|
||||
return `## ${children}\n\n`;
|
||||
case 'h3':
|
||||
return `### ${children}\n\n`;
|
||||
case 'h4':
|
||||
return `#### ${children}\n\n`;
|
||||
case 'h5':
|
||||
return `##### ${children}\n\n`;
|
||||
case 'h6':
|
||||
return `###### ${children}\n\n`;
|
||||
case 'p':
|
||||
return `${children}\n\n`;
|
||||
case 'ul':
|
||||
return `${children}\n`;
|
||||
case 'ol':
|
||||
return `${children}\n`;
|
||||
case 'li':
|
||||
const parent = el.parentElement;
|
||||
const isOrdered = parent?.tagName.toLowerCase() === 'ol';
|
||||
if (isOrdered) {
|
||||
const index = Array.from(parent?.children || []).indexOf(el) + 1;
|
||||
return `${index}. ${children}\n`;
|
||||
}
|
||||
return `- ${children}\n`;
|
||||
case 'code':
|
||||
const isBlock = el.parentElement?.tagName.toLowerCase() === 'pre';
|
||||
if (isBlock) {
|
||||
const lang = el.className.replace('language-', '');
|
||||
return `\`\`\`${lang}\n${children}\n\`\`\`\n\n`;
|
||||
}
|
||||
return `\`${children}\``;
|
||||
case 'pre':
|
||||
return children; // Already handled by code block
|
||||
case 'blockquote':
|
||||
return children.split('\n').map(line => `> ${line}`).join('\n') + '\n\n';
|
||||
case 'a':
|
||||
const href = el.getAttribute('href') || '';
|
||||
return `[${children}](${href})`;
|
||||
case 'strong':
|
||||
case 'b':
|
||||
return `**${children}**`;
|
||||
case 'em':
|
||||
case 'i':
|
||||
return `*${children}*`;
|
||||
case 'br':
|
||||
return '\n';
|
||||
case 'hr':
|
||||
return '---\n\n';
|
||||
case 'table':
|
||||
return `${children}\n`;
|
||||
case 'thead':
|
||||
case 'tbody':
|
||||
return children;
|
||||
case 'tr':
|
||||
return `${children}|\n`;
|
||||
case 'th':
|
||||
case 'td':
|
||||
return `| ${children} `;
|
||||
case 'img':
|
||||
const src = el.getAttribute('src') || '';
|
||||
const alt = el.getAttribute('alt') || '';
|
||||
return ``;
|
||||
default:
|
||||
return children;
|
||||
}
|
||||
}
|
||||
|
||||
return '';
|
||||
};
|
||||
|
||||
Array.from(element.childNodes).forEach(node => {
|
||||
text += processNode(node);
|
||||
});
|
||||
|
||||
return text;
|
||||
};
|
||||
|
||||
// Skip the title h1 if it's already added
|
||||
const contentToCopy = Array.from(contentElement.children)
|
||||
.filter(child => !(child.tagName === 'H1' && child.textContent === title))
|
||||
.map(child => extractMarkdown(child))
|
||||
.join('');
|
||||
|
||||
markdown += contentToCopy;
|
||||
|
||||
// Clean up excessive newlines
|
||||
markdown = markdown.replace(/\n{3,}/g, '\n\n').trim();
|
||||
|
||||
// Copy to clipboard
|
||||
await navigator.clipboard.writeText(markdown);
|
||||
|
||||
setCopied(true);
|
||||
setTimeout(() => setCopied(false), 2000);
|
||||
} catch (error) {
|
||||
console.error('Failed to copy page content:', error);
|
||||
}
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<button
|
||||
className={`${styles.copyPageButton} ${copied ? styles.copied : ''}`}
|
||||
onClick={copyPageAsMarkdown}
|
||||
aria-label="Copy page as markdown"
|
||||
title="Copy page as markdown"
|
||||
>
|
||||
{copied ? (
|
||||
<>
|
||||
<svg width="16" height="16" viewBox="0 0 16 16" fill="currentColor">
|
||||
<path d="M12.736 3.97a.733.733 0 0 1 1.047 0c.286.289.29.756.01 1.05L7.88 12.01a.733.733 0 0 1-1.065.02L3.217 8.384a.757.757 0 0 1 0-1.06.733.733 0 0 1 1.047 0l3.052 3.093 5.4-6.425a.247.247 0 0 1 .02-.022Z"/>
|
||||
</svg>
|
||||
<span className={styles.buttonText}>Copied!</span>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<svg width="16" height="16" viewBox="0 0 16 16" fill="currentColor">
|
||||
<path d="M4 2a2 2 0 0 1 2-2h8a2 2 0 0 1 2 2v8a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V2zm2-1a1 1 0 0 0-1 1v8a1 1 0 0 0 1 1h8a1 1 0 0 0 1-1V2a1 1 0 0 0-1-1H6z"/>
|
||||
<path d="M2 5a1 1 0 0 0-1 1v8a1 1 0 0 0 1 1h8a1 1 0 0 0 1-1v-1h1v1a2 2 0 0 1-2 2H2a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2h1v1H2z"/>
|
||||
</svg>
|
||||
<span className={styles.buttonText}>Copy page</span>
|
||||
</>
|
||||
)}
|
||||
</button>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
.copyPageButton {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
padding: 6px 12px;
|
||||
background: transparent;
|
||||
border: 1px solid var(--ifm-color-emphasis-300);
|
||||
border-radius: 6px;
|
||||
color: var(--ifm-font-color-base);
|
||||
font-size: 14px;
|
||||
font-weight: 500;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s ease;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.copyPageButton:hover {
|
||||
background-color: var(--ifm-color-emphasis-100);
|
||||
border-color: var(--ifm-color-emphasis-400);
|
||||
}
|
||||
|
||||
.copyPageButton:active {
|
||||
transform: translateY(1px);
|
||||
}
|
||||
|
||||
.copyPageButton.copied {
|
||||
background-color: var(--ifm-color-success-contrast-background);
|
||||
border-color: var(--ifm-color-success);
|
||||
color: var(--ifm-color-success-darkest);
|
||||
}
|
||||
|
||||
.copyPageButton.copied:hover {
|
||||
background-color: var(--ifm-color-success-contrast-background);
|
||||
border-color: var(--ifm-color-success);
|
||||
}
|
||||
|
||||
.buttonText {
|
||||
margin: 0 4px;
|
||||
}
|
||||
|
||||
/* Dark mode adjustments */
|
||||
[data-theme='dark'] .copyPageButton {
|
||||
border-color: var(--ifm-color-emphasis-400);
|
||||
}
|
||||
|
||||
[data-theme='dark'] .copyPageButton:hover {
|
||||
background-color: var(--ifm-color-emphasis-200);
|
||||
border-color: var(--ifm-color-emphasis-500);
|
||||
}
|
||||
|
||||
[data-theme='dark'] .copyPageButton.copied {
|
||||
background-color: var(--ifm-color-success-dark);
|
||||
border-color: var(--ifm-color-success);
|
||||
color: var(--ifm-color-success-contrast-foreground);
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import React from 'react';
|
||||
import DocItemContent from '@theme-original/DocItem/Content';
|
||||
import type DocItemContentType from '@theme/DocItem/Content';
|
||||
import type { WrapperProps } from '@docusaurus/types';
|
||||
import CopyPageButton from '@site/src/components/CopyPageButton';
|
||||
import styles from './styles.module.css';
|
||||
|
||||
type Props = WrapperProps<typeof DocItemContentType>;
|
||||
|
||||
export default function DocItemContentWrapper(props: Props): JSX.Element {
|
||||
return (
|
||||
<>
|
||||
<div className={styles.docItemHeader}>
|
||||
<div className={styles.docItemActions}>
|
||||
<CopyPageButton />
|
||||
</div>
|
||||
</div>
|
||||
<DocItemContent {...props} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
.docItemHeader {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
align-items: center;
|
||||
margin-bottom: 1rem;
|
||||
padding-bottom: 0.5rem;
|
||||
border-bottom: 1px solid var(--ifm-color-emphasis-200);
|
||||
}
|
||||
|
||||
.docItemActions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
@media (max-width: 768px) {
|
||||
.docItemHeader {
|
||||
margin-bottom: 0.75rem;
|
||||
}
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
|
||||
|
||||
:::tip Don't want to manage infrastructure?
|
||||
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
|
||||
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
|
||||
:::
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
|
||||
|
||||
## Cloud Mode Setup
|
||||
|
||||
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
|
||||
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
|
||||
2. An API key from your team admin
|
||||
3. A bank ID for your project (e.g., `team-acme-frontend`)
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
|
||||
|
||||
:::tip Don't want to manage infrastructure?
|
||||
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
|
||||
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
|
||||
:::
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
|
||||
|
||||
## Cloud Mode Setup
|
||||
|
||||
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
|
||||
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
|
||||
2. An API key from your team admin
|
||||
3. A bank ID for your project (e.g., `team-acme-frontend`)
|
||||
|
||||
|
||||
@@ -2,19 +2,60 @@
|
||||
|
||||
E2E and integration tests for Hindsight API that require a running server.
|
||||
|
||||
## Running Tests
|
||||
## Test Types
|
||||
|
||||
1. Start the API server:
|
||||
```bash
|
||||
./scripts/dev/start-api.sh
|
||||
```
|
||||
### 1. Tests with External Server
|
||||
Tests like `test_mcp_e2e.py` expect a server to already be running.
|
||||
|
||||
2. Run the tests:
|
||||
```bash
|
||||
cd hindsight-integration-tests
|
||||
HINDSIGHT_API_URL=http://localhost:8888 uv run pytest tests/ -v
|
||||
```
|
||||
**Running:**
|
||||
```bash
|
||||
# Start the API server
|
||||
./scripts/dev/start-api.sh
|
||||
|
||||
# Run tests
|
||||
cd hindsight-integration-tests
|
||||
HINDSIGHT_API_URL=http://localhost:8888 uv run pytest tests/test_mcp_e2e.py -v
|
||||
```
|
||||
|
||||
### 2. Self-Contained Tests
|
||||
Tests like `test_base_path_deployment.py` manage their own server lifecycle and use docker-compose.
|
||||
|
||||
**Running:**
|
||||
```bash
|
||||
cd hindsight-integration-tests
|
||||
|
||||
# Run with pytest
|
||||
uv run pytest tests/test_base_path_deployment.py -v
|
||||
|
||||
# Or run directly for nice output
|
||||
uv run python tests/test_base_path_deployment.py
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- Docker and docker-compose installed (for reverse proxy test)
|
||||
- No nginx required on host!
|
||||
|
||||
**What it tests:**
|
||||
- ✅ API with base path (direct server)
|
||||
- ✅ Full reverse proxy via docker-compose + Nginx
|
||||
- ✅ Regression: API without base path
|
||||
- ✅ Full retain/recall workflow
|
||||
|
||||
These tests:
|
||||
- Start their own API servers on dedicated ports (18888-18891)
|
||||
- Use docker-compose to test actual deployment scenarios
|
||||
- Run in parallel with other tests (no port conflicts)
|
||||
- Clean up automatically
|
||||
|
||||
## Running All Tests
|
||||
|
||||
```bash
|
||||
cd hindsight-integration-tests
|
||||
uv run pytest tests/ -v
|
||||
```
|
||||
|
||||
This runs both types. Self-contained tests won't conflict with the external server.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `HINDSIGHT_API_URL` - Base URL of the running Hindsight API (default: `http://localhost:8888`)
|
||||
- `HINDSIGHT_API_URL` - Base URL for external-server tests (default: `http://localhost:8888`)
|
||||
|
||||
@@ -0,0 +1,494 @@
|
||||
"""
|
||||
Integration test for base path deployment using Docker Compose.
|
||||
|
||||
This test validates that Hindsight works correctly when deployed
|
||||
behind a reverse proxy with path-based routing using the actual
|
||||
Docker Compose examples from docker/compose-examples/.
|
||||
|
||||
Tests:
|
||||
1. API with base path (direct, no proxy)
|
||||
2. Full stack via docker-compose with Nginx reverse proxy
|
||||
3. Regression: API without base path still works
|
||||
|
||||
Requirements:
|
||||
- Docker and docker-compose installed
|
||||
- No nginx required on host!
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
# Paths
|
||||
REPO_ROOT = Path(__file__).parent.parent.parent
|
||||
API_PATH = REPO_ROOT / "hindsight-api"
|
||||
COMPOSE_EXAMPLES_PATH = REPO_ROOT / "docker" / "docker-compose" / "nginx"
|
||||
|
||||
# Add hindsight-api to path for direct API testing
|
||||
sys.path.insert(0, str(API_PATH))
|
||||
|
||||
|
||||
def run_command(cmd: list[str], cwd: str | Path | None = None, env: dict | None = None) -> subprocess.CompletedProcess:
|
||||
"""Run a command and return the result."""
|
||||
return subprocess.run(
|
||||
cmd,
|
||||
cwd=cwd,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
def check_docker_available() -> bool:
|
||||
"""Check if Docker is available."""
|
||||
result = run_command(["docker", "info"])
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def get_docker_compose_command() -> list[str]:
|
||||
"""Get the docker-compose command (modern or legacy)."""
|
||||
import shutil
|
||||
# Try modern docker compose plugin first
|
||||
if shutil.which("docker"):
|
||||
result = run_command(["docker", "compose", "version"])
|
||||
if result.returncode == 0:
|
||||
return ["docker", "compose"]
|
||||
# Fall back to legacy docker-compose
|
||||
if shutil.which("docker-compose"):
|
||||
return ["docker-compose"]
|
||||
raise RuntimeError("docker-compose not available")
|
||||
|
||||
|
||||
def check_docker_compose_available() -> bool:
|
||||
"""Check if docker-compose is available."""
|
||||
try:
|
||||
get_docker_compose_command()
|
||||
return True
|
||||
except RuntimeError:
|
||||
return False
|
||||
|
||||
|
||||
class APIServer:
|
||||
"""Helper to manage API server lifecycle for direct testing."""
|
||||
|
||||
def __init__(self, base_path: str | None = None, port: int = 18888):
|
||||
self.base_path = base_path
|
||||
self.port = port
|
||||
self.process = None
|
||||
self.env = os.environ.copy()
|
||||
if base_path:
|
||||
self.env["HINDSIGHT_API_BASE_PATH"] = base_path
|
||||
|
||||
def start(self):
|
||||
"""Start the API server."""
|
||||
cmd = [
|
||||
"uv",
|
||||
"run",
|
||||
"--directory",
|
||||
str(API_PATH),
|
||||
"hindsight-api",
|
||||
"--host",
|
||||
"0.0.0.0",
|
||||
"--port",
|
||||
str(self.port),
|
||||
]
|
||||
|
||||
log_file = f"/tmp/hindsight-api-{self.port}.log"
|
||||
self.log_file = open(log_file, "w")
|
||||
|
||||
self.process = subprocess.Popen(
|
||||
cmd, env=self.env, stdout=self.log_file, stderr=subprocess.STDOUT
|
||||
)
|
||||
|
||||
# Wait for server to be ready
|
||||
base_url = f"http://localhost:{self.port}"
|
||||
if self.base_path:
|
||||
health_url = f"{base_url}{self.base_path}/health"
|
||||
else:
|
||||
health_url = f"{base_url}/health"
|
||||
|
||||
for _ in range(60): # 60 second timeout
|
||||
try:
|
||||
response = httpx.get(health_url, timeout=2.0)
|
||||
if response.status_code == 200:
|
||||
return
|
||||
except (httpx.ConnectError, httpx.ReadTimeout):
|
||||
pass
|
||||
time.sleep(1)
|
||||
|
||||
# Failed to start
|
||||
self.log_file.flush()
|
||||
with open(log_file) as f:
|
||||
print(f"API server failed to start. Logs:\n{f.read()}")
|
||||
raise RuntimeError(f"API server failed to start on port {self.port}")
|
||||
|
||||
def stop(self):
|
||||
"""Stop the API server."""
|
||||
if self.process:
|
||||
self.process.terminate()
|
||||
try:
|
||||
self.process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.process.kill()
|
||||
self.process.wait()
|
||||
self.process = None
|
||||
|
||||
if hasattr(self, "log_file") and self.log_file:
|
||||
self.log_file.close()
|
||||
|
||||
def __enter__(self):
|
||||
self.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
self.stop()
|
||||
|
||||
|
||||
class DockerComposeStack:
|
||||
"""Helper to manage docker-compose stack lifecycle."""
|
||||
|
||||
def __init__(self, compose_file: Path, project_name: str = "hindsight-test"):
|
||||
self.compose_file = compose_file
|
||||
self.project_name = project_name
|
||||
self.compose_cmd = get_docker_compose_command()
|
||||
self.env = os.environ.copy()
|
||||
# Set required env vars for docker-compose
|
||||
self.env["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "test-key")
|
||||
self.env["HINDSIGHT_API_LLM_PROVIDER"] = os.environ.get("HINDSIGHT_API_LLM_PROVIDER", "mock")
|
||||
self.env["HINDSIGHT_API_LLM_MODEL"] = os.environ.get("HINDSIGHT_API_LLM_MODEL", "mock-model")
|
||||
|
||||
def start(self, timeout: int = 120):
|
||||
"""Start the docker-compose stack."""
|
||||
print(f"Starting docker-compose stack: {self.compose_file.name}")
|
||||
|
||||
# Pull images first (but don't fail if it doesn't work)
|
||||
run_command(
|
||||
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "pull"],
|
||||
cwd=self.compose_file.parent,
|
||||
env=self.env,
|
||||
)
|
||||
|
||||
# Start services
|
||||
result = run_command(
|
||||
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "up", "-d", "--build"],
|
||||
cwd=self.compose_file.parent,
|
||||
env=self.env,
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f"Failed to start docker-compose:\nSTDOUT: {result.stdout}\nSTDERR: {result.stderr}")
|
||||
raise RuntimeError("Failed to start docker-compose stack")
|
||||
|
||||
# Wait for services to be healthy
|
||||
start_time = time.time()
|
||||
while time.time() - start_time < timeout:
|
||||
result = run_command(
|
||||
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "ps", "--format", "json"],
|
||||
cwd=self.compose_file.parent,
|
||||
env=self.env,
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
# Give it a few more seconds to fully initialize
|
||||
time.sleep(5)
|
||||
return
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
# Timeout - show logs and fail
|
||||
self.show_logs()
|
||||
raise RuntimeError(f"Docker compose stack failed to start within {timeout}s")
|
||||
|
||||
def show_logs(self):
|
||||
"""Show docker-compose logs."""
|
||||
result = run_command(
|
||||
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "logs", "--tail=100"],
|
||||
cwd=self.compose_file.parent,
|
||||
env=self.env,
|
||||
)
|
||||
print(f"Docker compose logs:\n{result.stdout}\n{result.stderr}")
|
||||
|
||||
def stop(self):
|
||||
"""Stop and remove the docker-compose stack."""
|
||||
print(f"Stopping docker-compose stack: {self.compose_file.name}")
|
||||
result = run_command(
|
||||
self.compose_cmd + ["-f", str(self.compose_file), "-p", self.project_name, "down", "-v"],
|
||||
cwd=self.compose_file.parent,
|
||||
env=self.env,
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
print(f"Warning: Failed to stop docker-compose:\n{result.stderr}")
|
||||
|
||||
def __enter__(self):
|
||||
self.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
self.stop()
|
||||
|
||||
|
||||
def test_api_without_base_path():
|
||||
"""Regression test: API works at root path (default behavior)."""
|
||||
with APIServer(base_path=None, port=18888) as server:
|
||||
base_url = f"http://localhost:{server.port}"
|
||||
|
||||
# Health check
|
||||
response = httpx.get(f"{base_url}/health")
|
||||
assert response.status_code == 200
|
||||
assert "status" in response.json()
|
||||
|
||||
# API endpoints
|
||||
response = httpx.get(f"{base_url}/v1/default/banks")
|
||||
assert response.status_code == 200
|
||||
assert "banks" in response.json()
|
||||
|
||||
# OpenAPI docs
|
||||
response = httpx.get(f"{base_url}/docs")
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_api_with_base_path_direct():
|
||||
"""Test API with base path configuration (direct, no proxy)."""
|
||||
base_path = "/hindsight"
|
||||
|
||||
with APIServer(base_path=base_path, port=18889) as server:
|
||||
base_url = f"http://localhost:{server.port}"
|
||||
|
||||
# Base path SHOULD work
|
||||
response = httpx.get(f"{base_url}{base_path}/health")
|
||||
assert response.status_code == 200
|
||||
assert "status" in response.json()
|
||||
|
||||
# API endpoints with base path
|
||||
response = httpx.get(f"{base_url}{base_path}/v1/default/banks")
|
||||
assert response.status_code == 200
|
||||
assert "banks" in response.json()
|
||||
|
||||
# OpenAPI docs with base path
|
||||
response = httpx.get(f"{base_url}{base_path}/docs")
|
||||
assert response.status_code == 200
|
||||
|
||||
# OpenAPI schema should have correct server URL
|
||||
response = httpx.get(f"{base_url}{base_path}/openapi.json")
|
||||
assert response.status_code == 200
|
||||
openapi = response.json()
|
||||
assert "servers" in openapi
|
||||
assert openapi["servers"][0]["url"] == base_path
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not check_docker_compose_available(),
|
||||
reason="docker-compose not available"
|
||||
)
|
||||
def test_reverse_proxy_simple_config():
|
||||
"""
|
||||
Test reverse proxy deployment using docker-compose with Nginx.
|
||||
|
||||
This creates a minimal test setup with:
|
||||
- API server running on HOST via uv (not Docker - faster, no image build needed!)
|
||||
- Nginx container that proxies to the host API
|
||||
|
||||
This tests the actual reverse proxy scenario without requiring the
|
||||
heavy Hindsight Docker image.
|
||||
"""
|
||||
base_path = "/hindsight"
|
||||
api_port = 18890
|
||||
|
||||
# Start API on host with base path
|
||||
with APIServer(base_path=base_path, port=api_port):
|
||||
# Create test docker-compose file (nginx only)
|
||||
test_compose = COMPOSE_EXAMPLES_PATH / "test-reverse-proxy.yml"
|
||||
|
||||
# Determine host address for nginx to reach host machine
|
||||
# host.docker.internal works on Docker Desktop (Mac/Windows)
|
||||
# On Linux, we use host network mode
|
||||
import platform
|
||||
if platform.system() == "Linux":
|
||||
network_mode = "host"
|
||||
api_host = "localhost"
|
||||
nginx_port = 18080 # With host mode, nginx must listen on 18080 directly
|
||||
port_mapping = "" # No port mapping with host mode
|
||||
else:
|
||||
network_mode = "bridge"
|
||||
api_host = "host.docker.internal"
|
||||
nginx_port = 80 # With bridge mode, nginx listens on 80 and is mapped
|
||||
port_mapping = """ ports:
|
||||
- "18080:80"
|
||||
"""
|
||||
|
||||
compose_content = f"""version: '3.8'
|
||||
|
||||
services:
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
{port_mapping} volumes:
|
||||
- ./test-nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
network_mode: {network_mode}
|
||||
"""
|
||||
|
||||
# Create test nginx config
|
||||
nginx_config = f"""events {{
|
||||
worker_connections 1024;
|
||||
}}
|
||||
|
||||
http {{
|
||||
server {{
|
||||
listen {nginx_port};
|
||||
server_name localhost;
|
||||
|
||||
location {base_path}/ {{
|
||||
proxy_pass http://{api_host}:{api_port};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
"""
|
||||
|
||||
# Write test files
|
||||
test_compose.write_text(compose_content)
|
||||
test_nginx_conf = COMPOSE_EXAMPLES_PATH / "test-nginx.conf"
|
||||
test_nginx_conf.write_text(nginx_config)
|
||||
|
||||
try:
|
||||
# Start nginx via docker-compose
|
||||
with DockerComposeStack(test_compose, project_name="hindsight-base-path-test"):
|
||||
proxy_url = "http://localhost:18080"
|
||||
|
||||
# Give nginx a moment to start
|
||||
time.sleep(2)
|
||||
|
||||
# Test through nginx proxy
|
||||
response = httpx.get(f"{proxy_url}{base_path}/health", timeout=10.0)
|
||||
assert response.status_code == 200
|
||||
assert "status" in response.json()
|
||||
|
||||
# API endpoints through proxy
|
||||
response = httpx.get(f"{proxy_url}{base_path}/v1/default/banks", timeout=10.0)
|
||||
assert response.status_code == 200
|
||||
assert "banks" in response.json()
|
||||
|
||||
# OpenAPI docs through proxy
|
||||
response = httpx.get(f"{proxy_url}{base_path}/docs", timeout=10.0)
|
||||
assert response.status_code == 200
|
||||
|
||||
finally:
|
||||
# Cleanup test files
|
||||
test_compose.unlink(missing_ok=True)
|
||||
test_nginx_conf.unlink(missing_ok=True)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_full_workflow_with_base_path():
|
||||
"""Test full retain/recall workflow through base path."""
|
||||
base_path = "/hindsight"
|
||||
bank_id = "integration_test_bank"
|
||||
|
||||
with APIServer(base_path=base_path, port=18891) as server:
|
||||
base_url = f"http://localhost:{server.port}{base_path}"
|
||||
|
||||
async with httpx.AsyncClient(base_url=base_url, timeout=30.0) as client:
|
||||
# 1. Get bank profile (creates if needed)
|
||||
response = await client.get(f"/v1/default/banks/{bank_id}/profile")
|
||||
assert response.status_code == 200
|
||||
|
||||
# 2. Store a memory
|
||||
response = await client.post(
|
||||
f"/v1/default/banks/{bank_id}/memories",
|
||||
json={
|
||||
"items": [
|
||||
{
|
||||
"content": "Hindsight supports deployment under custom base paths for reverse proxy scenarios.",
|
||||
"context": "integration test"
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
result = response.json()
|
||||
assert result["success"] is True
|
||||
|
||||
# 3. Recall the memory
|
||||
response = await client.post(
|
||||
f"/v1/default/banks/{bank_id}/memories/recall",
|
||||
json={"query": "base path deployment"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
recall_result = response.json()
|
||||
assert "results" in recall_result
|
||||
# Should find our memory
|
||||
assert len(recall_result["results"]) > 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
"""Run tests directly with python."""
|
||||
import sys
|
||||
|
||||
print("=" * 70)
|
||||
print("Hindsight Base Path Integration Tests")
|
||||
print("=" * 70)
|
||||
print()
|
||||
|
||||
all_passed = True
|
||||
|
||||
# Test 1: Without base path
|
||||
print("Test 1: API without base path (regression test)")
|
||||
print("-" * 70)
|
||||
try:
|
||||
test_api_without_base_path()
|
||||
print("✅ PASSED\n")
|
||||
except Exception as e:
|
||||
print(f"❌ FAILED: {e}\n")
|
||||
all_passed = False
|
||||
|
||||
# Test 2: With base path (direct)
|
||||
print("Test 2: API with base path (direct, no proxy)")
|
||||
print("-" * 70)
|
||||
try:
|
||||
test_api_with_base_path_direct()
|
||||
print("✅ PASSED\n")
|
||||
except Exception as e:
|
||||
print(f"❌ FAILED: {e}\n")
|
||||
all_passed = False
|
||||
|
||||
# Test 3: Docker compose reverse proxy
|
||||
print("Test 3: Reverse proxy via docker-compose")
|
||||
print("-" * 70)
|
||||
if check_docker_available() and check_docker_compose_available():
|
||||
try:
|
||||
test_reverse_proxy_simple_config()
|
||||
print("✅ PASSED\n")
|
||||
except Exception as e:
|
||||
print(f"❌ FAILED: {e}\n")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
all_passed = False
|
||||
else:
|
||||
print("⚠️ SKIPPED: Docker or docker-compose not available\n")
|
||||
|
||||
# Test 4: Full workflow
|
||||
print("Test 4: Full retain/recall workflow with base path")
|
||||
print("-" * 70)
|
||||
try:
|
||||
asyncio.run(test_full_workflow_with_base_path())
|
||||
print("✅ PASSED\n")
|
||||
except Exception as e:
|
||||
print(f"❌ FAILED: {e}\n")
|
||||
all_passed = False
|
||||
|
||||
# Summary
|
||||
print("=" * 70)
|
||||
if all_passed:
|
||||
print("✅ All tests passed!")
|
||||
sys.exit(0)
|
||||
else:
|
||||
print("❌ Some tests failed")
|
||||
sys.exit(1)
|
||||
@@ -3,7 +3,7 @@
|
||||
Hindsight can be deployed in several ways depending on your infrastructure and requirements.
|
||||
|
||||
:::tip Don't want to manage infrastructure?
|
||||
**[Hindsight Cloud](https://vectorize.io/hindsight/cloud)** is a fully managed service that handles all infrastructure, scaling, and maintenance. We're onboarding design partners now — [request early access](https://vectorize.io/hindsight/cloud).
|
||||
**[Hindsight Cloud](https://ui.hindsight.vectorize.io/signup)** is a fully managed service that handles all infrastructure, scaling, and maintenance — [sign up here](https://ui.hindsight.vectorize.io/signup).
|
||||
:::
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -161,11 +161,11 @@ uvx hindsight-embed configure
|
||||
|
||||
## Cloud Mode Setup
|
||||
|
||||
Cloud mode connects to [Hindsight Cloud](https://vectorize.io/hindsight/cloud), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
Cloud mode connects to [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup), allowing teams to share memories about a codebase. When one team member learns something, everyone benefits.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
1. A Hindsight Cloud account ([request access](https://vectorize.io/hindsight/cloud))
|
||||
1. A Hindsight Cloud account ([sign up](https://ui.hindsight.vectorize.io/signup))
|
||||
2. An API key from your team admin
|
||||
3. A bank ID for your project (e.g., `team-acme-frontend`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user