Compare commits

...
6 Commits
Author SHA1 Message Date
Nicolò Boschi 7972fd3906 feat: support litellm-sdk for reranker endpoint 2026-02-12 14:43:35 +01:00
Nicolò Boschi 86b698460e chore: remove dead code 2026-02-12 14:21:53 +01:00
Nicolò Boschi 6f9cef674b chore: remove dead code 2026-02-12 14:21:26 +01:00
Nicolò Boschi c029807add feat: support for other text and vector search pg extensions (#355)
* feat: support for other text and vector search pg extensions

* test: increase timeout for test_batch_chunking_behavior to account for VectorChord BM25 tokenization overhead

* feat: support for other text and vector search pg extensions
2026-02-12 14:13:04 +01:00
Nicolò Boschi 8d731f2e5f feat: implement hierarchical configuration (system, tenant, bank) (#329)
* feat: implement hierarchical configuration (system, tenant, bank)

* feat: implement hierarchical configuration (system, tenant, bank)

* docs: add instructions for hierarchical config in CLAUDE.md

* feat: add ENABLE_BANK_CONFIG_API flag (disabled by default)

- Add HINDSIGHT_API_ENABLE_BANK_CONFIG_API env var (default: false)
- Return 403 Forbidden from bank config endpoints when disabled
- Update tests to enable the flag
- Update CLAUDE.md documentation

This provides security control over the bank configuration API,
ensuring it's only accessible when explicitly enabled.

* docs: add hierarchical configuration section

* feat(cli): add bank config commands (config, set-config, reset-config)

- Add 'hindsight bank config' to view bank configuration
- Add 'hindsight bank set-config' to update LLM settings per bank
- Add 'hindsight bank reset-config' to reset to defaults
- Implements client API calls to new bank config endpoints

* fix(cli): fix compilation errors in bank config commands

- Fix type signature: use ApiClient instead of api::Client
- Fix confirmation: use ui::prompt_confirmation instead of ui::confirm
- Fix error handling: use anyhow! macro instead of errors::Error
- Fix type conversion: convert HashMap to serde_json::Map for API call

* feat: implement type-safe hierarchical config with bank overrides

Implements a production-ready hierarchical configuration system that prevents
accidentally using global defaults when bank-specific overrides exist.

- Created StaticConfigProxy that wraps HindsightConfig
- get_config() now returns proxy that blocks access to bank-configurable fields
- Raises ConfigFieldAccessError with clear message when accessing configurable fields
- Added _get_raw_config() for internal use only
- Forces developers to use resolve_full_config(bank_id, context) for bank settings

- Added resolve_full_config() method that returns complete HindsightConfig
- Resolves hierarchy: Global (env) → Tenant → Bank
- No caching to support multi-server deployments (always fresh from DB)
- LLM provider pooling handles expensive operations separately

- Updated entire retain pipeline to pass resolved config through call chain
- memory_engine.py: Resolves config at top level where bank_id/context available
- orchestrator.py: Accepts and passes config to fact_extraction
- fact_extraction.py: Uses passed config instead of get_config()
- utils.py: Added optional config param for backward compatibility

- consolidator.py: Uses resolve_full_config() for enable_observations check
- memory_engine.py: Resolves config before triggering consolidation

- Renamed "Memory Bank" to "Bank Configuration" with tabs
- Combined Stats and Operations into "General" tab
- Consolidated Profile and Configuration into "Configuration" tab
- Moved Actions dropdown to page level (outside tabs)

- Created new component for managing bank-specific config
- Displays configurable fields: retain_chunk_size, retain_extraction_mode, etc.
- Edit via dialog with form validation
- Reset to defaults via AlertDialog confirmation
- Shows field IDs in monospace for clarity
- Visual separation with borders and hover effects

- Removed inline edit mode, switched to dialog-based editing
- Separate dialogs for Disposition and Mission editing
- Read-only display with clear edit buttons
- Removed duplicate stats cards and operations

- bank-stats-view.tsx: Overview statistics (memories, links, documents, pending ops)
- bank-operations-view.tsx: Background operations table with filtering

**Problem**: Consolidation always used global enable_observations, ignoring bank overrides
**Root Cause**: consolidator.py called get_config() instead of resolving bank-specific config
**Solution**: Pass resolved config through the entire pipeline

**Problem**: asyncpg returning JSONB as JSON string instead of parsed dict
**Solution**: Explicit JSON parsing in config_resolver.py with type checking

- All 19 API integration tests pass
- All 10 hierarchical config tests pass
- Retain operations work correctly with bank-specific config
- Consolidation respects bank-specific enable_observations setting

- Updated developer/configuration.md with type-safe config access pattern
- Added examples showing correct usage patterns
- Documented ConfigFieldAccessError and resolution methods

- get_config() now returns StaticConfigProxy (blocks configurable field access)
- Code accessing bank-configurable fields must use resolve_full_config()
- Clear migration path with helpful error messages

Fixes hierarchical configuration to be production-ready with proper type safety.

* refactor: remove LLM client pool and simplify config resolver

Since LLM config (provider, model, api_key) is now static and not
bank-configurable, the LLMClientPool is no longer needed.

Changes:
- Remove hindsight_api/llm_client_pool.py (no longer needed)
- Remove memory_engine._get_bank_llm_config() (dead code, never called)
- Simplify config_resolver.py by eliminating duplication between
  resolve_full_config() and get_bank_config()
- get_bank_config() now calls resolve_full_config() and filters results
- Remove outdated "LLM provider pooling" comments from docstrings

All tests pass (10 hierarchical config tests, 19 API integration tests)

* fix: update tests to use _get_raw_config() for configurable fields

Fixed test fixtures that were accessing configurable fields (like
enable_observations) from get_config(), which now raises
ConfigFieldAccessError due to type-safe config access.

Changes:
- test_consolidation.py: Changed enable_observations fixture to use
  _get_raw_config() instead of get_config()
- test_consolidation.py: Updated test_consolidation_returns_disabled_status
  to set bank config instead of mocking get_config()
- test_link_expansion_retrieval.py: Changed fixture to use _get_raw_config()
- test_observations.py: Changed disable_observations fixture to use
  _get_raw_config()
- Regenerated OpenAPI spec and clients

All 39 previously failing tests now pass.

* fix: add missing config parameter to test calls of extract_facts_from_text()

Fixed 45 test failures where tests were calling extract_facts_from_text()
without the new required config parameter.

Changes:
- Added config=_get_raw_config() to all extract_facts_from_text() calls
- Fixed test_main_module.py to patch _get_raw_config instead of get_config
- Updated 6 test files with 37 function call sites

All tests should now pass.

* fix: add missing config parameter to test_skip_podcast_meta_commentary

One more test was missing the config parameter for extract_facts_from_text().
2026-02-12 13:14:57 +01:00
Nicolò Boschi f9a8a8e01e fix: resolve based_on schema/serialization issues in reflect API (#348)
* fix: add default values to OpenAPI schema for default_factory fields

This commit fixes the OpenAPI schema to include default values for fields
using default_factory, which improves schema accuracy and client generation.

Changes:
1. Added FieldWithDefault() helper to inject default values into OpenAPI schema
2. Updated 14 fields using default_factory to include defaults in schema:
   - ReflectBasedOn.{memories, mental_models, directives}
   - ReflectTrace.{tool_calls, llm_calls}
   - All tags fields
   - All trigger fields
   - All include fields

3. Regenerated OpenAPI spec with proper defaults

4. Added tests to verify API returns correct format with empty banks

Note: This fixes the schema but doesn't change the v0.3.0 -> v0.4.0 breaking
change where based_on went from list to object. Clients should handle both
formats for backward compatibility.

* fix: remove client imports from API test

The test was failing in CI because it imported the client library
which isn't installed in the API test environment.

Changed to test only API JSON response format, not client parsing.
This is more appropriate for an API test anyway.

* test: add client tests for ReflectResponse parsing

Added comprehensive tests in hindsight-clients/python/tests to verify:
- v0.4.0+ format with empty based_on object
- v0.4.0+ format with null based_on
- v0.4.0+ format with populated facts
- v0.3.0 format (list) correctly fails validation
- Missing based_on field handling

These tests document the v0.3.0 -> v0.4.0 breaking change where
based_on changed from list to object.
2026-02-12 11:26:12 +01:00
59 changed files with 6207 additions and 1549 deletions
+42 -6
View File
@@ -238,26 +238,61 @@ def process(data: UserData) -> str:
### Adding New API Configuration Flags
When adding a new environment variable configuration:
Configuration follows a hierarchical system: **Global (env vars) → Tenant (via extension) → Bank (database)**.
Fields must be categorized as either **hierarchical** (can be overridden per-tenant/bank) or **static** (server-level only).
#### Adding a New Configuration Field
1. **config.py** (`hindsight-api/hindsight_api/config.py`):
- Add `ENV_*` constant for the environment variable name
- Add `ENV_*` constant for the environment variable name (e.g., `ENV_MY_SETTING = "HINDSIGHT_API_MY_SETTING"`)
- Add `DEFAULT_*` constant for the default value
- Add field to `HindsightConfig` dataclass
- Add field to `HindsightConfig` dataclass with type annotation
- **Mark as hierarchical or static** by adding to `_HIERARCHICAL_FIELDS` set (hierarchical) or leaving it out (static)
- Add initialization in `from_env()` method
```python
# Hierarchical field (can be overridden per-bank)
_HIERARCHICAL_FIELDS = {
...,
"my_setting", # Add here for hierarchical
}
# Static field - just don't add to _HIERARCHICAL_FIELDS
```
2. **main.py** (`hindsight-api/hindsight_api/main.py`):
- Add field to the manual `HindsightConfig()` constructor call (search for "CLI override")
3. **Use the config** in code:
3. **Use hierarchical config in MemoryEngine**:
```python
# Config is resolved automatically per bank via ConfigResolver
config_dict = await self._config_resolver.get_bank_config(bank_id, context)
value = config_dict["my_setting"]
```
4. **Use static config** (non-hierarchical):
```python
from ...config import get_config
config = get_config()
value = config.your_new_field
value = config.my_static_field
```
4. **Documentation** (`hindsight-docs/docs/developer/configuration.md`):
5. **Documentation** (`hindsight-docs/docs/developer/configuration.md`):
- Add to appropriate section table with Variable, Description, Default
- Mark if it's hierarchical (can be overridden per-bank)
#### Hierarchical vs Static Guidelines
**Hierarchical** (per-bank overridable):
- LLM settings (provider, model, API key, base URL)
- Operation-specific settings (retain mode, chunk size, etc.)
- Feature flags that vary by customer/bank
**Static** (server-level only):
- Infrastructure settings (database URL, port, host)
- Global limits (max concurrent operations)
- System-wide feature flags
## Environment Setup
@@ -281,3 +316,4 @@ Optional (uses local models by default):
- `HINDSIGHT_API_EMBEDDINGS_PROVIDER`: local (default) or tei
- `HINDSIGHT_API_RERANKER_PROVIDER`: local (default) or tei
- `HINDSIGHT_API_DATABASE_URL`: External PostgreSQL (uses embedded pg0 by default)
- `HINDSIGHT_API_ENABLE_BANK_CONFIG_API`: Enable per-bank config API (default: false, disabled for security)
@@ -52,6 +52,8 @@ services:
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord CASCADE;';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pg_tokenizer CASCADE;';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE;';
echo 'Creating llmlingua2 tokenizer';
psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c \"SELECT create_tokenizer('llmlingua2', \\$\\$ model = \\\"llmlingua2\\\" \\$\\$);\" 2>/dev/null || echo 'Tokenizer already exists or creation skipped';
echo 'Database and extensions created successfully';
"
restart: "no"
@@ -65,25 +67,18 @@ services:
- "8888:8888"
- "9999:9999"
environment:
# LLM Configuration
- HINDSIGHT_API_LLM_PROVIDER=openai
- HINDSIGHT_API_LLM_MODEL=gpt-5-mini
# LiteLLM Configuration (shared by embeddings and reranker)
# Embeddings Configuration
# NOTE: OpenRouter does support embeddings endpoints
- HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
- HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-large
- DEFAULT_EMBEDDING_DIMENSION=3072
# Reranker Configuration
- HINDSIGHT_API_RERANKER_PROVIDER=litellm
- HINDSIGHT_API_RERANKER_LITELLM_MODEL=deepinfra/Qwen3-Reranker-8B
# LLM Configuration (uses OpenAI for testing vchord)
# LLM configuration
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
# Database Configuration
- HINDSIGHT_API_DATABASE_URL=postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
- HINDSIGHT_API_OTEL_TRACES_ENABLED=false
HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
# Vector and Text Search Extensions
HINDSIGHT_API_VECTOR_EXTENSION: vchord
HINDSIGHT_API_TEXT_SEARCH_EXTENSION: vchord
depends_on:
- db
networks:
@@ -6,6 +6,7 @@ Create Date: 2025-11-27 11:54:19.228030
"""
import os
from collections.abc import Sequence
import sqlalchemy as sa
@@ -23,21 +24,57 @@ depends_on: str | Sequence[str] | None = None
def _detect_vector_extension() -> str:
"""
Detect available vector extension: 'vchord' or 'pgvector'.
Prefers vchord if both available. Raises error if neither found.
Detect or validate vector extension: 'vchord' or 'pgvector'.
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
"""
conn = op.get_bind()
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if vchord_check:
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
# Validate configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
return "vchord"
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if pgvector_check:
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
return "pgvector"
else:
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
raise RuntimeError(
"Neither vchord nor pgvector extension found. Install one: CREATE EXTENSION vchord; or CREATE EXTENSION vector;"
)
def _detect_text_search_extension() -> str:
"""
Detect or validate text search extension: 'native' or 'vchord'.
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
Creates the extension if needed.
"""
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
if text_search_extension == "vchord":
# Create vchord_bm25 extension if not exists
try:
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
except Exception:
# Extension might already exist or user lacks permissions - verify it exists
conn = op.get_bind()
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
if not result:
# Extension truly doesn't exist - re-raise the error
raise
return "vchord"
elif text_search_extension == "native":
return "native"
else:
raise ValueError(
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
)
def upgrade() -> None:
@@ -185,11 +222,23 @@ def upgrade() -> None:
)
# Add search_vector column for full-text search
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
""")
# Type depends on configured text search backend
text_search_ext = _detect_text_search_extension()
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector bm25_catalog.bm25vector
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute("""
ALTER TABLE memory_units
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))) STORED
""")
op.create_index("idx_memory_units_bank_id", "memory_units", ["bank_id"])
op.create_index("idx_memory_units_document_id", "memory_units", ["document_id"])
@@ -238,11 +287,20 @@ def upgrade() -> None:
postgresql_ops={"embedding": "vector_cosine_ops"},
)
# Create BM25 full-text search index on search_vector
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING gin(search_vector)
""")
# Create full-text search index on search_vector
# Index type depends on text search backend
if text_search_ext == "vchord":
# VectorChord BM25 index
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL GIN index
op.execute("""
CREATE INDEX idx_memory_units_text_search ON memory_units
USING gin(search_vector)
""")
op.execute("""
CREATE MATERIALIZED VIEW memory_units_bm25 AS
@@ -10,6 +10,7 @@ This migration:
3. Adds consolidation tracking columns to the 'banks' table
"""
import os
from collections.abc import Sequence
from alembic import context, op
@@ -30,21 +31,57 @@ def _get_schema_prefix() -> str:
def _detect_vector_extension() -> str:
"""
Detect available vector extension: 'vchord' or 'pgvector'.
Prefers vchord if both available. Raises error if neither found.
Detect or validate vector extension: 'vchord' or 'pgvector'.
Respects HINDSIGHT_API_VECTOR_EXTENSION env var if set.
"""
conn = op.get_bind()
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if vchord_check:
vector_extension = os.getenv("HINDSIGHT_API_VECTOR_EXTENSION", "pgvector").lower()
# Validate configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
return "vchord"
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if pgvector_check:
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
return "pgvector"
else:
raise ValueError(f"Invalid HINDSIGHT_API_VECTOR_EXTENSION: {vector_extension}. Must be 'pgvector' or 'vchord'")
raise RuntimeError(
"Neither vchord nor pgvector extension found. Install one: CREATE EXTENSION vchord; or CREATE EXTENSION vector;"
)
def _detect_text_search_extension() -> str:
"""
Detect or validate text search extension: 'native' or 'vchord'.
Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
Creates the extension if needed.
"""
text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
if text_search_extension == "vchord":
# Create vchord_bm25 extension if not exists
try:
op.execute("CREATE EXTENSION IF NOT EXISTS vchord_bm25 CASCADE")
except Exception:
# Extension might already exist or user lacks permissions - verify it exists
conn = op.get_bind()
result = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord_bm25'")).fetchone()
if not result:
# Extension truly doesn't exist - re-raise the error
raise
return "vchord"
elif text_search_extension == "native":
return "native"
else:
raise ValueError(
f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native' or 'vchord'"
)
def upgrade() -> None:
@@ -54,6 +91,9 @@ def upgrade() -> None:
# Detect which vector extension is available
vector_ext = _detect_vector_extension()
# Detect which text search extension to use
text_search_ext = _detect_text_search_extension()
# 1. Create learnings table
op.execute(f"""
CREATE TABLE {schema}learnings (
@@ -96,11 +136,23 @@ def upgrade() -> None:
op.execute(f"CREATE INDEX idx_learnings_tags ON {schema}learnings USING GIN(tags)")
# Full-text search for learnings
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
""")
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector bm25_catalog.bm25vector
""")
op.execute(f"""
CREATE INDEX idx_learnings_text_search ON {schema}learnings
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute(f"""
ALTER TABLE {schema}learnings ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', text)) STORED
""")
op.execute(f"CREATE INDEX idx_learnings_text_search ON {schema}learnings USING gin(search_vector)")
# 2. Create pinned_reflections table
op.execute(f"""
@@ -142,14 +194,26 @@ def upgrade() -> None:
op.execute(f"CREATE INDEX idx_pinned_reflections_tags ON {schema}pinned_reflections USING GIN(tags)")
# Full-text search for pinned_reflections
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING gin(search_vector)
""")
if text_search_ext == "vchord":
# VectorChord BM25: bm25vector type (no GENERATED - tokenization happens on INSERT/UPDATE)
# Note: vchord_bm25 extension creates types in bm25_catalog schema
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector bm25_catalog.bm25vector
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
else: # native
# Native PostgreSQL: tsvector with automatic generation
op.execute(f"""
ALTER TABLE {schema}pinned_reflections ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('english', COALESCE(name, '') || ' ' || content)) STORED
""")
op.execute(f"""
CREATE INDEX idx_pinned_reflections_text_search ON {schema}pinned_reflections
USING gin(search_vector)
""")
# 3. Add consolidation tracking columns to banks table
op.execute(f"""
@@ -0,0 +1,64 @@
"""Add config JSONB column to banks table for hierarchical configuration
Revision ID: x9s0t1u2v3w4
Revises: w8r9s0t1u2v3
Create Date: 2026-02-09
This migration adds a `config` JSONB column to the banks table to support
per-bank configuration overrides. This enables hierarchical configuration where:
- Global config is loaded from environment variables
- Tenant config is provided via TenantExtension
- Bank config overrides are stored in banks.config JSONB column
The config column stores overrides for hierarchical fields (LLM settings,
retention parameters, retrieval settings, etc.) in Python field name format.
"""
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import context, op
from sqlalchemy.dialects.postgresql import JSONB
revision: str = "x9s0t1u2v3w4"
down_revision: str | Sequence[str] | None = "w8r9s0t1u2v3"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _get_schema_prefix() -> str:
"""Get schema prefix for table names (required for multi-tenant support)."""
schema = context.config.get_main_option("target_schema")
return f'"{schema}".' if schema else ""
def upgrade() -> None:
"""Add config JSONB column to banks table with GIN index."""
schema = _get_schema_prefix()
# Add config column to banks table
op.execute(f"""
ALTER TABLE {schema}banks
ADD COLUMN config JSONB NOT NULL DEFAULT '{{}}'::jsonb
""")
# Add GIN index for efficient JSONB queries
op.execute(f"""
CREATE INDEX idx_banks_config
ON {schema}banks
USING gin(config)
""")
def downgrade() -> None:
"""Remove config column and index from banks table."""
schema = _get_schema_prefix()
# Drop index first
op.execute(f"DROP INDEX IF EXISTS {schema}idx_banks_config")
# Drop column
op.execute(f"""
ALTER TABLE {schema}banks
DROP COLUMN IF EXISTS config
""")
+213 -18
View File
@@ -32,9 +32,45 @@ def _parse_metadata(metadata: Any) -> dict[str, Any]:
return {}
from typing import Callable
from pydantic import BaseModel, ConfigDict, Field, field_validator
from hindsight_api import MemoryEngine
def FieldWithDefault(default_factory: Callable, **kwargs) -> Any:
"""
Field wrapper that ensures default_factory values appear in OpenAPI schema.
Pydantic doesn't include default_factory in OpenAPI schemas, causing OpenAPI
Generator to make fields Optional with default=None instead of non-optional
with the correct default value.
This wrapper adds json_schema_extra to include the default in the schema.
"""
# Determine the default value for the schema based on the factory
if default_factory is list:
schema_default = []
elif default_factory is dict:
schema_default = {}
else:
# For custom factories (like IncludeOptions), use empty dict as placeholder
schema_default = {}
# Add or merge json_schema_extra
json_extra = kwargs.pop("json_schema_extra", {})
if isinstance(json_extra, dict):
json_extra["default"] = schema_default
else:
# If json_schema_extra was a function, we can't merge easily
# Fall back to just setting default
json_extra = {"default": schema_default}
return Field(default_factory=default_factory, json_schema_extra=json_extra, **kwargs)
from hindsight_api.config import get_config
from hindsight_api.engine.db_utils import acquire_with_retry
from hindsight_api.engine.memory_engine import Budget, _get_tiktoken_encoding, fq_table
from hindsight_api.engine.reflect.observations import Observation
@@ -103,8 +139,8 @@ class RecallRequest(BaseModel):
query_timestamp: str | None = Field(
default=None, description="ISO format date string (e.g., '2023-05-30T23:40:00')"
)
include: IncludeOptions = Field(
default_factory=IncludeOptions,
include: IncludeOptions = FieldWithDefault(
IncludeOptions,
description="Options for including additional data (entities are included by default)",
)
tags: list[str] | None = Field(
@@ -570,18 +606,16 @@ class ReflectLLMCall(BaseModel):
class ReflectBasedOn(BaseModel):
"""Evidence the response is based on: memories, mental models, and directives."""
memories: list[ReflectFact] = Field(default_factory=list, description="Memory facts used to generate the response")
mental_models: list[ReflectMentalModel] = Field(
default_factory=list, description="Mental models used during reflection"
)
directives: list[ReflectDirective] = Field(default_factory=list, description="Directives applied during reflection")
memories: list[ReflectFact] = FieldWithDefault(list, description="Memory facts used to generate the response")
mental_models: list[ReflectMentalModel] = FieldWithDefault(list, description="Mental models used during reflection")
directives: list[ReflectDirective] = FieldWithDefault(list, description="Directives applied during reflection")
class ReflectTrace(BaseModel):
"""Execution trace of LLM and tool calls during reflection."""
tool_calls: list[ReflectToolCall] = Field(default_factory=list, description="Tool calls made during reflection")
llm_calls: list[ReflectLLMCall] = Field(default_factory=list, description="LLM calls made during reflection")
tool_calls: list[ReflectToolCall] = FieldWithDefault(list, description="Tool calls made during reflection")
llm_calls: list[ReflectLLMCall] = FieldWithDefault(list, description="LLM calls made during reflection")
class ReflectResponse(BaseModel):
@@ -793,6 +827,55 @@ class CreateBankRequest(BaseModel):
background: str | None = Field(default=None, description="Deprecated: use mission instead")
class BankConfigUpdate(BaseModel):
"""Request model for updating bank configuration."""
model_config = ConfigDict(
json_schema_extra={
"example": {
"updates": {
"llm_model": "claude-sonnet-4-5",
"retain_extraction_mode": "verbose",
"retain_custom_instructions": "Extract technical details carefully",
}
}
}
)
updates: dict[str, Any] = Field(
description="Configuration overrides. Keys can be in Python field format (llm_provider) "
"or environment variable format (HINDSIGHT_API_LLM_PROVIDER). "
"Only hierarchical fields can be overridden per-bank."
)
class BankConfigResponse(BaseModel):
"""Response model for bank configuration."""
model_config = ConfigDict(
json_schema_extra={
"example": {
"bank_id": "my-bank",
"config": {
"llm_provider": "openai",
"llm_model": "gpt-4",
"retain_extraction_mode": "verbose",
},
"overrides": {
"llm_model": "gpt-4",
"retain_extraction_mode": "verbose",
},
}
}
)
bank_id: str = Field(description="Bank identifier")
config: dict[str, Any] = Field(
description="Fully resolved configuration with all hierarchical overrides applied (Python field names)"
)
overrides: dict[str, Any] = Field(description="Bank-specific configuration overrides only (Python field names)")
class GraphDataResponse(BaseModel):
"""Response model for graph data endpoint."""
@@ -942,7 +1025,7 @@ class DocumentResponse(BaseModel):
created_at: str
updated_at: str
memory_unit_count: int
tags: list[str] = Field(default_factory=list, description="Tags associated with this document")
tags: list[str] = FieldWithDefault(list, description="Tags associated with this document")
class DeleteDocumentResponse(BaseModel):
@@ -1066,7 +1149,7 @@ class DirectiveResponse(BaseModel):
content: str
priority: int = 0
is_active: bool = True
tags: list[str] = Field(default_factory=list)
tags: list[str] = FieldWithDefault(list)
created_at: str | None = None
updated_at: str | None = None
@@ -1084,7 +1167,7 @@ class CreateDirectiveRequest(BaseModel):
content: str = Field(description="The directive text to inject into prompts")
priority: int = Field(default=0, description="Higher priority directives are injected first")
is_active: bool = Field(default=True, description="Whether this directive is active")
tags: list[str] = Field(default_factory=list, description="Tags for filtering")
tags: list[str] = FieldWithDefault(list, description="Tags for filtering")
class UpdateDirectiveRequest(BaseModel):
@@ -1121,9 +1204,9 @@ class MentalModelResponse(BaseModel):
content: str = Field(
description="The mental model content as well-formatted markdown (auto-generated from reflect endpoint)"
)
tags: list[str] = Field(default_factory=list)
tags: list[str] = FieldWithDefault(list)
max_tokens: int = Field(default=2048)
trigger: MentalModelTrigger = Field(default_factory=MentalModelTrigger)
trigger: MentalModelTrigger = FieldWithDefault(MentalModelTrigger)
last_refreshed_at: str | None = None
created_at: str | None = None
reflect_response: dict | None = Field(
@@ -1159,9 +1242,9 @@ class CreateMentalModelRequest(BaseModel):
)
name: str = Field(description="Human-readable name for the mental model")
source_query: str = Field(description="The query to run to generate content")
tags: list[str] = Field(default_factory=list, description="Tags for scoped visibility")
tags: list[str] = FieldWithDefault(list, description="Tags for scoped visibility")
max_tokens: int = Field(default=2048, ge=256, le=8192, description="Maximum tokens for generated content")
trigger: MentalModelTrigger = Field(default_factory=MentalModelTrigger, description="Trigger settings")
trigger: MentalModelTrigger = FieldWithDefault(MentalModelTrigger, description="Trigger settings")
class CreateMentalModelResponse(BaseModel):
@@ -1322,6 +1405,7 @@ class FeaturesInfo(BaseModel):
observations: bool = Field(description="Whether observations (auto-consolidation) are enabled")
mcp: bool = Field(description="Whether MCP (Model Context Protocol) server is enabled")
worker: bool = Field(description="Whether the background worker is enabled")
bank_config_api: bool = Field(description="Whether per-bank configuration API is enabled")
class VersionResponse(BaseModel):
@@ -1335,6 +1419,7 @@ class VersionResponse(BaseModel):
"observations": False,
"mcp": True,
"worker": True,
"bank_config_api": False,
},
}
}
@@ -1614,17 +1699,21 @@ def _register_routes(app: FastAPI):
Returns version info and feature flags that can be used by clients
to determine which capabilities are available.
Note: observations flag shows the global default. Individual banks
may override this setting via bank-specific configuration.
"""
from hindsight_api import __version__
from hindsight_api.config import get_config
from hindsight_api.config import _get_raw_config
config = get_config()
config = _get_raw_config()
return VersionResponse(
api_version=__version__,
features=FeaturesInfo(
observations=config.enable_observations,
mcp=config.mcp_enabled,
worker=config.worker_enabled,
bank_config_api=config.enable_bank_config_api,
),
)
@@ -3278,6 +3367,112 @@ def _register_routes(app: FastAPI):
logger.error(f"Error in DELETE /v1/default/banks/{bank_id}/observations: {error_detail}")
raise HTTPException(status_code=500, detail=str(e))
@app.get(
"/v1/default/banks/{bank_id}/config",
response_model=BankConfigResponse,
summary="Get bank configuration",
description="Get fully resolved configuration for a bank including all hierarchical overrides (global → tenant → bank). "
"The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.",
operation_id="get_bank_config",
tags=["Banks"],
)
async def api_get_bank_config(bank_id: str, request_context: RequestContext = Depends(get_request_context)):
"""Get configuration for a bank with all hierarchical overrides applied."""
if not get_config().enable_bank_config_api:
raise HTTPException(
status_code=404,
detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to enable.",
)
try:
# Get resolved config from config resolver
config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
# Get bank-specific overrides only
bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
except (AuthenticationError, HTTPException):
raise
except Exception as e:
import traceback
error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
logger.error(f"Error in GET /v1/default/banks/{bank_id}/config: {error_detail}")
raise HTTPException(status_code=500, detail=str(e))
@app.patch(
"/v1/default/banks/{bank_id}/config",
response_model=BankConfigResponse,
summary="Update bank configuration",
description="Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). "
"Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).",
operation_id="update_bank_config",
tags=["Banks"],
)
async def api_update_bank_config(
bank_id: str, request: BankConfigUpdate, request_context: RequestContext = Depends(get_request_context)
):
"""Update configuration overrides for a bank."""
if not get_config().enable_bank_config_api:
raise HTTPException(
status_code=404,
detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to enable.",
)
try:
# Update config via config resolver (validates configurable fields and permissions)
await app.state.memory._config_resolver.update_bank_config(bank_id, request.updates, request_context)
# Return updated config
config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
except ValueError as e:
# Validation error (e.g., trying to override static field)
raise HTTPException(status_code=400, detail=str(e))
except (AuthenticationError, HTTPException):
raise
except Exception as e:
import traceback
error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
logger.error(f"Error in PATCH /v1/default/banks/{bank_id}/config: {error_detail}")
raise HTTPException(status_code=500, detail=str(e))
@app.delete(
"/v1/default/banks/{bank_id}/config",
response_model=BankConfigResponse,
summary="Reset bank configuration",
description="Reset bank configuration to defaults by removing all bank-specific overrides. "
"The bank will then use global and tenant-level configuration only.",
operation_id="reset_bank_config",
tags=["Banks"],
)
async def api_reset_bank_config(bank_id: str, request_context: RequestContext = Depends(get_request_context)):
"""Reset bank configuration to defaults (remove all overrides)."""
if not get_config().enable_bank_config_api:
raise HTTPException(
status_code=404,
detail="Bank configuration API is disabled. Set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true to enable.",
)
try:
# Reset config via config resolver
await app.state.memory._config_resolver.reset_bank_config(bank_id)
# Return updated config (should match defaults now)
config_dict = await app.state.memory._config_resolver.get_bank_config(bank_id, request_context)
bank_overrides = await app.state.memory._config_resolver._load_bank_config(bank_id)
return BankConfigResponse(bank_id=bank_id, config=config_dict, overrides=bank_overrides)
except (AuthenticationError, HTTPException):
raise
except Exception as e:
import traceback
error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
logger.error(f"Error in DELETE /v1/default/banks/{bank_id}/config: {error_detail}")
raise HTTPException(status_code=500, detail=str(e))
@app.post(
"/v1/default/banks/{bank_id}/consolidate",
response_model=ConsolidationResponse,
+246 -3
View File
@@ -8,8 +8,9 @@ import json
import logging
import os
import sys
from dataclasses import dataclass
from dataclasses import dataclass, field, fields
from datetime import datetime, timezone
from typing import Any
from dotenv import find_dotenv, load_dotenv
@@ -18,6 +19,103 @@ load_dotenv(find_dotenv(usecwd=True), override=True)
logger = logging.getLogger(__name__)
class ConfigFieldAccessError(AttributeError):
"""Raised when trying to access a bank-configurable field from global config."""
pass
class StaticConfigProxy:
"""
Proxy that wraps HindsightConfig and only allows access to static (non-configurable) fields.
Raises ConfigFieldAccessError when trying to access configurable fields that vary per-bank.
Forces developers to use get_resolved_config(bank_id, context) for bank-specific settings.
"""
def __init__(self, config: "HindsightConfig"):
object.__setattr__(self, "_config", config)
object.__setattr__(self, "_configurable_fields", HindsightConfig.get_configurable_fields())
def __getattribute__(self, name: str):
if name.startswith("_"):
return object.__getattribute__(self, name)
configurable_fields = object.__getattribute__(self, "_configurable_fields")
if name in configurable_fields:
raise ConfigFieldAccessError(
f"Field '{name}' is bank-configurable and cannot be accessed from global config. "
f"Use ConfigResolver.resolve_full_config(bank_id, context) to get bank-specific config. "
f"This prevents accidentally using global defaults when bank-specific overrides exist."
)
config = object.__getattribute__(self, "_config")
return getattr(config, name)
def __setattr__(self, name: str, value):
raise AttributeError("Config is read-only. Modifications must go through ConfigResolver.")
# Configuration field markers for hierarchical configuration
def hierarchical(default_value):
"""
Mark a config field as hierarchical (can be overridden per-tenant/bank).
Hierarchical fields can be customized at the tenant or bank level via database
configuration. Examples: LLM settings, retention parameters, retrieval settings.
"""
return field(default=default_value, metadata={"hierarchical": True})
def static(default_value):
"""
Mark a config field as static (server-level only, cannot be overridden).
Static fields are infrastructure-level settings that affect the entire server
and cannot vary per tenant or bank. Examples: database URL, API port, worker settings.
"""
return field(default=default_value, metadata={"hierarchical": False})
# Configuration key normalization utilities
def normalize_config_key(key: str) -> str:
"""
Convert environment variable format to Python field name format.
Examples:
HINDSIGHT_API_LLM_PROVIDER -> llm_provider
LLM_MODEL -> llm_model
llm_model -> llm_model (already normalized)
Args:
key: Environment variable name or Python field name
Returns:
Normalized Python field name (lowercase snake_case)
"""
if key.startswith("HINDSIGHT_API_"):
key = key[len("HINDSIGHT_API_") :]
return key.lower()
def normalize_config_dict(config: dict[str, Any]) -> dict[str, Any]:
"""
Normalize all keys in a config dict to Python field names.
Allows users to provide config overrides in either format:
- Python field format: {"llm_provider": "openai"}
- Env var format: {"HINDSIGHT_API_LLM_PROVIDER": "openai"}
Args:
config: Dict with env var or Python field names as keys
Returns:
Dict with all keys normalized to Python field names
"""
return {normalize_config_key(k): v for k, v in config.items()}
# Environment variable names
ENV_DATABASE_URL = "HINDSIGHT_API_DATABASE_URL"
ENV_DATABASE_SCHEMA = "HINDSIGHT_API_DATABASE_SCHEMA"
@@ -107,6 +205,9 @@ ENV_RERANKER_MAX_CANDIDATES = "HINDSIGHT_API_RERANKER_MAX_CANDIDATES"
ENV_RERANKER_FLASHRANK_MODEL = "HINDSIGHT_API_RERANKER_FLASHRANK_MODEL"
ENV_RERANKER_FLASHRANK_CACHE_DIR = "HINDSIGHT_API_RERANKER_FLASHRANK_CACHE_DIR"
ENV_VECTOR_EXTENSION = "HINDSIGHT_API_VECTOR_EXTENSION"
ENV_TEXT_SEARCH_EXTENSION = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION"
ENV_HOST = "HINDSIGHT_API_HOST"
ENV_PORT = "HINDSIGHT_API_PORT"
ENV_BASE_PATH = "HINDSIGHT_API_BASE_PATH"
@@ -114,6 +215,7 @@ ENV_LOG_LEVEL = "HINDSIGHT_API_LOG_LEVEL"
ENV_LOG_FORMAT = "HINDSIGHT_API_LOG_FORMAT"
ENV_WORKERS = "HINDSIGHT_API_WORKERS"
ENV_MCP_ENABLED = "HINDSIGHT_API_MCP_ENABLED"
ENV_ENABLE_BANK_CONFIG_API = "HINDSIGHT_API_ENABLE_BANK_CONFIG_API"
ENV_GRAPH_RETRIEVER = "HINDSIGHT_API_GRAPH_RETRIEVER"
ENV_MPFP_TOP_K_NEIGHBORS = "HINDSIGHT_API_MPFP_TOP_K_NEIGHBORS"
ENV_RECALL_MAX_CONCURRENT = "HINDSIGHT_API_RECALL_MAX_CONCURRENT"
@@ -224,6 +326,12 @@ DEFAULT_RERANKER_FLASHRANK_CACHE_DIR = None # Use default cache directory
DEFAULT_EMBEDDINGS_COHERE_MODEL = "embed-english-v3.0"
DEFAULT_RERANKER_COHERE_MODEL = "rerank-english-v3.0"
# Vector extension (pgvector vs vchord)
DEFAULT_VECTOR_EXTENSION = "pgvector" # Options: "pgvector", "vchord"
# Text search extension (native PostgreSQL vs vchord BM25)
DEFAULT_TEXT_SEARCH_EXTENSION = "native" # Options: "native", "vchord"
# LiteLLM defaults
DEFAULT_LITELLM_API_BASE = "http://localhost:4000"
DEFAULT_EMBEDDINGS_LITELLM_MODEL = "text-embedding-3-small"
@@ -236,6 +344,7 @@ DEFAULT_LOG_LEVEL = "info"
DEFAULT_LOG_FORMAT = "text" # Options: "text", "json"
DEFAULT_WORKERS = 1
DEFAULT_MCP_ENABLED = True
DEFAULT_ENABLE_BANK_CONFIG_API = False # Disabled by default for security
DEFAULT_GRAPH_RETRIEVER = "link_expansion" # Options: "link_expansion", "mpfp", "bfs"
DEFAULT_MPFP_TOP_K_NEIGHBORS = 20 # Fan-out limit per node in MPFP graph traversal
DEFAULT_RECALL_MAX_CONCURRENT = 32 # Max concurrent recall operations per worker
@@ -360,6 +469,8 @@ class HindsightConfig:
# Database
database_url: str
database_schema: str
vector_extension: str # "pgvector" or "vchord"
text_search_extension: str # "native" or "vchord"
# LLM (default, used as fallback for per-operation config)
llm_provider: str
@@ -446,6 +557,7 @@ class HindsightConfig:
log_level: str
log_format: str
mcp_enabled: bool
enable_bank_config_api: bool
# Recall
graph_retriever: str
@@ -498,8 +610,108 @@ class HindsightConfig:
otel_service_name: str
otel_deployment_environment: str
# Class-level sets for configuration categorization
# CREDENTIAL_FIELDS: Never exposed via API, never configurable per-tenant/bank
_CREDENTIAL_FIELDS = {
# API Keys
"llm_api_key",
"retain_llm_api_key",
"reflect_llm_api_key",
"consolidation_llm_api_key",
# Base URLs (could expose infrastructure)
"llm_base_url",
"retain_llm_base_url",
"reflect_llm_base_url",
"consolidation_llm_base_url",
"embeddings_tei_base_url",
"reranker_tei_base_url",
"reranker_cohere_base_url",
# Service Account Keys
"llm_vertexai_service_account_key",
}
# CONFIGURABLE_FIELDS: Safe behavioral settings that can be customized per-tenant/bank
# These fields are manually tagged as safe to expose and modify.
# Excludes credentials, infrastructure config, provider/model selection, and performance tuning.
_CONFIGURABLE_FIELDS = {
# Retention settings (behavioral)
"retain_chunk_size",
"retain_extraction_mode",
"retain_custom_instructions",
# Consolidation settings
"enable_observations",
}
@classmethod
def get_configurable_fields(cls) -> set[str]:
"""
Get set of field names that are configurable per-tenant/bank via API.
Configurable fields are manually tagged behavioral settings that are safe
to expose and modify (e.g., retain_chunk_size, custom_instructions).
Excludes credentials, infrastructure config, and provider/model selection.
Returns:
Set of configurable field names
"""
return cls._CONFIGURABLE_FIELDS.copy()
@classmethod
def get_credential_fields(cls) -> set[str]:
"""
Get set of field names that are credentials (NEVER exposed via API).
Credential fields include API keys, base URLs, and service account keys.
These must never be returned in API responses or accepted in updates.
Returns:
Set of credential field names
"""
return cls._CREDENTIAL_FIELDS.copy()
@classmethod
def get_hierarchical_fields(cls) -> set[str]:
"""
DEPRECATED: Use get_configurable_fields() instead.
Kept for backward compatibility during migration.
"""
return cls.get_configurable_fields()
@classmethod
def get_static_fields(cls) -> set[str]:
"""
Get set of field names that are static (server-level only).
Static fields are infrastructure-level settings that cannot vary
per tenant or bank. These include database config, API port, worker settings, etc.
Also includes credential fields which are never configurable.
Returns:
Set of static field names
"""
# Get all field names from dataclass
all_fields = {f.name for f in fields(cls)}
# Static fields = all fields - configurable fields
return all_fields - cls._CONFIGURABLE_FIELDS
def validate(self) -> None:
"""Validate configuration values and raise errors for invalid combinations."""
# Validate vector_extension
valid_extensions = ("pgvector", "vchord")
if self.vector_extension not in valid_extensions:
raise ValueError(
f"Invalid vector_extension: {self.vector_extension}. Must be one of: {', '.join(valid_extensions)}"
)
# Validate text_search_extension
valid_text_search = ("native", "vchord")
if self.text_search_extension not in valid_text_search:
raise ValueError(
f"Invalid text_search_extension: {self.text_search_extension}. Must be one of: {', '.join(valid_text_search)}"
)
# RETAIN_MAX_COMPLETION_TOKENS must be greater than RETAIN_CHUNK_SIZE
# to ensure the LLM has enough output capacity to extract facts from chunks
if self.retain_max_completion_tokens <= self.retain_chunk_size:
@@ -525,6 +737,8 @@ class HindsightConfig:
# Database
database_url=os.getenv(ENV_DATABASE_URL, DEFAULT_DATABASE_URL),
database_schema=os.getenv(ENV_DATABASE_SCHEMA, DEFAULT_DATABASE_SCHEMA),
vector_extension=os.getenv(ENV_VECTOR_EXTENSION, DEFAULT_VECTOR_EXTENSION).lower(),
text_search_extension=os.getenv(ENV_TEXT_SEARCH_EXTENSION, DEFAULT_TEXT_SEARCH_EXTENSION).lower(),
# LLM
llm_provider=llm_provider,
llm_api_key=os.getenv(ENV_LLM_API_KEY),
@@ -669,6 +883,8 @@ class HindsightConfig:
log_level=os.getenv(ENV_LOG_LEVEL, DEFAULT_LOG_LEVEL),
log_format=os.getenv(ENV_LOG_FORMAT, DEFAULT_LOG_FORMAT).lower(),
mcp_enabled=os.getenv(ENV_MCP_ENABLED, str(DEFAULT_MCP_ENABLED)).lower() == "true",
enable_bank_config_api=os.getenv(ENV_ENABLE_BANK_CONFIG_API, str(DEFAULT_ENABLE_BANK_CONFIG_API)).lower()
== "true",
# Recall
graph_retriever=os.getenv(ENV_GRAPH_RETRIEVER, DEFAULT_GRAPH_RETRIEVER),
mpfp_top_k_neighbors=int(os.getenv(ENV_MPFP_TOP_K_NEIGHBORS, str(DEFAULT_MPFP_TOP_K_NEIGHBORS))),
@@ -809,8 +1025,35 @@ class HindsightConfig:
_config_cache: HindsightConfig | None = None
def get_config() -> HindsightConfig:
"""Get the cached configuration, loading from environment on first call."""
def get_config() -> StaticConfigProxy:
"""
Get global configuration with ONLY static (non-configurable) fields accessible.
This returns a proxy that prevents access to bank-configurable fields
(like enable_observations, retain_chunk_size, etc.).
For bank-specific configuration, use:
config_resolver.resolve_full_config(bank_id, context)
This design prevents accidentally using global defaults when bank-specific
overrides exist.
Returns:
StaticConfigProxy that only exposes static infrastructure fields
Raises:
ConfigFieldAccessError: If you try to access a bank-configurable field
"""
return StaticConfigProxy(_get_raw_config())
def _get_raw_config() -> HindsightConfig:
"""
Get raw config (internal use only).
INTERNAL USE ONLY. Do not use this directly in application code.
Use get_config() for static fields or ConfigResolver.resolve_full_config() for bank-specific config.
"""
global _config_cache
if _config_cache is None:
_config_cache = HindsightConfig.from_env()
@@ -0,0 +1,274 @@
"""
Configuration resolution with hierarchical overrides.
Resolves config values through the hierarchy:
Global (env vars) → Tenant config (via extension) → Bank config (database)
Config values are resolved on every request to ensure consistency across
multiple API servers.
"""
import json
import logging
from dataclasses import asdict
from typing import Any
import asyncpg
from hindsight_api.config import HindsightConfig, _get_raw_config, normalize_config_dict
from hindsight_api.extensions.tenant import TenantExtension
from hindsight_api.models import RequestContext
logger = logging.getLogger(__name__)
class ConfigResolver:
"""Resolves hierarchical configuration with tenant/bank overrides."""
def __init__(self, pool: asyncpg.Pool, tenant_extension: TenantExtension | None = None):
"""
Initialize config resolver.
Args:
pool: Database connection pool
tenant_extension: Optional tenant extension for tenant-level config and permissions
"""
self.pool = pool
self.tenant_extension = tenant_extension
self._global_config = _get_raw_config()
self._configurable_fields = HindsightConfig.get_configurable_fields()
self._credential_fields = HindsightConfig.get_credential_fields()
async def resolve_full_config(self, bank_id: str, context: RequestContext | None = None) -> HindsightConfig:
"""
Resolve full HindsightConfig for a bank with hierarchical overrides applied.
This is for INTERNAL USE ONLY. Returns the complete config object with all fields
including credentials and static fields. Use get_bank_config() for API responses.
Resolution order:
1. Global config (from environment variables)
2. Tenant config overrides (from TenantExtension.get_tenant_config())
3. Bank config overrides (from banks.config JSONB)
Args:
bank_id: Bank identifier
context: Request context for tenant config resolution
Returns:
Complete HindsightConfig with hierarchical overrides applied
"""
# Start with global config (all fields)
config_dict = asdict(self._global_config)
# Load tenant config overrides (if tenant extension available)
if self.tenant_extension and context:
try:
tenant_overrides = await self.tenant_extension.get_tenant_config(context)
if tenant_overrides:
# Normalize keys and filter to configurable fields only
normalized_tenant = normalize_config_dict(tenant_overrides)
configurable_tenant = {k: v for k, v in normalized_tenant.items() if k in self._configurable_fields}
config_dict.update(configurable_tenant)
logger.debug(
f"Applied tenant config overrides for bank {bank_id}: {list(configurable_tenant.keys())}"
)
except Exception as e:
logger.warning(f"Failed to load tenant config for bank {bank_id}: {e}")
# Load bank config overrides
bank_overrides = await self._load_bank_config(bank_id)
if bank_overrides:
config_dict.update(bank_overrides)
logger.debug(f"Applied bank config overrides for bank {bank_id}: {list(bank_overrides.keys())}")
# Return full config object (dataclass doesn't have __init__ that accepts kwargs, so we update the object)
# Create a new config instance by copying the global config and updating fields
resolved_config = HindsightConfig(**config_dict)
return resolved_config
async def get_bank_config(self, bank_id: str, context: RequestContext | None = None) -> dict[str, Any]:
"""
Get fully resolved config for a bank (filtered by permissions).
Resolution order:
1. Global config (from environment variables)
2. Tenant config overrides (from TenantExtension.get_tenant_config())
3. Bank config overrides (from banks.config JSONB)
Note: Config is resolved on every call (not cached) to ensure consistency
across multiple API servers.
SECURITY:
- Only returns configurable fields (excludes static/infrastructure fields)
- Filters out ALL credential fields (API keys, base URLs, etc.)
- Further filtered by tenant/bank permissions if extension provides them
Args:
bank_id: Bank identifier
context: Request context for tenant config resolution and permissions
Returns:
Dict of allowed configurable fields only (never includes credentials or static fields)
"""
# Resolve full config with all hierarchical overrides
resolved_config = await self.resolve_full_config(bank_id, context)
config_dict = asdict(resolved_config)
# SECURITY: Filter to only configurable fields (exclude static/infrastructure)
filtered = {k: v for k, v in config_dict.items() if k in self._configurable_fields}
# SECURITY: Remove ALL credential fields (API keys, base URLs, etc.)
filtered = {k: v for k, v in filtered.items() if k not in self._credential_fields}
# PERMISSIONS: Further filter based on tenant/bank permissions
if self.tenant_extension and context:
try:
allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
if allowed_fields is not None: # None means "allow all"
filtered = {k: v for k, v in filtered.items() if k in allowed_fields}
logger.debug(
f"Applied permission filter for bank {bank_id}: allowed={len(allowed_fields)} fields, "
f"returned={len(filtered)} fields"
)
except Exception as e:
logger.warning(f"Failed to load permissions for bank {bank_id}: {e}")
return filtered
async def _load_bank_config(self, bank_id: str) -> dict[str, Any]:
"""
Load bank config overrides from banks.config JSONB column.
Args:
bank_id: Bank identifier
Returns:
Dict of config overrides (only configurable fields, normalized keys)
"""
try:
async with self.pool.acquire() as conn:
row = await conn.fetchrow(
"""
SELECT config FROM banks WHERE bank_id = $1
""",
bank_id,
)
if row and row["config"]:
config_data = row["config"]
# Handle case where JSONB is returned as JSON string
if isinstance(config_data, str):
config_data = json.loads(config_data)
# Normalize keys (handle both env var format and Python field format)
normalized = normalize_config_dict(config_data)
# Only return overrides for configurable fields
return {k: v for k, v in normalized.items() if k in self._configurable_fields}
except Exception as e:
logger.error(f"Failed to load bank config for {bank_id}: {e}")
return {}
async def update_bank_config(
self, bank_id: str, updates: dict[str, Any], context: RequestContext | None = None
) -> None:
"""
Update bank configuration overrides (with permission checking).
Args:
bank_id: Bank identifier
updates: Dict of config field names to new values.
Keys can be in env var format (HINDSIGHT_API_LLM_PROVIDER)
or Python field format (llm_provider).
Only configurable fields are allowed.
context: Request context for permission checking
Raises:
ValueError: If attempting to override invalid/disallowed fields
"""
# Normalize keys
normalized_updates = normalize_config_dict(updates)
# SECURITY: Reject credential fields explicitly
credential_attempts = set(normalized_updates.keys()) & self._credential_fields
if credential_attempts:
raise ValueError(
f"Cannot set credential fields via API: {sorted(credential_attempts)}. "
f"Credentials (API keys, base URLs) must be set at server level only."
)
# Validate all fields are configurable
invalid_fields = set(normalized_updates.keys()) - self._configurable_fields
if invalid_fields:
static_fields = HindsightConfig.get_static_fields()
invalid_static = invalid_fields & static_fields
if invalid_static:
raise ValueError(
f"Cannot override static (server-level) fields: {sorted(invalid_static)}. "
f"Only configurable fields can be overridden per-bank. "
f"Configurable fields include: {sorted(list(self._configurable_fields)[:10])}... "
f"(total: {len(self._configurable_fields)} fields)"
)
else:
raise ValueError(
f"Unknown configuration fields: {sorted(invalid_fields)}. "
f"Valid configurable fields: {sorted(list(self._configurable_fields)[:10])}..."
)
# PERMISSIONS: Check tenant/bank permissions
if self.tenant_extension and context:
try:
allowed_fields = await self.tenant_extension.get_allowed_config_fields(context, bank_id)
if allowed_fields is not None: # None means "allow all"
disallowed = set(normalized_updates.keys()) - allowed_fields
if disallowed:
raise ValueError(
f"Not allowed to modify fields: {sorted(disallowed)}. "
f"Your permissions allow: {sorted(list(allowed_fields)[:10])}..."
if allowed_fields
else "Not allowed to modify fields: {sorted(disallowed)}. "
"Your permissions do not allow any config modifications."
)
except ValueError:
raise # Re-raise permission errors
except Exception as e:
logger.warning(f"Failed to check permissions for bank {bank_id}: {e}")
# Continue without permission check (fail open for backward compatibility)
# Merge with existing config (JSONB || operator)
async with self.pool.acquire() as conn:
await conn.execute(
"""
UPDATE banks
SET config = config || $1::jsonb,
updated_at = now()
WHERE bank_id = $2
""",
json.dumps(normalized_updates),
bank_id,
)
logger.info(f"Updated bank config for {bank_id}: {list(normalized_updates.keys())}")
async def reset_bank_config(self, bank_id: str) -> None:
"""
Reset bank configuration to defaults (remove all overrides).
Args:
bank_id: Bank identifier
"""
async with self.pool.acquire() as conn:
await conn.execute(
"""
UPDATE banks
SET config = '{}'::jsonb,
updated_at = now()
WHERE bank_id = $1
""",
bank_id,
)
logger.info(f"Reset bank config for {bank_id} to defaults")
@@ -18,6 +18,7 @@ import uuid
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Any
from ...config import get_config
from ..memory_engine import fq_table
from ..retain import embedding_utils
from .prompts import (
@@ -82,9 +83,8 @@ async def run_consolidation_job(
Returns:
Dict with consolidation results
"""
from ...config import get_config
config = get_config()
# Resolve bank-specific config with hierarchical overrides
config = await memory_engine._config_resolver.resolve_full_config(bank_id, request_context)
perf = ConsolidationPerfLog(bank_id)
max_memories_per_batch = config.consolidation_batch_size
@@ -1016,15 +1016,33 @@ async def _create_observation_directly(
t0 = time.time()
observation_id = uuid.uuid4()
# Query varies based on text search backend
config = get_config()
if config.text_search_extension == "vchord":
# VectorChord: manually tokenize and insert search_vector
query = f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at, search_vector
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10,
tokenize($3, 'llmlingua2')::bm25_catalog.bm25vector)
RETURNING id
"""
else: # native
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
query = f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
RETURNING id
"""
row = await conn.fetchrow(
f"""
INSERT INTO {fq_table("memory_units")} (
id, bank_id, text, fact_type, embedding, proof_count, source_memory_ids, history,
tags, event_date, occurred_start, occurred_end, mentioned_at
)
VALUES ($1, $2, $3, 'observation', $4::vector, 1, $5, '[]'::jsonb, $6, $7, $8, $9, $10)
RETURNING id
""",
query,
observation_id,
bank_id,
observation_text,
@@ -968,7 +968,12 @@ class MemoryEngine(MemoryEngineInterface):
# Run database migrations if enabled
if self._run_migrations:
from ..migrations import ensure_embedding_dimension, run_migrations
from ..migrations import (
ensure_embedding_dimension,
ensure_text_search_extension,
ensure_vector_extension,
run_migrations,
)
if not self.db_url:
raise ValueError("Database URL is required for migrations")
@@ -976,30 +981,43 @@ class MemoryEngine(MemoryEngineInterface):
# Migrate all schemas from the tenant extension
# The tenant extension is the single source of truth for which schemas exist
logger.info("Running database migrations...")
try:
tenants = await self._tenant_extension.list_tenants()
if tenants:
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
for tenant in tenants:
schema = tenant.schema
if schema:
try:
run_migrations(self.db_url, schema=schema)
except Exception as e:
logger.warning(f"Failed to migrate schema {schema}: {e}")
logger.info("Schema migrations completed")
tenants = await self._tenant_extension.list_tenants()
if tenants:
logger.info(f"Running migrations on {len(tenants)} schema(s)...")
for tenant in tenants:
schema = tenant.schema
if schema:
run_migrations(self.db_url, schema=schema)
logger.info("Schema migrations completed")
# Ensure embedding column dimension matches the model's dimension
# This is done after migrations and after embeddings.initialize()
for tenant in tenants:
schema = tenant.schema
if schema:
try:
ensure_embedding_dimension(self.db_url, self.embeddings.dimension, schema=schema)
except Exception as e:
logger.warning(f"Failed to ensure embedding dimension for schema {schema}: {e}")
except Exception as e:
logger.warning(f"Failed to run schema migrations: {e}")
# Get config for vector extension setting
config = get_config()
# Ensure embedding column dimension matches the model's dimension
# This is done after migrations and after embeddings.initialize()
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_embedding_dimension(
self.db_url,
self.embeddings.dimension,
schema=schema,
vector_extension=config.vector_extension,
)
# Ensure vector indexes match the configured extension
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_vector_extension(self.db_url, vector_extension=config.vector_extension, schema=schema)
# Ensure text search columns/indexes match the configured extension
for tenant in tenants:
schema = tenant.schema
if schema:
ensure_text_search_extension(
self.db_url, text_search_extension=config.text_search_extension, schema=schema
)
logger.info(f"Connecting to PostgreSQL at {mask_network_location(self.db_url)}")
@@ -1018,6 +1036,12 @@ class MemoryEngine(MemoryEngineInterface):
# Initialize entity resolver with pool
self.entity_resolver = EntityResolver(self._pool)
# Initialize config resolver for hierarchical configuration
from ..config_resolver import ConfigResolver
self._config_resolver = ConfigResolver(pool=self._pool, tenant_extension=self._tenant_extension)
logger.debug("Config resolver initialized for hierarchical configuration")
# Set executor for task backend and initialize
self._task_backend.set_executor(self.execute_task)
await self._task_backend.initialize()
@@ -1447,6 +1471,7 @@ class MemoryEngine(MemoryEngineInterface):
sub_results, sub_usage = await self._retain_batch_async_internal(
bank_id=bank_id,
contents=sub_batch,
request_context=request_context,
document_id=document_id,
is_first_batch=i == 1, # Only upsert on first batch
fact_type_override=fact_type_override,
@@ -1466,6 +1491,7 @@ class MemoryEngine(MemoryEngineInterface):
result, total_usage = await self._retain_batch_async_internal(
bank_id=bank_id,
contents=contents,
request_context=request_context,
document_id=document_id,
is_first_batch=True,
fact_type_override=fact_type_override,
@@ -1497,9 +1523,8 @@ class MemoryEngine(MemoryEngineInterface):
logger.warning(f"Post-retain hook error (non-fatal): {e}")
# Trigger consolidation as a tracked async operation if enabled
from ..config import get_config
config = get_config()
# Resolve bank-specific config to check if observations are enabled for this bank
config = await self._config_resolver.resolve_full_config(bank_id, request_context)
if config.enable_observations:
try:
await self.submit_async_consolidation(bank_id=bank_id, request_context=request_context)
@@ -1515,6 +1540,7 @@ class MemoryEngine(MemoryEngineInterface):
self,
bank_id: str,
contents: list[RetainContentDict],
request_context: "RequestContext",
document_id: str | None = None,
is_first_batch: bool = True,
fact_type_override: str | None = None,
@@ -1532,6 +1558,7 @@ class MemoryEngine(MemoryEngineInterface):
Args:
bank_id: Unique identifier for the bank
contents: List of dicts with content, context, event_date
request_context: Request context for config resolution
document_id: Optional document ID (always upserts if exists)
is_first_batch: Whether this is the first batch (for chunked operations, only delete on first batch)
fact_type_override: Override fact type for all facts
@@ -1548,6 +1575,9 @@ class MemoryEngine(MemoryEngineInterface):
pool = await self._get_pool()
# Resolve bank-specific config for this operation
resolved_config = await self._config_resolver.resolve_full_config(bank_id, request_context)
# Create parent span for retain operation
with create_operation_span("retain", bank_id):
return await orchestrator.retain_batch(
@@ -1564,6 +1594,7 @@ class MemoryEngine(MemoryEngineInterface):
fact_type_override=fact_type_override,
confidence_score=confidence_score,
document_tags=document_tags,
config=resolved_config,
)
def recall(
@@ -702,6 +702,7 @@ async def _extract_facts_from_chunk(
event_date: datetime,
context: str,
llm_config: "LLMConfig",
config,
agent_name: str = None,
) -> tuple[list[dict[str, str]], TokenUsage]:
"""
@@ -721,7 +722,6 @@ async def _extract_facts_from_chunk(
fact_types_instruction = "Extract ONLY 'world' and 'assistant' type facts."
# Check config for extraction mode and causal link extraction
config = get_config()
extraction_mode = config.retain_extraction_mode
extract_causal_links = config.retain_extract_causal_links
@@ -1055,6 +1055,7 @@ async def _extract_facts_with_auto_split(
event_date: datetime,
context: str,
llm_config: LLMConfig,
config,
agent_name: str = None,
) -> tuple[list[dict[str, str]], TokenUsage]:
"""
@@ -1070,6 +1071,7 @@ async def _extract_facts_with_auto_split(
event_date: Reference date for temporal information
context: Context about the conversation/document
llm_config: LLM configuration to use
config: Resolved HindsightConfig for this bank
agent_name: Optional agent name (memory owner)
Returns:
@@ -1088,6 +1090,7 @@ async def _extract_facts_with_auto_split(
event_date=event_date,
context=context,
llm_config=llm_config,
config=config,
agent_name=agent_name,
)
except OutputTooLongError:
@@ -1132,6 +1135,7 @@ async def _extract_facts_with_auto_split(
event_date=event_date,
context=context,
llm_config=llm_config,
config=config,
agent_name=agent_name,
),
_extract_facts_with_auto_split(
@@ -1141,6 +1145,7 @@ async def _extract_facts_with_auto_split(
event_date=event_date,
context=context,
llm_config=llm_config,
config=config,
agent_name=agent_name,
),
]
@@ -1164,6 +1169,7 @@ async def extract_facts_from_text(
event_date: datetime,
llm_config: LLMConfig,
agent_name: str,
config,
context: str = "",
) -> tuple[list[Fact], list[tuple[str, int]], TokenUsage]:
"""
@@ -1178,9 +1184,10 @@ async def extract_facts_from_text(
Args:
text: Input text (conversation, article, etc.)
event_date: Reference date for resolving relative times
context: Context about the conversation/document
llm_config: LLM configuration to use
agent_name: Agent name (memory owner)
config: Resolved HindsightConfig for this bank
context: Context about the conversation/document
Returns:
Tuple of (facts, chunks, usage) where:
@@ -1188,7 +1195,6 @@ async def extract_facts_from_text(
- chunks: List of tuples (chunk_text, fact_count) for each chunk
- usage: Aggregated token usage across all LLM calls
"""
config = get_config()
chunks = chunk_text(text, max_chars=config.retain_chunk_size)
# Log chunk count before starting LLM requests
@@ -1207,6 +1213,7 @@ async def extract_facts_from_text(
event_date=event_date,
context=context,
llm_config=llm_config,
config=config,
agent_name=agent_name,
)
for i, chunk in enumerate(chunks)
@@ -1239,7 +1246,7 @@ SECONDS_PER_FACT = 10
async def extract_facts_from_contents(
contents: list[RetainContent], llm_config, agent_name: str
contents: list[RetainContent], llm_config, agent_name: str, config
) -> tuple[list[ExtractedFactType], list[ChunkMetadata], TokenUsage]:
"""
Extract facts from multiple content items in parallel.
@@ -1254,6 +1261,7 @@ async def extract_facts_from_contents(
contents: List of RetainContent objects to process
llm_config: LLM configuration for fact extraction
agent_name: Name of the agent (for agent-related fact detection)
config: Resolved HindsightConfig for this bank
Returns:
Tuple of (extracted_facts, chunks_metadata, usage)
@@ -1272,6 +1280,7 @@ async def extract_facts_from_contents(
context=item.context,
llm_config=llm_config,
agent_name=agent_name,
config=config,
)
fact_extraction_tasks.append(task)
@@ -7,6 +7,7 @@ Handles insertion of facts into the database.
import json
import logging
from ...config import get_config
from ..memory_engine import fq_table
from .fact_extraction import _sanitize_text
from .types import ProcessedFact
@@ -70,28 +71,58 @@ async def insert_facts_batch(
# Batch insert all facts
# Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
results = await conn.fetch(
f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
# Query varies based on text search backend
config = get_config()
if config.text_search_extension == "vchord":
# VectorChord: manually tokenize and insert search_vector
query = f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
FROM input_data
RETURNING id
""",
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags, search_vector)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
),
tokenize(COALESCE(text, '') || ' ' || COALESCE(context, ''), 'llmlingua2')::bm25_catalog.bm25vector
FROM input_data
RETURNING id
"""
else: # native
# Native PostgreSQL: search_vector is GENERATED ALWAYS, don't include it
query = f"""
WITH input_data AS (
SELECT * FROM unnest(
$2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
$8::text[], $9::text[], $10::float[], $11::jsonb[], $12::text[], $13::text[], $14::jsonb[]
) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags_json)
)
INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id, tags)
SELECT
$1,
text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
context, fact_type, confidence_score, metadata, chunk_id, document_id,
COALESCE(
(SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
'{{}}'::varchar[]
)
FROM input_data
RETURNING id
"""
results = await conn.fetch(
query,
bank_id,
fact_texts,
embeddings,
@@ -76,6 +76,7 @@ async def retain_batch(
duplicate_checker_fn,
bank_id: str,
contents_dicts: list[RetainContentDict],
config,
document_id: str | None = None,
is_first_batch: bool = True,
fact_type_override: str | None = None,
@@ -94,6 +95,7 @@ async def retain_batch(
duplicate_checker_fn: Function to check for duplicate facts
bank_id: Bank identifier
contents_dicts: List of content dictionaries
config: Resolved HindsightConfig for this bank
document_id: Optional document ID
is_first_batch: Whether this is the first batch
fact_type_override: Override fact type for all facts
@@ -144,7 +146,9 @@ async def retain_batch(
# Step 1: Extract facts from all contents
step_start = time.time()
extracted_facts, chunks, usage = await fact_extraction.extract_facts_from_contents(contents, llm_config, agent_name)
extracted_facts, chunks, usage = await fact_extraction.extract_facts_from_contents(
contents, llm_config, agent_name, config
)
log_buffer.append(
f"[1] Extract facts: {len(extracted_facts)} facts, {len(chunks)} chunks from {len(contents)} contents in {time.time() - step_start:.3f}s"
)
@@ -13,12 +13,10 @@ from .reranking import CrossEncoderReranker
from .retrieval import (
ParallelRetrievalResult,
get_default_graph_retriever,
retrieve_parallel,
set_default_graph_retriever,
)
__all__ = [
"retrieve_parallel",
"get_default_graph_retriever",
"set_default_graph_retriever",
"ParallelRetrievalResult",
@@ -85,116 +85,6 @@ def set_default_graph_retriever(retriever: GraphRetriever) -> None:
_default_graph_retriever = retriever
async def retrieve_semantic(
conn,
query_emb_str: str,
bank_id: str,
fact_type: str,
limit: int,
tags: list[str] | None = None,
) -> list[RetrievalResult]:
"""
Semantic retrieval via vector similarity.
Args:
conn: Database connection
query_emb_str: Query embedding as string
agent_id: bank ID
fact_type: Fact type to filter
limit: Maximum results to return
tags: Optional list of tags for visibility filtering (OR matching)
Returns:
List of RetrievalResult objects
"""
from .tags import TagsMatch, build_tags_where_clause_simple
tags_clause = build_tags_where_clause_simple(tags, 5)
params = [query_emb_str, bank_id, fact_type, limit]
if tags:
params.append(tags)
results = await conn.fetch(
f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = $3
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
ORDER BY embedding <=> $1::vector
LIMIT $4
""",
*params,
)
return [RetrievalResult.from_db_row(dict(r)) for r in results]
async def retrieve_bm25(
conn,
query_text: str,
bank_id: str,
fact_type: str,
limit: int,
tags: list[str] | None = None,
) -> list[RetrievalResult]:
"""
BM25 keyword retrieval via full-text search.
Args:
conn: Database connection
query_text: Query text
agent_id: bank ID
fact_type: Fact type to filter
limit: Maximum results to return
tags: Optional list of tags for visibility filtering (OR matching)
Returns:
List of RetrievalResult objects
"""
import re
from .tags import TagsMatch, build_tags_where_clause_simple
# Sanitize query text: remove special characters that have meaning in tsquery
# Keep only alphanumeric characters and spaces
sanitized_text = re.sub(r"[^\w\s]", " ", query_text.lower())
# Split and filter empty strings
tokens = [token for token in sanitized_text.split() if token]
if not tokens:
# If no valid tokens, return empty results
return []
# Convert query to tsquery using OR for more flexible matching
# This prevents empty results when some terms are missing
query_tsquery = " | ".join(tokens)
tags_clause = build_tags_where_clause_simple(tags, 5)
params = [query_tsquery, bank_id, fact_type, limit]
if tags:
params.append(tags)
results = await conn.fetch(
f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = $3
AND search_vector @@ to_tsquery('english', $1)
{tags_clause}
ORDER BY bm25_score DESC
LIMIT $4
""",
*params,
)
return [RetrievalResult.from_db_row(dict(r)) for r in results]
async def retrieve_semantic_bm25_combined(
conn,
query_emb_str: str,
@@ -268,59 +158,109 @@ async def retrieve_semantic_bm25_combined(
result_dict[ft][0].append(RetrievalResult.from_db_row(row))
return result_dict
query_tsquery = " | ".join(tokens)
# Build BM25 query based on text search backend
config = get_config()
# Build tags clause - param 6 if tags provided
tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
if tags:
params.append(tags)
if config.text_search_extension == "vchord":
# VectorChord BM25: use <&> operator with to_bm25query and tokenize
# Note: VectorChord scores are negative (higher = better, so -1 > -10)
params = [query_emb_str, bank_id, fact_types, limit, query_text] # Pass raw query_text for tokenization
if tags:
params.append(tags)
query = f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY search_vector <&> to_bm25query('idx_memory_units_text_search', tokenize($5, 'llmlingua2')) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
"""
else: # native
# Native PostgreSQL: use ts_rank_cd with to_tsquery
query_tsquery = " | ".join(tokens)
params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
if tags:
params.append(tags)
query = f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
AND search_vector @@ to_tsquery('english', $5)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
"""
# Combined CTE query for both semantic and BM25 across all fact types
# Uses window functions to limit per fact_type per method
results = await conn.fetch(
f"""
WITH semantic_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity,
NULL::float AS bm25_score,
'semantic' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1::vector) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND embedding IS NOT NULL
AND fact_type = ANY($3)
AND (1 - (embedding <=> $1::vector)) >= 0.3
{tags_clause}
),
bm25_ranked AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
NULL::float AS similarity,
ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
'bm25' AS source,
ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY ts_rank_cd(search_vector, to_tsquery('english', $5)) DESC) AS rn
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = ANY($3)
AND search_vector @@ to_tsquery('english', $5)
{tags_clause}
),
semantic AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM semantic_ranked WHERE rn <= $4
),
bm25 AS (
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
similarity, bm25_score, source
FROM bm25_ranked WHERE rn <= $4
)
SELECT * FROM semantic
UNION ALL
SELECT * FROM bm25
""",
*params,
)
results = await conn.fetch(query, *params)
# Group results by fact_type and source
result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {ft: ([], []) for ft in fact_types}
@@ -561,623 +501,6 @@ async def retrieve_temporal_combined(
return results_by_ft
async def retrieve_temporal(
conn,
query_emb_str: str,
bank_id: str,
fact_type: str,
start_date: datetime,
end_date: datetime,
budget: int,
semantic_threshold: float = 0.1,
tags: list[str] | None = None,
) -> list[RetrievalResult]:
"""
Temporal retrieval with spreading activation.
Strategy:
1. Find entry points (facts in date range with semantic relevance)
2. Spread through temporal links to related facts
3. Score by temporal proximity + semantic similarity + link weight
Args:
conn: Database connection
query_emb_str: Query embedding as string
agent_id: bank ID
fact_type: Fact type to filter
start_date: Start of time range
end_date: End of time range
budget: Node budget for spreading
semantic_threshold: Minimum semantic similarity to include
tags: Optional list of tags for visibility filtering (OR matching)
Returns:
List of RetrievalResult objects with temporal scores
"""
# Ensure start_date and end_date are timezone-aware (UTC) to match database datetimes
if start_date.tzinfo is None:
start_date = start_date.replace(tzinfo=UTC)
if end_date.tzinfo is None:
end_date = end_date.replace(tzinfo=UTC)
from .tags import TagsMatch, build_tags_where_clause_simple
tags_clause = build_tags_where_clause_simple(tags, 7)
params = [query_emb_str, bank_id, fact_type, start_date, end_date, semantic_threshold]
if tags:
params.append(tags)
entry_points = await conn.fetch(
f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, embedding, fact_type, document_id, chunk_id, tags,
1 - (embedding <=> $1::vector) AS similarity
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = $3
AND embedding IS NOT NULL
AND (
-- Match if occurred range overlaps with query range
(occurred_start IS NOT NULL AND occurred_end IS NOT NULL
AND occurred_start <= $5 AND occurred_end >= $4)
OR
-- Match if mentioned_at falls within query range
(mentioned_at IS NOT NULL AND mentioned_at BETWEEN $4 AND $5)
OR
-- Match if any occurred date is set and overlaps (even if only start or end is set)
(occurred_start IS NOT NULL AND occurred_start BETWEEN $4 AND $5)
OR
(occurred_end IS NOT NULL AND occurred_end BETWEEN $4 AND $5)
)
AND (1 - (embedding <=> $1::vector)) >= $6
{tags_clause}
ORDER BY COALESCE(occurred_start, mentioned_at, occurred_end) DESC, (embedding <=> $1::vector) ASC
LIMIT 10
""",
*params,
)
if not entry_points:
return []
# Calculate temporal scores for entry points
total_days = (end_date - start_date).total_seconds() / 86400
mid_date = start_date + (end_date - start_date) / 2 # Calculate once for all comparisons
results = []
visited = set()
for ep in entry_points:
unit_id = str(ep["id"])
visited.add(unit_id)
# Calculate temporal proximity using the most relevant date
# Priority: occurred_start/end (event time) > mentioned_at (mention time)
best_date = None
if ep["occurred_start"] is not None and ep["occurred_end"] is not None:
# Use midpoint of occurred range
best_date = ep["occurred_start"] + (ep["occurred_end"] - ep["occurred_start"]) / 2
elif ep["occurred_start"] is not None:
best_date = ep["occurred_start"]
elif ep["occurred_end"] is not None:
best_date = ep["occurred_end"]
elif ep["mentioned_at"] is not None:
best_date = ep["mentioned_at"]
# Temporal proximity score (closer to range center = higher score)
if best_date:
days_from_mid = abs((best_date - mid_date).total_seconds() / 86400)
temporal_proximity = 1.0 - min(days_from_mid / (total_days / 2), 1.0) if total_days > 0 else 1.0
else:
temporal_proximity = 0.5 # Fallback if no dates (shouldn't happen due to WHERE clause)
# Create RetrievalResult with temporal scores
ep_result = RetrievalResult.from_db_row(dict(ep))
ep_result.temporal_score = temporal_proximity
ep_result.temporal_proximity = temporal_proximity
results.append(ep_result)
# Spread through temporal links using BATCHED neighbor fetching
# Map node_id -> (semantic_sim, temporal_score) for propagation
node_scores = {str(ep["id"]): (ep["similarity"], 1.0) for ep in entry_points}
frontier = list(node_scores.keys()) # Current batch of nodes to expand
budget_remaining = budget - len(entry_points)
batch_size = 20 # Process this many nodes per DB query
while frontier and budget_remaining > 0:
# Take a batch from frontier
batch_ids = frontier[:batch_size]
frontier = frontier[batch_size:]
# Batch fetch all neighbors for this batch of nodes
neighbors = await conn.fetch(
f"""
SELECT mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start, mu.occurred_end, mu.mentioned_at, mu.embedding, mu.fact_type, mu.document_id, mu.chunk_id,
ml.weight, ml.link_type, ml.from_unit_id,
1 - (mu.embedding <=> $1::vector) AS similarity
FROM {fq_table("memory_links")} ml
JOIN {fq_table("memory_units")} mu ON ml.to_unit_id = mu.id
WHERE ml.from_unit_id = ANY($2::uuid[])
AND ml.link_type IN ('temporal', 'causes', 'caused_by', 'enables', 'prevents')
AND ml.weight >= 0.1
AND mu.fact_type = $3
AND mu.embedding IS NOT NULL
AND (1 - (mu.embedding <=> $1::vector)) >= $4
ORDER BY ml.weight DESC
LIMIT $5
""",
query_emb_str,
batch_ids,
fact_type,
semantic_threshold,
batch_size * 10, # Allow up to 10 neighbors per node in batch
)
for n in neighbors:
neighbor_id = str(n["id"])
if neighbor_id in visited:
continue
visited.add(neighbor_id)
budget_remaining -= 1
# Get parent's scores for propagation
parent_id = str(n["from_unit_id"])
_, parent_temporal_score = node_scores.get(parent_id, (0.5, 0.5))
# Calculate temporal score for neighbor using best available date
neighbor_best_date = None
if n["occurred_start"] is not None and n["occurred_end"] is not None:
neighbor_best_date = n["occurred_start"] + (n["occurred_end"] - n["occurred_start"]) / 2
elif n["occurred_start"] is not None:
neighbor_best_date = n["occurred_start"]
elif n["occurred_end"] is not None:
neighbor_best_date = n["occurred_end"]
elif n["mentioned_at"] is not None:
neighbor_best_date = n["mentioned_at"]
if neighbor_best_date:
days_from_mid = abs((neighbor_best_date - mid_date).total_seconds() / 86400)
neighbor_temporal_proximity = (
1.0 - min(days_from_mid / (total_days / 2), 1.0) if total_days > 0 else 1.0
)
else:
neighbor_temporal_proximity = 0.3 # Lower score if no temporal data
# Boost causal links (same as graph retrieval)
link_type = n["link_type"]
if link_type in ("causes", "caused_by"):
causal_boost = 2.0
elif link_type in ("enables", "prevents"):
causal_boost = 1.5
else:
causal_boost = 1.0
# Propagate temporal score through links (decay, with causal boost)
propagated_temporal = parent_temporal_score * n["weight"] * causal_boost * 0.7
# Combined temporal score
combined_temporal = max(neighbor_temporal_proximity, propagated_temporal)
# Create RetrievalResult with temporal scores
neighbor_result = RetrievalResult.from_db_row(dict(n))
neighbor_result.temporal_score = combined_temporal
neighbor_result.temporal_proximity = neighbor_temporal_proximity
results.append(neighbor_result)
# Track scores for propagation and add to frontier
if budget_remaining > 0 and combined_temporal > 0.2:
node_scores[neighbor_id] = (n["similarity"], combined_temporal)
frontier.append(neighbor_id)
if budget_remaining <= 0:
break
return results
async def retrieve_parallel(
pool,
query_text: str,
query_embedding_str: str,
bank_id: str,
fact_type: str,
thinking_budget: int,
question_date: datetime | None = None,
query_analyzer: Optional["QueryAnalyzer"] = None,
graph_retriever: GraphRetriever | None = None,
temporal_constraint: tuple | None = None, # Pre-extracted temporal constraint
tags: list[str] | None = None, # Visibility scope tags for filtering
) -> ParallelRetrievalResult:
"""
Run 3-way or 4-way parallel retrieval (adds temporal if detected).
Args:
pool: Database connection pool
query_text: Query text
query_embedding_str: Query embedding as string
bank_id: Bank ID
fact_type: Fact type to filter
thinking_budget: Budget for graph traversal and retrieval limits
question_date: Optional date when question was asked (for temporal filtering)
query_analyzer: Query analyzer to use (defaults to TransformerQueryAnalyzer)
graph_retriever: Graph retrieval strategy (defaults to configured retriever)
temporal_constraint: Pre-extracted temporal constraint (optional)
tags: Optional list of tags for visibility filtering (OR matching)
Returns:
ParallelRetrievalResult with semantic, bm25, graph, temporal results and timings
"""
retriever = graph_retriever or get_default_graph_retriever()
# Use optimized parallel path for MPFP and LinkExpansion (runs all methods truly in parallel)
# BFS uses legacy path that extracts temporal constraint upfront
if retriever.name in ("mpfp", "link_expansion"):
return await _retrieve_parallel_mpfp(
pool,
query_text,
query_embedding_str,
bank_id,
fact_type,
thinking_budget,
temporal_constraint,
retriever,
question_date,
query_analyzer,
tags=tags,
)
else:
# For BFS, extract temporal constraint upfront (legacy path)
if temporal_constraint is None:
from .temporal_extraction import extract_temporal_constraint
temporal_constraint = extract_temporal_constraint(
query_text, reference_date=question_date, analyzer=query_analyzer
)
return await _retrieve_parallel_bfs(
pool,
query_text,
query_embedding_str,
bank_id,
fact_type,
thinking_budget,
temporal_constraint,
retriever,
tags=tags,
)
@dataclass
class _TimedResult:
"""Internal result with timing."""
results: list[RetrievalResult]
time: float
conn_wait: float = 0.0 # Connection acquisition wait time
async def _retrieve_parallel_mpfp(
pool,
query_text: str,
query_embedding_str: str,
bank_id: str,
fact_type: str,
thinking_budget: int,
temporal_constraint: tuple | None,
retriever: GraphRetriever,
question_date: datetime | None = None,
query_analyzer=None,
tags: list[str] | None = None,
) -> ParallelRetrievalResult:
"""
MPFP retrieval with true parallelization.
All methods run independently in parallel:
- Semantic: vector similarity search
- BM25: keyword search
- Graph: MPFP traversal (does its own semantic seeds internally)
- Temporal: date extraction (if needed) + date-range search
Temporal extraction runs IN PARALLEL with other retrievals, so even if
dateparser is slow, it doesn't block semantic/BM25/graph.
"""
import time
async def run_semantic() -> _TimedResult:
"""Independent semantic retrieval."""
start = time.time()
acquire_start = time.time()
async with acquire_with_retry(pool) as conn:
conn_wait = time.time() - acquire_start
results = await retrieve_semantic(
conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget, tags=tags
)
return _TimedResult(results, time.time() - start, conn_wait)
async def run_bm25() -> _TimedResult:
"""Independent BM25 retrieval."""
start = time.time()
acquire_start = time.time()
async with acquire_with_retry(pool) as conn:
conn_wait = time.time() - acquire_start
results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget, tags=tags)
return _TimedResult(results, time.time() - start, conn_wait)
async def run_graph() -> tuple[list[RetrievalResult], float, MPFPTimings | None]:
"""Independent graph retrieval - does its own semantic seeds."""
start = time.time()
# MPFP does its own semantic seeds via _find_semantic_seeds
# Note: temporal_seeds not used here to avoid dependency on temporal extraction
results, mpfp_timing = await retriever.retrieve(
pool=pool,
query_embedding_str=query_embedding_str,
bank_id=bank_id,
fact_type=fact_type,
budget=thinking_budget,
query_text=query_text,
semantic_seeds=None, # Let MPFP find its own seeds
temporal_seeds=None, # Don't wait for temporal extraction
tags=tags,
)
return results, time.time() - start, mpfp_timing
@dataclass
class _TemporalWithConstraint:
"""Temporal results with the extracted constraint."""
results: list[RetrievalResult]
time: float
constraint: tuple | None
extraction_time: float # Time spent in query analyzer (dateparser)
conn_wait: float = 0.0 # Connection acquisition wait time
async def run_temporal_with_extraction() -> _TemporalWithConstraint:
"""
Extract temporal constraint AND run temporal retrieval.
This runs in parallel with semantic/BM25/graph, so dateparser
latency doesn't block other retrievals.
"""
start = time.time()
# Use pre-provided constraint if available
tc = temporal_constraint
extraction_time = 0.0
# Otherwise extract from query (this is the potentially slow dateparser call)
if tc is None:
from .temporal_extraction import extract_temporal_constraint
extraction_start = time.time()
tc = extract_temporal_constraint(query_text, reference_date=question_date, analyzer=query_analyzer)
extraction_time = time.time() - extraction_start
# If no temporal constraint found, return empty (but still report extraction time)
if tc is None:
return _TemporalWithConstraint([], time.time() - start, None, extraction_time, 0.0)
# Run temporal retrieval with the extracted constraint
tc_start, tc_end = tc
acquire_start = time.time()
async with acquire_with_retry(pool) as conn:
conn_wait = time.time() - acquire_start
results = await retrieve_temporal(
conn,
query_embedding_str,
bank_id,
fact_type,
tc_start,
tc_end,
budget=thinking_budget,
semantic_threshold=0.1,
)
return _TemporalWithConstraint(results, time.time() - start, tc, extraction_time, conn_wait)
# Run ALL methods in parallel (including temporal extraction!)
semantic_result, bm25_result, graph_result, temporal_result = await asyncio.gather(
run_semantic(),
run_bm25(),
run_graph(),
run_temporal_with_extraction(),
)
graph_results, graph_time, mpfp_timing = graph_result
# Compute max connection wait across all methods (graph handles its own connections)
max_conn_wait = max(semantic_result.conn_wait, bm25_result.conn_wait, temporal_result.conn_wait)
return ParallelRetrievalResult(
semantic=semantic_result.results,
bm25=bm25_result.results,
graph=graph_results,
temporal=temporal_result.results if temporal_result.results else None,
timings={
"semantic": semantic_result.time,
"bm25": bm25_result.time,
"graph": graph_time,
"temporal": temporal_result.time,
"temporal_extraction": temporal_result.extraction_time,
},
temporal_constraint=temporal_result.constraint,
mpfp_timings=[mpfp_timing] if mpfp_timing else [],
max_conn_wait=max_conn_wait,
)
async def _get_temporal_entry_points(
conn,
query_embedding_str: str,
bank_id: str,
fact_type: str,
start_date: datetime,
end_date: datetime,
limit: int = 20,
semantic_threshold: float = 0.1,
) -> list[RetrievalResult]:
"""Get temporal entry points (facts in date range with semantic relevance)."""
if start_date.tzinfo is None:
start_date = start_date.replace(tzinfo=UTC)
if end_date.tzinfo is None:
end_date = end_date.replace(tzinfo=UTC)
rows = await conn.fetch(
f"""
SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at,
embedding, fact_type, document_id, chunk_id,
1 - (embedding <=> $1::vector) AS similarity
FROM {fq_table("memory_units")}
WHERE bank_id = $2
AND fact_type = $3
AND embedding IS NOT NULL
AND (
(occurred_start IS NOT NULL AND occurred_end IS NOT NULL
AND occurred_start <= $5 AND occurred_end >= $4)
OR (mentioned_at IS NOT NULL AND mentioned_at BETWEEN $4 AND $5)
OR (occurred_start IS NOT NULL AND occurred_start BETWEEN $4 AND $5)
OR (occurred_end IS NOT NULL AND occurred_end BETWEEN $4 AND $5)
)
AND (1 - (embedding <=> $1::vector)) >= $6
ORDER BY COALESCE(occurred_start, mentioned_at, occurred_end) DESC,
(embedding <=> $1::vector) ASC
LIMIT $7
""",
query_embedding_str,
bank_id,
fact_type,
start_date,
end_date,
semantic_threshold,
limit,
)
results = []
total_days = max((end_date - start_date).total_seconds() / 86400, 1)
mid_date = start_date + (end_date - start_date) / 2
for row in rows:
result = RetrievalResult.from_db_row(dict(row))
# Calculate temporal proximity score
best_date = None
if row["occurred_start"] and row["occurred_end"]:
best_date = row["occurred_start"] + (row["occurred_end"] - row["occurred_start"]) / 2
elif row["occurred_start"]:
best_date = row["occurred_start"]
elif row["occurred_end"]:
best_date = row["occurred_end"]
elif row["mentioned_at"]:
best_date = row["mentioned_at"]
if best_date:
days_from_mid = abs((best_date - mid_date).total_seconds() / 86400)
result.temporal_proximity = 1.0 - min(days_from_mid / (total_days / 2), 1.0)
else:
result.temporal_proximity = 0.5
result.temporal_score = result.temporal_proximity
results.append(result)
return results
async def _retrieve_parallel_bfs(
pool,
query_text: str,
query_embedding_str: str,
bank_id: str,
fact_type: str,
thinking_budget: int,
temporal_constraint: tuple | None,
retriever: GraphRetriever,
tags: list[str] | None = None,
) -> ParallelRetrievalResult:
"""BFS retrieval: all methods run in parallel (original behavior)."""
import time
async def run_semantic() -> _TimedResult:
start = time.time()
async with acquire_with_retry(pool) as conn:
results = await retrieve_semantic(
conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget, tags=tags
)
return _TimedResult(results, time.time() - start)
async def run_bm25() -> _TimedResult:
start = time.time()
async with acquire_with_retry(pool) as conn:
results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget, tags=tags)
return _TimedResult(results, time.time() - start)
async def run_graph() -> _TimedResult:
start = time.time()
results, _ = await retriever.retrieve(
pool=pool,
query_embedding_str=query_embedding_str,
bank_id=bank_id,
fact_type=fact_type,
budget=thinking_budget,
query_text=query_text,
tags=tags,
)
return _TimedResult(results, time.time() - start)
async def run_temporal(tc_start, tc_end) -> _TimedResult:
start = time.time()
async with acquire_with_retry(pool) as conn:
results = await retrieve_temporal(
conn,
query_embedding_str,
bank_id,
fact_type,
tc_start,
tc_end,
budget=thinking_budget,
semantic_threshold=0.1,
tags=tags,
)
return _TimedResult(results, time.time() - start)
if temporal_constraint:
tc_start, tc_end = temporal_constraint
semantic_r, bm25_r, graph_r, temporal_r = await asyncio.gather(
run_semantic(),
run_bm25(),
run_graph(),
run_temporal(tc_start, tc_end),
)
return ParallelRetrievalResult(
semantic=semantic_r.results,
bm25=bm25_r.results,
graph=graph_r.results,
temporal=temporal_r.results,
timings={
"semantic": semantic_r.time,
"bm25": bm25_r.time,
"graph": graph_r.time,
"temporal": temporal_r.time,
},
temporal_constraint=temporal_constraint,
)
else:
semantic_r, bm25_r, graph_r = await asyncio.gather(
run_semantic(),
run_bm25(),
run_graph(),
)
return ParallelRetrievalResult(
semantic=semantic_r.results,
bm25=bm25_r.results,
graph=graph_r.results,
temporal=None,
timings={
"semantic": semantic_r.time,
"bm25": bm25_r.time,
"graph": graph_r.time,
},
temporal_constraint=None,
)
async def retrieve_all_fact_types_parallel(
pool,
query_text: str,
+10 -1
View File
@@ -19,6 +19,7 @@ async def extract_facts(
context: str = "",
llm_config: "LLMConfig" = None,
agent_name: str = None,
config=None,
) -> tuple[list["Fact"], list[tuple[str, int]]]:
"""
Extract semantic facts from text using LLM.
@@ -35,6 +36,7 @@ async def extract_facts(
context: Context about the conversation/document
llm_config: LLM configuration to use
agent_name: Optional agent name to help identify agent-related facts
config: HindsightConfig to use (defaults to global config if not provided)
Returns:
Tuple of (facts, chunks) where:
@@ -47,12 +49,19 @@ async def extract_facts(
if not text or not text.strip():
return [], []
# Use provided config or fall back to global config
if config is None:
from ..config import _get_raw_config
config = _get_raw_config()
facts, chunks, _ = await extract_facts_from_text(
text,
event_date,
context=context,
llm_config=llm_config,
agent_name=agent_name,
config=config,
context=context,
)
if not facts:
@@ -96,7 +96,13 @@ class DefaultExtensionContext(ExtensionContext):
async def run_migration(self, schema: str) -> None:
"""Run migrations for a specific schema."""
from hindsight_api.migrations import ensure_embedding_dimension, run_migrations
from hindsight_api.config import get_config
from hindsight_api.migrations import (
ensure_embedding_dimension,
ensure_text_search_extension,
ensure_vector_extension,
run_migrations,
)
# Prefer getting URL from memory engine (handles pg0 case where URL is set after init)
db_url = self._database_url
@@ -107,6 +113,9 @@ class DefaultExtensionContext(ExtensionContext):
run_migrations(db_url, schema=schema)
# Get config for vector extension setting
config = get_config()
# Ensure embedding column dimension matches the model's dimension
# This is needed because migrations create columns with default dimension
if self._memory_engine is not None:
@@ -114,7 +123,15 @@ class DefaultExtensionContext(ExtensionContext):
if embeddings is not None:
dimension = getattr(embeddings, "dimension", None)
if dimension is not None:
ensure_embedding_dimension(db_url, dimension, schema=schema)
ensure_embedding_dimension(
db_url, dimension, schema=schema, vector_extension=config.vector_extension
)
# Ensure vector indexes match the configured extension
ensure_vector_extension(db_url, vector_extension=config.vector_extension, schema=schema)
# Ensure text search columns/indexes match the configured extension
ensure_text_search_extension(db_url, text_search_extension=config.text_search_extension, schema=schema)
def get_memory_engine(self) -> "MemoryEngineInterface":
"""Get the memory engine interface."""
@@ -2,6 +2,7 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any
from hindsight_api.extensions.base import Extension
from hindsight_api.models import RequestContext
@@ -88,6 +89,54 @@ class TenantExtension(Extension, ABC):
"""
...
async def get_tenant_config(self, context: RequestContext) -> dict[str, Any]:
"""
Get tenant-specific configuration overrides.
This method is called during hierarchical configuration resolution to get
tenant-level config overrides. The returned dict should contain Python field
names (lowercase snake_case) as keys, not environment variable names.
Example:
{"llm_model": "gpt-4", "retain_extraction_mode": "verbose"}
The default implementation returns an empty dict (no tenant-specific config).
Override this method in custom extensions to provide tenant-specific configuration.
Args:
context: The request context containing tenant information.
Returns:
Dict of config field names to values (only configurable fields).
Empty dict if no tenant-specific config.
"""
return {}
async def get_allowed_config_fields(self, context: RequestContext, bank_id: str) -> set[str] | None:
"""
Get set of config fields that this tenant/bank is allowed to modify.
This method controls which configurable fields can be modified via the bank config API.
It enables fine-grained permission control per tenant or per bank.
Examples:
- Return None: Allow all configurable fields (default)
- Return {"retain_chunk_size", "retain_custom_instructions"}: Allow only these fields
- Return set(): Allow no modifications (read-only)
The default implementation returns None (all configurable fields allowed).
Override this method in custom extensions to implement custom permission logic.
Args:
context: The request context containing tenant information.
bank_id: The bank identifier for per-bank permissions.
Returns:
Set of allowed field names, or None to allow all configurable fields.
Returned fields must be a subset of HindsightConfig.get_configurable_fields().
"""
return None
async def authenticate_mcp(self, context: RequestContext) -> TenantContext:
"""
Authenticate MCP requests.
+5 -2
View File
@@ -23,7 +23,7 @@ import uvicorn
from . import MemoryEngine, __version__
from .api import create_app
from .banner import print_banner
from .config import DEFAULT_WORKERS, ENV_WORKERS, HindsightConfig, get_config
from .config import DEFAULT_WORKERS, ENV_WORKERS, HindsightConfig, _get_raw_config
from .daemon import (
DEFAULT_DAEMON_PORT,
DEFAULT_IDLE_TIMEOUT,
@@ -68,7 +68,7 @@ def main():
global _memory
# Load configuration from environment (for CLI args defaults)
config = get_config()
config = _get_raw_config()
parser = argparse.ArgumentParser(
prog="hindsight-api",
@@ -155,6 +155,8 @@ def main():
config = HindsightConfig(
database_url=config.database_url,
database_schema=config.database_schema,
vector_extension=config.vector_extension,
text_search_extension=config.text_search_extension,
llm_provider=config.llm_provider,
llm_api_key=config.llm_api_key,
llm_model=config.llm_model,
@@ -227,6 +229,7 @@ def main():
log_level=args.log_level,
log_format=config.log_format,
mcp_enabled=config.mcp_enabled,
enable_bank_config_api=config.enable_bank_config_api,
graph_retriever=config.graph_retriever,
mpfp_top_k_neighbors=config.mpfp_top_k_neighbors,
recall_max_concurrent=config.recall_max_concurrent,
+376 -19
View File
@@ -33,35 +33,39 @@ logger = logging.getLogger(__name__)
MIGRATION_LOCK_ID = 123456789
def _detect_vector_extension(conn) -> str:
def _detect_vector_extension(conn, vector_extension: str = "pgvector") -> str:
"""
Detect available vector extension: 'vchord' or 'pgvector'.
Prefers vchord if both available. Raises error if neither found.
Validate vector extension: 'vchord' or 'pgvector'.
Args:
conn: SQLAlchemy connection object
vector_extension: Configured extension ("pgvector" or "vchord")
Returns:
"vchord" or "pgvector"
Raises:
RuntimeError: If neither extension is installed
RuntimeError: If configured extension is not installed
"""
# Check vchord first (preferred for high-dimensional embeddings)
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if vchord_check:
logger.debug("Detected vector extension: vchord")
# Verify the configured extension is installed
if vector_extension == "vchord":
vchord_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vchord'")).scalar()
if not vchord_check:
raise RuntimeError(
"Configured vector extension 'vchord' not found. Install it with: CREATE EXTENSION vchord CASCADE;"
)
logger.debug("Using configured vector extension: vchord")
return "vchord"
# Fall back to pgvector
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if pgvector_check:
logger.debug("Detected vector extension: pgvector")
elif vector_extension == "pgvector":
pgvector_check = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'")).scalar()
if not pgvector_check:
raise RuntimeError(
"Configured vector extension 'pgvector' not found. Install it with: CREATE EXTENSION vector;"
)
logger.debug("Using configured vector extension: pgvector")
return "pgvector"
raise RuntimeError(
"Neither vchord nor pgvector extension found. Install one: CREATE EXTENSION vchord; or CREATE EXTENSION vector;"
)
else:
raise ValueError(f"Invalid vector_extension: {vector_extension}. Must be 'pgvector' or 'vchord'")
def _get_schema_lock_id(schema: str) -> int:
@@ -355,6 +359,7 @@ def ensure_embedding_dimension(
database_url: str,
required_dimension: int,
schema: str | None = None,
vector_extension: str = "pgvector",
) -> None:
"""
Ensure the embedding column dimension matches the model's dimension.
@@ -369,6 +374,7 @@ def ensure_embedding_dimension(
database_url: SQLAlchemy database URL
required_dimension: The embedding dimension required by the model
schema: Target PostgreSQL schema name (None for public)
vector_extension: Configured vector extension ("pgvector" or "vchord")
Raises:
RuntimeError: If dimension mismatch with existing data
@@ -393,8 +399,8 @@ def ensure_embedding_dimension(
return
# Detect which vector extension is available
vector_ext = _detect_vector_extension(conn)
logger.info(f"Detected vector extension: {vector_ext}")
vector_ext = _detect_vector_extension(conn, vector_extension)
logger.info(f"Using vector extension: {vector_ext}")
# Get current column dimension from pg_attribute
# pgvector stores dimension in atttypmod
@@ -491,3 +497,354 @@ def ensure_embedding_dimension(
conn.commit()
logger.info(f"Successfully changed embedding dimension to {required_dimension}")
def ensure_vector_extension(
database_url: str,
vector_extension: str = "pgvector",
schema: str | None = None,
) -> None:
"""
Ensure the vector indexes match the configured vector extension.
This function checks the current vector index type in the database
and adjusts it if necessary:
- If index type matches configured extension: no action needed
- If they differ and tables are empty: drop old indexes, recreate with new type
- If they differ and tables have data: raise error with migration guidance
Args:
database_url: SQLAlchemy database URL
vector_extension: Configured vector extension ("pgvector" or "vchord")
schema: Target PostgreSQL schema name (None for public)
Raises:
RuntimeError: If extension mismatch with existing data
"""
schema_name = schema or "public"
engine = create_engine(database_url)
with engine.connect() as conn:
# Detect which vector extension should be used
target_ext = _detect_vector_extension(conn, vector_extension)
logger.info(f"Target vector extension: {target_ext}")
# Tables with vector indexes to check
tables_to_check = [
("memory_units", "idx_memory_units_embedding"),
("learnings", "idx_learnings_embedding"),
("pinned_reflections", "idx_pinned_reflections_embedding"),
]
# Determine target index type
target_index_type = "vchordrq" if target_ext == "vchord" else "hnsw"
mismatched_tables = []
tables_with_data = []
for table_name, index_name in tables_to_check:
# Check if table exists
table_exists = conn.execute(
text("""
SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_schema = :schema AND table_name = :table_name
)
"""),
{"schema": schema_name, "table_name": table_name},
).scalar()
if not table_exists:
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
continue
# Check current index type by querying pg_indexes
current_index_info = conn.execute(
text("""
SELECT indexdef
FROM pg_indexes
WHERE schemaname = :schema
AND tablename = :table_name
AND indexname LIKE :index_pattern
"""),
{"schema": schema_name, "table_name": table_name, "index_pattern": "%embedding%"},
).fetchone()
if not current_index_info:
logger.warning(f"No embedding index found for {table_name}, will create it")
mismatched_tables.append((table_name, index_name, None))
continue
indexdef = current_index_info[0].lower()
if "vchordrq" in indexdef:
current_index_type = "vchordrq"
elif "hnsw" in indexdef:
current_index_type = "hnsw"
else:
logger.warning(f"Unknown index type for {table_name}: {indexdef}")
continue
# Check if index type matches target
if current_index_type != target_index_type:
logger.info(
f"Index type mismatch on {table_name}: current={current_index_type}, target={target_index_type}"
)
mismatched_tables.append((table_name, index_name, current_index_type))
# Check if table has data
row_count = conn.execute(
text(f"SELECT COUNT(*) FROM {schema_name}.{table_name} WHERE embedding IS NOT NULL")
).scalar()
if row_count > 0:
tables_with_data.append((table_name, row_count))
else:
logger.debug(f"Index type OK for {table_name}: {current_index_type}")
# If no mismatches, we're done
if not mismatched_tables:
logger.debug(f"All vector indexes match configured extension: {target_ext}")
return
# If there's data in any mismatched table, raise error
if tables_with_data:
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
raise RuntimeError(
f"Cannot change vector extension from {current_index_type} to {target_index_type}: "
f"the following tables contain data: {table_list}. "
f"To change vector extension, you must either:\n"
f" 1. Re-embed all data: DELETE FROM {schema_name}.memory_units; "
f"DELETE FROM {schema_name}.learnings; DELETE FROM {schema_name}.pinned_reflections; then restart\n"
f" 2. Use the current vector extension (set HINDSIGHT_API_VECTOR_EXTENSION='{current_index_type.replace('vchordrq', 'vchord').replace('hnsw', 'pgvector')}')"
)
# Tables are empty, safe to recreate indexes
logger.info(f"Recreating vector indexes for {target_ext}")
for table_name, index_name, current_type in mismatched_tables:
# Drop existing index if it exists
if current_type:
logger.info(f"Dropping {current_type} index on {table_name}")
conn.execute(text(f"DROP INDEX IF EXISTS {schema_name}.{index_name}"))
# Create new index with appropriate type
if target_ext == "vchord":
logger.info(f"Creating vchordrq index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS {index_name}
ON {schema_name}.{table_name}
USING vchordrq (embedding vector_l2_ops)
""")
)
else: # pgvector
logger.info(f"Creating HNSW index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX IF NOT EXISTS {index_name}
ON {schema_name}.{table_name}
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64)
""")
)
conn.commit()
logger.info(f"Successfully migrated vector indexes to {target_ext}")
def ensure_text_search_extension(
database_url: str,
text_search_extension: str = "native",
schema: str | None = None,
) -> None:
"""
Ensure the text search columns and indexes match the configured extension.
This function checks the current search_vector column type and index type
in the database and adjusts them if necessary:
- If they match configured extension: no action needed
- If they differ and tables are empty: drop old column/index, recreate with new type
- If they differ and tables have data: raise error with migration guidance
Args:
database_url: SQLAlchemy database URL
text_search_extension: Configured text search extension ("native" or "vchord")
schema: Target PostgreSQL schema name (None for public)
Raises:
RuntimeError: If extension mismatch with existing data
"""
schema_name = schema or "public"
engine = create_engine(database_url)
with engine.connect() as conn:
# Tables with search_vector columns to check
tables_to_check = [
"memory_units",
"reflections", # Renamed from pinned_reflections in p1k2l3m4n5o6 migration
]
# Determine target column type and index type
if text_search_extension == "vchord":
target_column_type = "bm25vector"
target_index_type = "bm25"
else: # native
target_column_type = "tsvector"
target_index_type = "gin"
mismatched_tables = []
tables_with_data = []
for table_name in tables_to_check:
# Check if table exists
table_exists = conn.execute(
text("""
SELECT EXISTS (
SELECT 1 FROM information_schema.tables
WHERE table_schema = :schema AND table_name = :table_name
)
"""),
{"schema": schema_name, "table_name": table_name},
).scalar()
if not table_exists:
logger.debug(f"Table {table_name} does not exist in schema '{schema_name}', skipping")
continue
# Get current column type from information_schema
current_column_info = conn.execute(
text("""
SELECT data_type, udt_name
FROM information_schema.columns
WHERE table_schema = :schema
AND table_name = :table_name
AND column_name = 'search_vector'
"""),
{"schema": schema_name, "table_name": table_name},
).fetchone()
if not current_column_info:
logger.warning(f"No search_vector column found for {table_name}, will create it")
mismatched_tables.append((table_name, None, None))
continue
# Check column type (udt_name contains the actual type: tsvector, bm25vector, etc.)
current_column_type = current_column_info[1] # udt_name
# Get current index type
current_index_info = conn.execute(
text("""
SELECT am.amname
FROM pg_indexes pi
JOIN pg_class c ON c.relname = pi.indexname
JOIN pg_am am ON am.oid = c.relam
WHERE pi.schemaname = :schema
AND pi.tablename = :table_name
AND pi.indexname LIKE '%text_search%'
"""),
{"schema": schema_name, "table_name": table_name},
).fetchone()
current_index_type = current_index_info[0] if current_index_info else None
# Check if column and index types match target
column_matches = current_column_type == target_column_type
index_matches = current_index_type == target_index_type if current_index_type else False
if not (column_matches and index_matches):
logger.info(
f"Text search mismatch on {table_name}: "
f"column={current_column_type} (want {target_column_type}), "
f"index={current_index_type} (want {target_index_type})"
)
mismatched_tables.append((table_name, current_column_type, current_index_type))
# Check if table has data
row_count = conn.execute(text(f"SELECT COUNT(*) FROM {schema_name}.{table_name}")).scalar()
if row_count > 0:
tables_with_data.append((table_name, row_count))
else:
logger.debug(f"Text search OK for {table_name}: {current_column_type}/{current_index_type}")
# If no mismatches, we're done
if not mismatched_tables:
logger.debug(f"All text search columns/indexes match configured extension: {text_search_extension}")
return
# If there's data in any mismatched table, raise error
if tables_with_data:
table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
current_ext = "native" if mismatched_tables[0][1] == "tsvector" else "vchord"
raise RuntimeError(
f"Cannot change text search extension from {current_ext} to {text_search_extension}: "
f"the following tables contain data: {table_list}. "
f"To change text search extension, you must either:\n"
f" 1. Clear all data: DELETE FROM {schema_name}.memory_units; "
f"DELETE FROM {schema_name}.reflections; then restart\n"
f" 2. Use the current text search extension (set HINDSIGHT_API_TEXT_SEARCH_EXTENSION='{current_ext}')"
)
# Tables are empty, safe to recreate columns/indexes
logger.info(f"Recreating text search columns/indexes for {text_search_extension}")
for table_name, current_col_type, current_idx_type in mismatched_tables:
# Drop existing index if it exists
if current_idx_type:
logger.info(f"Dropping {current_idx_type} index on {table_name}")
conn.execute(
text(f"""
DROP INDEX IF EXISTS {schema_name}.idx_{table_name.replace(".", "_")}_text_search
""")
)
# Drop existing column if it exists
if current_col_type:
logger.info(f"Dropping {current_col_type} column on {table_name}")
conn.execute(text(f"ALTER TABLE {schema_name}.{table_name} DROP COLUMN IF EXISTS search_vector"))
# Create new column with appropriate type
if text_search_extension == "vchord":
logger.info(f"Creating bm25vector column on {table_name}")
# Note: vchord_bm25 extension creates types in bm25_catalog schema
conn.execute(
text(f"ALTER TABLE {schema_name}.{table_name} ADD COLUMN search_vector bm25_catalog.bm25vector")
)
# Create BM25 index
logger.info(f"Creating BM25 index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
ON {schema_name}.{table_name}
USING bm25 (search_vector bm25_catalog.bm25_ops)
""")
)
else: # native
logger.info(f"Creating tsvector column on {table_name}")
# Different GENERATED expression for each table
if table_name == "memory_units":
generated_expr = "to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))"
else: # reflections
generated_expr = "to_tsvector('english', COALESCE(name, '') || ' ' || content)"
conn.execute(
text(f"""
ALTER TABLE {schema_name}.{table_name}
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS ({generated_expr}) STORED
""")
)
# Create GIN index
logger.info(f"Creating GIN index on {table_name}")
conn.execute(
text(f"""
CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
ON {schema_name}.{table_name}
USING gin(search_vector)
""")
)
conn.commit()
logger.info(f"Successfully migrated text search to {text_search_extension}")
@@ -12,6 +12,7 @@ from datetime import datetime
import pytest
from hindsight_api import LLMConfig
from hindsight_api.config import _get_raw_config
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
@@ -44,6 +45,7 @@ class TestCausalRelationsValidation:
context=context,
llm_config=llm_config,
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -88,6 +90,7 @@ class TestCausalRelationsValidation:
context=context,
llm_config=llm_config,
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -124,6 +127,7 @@ class TestCausalRelationsValidation:
context=context,
llm_config=llm_config,
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract facts about the causal chain"
@@ -173,6 +177,7 @@ class TestCausalRelationsValidation:
context=context,
llm_config=llm_config,
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract facts"
@@ -209,6 +214,7 @@ class TestCausalRelationsValidation:
context=context,
llm_config=llm_config,
agent_name="TestUser",
config=_get_raw_config(),
)
# Verify relation types are all backward-looking
@@ -10,6 +10,7 @@ from datetime import datetime
import pytest
from hindsight_api import LLMConfig
from hindsight_api.config import _get_raw_config
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
@@ -37,7 +38,8 @@ After searching for weeks, I finally found a cheaper apartment in Brooklyn.
llm_config = LLMConfig.for_memory()
facts, _, _ = await extract_facts_from_text(
text=text, event_date=datetime(2024, 3, 15), context=context, llm_config=llm_config, agent_name="TestUser"
text=text, event_date=datetime(2024, 3, 15), context=context, llm_config=llm_config, agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) >= 3, f"Should extract at least 3 facts from the causal chain. Got {len(facts)}"
@@ -106,7 +108,8 @@ The renovation took three months and cost $15,000.
llm_config = LLMConfig.for_memory()
facts, _, _ = await extract_facts_from_text(
text=text, event_date=datetime(2024, 6, 1), context=context, llm_config=llm_config, agent_name="TestUser"
text=text, event_date=datetime(2024, 6, 1), context=context, llm_config=llm_config, agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) >= 4, f"Should extract at least 4 facts. Got {len(facts)}"
@@ -136,7 +139,8 @@ Machine learning fascinated me so much that I changed my career to data science.
llm_config = LLMConfig.for_memory()
facts, _, _ = await extract_facts_from_text(
text=text, event_date=datetime(2024, 1, 1), context=context, llm_config=llm_config, agent_name="TestUser"
text=text, event_date=datetime(2024, 1, 1), context=context, llm_config=llm_config, agent_name="TestUser",
config=_get_raw_config(),
)
# Check no fact references itself
@@ -163,7 +167,8 @@ The new role enabled me to lead a team of engineers.
llm_config = LLMConfig.for_memory()
facts, _, _ = await extract_facts_from_text(
text=text, event_date=datetime(2024, 2, 15), context=context, llm_config=llm_config, agent_name="TestUser"
text=text, event_date=datetime(2024, 2, 15), context=context, llm_config=llm_config, agent_name="TestUser",
config=_get_raw_config(),
)
# Validate all indices (must reference PREVIOUS facts only)
@@ -190,7 +195,8 @@ Reduced spending somewhat affected local businesses.
llm_config = LLMConfig.for_memory()
facts, _, _ = await extract_facts_from_text(
text=text, event_date=datetime(2024, 4, 1), context=context, llm_config=llm_config, agent_name="TestUser"
text=text, event_date=datetime(2024, 4, 1), context=context, llm_config=llm_config, agent_name="TestUser",
config=_get_raw_config(),
)
for i, fact in enumerate(facts):
+15 -14
View File
@@ -21,9 +21,9 @@ from hindsight_api.engine.reflect.tools import (
@pytest.fixture(autouse=True)
def enable_observations():
"""Enable observations for all tests in this module."""
from hindsight_api.config import get_config
from hindsight_api.config import _get_raw_config
config = get_config()
config = _get_raw_config()
original_value = config.enable_observations
config.enable_observations = True
yield
@@ -563,25 +563,26 @@ class TestConsolidationDisabled:
self, memory: MemoryEngine, request_context
):
"""Test that consolidation returns disabled status when enable_observations is False."""
from unittest.mock import patch
bank_id = f"test-consolidation-disabled-{uuid.uuid4().hex[:8]}"
# Create the bank
await memory.get_bank_profile(bank_id=bank_id, request_context=request_context)
# Disable observations via config
with patch("hindsight_api.config.get_config") as mock_config:
mock_config.return_value.enable_observations = False
# Disable observations for this bank via bank config
await memory._config_resolver.update_bank_config(
bank_id=bank_id,
updates={"enable_observations": False},
context=request_context,
)
result = await run_consolidation_job(
memory_engine=memory,
bank_id=bank_id,
request_context=request_context,
)
result = await run_consolidation_job(
memory_engine=memory,
bank_id=bank_id,
request_context=request_context,
)
assert result["status"] == "disabled"
assert result["bank_id"] == bank_id
assert result["status"] == "disabled"
assert result["bank_id"] == bank_id
# Cleanup
await memory.delete_bank(bank_id, request_context=request_context)
@@ -8,7 +8,7 @@ from datetime import datetime
import pytest
from hindsight_api.config import get_config, clear_config_cache
from hindsight_api.config import get_config, clear_config_cache, _get_raw_config
from hindsight_api.engine.llm_wrapper import LLMConfig
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
@@ -58,6 +58,7 @@ async def test_fact_extraction_basic_analysis(llm_config):
llm_config=llm_config,
agent_name="test-agent",
context="Friday Standup meeting",
config=_get_raw_config(),
)
duration = time.time() - start_time
@@ -11,6 +11,7 @@ from datetime import datetime
import pytest
from hindsight_api import LLMConfig
from hindsight_api.config import _get_raw_config
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
@@ -44,7 +45,8 @@ I ran into my neighbor Sarah who mentioned she's planning a trip to Italy next m
event_date=datetime(2024, 6, 15),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
input_length = len(text)
@@ -88,7 +90,8 @@ User: Perfect, I'll make a reservation for Saturday at 7pm.
event_date=datetime(2024, 6, 15),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
input_length = len(text)
@@ -144,7 +147,8 @@ I edited about 20 photos from my recent trip to the mountains.
event_date=datetime(2024, 4, 15),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
input_length = len(text)
@@ -208,7 +212,8 @@ I edited about 20 photos from my recent trip to the mountains.
event_date=datetime(2023, 5, 8), # Date from locomo dataset
context=context,
llm_config=llm_config,
agent_name=data["conversation"]["speaker_a"]
agent_name=data["conversation"]["speaker_a"],
config=_get_raw_config(),
)
# Calculate ratios
@@ -269,7 +274,8 @@ I'm planning to visit Japan next year.
event_date=datetime(2024, 6, 15),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
# Count approximate number of statements (sentences)
@@ -17,6 +17,7 @@ from datetime import UTC, datetime
import pytest
from hindsight_api import LLMConfig
from hindsight_api.config import _get_raw_config
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
# =============================================================================
@@ -48,7 +49,8 @@ Marcus felt anxious about the upcoming interview.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -80,7 +82,8 @@ The music was so loud I could barely hear myself think.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -113,7 +116,8 @@ Maybe we should reconsider the timeline.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -146,7 +150,8 @@ I'm unable to attend the conference due to scheduling conflicts.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -178,7 +183,8 @@ Unlike last year, we're ahead of schedule.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -211,7 +217,8 @@ She's enthusiastic about the opportunity.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -244,7 +251,8 @@ I'm planning to switch careers because I'm not fulfilled in my current role.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -281,7 +289,8 @@ Family is the most important thing to her.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -315,7 +324,8 @@ I prefer presenting in person rather than virtually because I can read the room
event_date=event_date,
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -372,7 +382,8 @@ I'm planning to visit Tokyo next month.
event_date=event_date,
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -423,7 +434,8 @@ with a concert surrounded by music, joy and the warm summer breeze.
event_date=event_date,
context=context,
llm_config=llm_config,
agent_name="Melanie"
agent_name="Melanie",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -493,7 +505,8 @@ It was a beautiful day and I plan to make this a regular habit.
event_date=event_date,
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -547,7 +560,8 @@ It was a beautiful day and I plan to make this a regular habit.
event_date=reference_date,
llm_config=llm_config,
agent_name="TestUser",
context="Personal diary"
context="Personal diary",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -577,7 +591,8 @@ It was a beautiful day and I plan to make this a regular habit.
event_date=reference_date,
llm_config=llm_config,
agent_name="TestUser",
context="General info"
context="General info",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -604,7 +619,8 @@ It was a beautiful day and I plan to make this a regular habit.
event_date=reference_date,
llm_config=llm_config,
agent_name="TestUser",
context="Calendar events"
context="Calendar events",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -655,7 +671,8 @@ great time! Every time I see it, I can't help but smile.
event_date=event_date,
context=context,
llm_config=llm_config,
agent_name="Deborah"
agent_name="Deborah",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -705,7 +722,8 @@ I've learned so much from it.
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -774,7 +792,8 @@ Jamie: Congratulations! I'd love to read it.
event_date=datetime(2024, 11, 13),
llm_config=llm_config,
agent_name="Marcus",
context=context
context=context,
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact from the transcript"
@@ -819,7 +838,8 @@ We presented our findings to the team yesterday.
event_date=datetime(2024, 11, 13),
llm_config=llm_config,
agent_name="TestUser",
context=context
context=context,
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract facts"
@@ -854,7 +874,8 @@ Jamie: [teasing] We'll see who's right, my Niners pick is solid.
event_date=datetime(2024, 11, 14),
context=context,
llm_config=llm_config,
agent_name=agent_name
agent_name=agent_name,
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -920,7 +941,8 @@ so the algorithm learns to box out. See you next week!
event_date=datetime(2024, 11, 13),
llm_config=llm_config,
agent_name="Marcus",
context=context
context=context,
config=_get_raw_config(),
)
assert len(facts) > 0, "Should extract at least one fact"
@@ -0,0 +1,491 @@
"""
Tests for hierarchical configuration system.
Tests config resolution hierarchy (global → tenant → bank),
key normalization, API endpoints, validation, and caching.
"""
import os
import pytest
from hindsight_api import MemoryEngine
from hindsight_api.config import HindsightConfig, normalize_config_dict, normalize_config_key
from hindsight_api.config_resolver import ConfigResolver
from hindsight_api.extensions.tenant import TenantExtension
from hindsight_api.models import RequestContext
# Enable bank config API for all tests in this module
os.environ["HINDSIGHT_API_ENABLE_BANK_CONFIG_API"] = "true"
class MockTenantExtension(TenantExtension):
"""Mock tenant extension for testing tenant-level config."""
def __init__(self, tenant_config: dict):
self.tenant_config = tenant_config
async def authenticate(self, context):
from hindsight_api.extensions.tenant import TenantContext
return TenantContext(schema_name="public")
async def list_tenants(self):
from hindsight_api.extensions.tenant import Tenant
return [Tenant(schema="public")]
async def get_tenant_config(self, context):
"""Return mock tenant config."""
return self.tenant_config
@pytest.mark.asyncio
async def test_config_key_normalization():
"""Test that env var keys are normalized to Python field names."""
# Test basic normalization
assert normalize_config_key("HINDSIGHT_API_LLM_PROVIDER") == "llm_provider"
assert normalize_config_key("HINDSIGHT_API_LLM_MODEL") == "llm_model"
assert normalize_config_key("HINDSIGHT_API_RETAIN_LLM_PROVIDER") == "retain_llm_provider"
# Test already normalized keys
assert normalize_config_key("llm_provider") == "llm_provider"
assert normalize_config_key("llm_model") == "llm_model"
# Test dict normalization
input_dict = {
"HINDSIGHT_API_LLM_PROVIDER": "openai",
"HINDSIGHT_API_LLM_MODEL": "gpt-4",
"llm_base_url": "https://api.openai.com",
}
expected = {"llm_provider": "openai", "llm_model": "gpt-4", "llm_base_url": "https://api.openai.com"}
assert normalize_config_dict(input_dict) == expected
@pytest.mark.asyncio
async def test_hierarchical_fields_categorization():
"""Test that fields are correctly categorized as configurable, credentials, or static."""
configurable = HindsightConfig.get_configurable_fields()
credentials = HindsightConfig.get_credential_fields()
static = HindsightConfig.get_static_fields()
# Verify no overlap between configurable and credentials
assert len(configurable & credentials) == 0, "Configurable fields should not include credentials"
# Verify configurable fields include behavioral settings (safe to modify)
assert "retain_extraction_mode" in configurable
assert "enable_observations" in configurable
assert "retain_chunk_size" in configurable
assert "retain_custom_instructions" in configurable
# Verify count is correct (only 4 fields)
assert len(configurable) == 4
# Verify credential fields (NEVER exposed)
assert "llm_api_key" in credentials
assert "llm_base_url" in credentials
assert "retain_llm_api_key" in credentials
assert "reflect_llm_api_key" in credentials
# Verify static fields include server settings AND non-configurable LLM fields
assert "database_url" in static
assert "port" in static
assert "host" in static
assert "embeddings_provider" in static
assert "reranker_provider" in static
assert "worker_enabled" in static
assert "llm_provider" in static # Not configurable (needs presets)
assert "llm_model" in static # Not configurable (needs presets)
assert "graph_retriever" in static # Performance tuning, not configurable
assert "llm_max_concurrent" in static # Performance tuning, not configurable
@pytest.mark.asyncio
async def test_config_hierarchy_resolution(memory, request_context):
"""Test that config resolution follows global → tenant → bank hierarchy."""
bank_id = "test-hierarchy-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
# Set up mock tenant extension with tenant-level config (use configurable fields only)
tenant_config = {"retain_chunk_size": 5000, "retain_extraction_mode": "tenant-mode"}
mock_tenant = MockTenantExtension(tenant_config)
# Create config resolver with mock tenant extension
resolver = ConfigResolver(pool=memory._pool, tenant_extension=mock_tenant)
# Test 1: Global config only (no overrides)
context = RequestContext(api_key=None, api_key_id=None, tenant_id=None, internal=False)
config = await resolver.get_bank_config(bank_id, context)
# Should have configurable fields from global config (NOT credentials or llm_provider/model)
assert "retain_chunk_size" in config # Configurable field
assert "llm_api_key" not in config # Credential - never exposed
assert "llm_provider" not in config # Not configurable (needs presets)
# Test 2: Add tenant-level overrides
config = await resolver.get_bank_config(bank_id, context)
# Should apply tenant overrides (only configurable fields)
assert config["retain_chunk_size"] == 5000 # Tenant override
assert config["retain_extraction_mode"] == "tenant-mode" # Tenant override
# Test 3: Add bank-level overrides (should take precedence)
await resolver.update_bank_config(
bank_id,
{"retain_chunk_size": 2000, "retain_extraction_mode": "bank-mode"}, # Override tenant settings
context,
)
# Config should reflect changes immediately (no caching)
config = await resolver.get_bank_config(bank_id, context)
# Bank overrides should take precedence over tenant
assert config["retain_chunk_size"] == 2000 # Bank override wins
assert config["retain_extraction_mode"] == "bank-mode" # Bank override wins
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_validation_rejects_static_fields(memory, request_context):
"""Test that attempting to override static fields raises ValueError."""
bank_id = "test-validation-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Test 1: Configurable fields should work
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 4000, "retain_extraction_mode": "verbose"})
# Test 2: Static fields should raise ValueError
with pytest.raises(ValueError, match="Cannot override static"):
await resolver.update_bank_config(bank_id, {"port": 9000})
with pytest.raises(ValueError, match="Cannot override static"):
await resolver.update_bank_config(bank_id, {"database_url": "postgresql://fake"})
with pytest.raises(ValueError, match="Cannot override static"):
await resolver.update_bank_config(bank_id, {"embeddings_provider": "openai"})
# Test 3: Credential fields should raise ValueError
with pytest.raises(ValueError, match="Cannot set credential fields"):
await resolver.update_bank_config(bank_id, {"llm_api_key": "sk-fake"})
# Test 4: Non-configurable LLM fields should raise ValueError (need presets)
with pytest.raises(ValueError, match="Cannot override static"):
await resolver.update_bank_config(bank_id, {"llm_model": "gpt-4"})
# Test 5: Mix of configurable and static should fail
with pytest.raises(ValueError, match="Cannot override static"):
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 4000, "port": 9000})
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_freshness_across_updates(memory, request_context):
"""Test that config changes are immediately visible (no stale cache)."""
bank1 = "freshness-test-1"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank1, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Test 1: Initial config reflects global defaults
config1 = await resolver.get_bank_config(bank1, None)
initial_chunk_size = config1["retain_chunk_size"]
# Test 2: Update config
await resolver.update_bank_config(bank1, {"retain_chunk_size": 4000})
# Test 3: Next call should see updated value immediately (no stale cache)
config2 = await resolver.get_bank_config(bank1, None)
assert config2["retain_chunk_size"] == 4000
# Test 4: Multiple updates are all immediately visible
await resolver.update_bank_config(bank1, {"retain_chunk_size": 4500})
config3 = await resolver.get_bank_config(bank1, None)
assert config3["retain_chunk_size"] == 4500
# Test 5: Reset restores global defaults immediately
await resolver.reset_bank_config(bank1)
config4 = await resolver.get_bank_config(bank1, None)
assert config4["retain_chunk_size"] == initial_chunk_size # Back to global default
# Test 6: Each call returns a fresh config dict (not a cached reference)
config5 = await resolver.get_bank_config(bank1, None)
config6 = await resolver.get_bank_config(bank1, None)
assert config5 is not config6 # Different object instances
finally:
await memory.delete_bank(bank1, request_context=request_context)
@pytest.mark.asyncio
async def test_config_reset_to_defaults(memory, request_context):
"""Test that resetting config removes all bank-specific overrides."""
bank_id = "test-reset-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Add bank-specific overrides
await resolver.update_bank_config(
bank_id,
{
"retain_chunk_size": 5500,
"retain_extraction_mode": "custom",
"retain_custom_instructions": "Custom instructions",
},
)
# Verify overrides applied
config = await resolver.get_bank_config(bank_id, None)
assert config["retain_chunk_size"] == 5500
assert config["retain_extraction_mode"] == "custom"
assert config["retain_custom_instructions"] == "Custom instructions"
# Reset to defaults
await resolver.reset_bank_config(bank_id)
# Verify overrides removed (back to global defaults)
config_reset = await resolver.get_bank_config(bank_id, None)
assert config_reset["retain_chunk_size"] != 5500 # Should be global default
assert config_reset["retain_extraction_mode"] != "custom" # Should be global default
# Verify bank_config is empty
bank_overrides = await resolver._load_bank_config(bank_id)
assert bank_overrides == {}
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_supports_both_key_formats(memory, request_context):
"""Test that API accepts both env var and Python field formats."""
bank_id = "test-key-format-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Test 1: Python field format
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 7000})
config = await resolver.get_bank_config(bank_id, None)
assert config["retain_chunk_size"] == 7000
# Test 2: Env var format (should be normalized)
await resolver.update_bank_config(bank_id, {"HINDSIGHT_API_RETAIN_CHUNK_SIZE": 8000})
config = await resolver.get_bank_config(bank_id, None)
assert config["retain_chunk_size"] == 8000
# Test 3: Mixed format in same request
await resolver.update_bank_config(
bank_id,
{
"retain_chunk_size": 9000, # Python format
"HINDSIGHT_API_RETAIN_EXTRACTION_MODE": "verbose", # Env format
},
)
config = await resolver.get_bank_config(bank_id, None)
assert config["retain_chunk_size"] == 9000
assert config["retain_extraction_mode"] == "verbose"
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_only_configurable_fields_stored(memory, request_context):
"""Test that only configurable fields are stored in bank config."""
bank_id = "test-filter-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Add valid configurable field
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 3500})
# Load bank config and verify only configurable fields present
bank_overrides = await resolver._load_bank_config(bank_id)
for key in bank_overrides.keys():
assert key in HindsightConfig.get_configurable_fields(), f"Non-configurable field {key} in bank config"
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_get_bank_config_no_static_or_credential_fields_leak(memory, request_context):
"""
SECURITY TEST: Verify get_bank_config() only returns configurable fields (no static/credentials).
This prevents leaking sensitive system configuration like database URLs,
API keys, LLM providers/models, worker counts, etc. when retrieving bank configuration.
"""
bank_id = "test-security-bank"
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
resolver = ConfigResolver(pool=memory._pool)
# Get bank config
config = await resolver.get_bank_config(bank_id, None)
# Get field categorizations
configurable_fields = HindsightConfig.get_configurable_fields()
credential_fields = HindsightConfig.get_credential_fields()
static_fields = HindsightConfig.get_static_fields()
# SECURITY: Verify ONLY configurable fields are returned (NO static, NO credentials)
for key in config.keys():
assert key in configurable_fields, (
f"SECURITY VIOLATION: Non-configurable field '{key}' returned by get_bank_config(). "
f"Only configurable fields should be returned to prevent leaking system config."
)
assert key not in credential_fields, (
f"SECURITY VIOLATION: Credential field '{key}' returned by get_bank_config(). "
f"Credentials must NEVER be exposed via API."
)
# SECURITY: Verify specific sensitive fields are NOT present
sensitive_fields = [
"database_url", "api_port", "host", "worker_count", # Infrastructure
"llm_api_key", "llm_base_url", # Credentials
"retain_llm_api_key", "reflect_llm_api_key", # More credentials
"llm_provider", "llm_model", # Not configurable (need presets)
]
for field in sensitive_fields:
assert field not in config, (
f"SECURITY VIOLATION: Sensitive field '{field}' returned by get_bank_config(). "
f"Must not be exposed via bank config API."
)
# Verify we have the expected configurable fields (small set)
expected_configurable = ["retain_chunk_size", "retain_extraction_mode", "enable_observations"]
for field in expected_configurable:
assert field in config, f"Expected configurable field '{field}' missing from config"
# Should have a small number of configurable fields (not hundreds)
assert len(config) < 20, f"Too many fields returned: {len(config)}"
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@pytest.mark.asyncio
async def test_config_permissions_system(memory, request_context):
"""
Test that tenant extension can control which fields banks are allowed to modify.
Tests get_allowed_config_fields() permission system.
"""
bank_id = "test-permissions-bank"
class PermissionTenantExtension(TenantExtension):
"""Mock tenant extension with configurable permissions."""
def __init__(self, allowed_fields: set[str] | None):
self.allowed_fields = allowed_fields
async def authenticate(self, context):
from hindsight_api.extensions.tenant import TenantContext
return TenantContext(schema_name="public")
async def list_tenants(self):
from hindsight_api.extensions.tenant import Tenant
return [Tenant(schema="public")]
async def get_allowed_config_fields(self, context, bank_id):
"""Return configured allowed fields."""
return self.allowed_fields
try:
# Ensure bank exists in database
await memory.get_bank_profile(bank_id, request_context=request_context)
# Test 1: None = allow all configurable fields
extension = PermissionTenantExtension(allowed_fields=None)
resolver = ConfigResolver(pool=memory._pool, tenant_extension=extension)
await resolver.update_bank_config(
bank_id, {"retain_chunk_size": 4000, "retain_extraction_mode": "verbose"}, request_context
)
config = await resolver.get_bank_config(bank_id, request_context)
assert config["retain_chunk_size"] == 4000
assert config["retain_extraction_mode"] == "verbose"
# Reset for next test
await resolver.reset_bank_config(bank_id)
# Test 2: Specific set = only those fields allowed
extension = PermissionTenantExtension(allowed_fields={"retain_chunk_size"})
resolver = ConfigResolver(pool=memory._pool, tenant_extension=extension)
# Should allow retain_chunk_size
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 5000}, request_context)
config = await resolver.get_bank_config(bank_id, request_context)
assert config["retain_chunk_size"] == 5000
# Should reject retain_extraction_mode (not in allowed list)
with pytest.raises(ValueError, match="Not allowed to modify fields"):
await resolver.update_bank_config(bank_id, {"retain_extraction_mode": "verbose"}, request_context)
# Should reject mix of allowed and disallowed
with pytest.raises(ValueError, match="Not allowed to modify fields"):
await resolver.update_bank_config(
bank_id, {"retain_chunk_size": 6000, "retain_extraction_mode": "verbose"}, request_context
)
# Reset for next test
await resolver.reset_bank_config(bank_id)
# Test 3: Empty set = no modifications allowed (read-only)
extension = PermissionTenantExtension(allowed_fields=set())
resolver = ConfigResolver(pool=memory._pool, tenant_extension=extension)
with pytest.raises(ValueError, match="Not allowed to modify fields"):
await resolver.update_bank_config(bank_id, {"retain_chunk_size": 7000}, request_context)
# Test 4: get_bank_config should filter response based on permissions
extension = PermissionTenantExtension(allowed_fields={"retain_chunk_size", "enable_observations"})
resolver = ConfigResolver(pool=memory._pool, tenant_extension=extension)
config = await resolver.get_bank_config(bank_id, request_context)
# Should only return allowed fields
assert "retain_chunk_size" in config
assert "enable_observations" in config
# Other configurable fields should be filtered out
assert "retain_extraction_mode" not in config
assert "retain_custom_instructions" not in config
finally:
await memory.delete_bank(bank_id, request_context=request_context)
@@ -12,9 +12,9 @@ import pytest
@pytest.fixture(autouse=True)
def enable_observations():
"""Enable observations for all tests in this module."""
from hindsight_api.config import get_config
from hindsight_api.config import _get_raw_config
config = get_config()
config = _get_raw_config()
original_value = config.enable_observations
config.enable_observations = True
yield
+1 -1
View File
@@ -209,7 +209,7 @@ class TestLargeBatchRetain:
raise
@pytest.mark.asyncio
@pytest.mark.timeout(120)
@pytest.mark.timeout(240) # Increased timeout for VectorChord BM25 tokenization
async def test_batch_chunking_behavior(self, memory_with_mock_llm, request_context):
"""
Test that large batches are properly chunked into sub-batches.
+7 -7
View File
@@ -45,7 +45,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.load_extension", side_effect=tracking_load_extension), \
patch("hindsight_api.main.DefaultExtensionContext"), \
patch("hindsight_api.main.print_banner"), \
@@ -96,7 +96,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.load_extension", side_effect=tracking_load_extension), \
patch("hindsight_api.main.DefaultExtensionContext"), \
patch("hindsight_api.main.print_banner"), \
@@ -143,7 +143,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.DefaultExtensionContext"), \
patch("hindsight_api.main.print_banner"), \
patch("uvicorn.run"):
@@ -200,7 +200,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.DefaultExtensionContext", side_effect=capture_context), \
patch("hindsight_api.main.print_banner"), \
patch("uvicorn.run"):
@@ -242,7 +242,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.print_banner"), \
patch("uvicorn.run"):
@@ -287,7 +287,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
patch("hindsight_api.main.create_app", return_value=mock_app), \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.print_banner"), \
patch("uvicorn.run", side_effect=capture_uvicorn_run):
@@ -327,7 +327,7 @@ class TestMainModuleExtensionLoading:
with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
patch("hindsight_api.main.create_app") as mock_create_app, \
patch("hindsight_api.main.get_config") as mock_get_config, \
patch("hindsight_api.main._get_raw_config") as mock_get_config, \
patch("hindsight_api.main.print_banner"), \
patch("uvicorn.run", side_effect=capture_uvicorn_run):
+2 -2
View File
@@ -8,14 +8,14 @@ populated from the summary for backwards compatibility.
import pytest
from hindsight_api.engine.memory_engine import Budget
from hindsight_api import RequestContext
from hindsight_api.config import get_config
from hindsight_api.config import _get_raw_config
from datetime import datetime, timezone
@pytest.fixture
def disable_observations():
"""Disable observations for a specific test."""
config = get_config()
config = _get_raw_config()
original_value = config.enable_observations
config.enable_observations = False
yield
@@ -0,0 +1,103 @@
"""
Test reflect endpoint with empty based_on (no memories scenario).
This test verifies that the API returns the correct based_on format:
- v0.3.0 (old): returned based_on as list []
- v0.4.0+ (current): returns based_on as object {"memories": [], "mental_models": [], "directives": []}
"""
import pytest
import pytest_asyncio
import httpx
from hindsight_api.api import create_app
@pytest_asyncio.fixture
async def api_client(memory):
"""Create an async test client for the FastAPI app."""
app = create_app(memory, initialize_memory=False)
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
yield client
@pytest.mark.asyncio
async def test_reflect_with_no_memories_empty_bank(api_client):
"""Test reflect on an empty bank (no memories) with include.facts enabled."""
bank_id = "test_empty_bank"
# Reflect on empty bank with facts requested
response = await api_client.post(
f"/v1/default/banks/{bank_id}/reflect",
json={
"query": "What do you know about machine learning?",
"budget": "low",
"include": {
"facts": {} # Request facts but bank is empty
}
}
)
assert response.status_code == 200
data = response.json()
# DEBUG: Print what the API actually returned
import json
print("\n" + "="*80)
print("API Response:")
print(json.dumps(data, indent=2))
print("="*80 + "\n")
# Verify response structure
assert "text" in data
assert "based_on" in data
# The API should return based_on as either:
# 1. null/None (if include.facts not set)
# 2. {"memories": [], "mental_models": [], "directives": []} (if include.facts set but empty)
# It should NEVER return based_on: []
based_on = data.get("based_on")
if based_on is not None:
assert isinstance(based_on, dict), f"based_on should be dict or null, got {type(based_on)}: {based_on}"
assert not isinstance(based_on, list), f"based_on should NEVER be a list! Got: {based_on}"
assert "memories" in based_on
assert "mental_models" in based_on
assert "directives" in based_on
# All should be empty lists
assert based_on["memories"] == []
assert based_on["mental_models"] == []
assert based_on["directives"] == []
# Verify the structure is parseable as proper types
assert isinstance(data["text"], str)
if based_on is not None:
# Verify it's the v0.4.0+ format (object with arrays)
assert isinstance(based_on["memories"], list)
assert isinstance(based_on["mental_models"], list)
assert isinstance(based_on["directives"], list)
@pytest.mark.asyncio
async def test_reflect_without_include_facts(api_client):
"""Test reflect without requesting facts (based_on should be None)."""
bank_id = "test_no_facts"
response = await api_client.post(
f"/v1/default/banks/{bank_id}/reflect",
json={
"query": "Hello world",
"budget": "low"
# No include.facts
}
)
assert response.status_code == 200
data = response.json()
# When include.facts is not set, based_on should not be in response (or be null)
based_on = data.get("based_on")
assert based_on is None, f"based_on should be None when not requested, got {type(based_on)}: {based_on}"
# Verify structure
assert isinstance(data["text"], str)
+3 -2
View File
@@ -2093,7 +2093,7 @@ async def test_custom_extraction_mode():
import os
from hindsight_api import LLMConfig
from hindsight_api.engine.retain.fact_extraction import extract_facts_from_text
from hindsight_api.config import clear_config_cache
from hindsight_api.config import clear_config_cache, _get_raw_config
# Save original env vars
original_mode = os.getenv("HINDSIGHT_API_RETAIN_EXTRACTION_MODE")
@@ -2135,7 +2135,8 @@ If the text contains both Italian and English content, extract ONLY the Italian
event_date=datetime(2024, 1, 15, tzinfo=timezone.utc),
context="team meeting notes",
llm_config=llm_config,
agent_name="TestUser"
agent_name="TestUser",
config=_get_raw_config(),
)
logger.info(f"\nExtracted {len(facts)} facts with custom mode (Italian only):")
+37
View File
@@ -387,6 +387,43 @@ impl ApiClient {
})
}
pub fn get_bank_config(
&self,
bank_id: &str,
_verbose: bool,
) -> Result<types::BankConfigResponse> {
self.runtime.block_on(async {
let response = self.client.get_bank_config(bank_id, None).await?;
Ok(response.into_inner())
})
}
pub fn update_bank_config(
&self,
bank_id: &str,
updates: std::collections::HashMap<String, serde_json::Value>,
_verbose: bool,
) -> Result<types::BankConfigResponse> {
self.runtime.block_on(async {
// Convert HashMap to serde_json::Map
let updates_map: serde_json::Map<String, serde_json::Value> = updates.into_iter().collect();
let request = types::BankConfigUpdate { updates: updates_map };
let response = self.client.update_bank_config(bank_id, None, &request).await?;
Ok(response.into_inner())
})
}
pub fn reset_bank_config(
&self,
bank_id: &str,
_verbose: bool,
) -> Result<types::BankConfigResponse> {
self.runtime.block_on(async {
let response = self.client.reset_bank_config(bank_id, None).await?;
Ok(response.into_inner())
})
}
// --- Tag Methods ---
pub fn list_tags(
+157 -1
View File
@@ -1,4 +1,4 @@
use anyhow::Result;
use anyhow::{anyhow, Result};
use crate::api::ApiClient;
use crate::output::{self, OutputFormat};
use crate::ui;
@@ -655,3 +655,159 @@ pub fn clear_observations(
Err(e) => Err(e),
}
}
pub fn config(
client: &ApiClient,
bank_id: &str,
overrides_only: bool,
verbose: bool,
output_format: OutputFormat,
) -> Result<()> {
let spinner = if output_format == OutputFormat::Pretty {
Some(ui::create_spinner("Fetching bank configuration..."))
} else {
None
};
let response = client.get_bank_config(bank_id, verbose);
if let Some(mut sp) = spinner {
sp.finish();
}
match response {
Ok(result) => {
if output_format == OutputFormat::Pretty {
ui::print_success(&format!("Configuration for bank '{}'", bank_id));
println!();
if overrides_only {
println!("Bank-specific overrides:");
if result.overrides.is_empty() {
println!(" (none - using defaults)");
} else {
for (key, value) in result.overrides.iter() {
println!(" {}: {:?}", key, value);
}
}
} else {
println!("Resolved configuration (with all overrides applied):");
for (key, value) in result.config.iter() {
println!(" {}: {:?}", key, value);
}
}
} else {
if overrides_only {
output::print_output(&result.overrides, output_format)?;
} else {
output::print_output(&result, output_format)?;
}
}
Ok(())
}
Err(e) => Err(e),
}
}
pub fn set_config(
client: &ApiClient,
bank_id: &str,
llm_provider: Option<String>,
llm_model: Option<String>,
llm_api_key: Option<String>,
llm_base_url: Option<String>,
verbose: bool,
output_format: OutputFormat,
) -> Result<()> {
use std::collections::HashMap;
let mut updates: HashMap<String, serde_json::Value> = HashMap::new();
if let Some(provider) = llm_provider {
updates.insert("llm_provider".to_string(), serde_json::Value::String(provider));
}
if let Some(model) = llm_model {
updates.insert("llm_model".to_string(), serde_json::Value::String(model));
}
if let Some(api_key) = llm_api_key {
updates.insert("llm_api_key".to_string(), serde_json::Value::String(api_key));
}
if let Some(base_url) = llm_base_url {
updates.insert("llm_base_url".to_string(), serde_json::Value::String(base_url));
}
if updates.is_empty() {
return Err(anyhow!("No config updates provided. Use --llm-provider, --llm-model, --llm-api-key, or --llm-base-url".to_string()));
}
let spinner = if output_format == OutputFormat::Pretty {
Some(ui::create_spinner("Updating bank configuration..."))
} else {
None
};
let response = client.update_bank_config(bank_id, updates, verbose);
if let Some(mut sp) = spinner {
sp.finish();
}
match response {
Ok(result) => {
if output_format == OutputFormat::Pretty {
ui::print_success(&format!("Configuration updated for bank '{}'", bank_id));
println!("\nUpdated overrides:");
for (key, value) in result.overrides.iter() {
println!(" {}: {:?}", key, value);
}
} else {
output::print_output(&result, output_format)?;
}
Ok(())
}
Err(e) => Err(e),
}
}
pub fn reset_config(
client: &ApiClient,
bank_id: &str,
yes: bool,
verbose: bool,
output_format: OutputFormat,
) -> Result<()> {
if !yes && output_format == OutputFormat::Pretty {
let confirmed = ui::prompt_confirmation(&format!(
"Reset all configuration overrides for bank '{}'?",
bank_id
))?;
if !confirmed {
ui::print_info("Operation cancelled");
return Ok(());
}
}
let spinner = if output_format == OutputFormat::Pretty {
Some(ui::create_spinner("Resetting bank configuration..."))
} else {
None
};
let response = client.reset_bank_config(bank_id, verbose);
if let Some(mut sp) = spinner {
sp.finish();
}
match response {
Ok(result) => {
if output_format == OutputFormat::Pretty {
ui::print_success(&format!("Configuration reset to defaults for bank '{}'", bank_id));
} else {
output::print_output(&result, output_format)?;
}
Ok(())
}
Err(e) => Err(e),
}
}
+33 -3
View File
@@ -58,8 +58,22 @@ fn format_error_message(err: &anyhow::Error, api_url: &str) -> String {
);
}
// 404 Not Found
// 404 Not Found - check for disabled features first
if err_str.contains("404") {
if err_str.contains("Bank configuration API is disabled") {
return format!(
"{} {}\n\n{}\n {}\n\n{}\n {}\n\n{}\n {}",
"".bright_red().bold(),
"Bank configuration API is disabled".bright_red().bold(),
"API URL:".bright_yellow(),
api_url.bright_white(),
"This feature is disabled by default for security.".bright_yellow(),
"To enable, set HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true on the API server".bright_white(),
"Note:".bright_cyan(),
"This allows per-bank LLM configuration overrides via API".bright_white()
);
}
return format!(
"{} {}\n\n{}\n {}\n\n{}\n • {}\n • {}\n\n{}\n {}",
"".bright_red().bold(),
@@ -74,8 +88,8 @@ fn format_error_message(err: &anyhow::Error, api_url: &str) -> String {
);
}
// 401/403 Authentication
if err_str.contains("401") || err_str.contains("403") {
// 401 Authentication failed
if err_str.contains("401") {
return format!(
"{} {}\n\n{}\n {}\n\n{}\n • {}\n • {}\n\n{}\n {}",
"".bright_red().bold(),
@@ -90,6 +104,22 @@ fn format_error_message(err: &anyhow::Error, api_url: &str) -> String {
);
}
// 403 Forbidden
if err_str.contains("403") {
return format!(
"{} {}\n\n{}\n {}\n\n{}\n • {}\n • {}\n\n{}\n {}",
"".bright_red().bold(),
"Permission denied (403)".bright_red().bold(),
"API URL:".bright_yellow(),
api_url.bright_white(),
"Possible causes:".bright_yellow(),
"This operation is not allowed".bright_white(),
"The feature may be disabled on the server".bright_white(),
"Try:".bright_green(),
"Check server configuration or contact your administrator".bright_white()
);
}
// 500 Server Error
if err_str.contains("500") || err_str.contains("502") || err_str.contains("503") {
return format!(
+51
View File
@@ -279,6 +279,48 @@ enum BankCommands {
#[arg(short = 'y', long)]
yes: bool,
},
/// Get bank configuration (hierarchical overrides)
Config {
/// Bank ID
bank_id: String,
/// Show only bank-specific overrides (not full resolved config)
#[arg(long)]
overrides_only: bool,
},
/// Update bank configuration (set hierarchical overrides)
SetConfig {
/// Bank ID
bank_id: String,
/// LLM provider override
#[arg(long)]
llm_provider: Option<String>,
/// LLM model override
#[arg(long)]
llm_model: Option<String>,
/// LLM API key override
#[arg(long)]
llm_api_key: Option<String>,
/// LLM base URL override
#[arg(long)]
llm_base_url: Option<String>,
},
/// Reset bank configuration to defaults (remove all overrides)
ResetConfig {
/// Bank ID
bank_id: String,
/// Skip confirmation prompt
#[arg(short = 'y', long)]
yes: bool,
},
}
#[derive(Subcommand)]
@@ -776,6 +818,15 @@ fn run() -> Result<()> {
BankCommands::ClearObservations { bank_id, yes } => {
commands::bank::clear_observations(&client, &bank_id, yes, verbose, output_format)
}
BankCommands::Config { bank_id, overrides_only } => {
commands::bank::config(&client, &bank_id, overrides_only, verbose, output_format)
}
BankCommands::SetConfig { bank_id, llm_provider, llm_model, llm_api_key, llm_base_url } => {
commands::bank::set_config(&client, &bank_id, llm_provider, llm_model, llm_api_key, llm_base_url, verbose, output_format)
}
BankCommands::ResetConfig { bank_id, yes } => {
commands::bank::reset_config(&client, &bank_id, yes, verbose, output_format)
}
},
// Memory commands
@@ -16,6 +16,8 @@ hindsight_client_api/models/__init__.py
hindsight_client_api/models/add_background_request.py
hindsight_client_api/models/async_operation_submit_response.py
hindsight_client_api/models/background_response.py
hindsight_client_api/models/bank_config_response.py
hindsight_client_api/models/bank_config_update.py
hindsight_client_api/models/bank_list_item.py
hindsight_client_api/models/bank_list_response.py
hindsight_client_api/models/bank_profile_response.py
@@ -41,6 +41,8 @@ from hindsight_client_api.exceptions import ApiException
from hindsight_client_api.models.add_background_request import AddBackgroundRequest
from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
from hindsight_client_api.models.background_response import BackgroundResponse
from hindsight_client_api.models.bank_config_response import BankConfigResponse
from hindsight_client_api.models.bank_config_update import BankConfigUpdate
from hindsight_client_api.models.bank_list_item import BankListItem
from hindsight_client_api.models.bank_list_response import BankListResponse
from hindsight_client_api.models.bank_profile_response import BankProfileResponse
@@ -20,6 +20,8 @@ from pydantic import StrictStr
from typing import Optional
from hindsight_client_api.models.add_background_request import AddBackgroundRequest
from hindsight_client_api.models.background_response import BackgroundResponse
from hindsight_client_api.models.bank_config_response import BankConfigResponse
from hindsight_client_api.models.bank_config_update import BankConfigUpdate
from hindsight_client_api.models.bank_list_response import BankListResponse
from hindsight_client_api.models.bank_profile_response import BankProfileResponse
from hindsight_client_api.models.bank_stats_response import BankStatsResponse
@@ -1495,6 +1497,284 @@ class BanksApi:
@validate_call
async def get_bank_config(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> BankConfigResponse:
"""Get bank configuration
Get fully resolved configuration for a bank including all hierarchical overrides (global tenant bank). The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._get_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
).data
@validate_call
async def get_bank_config_with_http_info(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> ApiResponse[BankConfigResponse]:
"""Get bank configuration
Get fully resolved configuration for a bank including all hierarchical overrides (global tenant bank). The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._get_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
)
@validate_call
async def get_bank_config_without_preload_content(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> RESTResponseType:
"""Get bank configuration
Get fully resolved configuration for a bank including all hierarchical overrides (global tenant bank). The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._get_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
return response_data.response
def _get_bank_config_serialize(
self,
bank_id,
authorization,
_request_auth,
_content_type,
_headers,
_host_index,
) -> RequestSerialized:
_host = None
_collection_formats: Dict[str, str] = {
}
_path_params: Dict[str, str] = {}
_query_params: List[Tuple[str, str]] = []
_header_params: Dict[str, Optional[str]] = _headers or {}
_form_params: List[Tuple[str, str]] = []
_files: Dict[
str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
] = {}
_body_params: Optional[bytes] = None
# process the path parameters
if bank_id is not None:
_path_params['bank_id'] = bank_id
# process the query parameters
# process the header parameters
if authorization is not None:
_header_params['authorization'] = authorization
# process the form parameters
# process the body parameter
# set the HTTP header `Accept`
if 'Accept' not in _header_params:
_header_params['Accept'] = self.api_client.select_header_accept(
[
'application/json'
]
)
# authentication setting
_auth_settings: List[str] = [
]
return self.api_client.param_serialize(
method='GET',
resource_path='/v1/default/banks/{bank_id}/config',
path_params=_path_params,
query_params=_query_params,
header_params=_header_params,
body=_body_params,
post_params=_form_params,
files=_files,
auth_settings=_auth_settings,
collection_formats=_collection_formats,
_host=_host,
_request_auth=_request_auth
)
@validate_call
async def get_bank_profile(
self,
@@ -2036,6 +2316,284 @@ class BanksApi:
@validate_call
async def reset_bank_config(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> BankConfigResponse:
"""Reset bank configuration
Reset bank configuration to defaults by removing all bank-specific overrides. The bank will then use global and tenant-level configuration only.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._reset_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
).data
@validate_call
async def reset_bank_config_with_http_info(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> ApiResponse[BankConfigResponse]:
"""Reset bank configuration
Reset bank configuration to defaults by removing all bank-specific overrides. The bank will then use global and tenant-level configuration only.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._reset_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
)
@validate_call
async def reset_bank_config_without_preload_content(
self,
bank_id: StrictStr,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> RESTResponseType:
"""Reset bank configuration
Reset bank configuration to defaults by removing all bank-specific overrides. The bank will then use global and tenant-level configuration only.
:param bank_id: (required)
:type bank_id: str
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._reset_bank_config_serialize(
bank_id=bank_id,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
return response_data.response
def _reset_bank_config_serialize(
self,
bank_id,
authorization,
_request_auth,
_content_type,
_headers,
_host_index,
) -> RequestSerialized:
_host = None
_collection_formats: Dict[str, str] = {
}
_path_params: Dict[str, str] = {}
_query_params: List[Tuple[str, str]] = []
_header_params: Dict[str, Optional[str]] = _headers or {}
_form_params: List[Tuple[str, str]] = []
_files: Dict[
str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
] = {}
_body_params: Optional[bytes] = None
# process the path parameters
if bank_id is not None:
_path_params['bank_id'] = bank_id
# process the query parameters
# process the header parameters
if authorization is not None:
_header_params['authorization'] = authorization
# process the form parameters
# process the body parameter
# set the HTTP header `Accept`
if 'Accept' not in _header_params:
_header_params['Accept'] = self.api_client.select_header_accept(
[
'application/json'
]
)
# authentication setting
_auth_settings: List[str] = [
]
return self.api_client.param_serialize(
method='DELETE',
resource_path='/v1/default/banks/{bank_id}/config',
path_params=_path_params,
query_params=_query_params,
header_params=_header_params,
body=_body_params,
post_params=_form_params,
files=_files,
auth_settings=_auth_settings,
collection_formats=_collection_formats,
_host=_host,
_request_auth=_request_auth
)
@validate_call
async def trigger_consolidation(
self,
@@ -2620,6 +3178,312 @@ class BanksApi:
@validate_call
async def update_bank_config(
self,
bank_id: StrictStr,
bank_config_update: BankConfigUpdate,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> BankConfigResponse:
"""Update bank configuration
Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).
:param bank_id: (required)
:type bank_id: str
:param bank_config_update: (required)
:type bank_config_update: BankConfigUpdate
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._update_bank_config_serialize(
bank_id=bank_id,
bank_config_update=bank_config_update,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
).data
@validate_call
async def update_bank_config_with_http_info(
self,
bank_id: StrictStr,
bank_config_update: BankConfigUpdate,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> ApiResponse[BankConfigResponse]:
"""Update bank configuration
Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).
:param bank_id: (required)
:type bank_id: str
:param bank_config_update: (required)
:type bank_config_update: BankConfigUpdate
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._update_bank_config_serialize(
bank_id=bank_id,
bank_config_update=bank_config_update,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
await response_data.read()
return self.api_client.response_deserialize(
response_data=response_data,
response_types_map=_response_types_map,
)
@validate_call
async def update_bank_config_without_preload_content(
self,
bank_id: StrictStr,
bank_config_update: BankConfigUpdate,
authorization: Optional[StrictStr] = None,
_request_timeout: Union[
None,
Annotated[StrictFloat, Field(gt=0)],
Tuple[
Annotated[StrictFloat, Field(gt=0)],
Annotated[StrictFloat, Field(gt=0)]
]
] = None,
_request_auth: Optional[Dict[StrictStr, Any]] = None,
_content_type: Optional[StrictStr] = None,
_headers: Optional[Dict[StrictStr, Any]] = None,
_host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
) -> RESTResponseType:
"""Update bank configuration
Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).
:param bank_id: (required)
:type bank_id: str
:param bank_config_update: (required)
:type bank_config_update: BankConfigUpdate
:param authorization:
:type authorization: str
:param _request_timeout: timeout setting for this request. If one
number provided, it will be total request
timeout. It can also be a pair (tuple) of
(connection, read) timeouts.
:type _request_timeout: int, tuple(int, int), optional
:param _request_auth: set to override the auth_settings for an a single
request; this effectively ignores the
authentication in the spec for a single request.
:type _request_auth: dict, optional
:param _content_type: force content-type for the request.
:type _content_type: str, Optional
:param _headers: set to override the headers for a single
request; this effectively ignores the headers
in the spec for a single request.
:type _headers: dict, optional
:param _host_index: set to override the host_index for a single
request; this effectively ignores the host_index
in the spec for a single request.
:type _host_index: int, optional
:return: Returns the result object.
""" # noqa: E501
_param = self._update_bank_config_serialize(
bank_id=bank_id,
bank_config_update=bank_config_update,
authorization=authorization,
_request_auth=_request_auth,
_content_type=_content_type,
_headers=_headers,
_host_index=_host_index
)
_response_types_map: Dict[str, Optional[str]] = {
'200': "BankConfigResponse",
'422': "HTTPValidationError",
}
response_data = await self.api_client.call_api(
*_param,
_request_timeout=_request_timeout
)
return response_data.response
def _update_bank_config_serialize(
self,
bank_id,
bank_config_update,
authorization,
_request_auth,
_content_type,
_headers,
_host_index,
) -> RequestSerialized:
_host = None
_collection_formats: Dict[str, str] = {
}
_path_params: Dict[str, str] = {}
_query_params: List[Tuple[str, str]] = []
_header_params: Dict[str, Optional[str]] = _headers or {}
_form_params: List[Tuple[str, str]] = []
_files: Dict[
str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
] = {}
_body_params: Optional[bytes] = None
# process the path parameters
if bank_id is not None:
_path_params['bank_id'] = bank_id
# process the query parameters
# process the header parameters
if authorization is not None:
_header_params['authorization'] = authorization
# process the form parameters
# process the body parameter
if bank_config_update is not None:
_body_params = bank_config_update
# set the HTTP header `Accept`
if 'Accept' not in _header_params:
_header_params['Accept'] = self.api_client.select_header_accept(
[
'application/json'
]
)
# set the HTTP header `Content-Type`
if _content_type:
_header_params['Content-Type'] = _content_type
else:
_default_content_type = (
self.api_client.select_header_content_type(
[
'application/json'
]
)
)
if _default_content_type is not None:
_header_params['Content-Type'] = _default_content_type
# authentication setting
_auth_settings: List[str] = [
]
return self.api_client.param_serialize(
method='PATCH',
resource_path='/v1/default/banks/{bank_id}/config',
path_params=_path_params,
query_params=_query_params,
header_params=_header_params,
body=_body_params,
post_params=_form_params,
files=_files,
auth_settings=_auth_settings,
collection_formats=_collection_formats,
_host=_host,
_request_auth=_request_auth
)
@validate_call
async def update_bank_disposition(
self,
@@ -17,6 +17,8 @@
from hindsight_client_api.models.add_background_request import AddBackgroundRequest
from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
from hindsight_client_api.models.background_response import BackgroundResponse
from hindsight_client_api.models.bank_config_response import BankConfigResponse
from hindsight_client_api.models.bank_config_update import BankConfigUpdate
from hindsight_client_api.models.bank_list_item import BankListItem
from hindsight_client_api.models.bank_list_response import BankListResponse
from hindsight_client_api.models.bank_profile_response import BankProfileResponse
@@ -0,0 +1,91 @@
# coding: utf-8
"""
Hindsight HTTP API
HTTP API for Hindsight
The version of the OpenAPI document: 0.4.10
Generated by OpenAPI Generator (https://openapi-generator.tech)
Do not edit the class manually.
""" # noqa: E501
from __future__ import annotations
import pprint
import re # noqa: F401
import json
from pydantic import BaseModel, ConfigDict, Field, StrictStr
from typing import Any, ClassVar, Dict, List
from typing import Optional, Set
from typing_extensions import Self
class BankConfigResponse(BaseModel):
"""
Response model for bank configuration.
""" # noqa: E501
bank_id: StrictStr = Field(description="Bank identifier")
config: Dict[str, Any] = Field(description="Fully resolved configuration with all hierarchical overrides applied (Python field names)")
overrides: Dict[str, Any] = Field(description="Bank-specific configuration overrides only (Python field names)")
__properties: ClassVar[List[str]] = ["bank_id", "config", "overrides"]
model_config = ConfigDict(
populate_by_name=True,
validate_assignment=True,
protected_namespaces=(),
)
def to_str(self) -> str:
"""Returns the string representation of the model using alias"""
return pprint.pformat(self.model_dump(by_alias=True))
def to_json(self) -> str:
"""Returns the JSON representation of the model using alias"""
# TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
return json.dumps(self.to_dict())
@classmethod
def from_json(cls, json_str: str) -> Optional[Self]:
"""Create an instance of BankConfigResponse from a JSON string"""
return cls.from_dict(json.loads(json_str))
def to_dict(self) -> Dict[str, Any]:
"""Return the dictionary representation of the model using alias.
This has the following differences from calling pydantic's
`self.model_dump(by_alias=True)`:
* `None` is only added to the output dict for nullable fields that
were set at model initialization. Other fields with value `None`
are ignored.
"""
excluded_fields: Set[str] = set([
])
_dict = self.model_dump(
by_alias=True,
exclude=excluded_fields,
exclude_none=True,
)
return _dict
@classmethod
def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
"""Create an instance of BankConfigResponse from a dict"""
if obj is None:
return None
if not isinstance(obj, dict):
return cls.model_validate(obj)
_obj = cls.model_validate({
"bank_id": obj.get("bank_id"),
"config": obj.get("config"),
"overrides": obj.get("overrides")
})
return _obj
@@ -0,0 +1,87 @@
# coding: utf-8
"""
Hindsight HTTP API
HTTP API for Hindsight
The version of the OpenAPI document: 0.4.10
Generated by OpenAPI Generator (https://openapi-generator.tech)
Do not edit the class manually.
""" # noqa: E501
from __future__ import annotations
import pprint
import re # noqa: F401
import json
from pydantic import BaseModel, ConfigDict, Field
from typing import Any, ClassVar, Dict, List
from typing import Optional, Set
from typing_extensions import Self
class BankConfigUpdate(BaseModel):
"""
Request model for updating bank configuration.
""" # noqa: E501
updates: Dict[str, Any] = Field(description="Configuration overrides. Keys can be in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER). Only hierarchical fields can be overridden per-bank.")
__properties: ClassVar[List[str]] = ["updates"]
model_config = ConfigDict(
populate_by_name=True,
validate_assignment=True,
protected_namespaces=(),
)
def to_str(self) -> str:
"""Returns the string representation of the model using alias"""
return pprint.pformat(self.model_dump(by_alias=True))
def to_json(self) -> str:
"""Returns the JSON representation of the model using alias"""
# TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
return json.dumps(self.to_dict())
@classmethod
def from_json(cls, json_str: str) -> Optional[Self]:
"""Create an instance of BankConfigUpdate from a JSON string"""
return cls.from_dict(json.loads(json_str))
def to_dict(self) -> Dict[str, Any]:
"""Return the dictionary representation of the model using alias.
This has the following differences from calling pydantic's
`self.model_dump(by_alias=True)`:
* `None` is only added to the output dict for nullable fields that
were set at model initialization. Other fields with value `None`
are ignored.
"""
excluded_fields: Set[str] = set([
])
_dict = self.model_dump(
by_alias=True,
exclude=excluded_fields,
exclude_none=True,
)
return _dict
@classmethod
def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
"""Create an instance of BankConfigUpdate from a dict"""
if obj is None:
return None
if not isinstance(obj, dict):
return cls.model_validate(obj)
_obj = cls.model_validate({
"updates": obj.get("updates")
})
return _obj
@@ -29,7 +29,8 @@ class FeaturesInfo(BaseModel):
observations: StrictBool = Field(description="Whether observations (auto-consolidation) are enabled")
mcp: StrictBool = Field(description="Whether MCP (Model Context Protocol) server is enabled")
worker: StrictBool = Field(description="Whether the background worker is enabled")
__properties: ClassVar[List[str]] = ["observations", "mcp", "worker"]
bank_config_api: StrictBool = Field(description="Whether per-bank configuration API is enabled")
__properties: ClassVar[List[str]] = ["observations", "mcp", "worker", "bank_config_api"]
model_config = ConfigDict(
populate_by_name=True,
@@ -84,7 +85,8 @@ class FeaturesInfo(BaseModel):
_obj = cls.model_validate({
"observations": obj.get("observations"),
"mcp": obj.get("mcp"),
"worker": obj.get("worker")
"worker": obj.get("worker"),
"bank_config_api": obj.get("bank_config_api")
})
return _obj
@@ -0,0 +1,125 @@
"""
Test ReflectResponse parsing for different API versions.
This tests the client's ability to parse reflect responses from:
- v0.3.0 API (based_on as list)
- v0.4.0+ API (based_on as object)
"""
import pytest
from hindsight_client_api.models.reflect_response import ReflectResponse
from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
def test_parse_v4_format_with_empty_based_on():
"""Test parsing v0.4.0+ format with empty based_on object."""
response_data = {
"text": "I don't have any information about that.",
"based_on": {
"memories": [],
"mental_models": [],
"directives": []
}
}
response = ReflectResponse.from_dict(response_data)
assert response is not None
assert response.text == "I don't have any information about that."
assert response.based_on is not None
assert isinstance(response.based_on, ReflectBasedOn)
assert response.based_on.memories == []
assert response.based_on.mental_models == []
assert response.based_on.directives == []
def test_parse_v4_format_with_null_based_on():
"""Test parsing v0.4.0+ format with null based_on (include.facts not set)."""
response_data = {
"text": "Hello!",
"based_on": None
}
response = ReflectResponse.from_dict(response_data)
assert response is not None
assert response.text == "Hello!"
assert response.based_on is None
def test_parse_v4_format_with_populated_based_on():
"""Test parsing v0.4.0+ format with actual facts."""
response_data = {
"text": "Based on my knowledge, AI is transformative.",
"based_on": {
"memories": [
{
"id": "mem-123",
"text": "AI is used in healthcare",
"type": "world",
"context": None,
"occurred_start": None,
"occurred_end": None
}
],
"mental_models": [
{
"id": "mm-456",
"text": "AI transforms industries",
"context": "technology trends"
}
],
"directives": [
{
"id": "dir-789",
"name": "Be concise",
"content": "Keep responses brief"
}
]
}
}
response = ReflectResponse.from_dict(response_data)
assert response is not None
assert response.text == "Based on my knowledge, AI is transformative."
assert response.based_on is not None
assert len(response.based_on.memories) == 1
assert response.based_on.memories[0].id == "mem-123"
assert len(response.based_on.mental_models) == 1
assert response.based_on.mental_models[0].id == "mm-456"
assert len(response.based_on.directives) == 1
assert response.based_on.directives[0].id == "dir-789"
def test_parse_v3_format_with_empty_list_fails():
"""
Test that v0.3.0 format (based_on as list) fails validation.
This is a BREAKING CHANGE from v0.3.0 to v0.4.0.
Clients using v0.4.x SDK cannot parse v0.3.0 API responses.
Users must either:
- Upgrade API to v0.4.0+
- Use v0.3.0 client with v0.3.0 API
"""
response_data = {
"text": "No information available.",
"based_on": [] # v0.3.0 format - incompatible with v0.4.0+ client
}
with pytest.raises(Exception) as exc_info:
ReflectResponse.from_dict(response_data)
# Should fail with validation error
assert "ValidationError" in str(type(exc_info.value).__name__) or "validation" in str(exc_info.value).lower()
def test_parse_missing_based_on_field():
"""Test parsing response when based_on field is omitted entirely."""
response_data = {
"text": "Hello!"
# based_on field not present
}
response = ReflectResponse.from_dict(response_data)
assert response is not None
assert response.text == "Hello!"
assert response.based_on is None
@@ -39,6 +39,9 @@ import type {
GetAgentStatsData,
GetAgentStatsErrors,
GetAgentStatsResponses,
GetBankConfigData,
GetBankConfigErrors,
GetBankConfigResponses,
GetBankProfileData,
GetBankProfileErrors,
GetBankProfileResponses,
@@ -108,12 +111,18 @@ import type {
RegenerateEntityObservationsData,
RegenerateEntityObservationsErrors,
RegenerateEntityObservationsResponses,
ResetBankConfigData,
ResetBankConfigErrors,
ResetBankConfigResponses,
RetainMemoriesData,
RetainMemoriesErrors,
RetainMemoriesResponses,
TriggerConsolidationData,
TriggerConsolidationErrors,
TriggerConsolidationResponses,
UpdateBankConfigData,
UpdateBankConfigErrors,
UpdateBankConfigResponses,
UpdateBankData,
UpdateBankDispositionData,
UpdateBankDispositionErrors,
@@ -808,6 +817,55 @@ export const clearObservations = <ThrowOnError extends boolean = false>(
ThrowOnError
>({ url: "/v1/default/banks/{bank_id}/observations", ...options });
/**
* Reset bank configuration
*
* Reset bank configuration to defaults by removing all bank-specific overrides. The bank will then use global and tenant-level configuration only.
*/
export const resetBankConfig = <ThrowOnError extends boolean = false>(
options: Options<ResetBankConfigData, ThrowOnError>,
) =>
(options.client ?? client).delete<
ResetBankConfigResponses,
ResetBankConfigErrors,
ThrowOnError
>({ url: "/v1/default/banks/{bank_id}/config", ...options });
/**
* Get bank configuration
*
* Get fully resolved configuration for a bank including all hierarchical overrides (global tenant bank). The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.
*/
export const getBankConfig = <ThrowOnError extends boolean = false>(
options: Options<GetBankConfigData, ThrowOnError>,
) =>
(options.client ?? client).get<
GetBankConfigResponses,
GetBankConfigErrors,
ThrowOnError
>({ url: "/v1/default/banks/{bank_id}/config", ...options });
/**
* Update bank configuration
*
* Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).
*/
export const updateBankConfig = <ThrowOnError extends boolean = false>(
options: Options<UpdateBankConfigData, ThrowOnError>,
) =>
(options.client ?? client).patch<
UpdateBankConfigResponses,
UpdateBankConfigErrors,
ThrowOnError
>({
url: "/v1/default/banks/{bank_id}/config",
...options,
headers: {
"Content-Type": "application/json",
...options.headers,
},
});
/**
* Trigger consolidation
*
@@ -59,6 +59,52 @@ export type BackgroundResponse = {
disposition?: DispositionTraits | null;
};
/**
* BankConfigResponse
*
* Response model for bank configuration.
*/
export type BankConfigResponse = {
/**
* Bank Id
*
* Bank identifier
*/
bank_id: string;
/**
* Config
*
* Fully resolved configuration with all hierarchical overrides applied (Python field names)
*/
config: {
[key: string]: unknown;
};
/**
* Overrides
*
* Bank-specific configuration overrides only (Python field names)
*/
overrides: {
[key: string]: unknown;
};
};
/**
* BankConfigUpdate
*
* Request model for updating bank configuration.
*/
export type BankConfigUpdate = {
/**
* Updates
*
* Configuration overrides. Keys can be in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER). Only hierarchical fields can be overridden per-bank.
*/
updates: {
[key: string]: unknown;
};
};
/**
* BankListItem
*
@@ -816,6 +862,12 @@ export type FeaturesInfo = {
* Whether the background worker is enabled
*/
worker: boolean;
/**
* Bank Config Api
*
* Whether per-bank configuration API is enabled
*/
bank_config_api: boolean;
};
/**
@@ -3426,6 +3478,119 @@ export type ClearObservationsResponses = {
export type ClearObservationsResponse =
ClearObservationsResponses[keyof ClearObservationsResponses];
export type ResetBankConfigData = {
body?: never;
headers?: {
/**
* Authorization
*/
authorization?: string | null;
};
path: {
/**
* Bank Id
*/
bank_id: string;
};
query?: never;
url: "/v1/default/banks/{bank_id}/config";
};
export type ResetBankConfigErrors = {
/**
* Validation Error
*/
422: HttpValidationError;
};
export type ResetBankConfigError =
ResetBankConfigErrors[keyof ResetBankConfigErrors];
export type ResetBankConfigResponses = {
/**
* Successful Response
*/
200: BankConfigResponse;
};
export type ResetBankConfigResponse =
ResetBankConfigResponses[keyof ResetBankConfigResponses];
export type GetBankConfigData = {
body?: never;
headers?: {
/**
* Authorization
*/
authorization?: string | null;
};
path: {
/**
* Bank Id
*/
bank_id: string;
};
query?: never;
url: "/v1/default/banks/{bank_id}/config";
};
export type GetBankConfigErrors = {
/**
* Validation Error
*/
422: HttpValidationError;
};
export type GetBankConfigError = GetBankConfigErrors[keyof GetBankConfigErrors];
export type GetBankConfigResponses = {
/**
* Successful Response
*/
200: BankConfigResponse;
};
export type GetBankConfigResponse =
GetBankConfigResponses[keyof GetBankConfigResponses];
export type UpdateBankConfigData = {
body: BankConfigUpdate;
headers?: {
/**
* Authorization
*/
authorization?: string | null;
};
path: {
/**
* Bank Id
*/
bank_id: string;
};
query?: never;
url: "/v1/default/banks/{bank_id}/config";
};
export type UpdateBankConfigErrors = {
/**
* Validation Error
*/
422: HttpValidationError;
};
export type UpdateBankConfigError =
UpdateBankConfigErrors[keyof UpdateBankConfigErrors];
export type UpdateBankConfigResponses = {
/**
* Successful Response
*/
200: BankConfigResponse;
};
export type UpdateBankConfigResponse =
UpdateBankConfigResponses[keyof UpdateBankConfigResponses];
export type TriggerConsolidationData = {
body?: never;
headers?: {
@@ -0,0 +1,77 @@
import { NextRequest, NextResponse } from "next/server";
import { lowLevelClient, sdk } from "@/lib/hindsight-client";
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ bankId: string }> }
) {
try {
const { bankId } = await params;
const response = await sdk.getBankConfig({
client: lowLevelClient,
path: { bank_id: bankId },
});
if (!response.data) {
console.error("[Bank Config API] No data in response", { response, error: response.error });
throw new Error(`API returned no data: ${JSON.stringify(response.error || "Unknown error")}`);
}
return NextResponse.json(response.data, { status: 200 });
} catch (error) {
console.error("Error fetching bank config:", error);
return NextResponse.json({ error: "Failed to fetch bank config" }, { status: 500 });
}
}
export async function PATCH(
request: NextRequest,
{ params }: { params: Promise<{ bankId: string }> }
) {
try {
const { bankId } = await params;
const body = await request.json();
const { updates } = body;
const response = await sdk.updateBankConfig({
client: lowLevelClient,
path: { bank_id: bankId },
body: { updates },
});
if (!response.data) {
console.error("[Bank Config API] No data in response", { response, error: response.error });
throw new Error(`API returned no data: ${JSON.stringify(response.error || "Unknown error")}`);
}
return NextResponse.json(response.data, { status: 200 });
} catch (error) {
console.error("Error updating bank config:", error);
return NextResponse.json({ error: "Failed to update bank config" }, { status: 500 });
}
}
export async function DELETE(
request: NextRequest,
{ params }: { params: Promise<{ bankId: string }> }
) {
try {
const { bankId } = await params;
const response = await sdk.resetBankConfig({
client: lowLevelClient,
path: { bank_id: bankId },
});
if (!response.data) {
console.error("[Bank Config API] No data in response", { response, error: response.error });
throw new Error(`API returned no data: ${JSON.stringify(response.error || "Unknown error")}`);
}
return NextResponse.json(response.data, { status: 200 });
} catch (error) {
console.error("Error resetting bank config:", error);
return NextResponse.json({ error: "Failed to reset bank config" }, { status: 500 });
}
}
@@ -1,5 +1,6 @@
"use client";
import { useState } from "react";
import { useParams, useRouter, useSearchParams } from "next/navigation";
import { BankSelector } from "@/components/bank-selector";
import { Sidebar } from "@/components/sidebar";
@@ -9,22 +10,56 @@ import { EntitiesView } from "@/components/entities-view";
import { ThinkView } from "@/components/think-view";
import { SearchDebugView } from "@/components/search-debug-view";
import { BankProfileView } from "@/components/bank-profile-view";
import { BankConfigView } from "@/components/bank-config-view";
import { BankStatsView } from "@/components/bank-stats-view";
import { BankOperationsView } from "@/components/bank-operations-view";
import { MentalModelsView } from "@/components/mental-models-view";
import { useFeatures } from "@/lib/features-context";
import { useBank } from "@/lib/bank-context";
import { client } from "@/lib/api";
import { Button } from "@/components/ui/button";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog";
import { Brain, Trash2, Loader2, MoreVertical, Pencil } from "lucide-react";
type NavItem = "recall" | "reflect" | "data" | "documents" | "entities" | "profile";
type DataSubTab = "world" | "experience" | "observations" | "mental-models";
type BankConfigTab = "general" | "configuration";
export default function BankPage() {
const params = useParams();
const router = useRouter();
const searchParams = useSearchParams();
const { features } = useFeatures();
const { currentBank: bankId, setCurrentBank, loadBanks } = useBank();
const bankId = params.bankId as string;
const view = (searchParams.get("view") || "profile") as NavItem;
const subTab = (searchParams.get("subTab") || "world") as DataSubTab;
const bankConfigTab = (searchParams.get("bankConfigTab") || "general") as BankConfigTab;
const observationsEnabled = features?.observations ?? false;
const bankConfigEnabled = features?.bank_config_api ?? false;
// Bank actions state
const [showDeleteDialog, setShowDeleteDialog] = useState(false);
const [isDeleting, setIsDeleting] = useState(false);
const [showClearObservationsDialog, setShowClearObservationsDialog] = useState(false);
const [isClearingObservations, setIsClearingObservations] = useState(false);
const [isConsolidating, setIsConsolidating] = useState(false);
const handleTabChange = (tab: NavItem) => {
router.push(`/banks/${bankId}?view=${tab}`);
@@ -34,6 +69,58 @@ export default function BankPage() {
router.push(`/banks/${bankId}?view=data&subTab=${newSubTab}`);
};
const handleBankConfigTabChange = (newTab: BankConfigTab) => {
router.push(`/banks/${bankId}?view=profile&bankConfigTab=${newTab}`);
};
const handleDeleteBank = async () => {
if (!bankId) return;
setIsDeleting(true);
try {
await client.deleteBank(bankId);
setShowDeleteDialog(false);
setCurrentBank(null);
await loadBanks();
router.push("/");
} catch (error) {
console.error("Error deleting bank:", error);
alert("Error deleting bank: " + (error as Error).message);
} finally {
setIsDeleting(false);
}
};
const handleClearObservations = async () => {
if (!bankId) return;
setIsClearingObservations(true);
try {
const result = await client.clearObservations(bankId);
setShowClearObservationsDialog(false);
alert(result.message || "Observations cleared successfully");
} catch (error) {
console.error("Error clearing observations:", error);
alert("Error clearing observations: " + (error as Error).message);
} finally {
setIsClearingObservations(false);
}
};
const handleTriggerConsolidation = async () => {
if (!bankId) return;
setIsConsolidating(true);
try {
await client.triggerConsolidation(bankId);
} catch (error) {
console.error("Error triggering consolidation:", error);
alert("Error triggering consolidation: " + (error as Error).message);
} finally {
setIsConsolidating(false);
}
};
return (
<div className="min-h-screen bg-background flex flex-col">
<BankSelector />
@@ -43,15 +130,125 @@ export default function BankPage() {
<main className="flex-1 overflow-y-auto">
<div className="p-6">
{/* Profile Tab */}
{/* Bank Configuration Tab */}
{view === "profile" && (
<div>
<h1 className="text-3xl font-bold mb-2 text-foreground">Bank Profile</h1>
<p className="text-muted-foreground mb-6">
View and edit the memory bank profile, disposition traits, and background
information.
</p>
<BankProfileView />
<div className="flex justify-between items-start mb-6">
<div>
<h1 className="text-3xl font-bold mb-2 text-foreground">Bank Configuration</h1>
<p className="text-muted-foreground">
Manage bank settings, profile, and operations.
</p>
</div>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm">
Actions
<MoreVertical className="w-4 h-4 ml-2" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
<DropdownMenuItem
onClick={handleTriggerConsolidation}
disabled={isConsolidating || !observationsEnabled}
title={
!observationsEnabled ? "Observations feature is not enabled" : undefined
}
>
{isConsolidating ? (
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
) : (
<Brain className="w-4 h-4 mr-2" />
)}
{isConsolidating ? "Consolidating..." : "Run Consolidation"}
{!observationsEnabled && (
<span className="ml-auto text-xs text-muted-foreground">Off</span>
)}
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => setShowClearObservationsDialog(true)}
disabled={!observationsEnabled}
className="text-amber-600 dark:text-amber-400 focus:text-amber-700 dark:focus:text-amber-300"
title={
!observationsEnabled ? "Observations feature is not enabled" : undefined
}
>
<Trash2 className="w-4 h-4 mr-2" />
Clear Observations
{!observationsEnabled && (
<span className="ml-auto text-xs text-muted-foreground">Off</span>
)}
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem
onClick={() => setShowDeleteDialog(true)}
className="text-red-600 dark:text-red-400 focus:text-red-700 dark:focus:text-red-300"
>
<Trash2 className="w-4 h-4 mr-2" />
Delete Bank
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
{/* Sub-tabs */}
<div className="mb-6 border-b border-border">
<div className="flex gap-1">
<button
onClick={() => handleBankConfigTabChange("general")}
className={`px-6 py-3 font-semibold text-sm transition-all relative ${
bankConfigTab === "general"
? "text-primary"
: "text-muted-foreground hover:text-foreground"
}`}
>
General
{bankConfigTab === "general" && (
<div className="absolute bottom-0 left-0 right-0 h-0.5 bg-primary" />
)}
</button>
<button
onClick={() => handleBankConfigTabChange("configuration")}
className={`px-6 py-3 font-semibold text-sm transition-all relative ${
bankConfigTab === "configuration"
? "text-primary"
: "text-muted-foreground hover:text-foreground"
}`}
>
Configuration
{bankConfigTab === "configuration" && (
<div className="absolute bottom-0 left-0 right-0 h-0.5 bg-primary" />
)}
</button>
</div>
</div>
{/* Tab content */}
<div>
{bankConfigTab === "general" && (
<div>
<p className="text-sm text-muted-foreground mb-4">
Overview statistics and background operations for this memory bank.
</p>
<div className="space-y-6">
<BankStatsView />
<BankOperationsView />
</div>
</div>
)}
{bankConfigTab === "configuration" && (
<div>
<p className="text-sm text-muted-foreground mb-6">
Configure disposition traits, mission, directives, and behavioral settings
for this bank.
</p>
<div className="space-y-6">
<BankProfileView />
{bankConfigEnabled && <BankConfigView />}
</div>
</div>
)}
</div>
</div>
)}
@@ -242,6 +439,88 @@ export default function BankPage() {
</div>
</main>
</div>
{/* Delete Bank Confirmation Dialog */}
<AlertDialog open={showDeleteDialog} onOpenChange={setShowDeleteDialog}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Delete Memory Bank</AlertDialogTitle>
<AlertDialogDescription asChild>
<div className="space-y-2 text-sm text-muted-foreground">
<p>
Are you sure you want to delete the memory bank{" "}
<span className="font-semibold text-foreground">{bankId}</span>?
</p>
<p className="text-red-600 dark:text-red-400 font-medium">
This action cannot be undone. All memories, entities, documents, and the bank
profile will be permanently deleted.
</p>
</div>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel disabled={isDeleting}>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleDeleteBank}
disabled={isDeleting}
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
>
{isDeleting ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Deleting...
</>
) : (
<>
<Trash2 className="w-4 h-4 mr-2" />
Delete Bank
</>
)}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
{/* Clear Observations Confirmation Dialog */}
<AlertDialog open={showClearObservationsDialog} onOpenChange={setShowClearObservationsDialog}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Clear Observations</AlertDialogTitle>
<AlertDialogDescription asChild>
<div className="space-y-2 text-sm text-muted-foreground">
<p>
Are you sure you want to clear all observations for{" "}
<span className="font-semibold text-foreground">{bankId}</span>?
</p>
<p className="text-amber-600 dark:text-amber-400 font-medium">
This will delete all consolidated knowledge. Observations will be regenerated the
next time consolidation runs.
</p>
</div>
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel disabled={isClearingObservations}>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={handleClearObservations}
disabled={isClearingObservations}
className="bg-amber-500 text-white hover:bg-amber-600"
>
{isClearingObservations ? (
<>
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
Clearing...
</>
) : (
<>
<Trash2 className="w-4 h-4 mr-2" />
Clear Observations
</>
)}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
);
}
@@ -0,0 +1,480 @@
"use client";
import { useState, useEffect } from "react";
import { useBank } from "@/lib/bank-context";
import { client } from "@/lib/api";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Textarea } from "@/components/ui/textarea";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog";
import { Alert, AlertDescription } from "@/components/ui/alert";
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuTrigger,
} from "@/components/ui/dropdown-menu";
import { Loader2, AlertCircle, CheckCircle2, Pencil, RotateCcw, MoreVertical } from "lucide-react";
import { Card, CardContent, CardHeader, CardTitle, CardDescription } from "@/components/ui/card";
// Field metadata for UI rendering
const FIELD_CATEGORIES = {
retention: {
title: "Retention Settings",
description: "Control how memories are extracted and stored",
fields: {
retain_chunk_size: {
label: "Chunk Size",
type: "number",
description: "Size of text chunks for processing (tokens)",
min: 500,
max: 8000,
},
retain_extraction_mode: {
label: "Extraction Mode",
type: "select",
description: "How to extract facts from content",
options: ["concise", "verbose", "custom"],
},
retain_custom_instructions: {
label: "Custom Instructions",
type: "textarea",
description:
"Custom instructions for fact extraction (requires retain_extraction_mode='custom')",
placeholder: "Focus on technical details and implementation specifics...",
rows: 3,
},
},
},
consolidation: {
title: "Consolidation Settings",
description: "Control observation synthesis",
fields: {
enable_observations: {
label: "Enable Observations",
type: "boolean",
description: "Enable automatic consolidation of facts into observations",
},
},
},
};
export function BankConfigView() {
const { currentBank: bankId } = useBank();
const [loading, setLoading] = useState(true);
const [config, setConfig] = useState<Record<string, any>>({});
const [overrides, setOverrides] = useState<Record<string, any>>({});
const [showEditDialog, setShowEditDialog] = useState(false);
const [showResetDialog, setShowResetDialog] = useState(false);
const [resetting, setResetting] = useState(false);
useEffect(() => {
if (bankId) {
loadConfig();
}
}, [bankId]);
const loadConfig = async () => {
if (!bankId) return;
setLoading(true);
try {
const response = await client.getBankConfig(bankId);
setConfig(response.config);
setOverrides(response.overrides);
} catch (err: any) {
console.error("Failed to load config:", err);
} finally {
setLoading(false);
}
};
const handleReset = () => {
setShowResetDialog(true);
};
const confirmReset = async () => {
if (!bankId) return;
setResetting(true);
try {
await client.resetBankConfig(bankId);
await loadConfig();
setShowResetDialog(false);
} catch (err: any) {
console.error("Failed to reset config:", err);
alert("Error resetting config: " + err.message);
} finally {
setResetting(false);
}
};
const renderReadOnlyField = (fieldKey: string, fieldMeta: any) => {
const value = config[fieldKey];
return (
<div
key={fieldKey}
className="flex items-start justify-between gap-4 p-3 border border-border rounded-lg bg-muted/30 hover:bg-muted/50 transition-colors"
>
<div className="flex-1 min-w-0">
<div className="text-sm font-medium font-mono">{fieldKey}</div>
{fieldMeta.description && (
<p className="text-xs text-muted-foreground mt-0.5">{fieldMeta.description}</p>
)}
</div>
<div className="text-sm text-foreground font-mono flex-shrink-0">
{fieldMeta.type === "boolean" ? (
<span className={value ? "text-green-600" : "text-muted-foreground"}>
{value ? "Enabled" : "Disabled"}
</span>
) : fieldMeta.type === "textarea" ? (
<span className="text-muted-foreground italic">
{value ? `${value.substring(0, 50)}${value.length > 50 ? "..." : ""}` : "Not set"}
</span>
) : (
value || <span className="text-muted-foreground italic">Not set</span>
)}
</div>
</div>
);
};
if (!bankId) {
return (
<div className="flex items-center justify-center py-12">
<p className="text-muted-foreground">No bank selected</p>
</div>
);
}
if (loading) {
return (
<div className="flex items-center justify-center py-12">
<Loader2 className="h-8 w-8 animate-spin text-muted-foreground" />
</div>
);
}
return (
<>
<Card>
<CardHeader className="pb-3">
<div className="flex items-center justify-between">
<div>
<CardTitle className="text-base">Configuration Settings</CardTitle>
<CardDescription className="text-xs">
Behavioral parameters for this memory bank
</CardDescription>
</div>
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="ghost" size="sm" disabled={resetting}>
{resetting ? (
<Loader2 className="h-4 w-4 animate-spin" />
) : (
<MoreVertical className="h-4 w-4" />
)}
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end">
<DropdownMenuItem onClick={() => setShowEditDialog(true)}>
<Pencil className="h-4 w-4 mr-2" />
Edit
</DropdownMenuItem>
<DropdownMenuItem onClick={handleReset}>
<RotateCcw className="h-4 w-4 mr-2" />
Reset to Defaults
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
</CardHeader>
<CardContent className="space-y-6">
{Object.entries(FIELD_CATEGORIES).map(([catKey, category]) => (
<div key={catKey}>
<div className="mb-3">
<h3 className="text-sm font-semibold">{category.title}</h3>
<p className="text-xs text-muted-foreground">{category.description}</p>
</div>
<div className="grid grid-cols-2 gap-x-8 gap-y-4">
{Object.entries(category.fields).map(([fieldKey, fieldMeta]) =>
renderReadOnlyField(fieldKey, fieldMeta)
)}
</div>
</div>
))}
</CardContent>
</Card>
{showEditDialog && (
<ConfigEditDialog
bankId={bankId}
initialConfig={config}
overrides={overrides}
onClose={() => setShowEditDialog(false)}
onSaved={() => {
loadConfig();
setShowEditDialog(false);
}}
/>
)}
<AlertDialog open={showResetDialog} onOpenChange={setShowResetDialog}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Reset Configuration</AlertDialogTitle>
<AlertDialogDescription>
Are you sure you want to reset all configuration overrides to defaults? This action
cannot be undone.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel disabled={resetting}>Cancel</AlertDialogCancel>
<AlertDialogAction onClick={confirmReset} disabled={resetting}>
{resetting ? (
<>
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
Resetting...
</>
) : (
"Reset to Defaults"
)}
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</>
);
}
// Edit dialog component
function ConfigEditDialog({
bankId,
initialConfig,
overrides,
onClose,
onSaved,
}: {
bankId: string;
initialConfig: Record<string, any>;
overrides: Record<string, any>;
onClose: () => void;
onSaved: () => void;
}) {
const [saving, setSaving] = useState(false);
const [error, setError] = useState<string | null>(null);
const [config, setConfig] = useState(initialConfig);
const handleFieldChange = (field: string, value: any) => {
setConfig({ ...config, [field]: value });
};
const handleSave = async () => {
setSaving(true);
setError(null);
try {
const updates: Record<string, any> = {};
Object.keys(config).forEach((key) => {
const isConfigurable = Object.values(FIELD_CATEGORIES).some((cat) =>
Object.keys(cat.fields).includes(key)
);
if (isConfigurable) {
updates[key] = config[key];
}
});
await client.updateBankConfig(bankId, updates);
onSaved();
} catch (err: any) {
console.error("Failed to save config:", err);
setError(err.message || "Failed to save configuration");
setSaving(false);
}
};
const renderField = (fieldKey: string, fieldMeta: any) => {
const value = config[fieldKey];
if (fieldMeta.type === "boolean") {
return (
<div key={fieldKey} className="space-y-2">
<div className="flex items-center justify-between">
<div>
<Label htmlFor={fieldKey} className="font-mono">
{fieldKey}
</Label>
{fieldMeta.description && (
<p className="text-xs text-muted-foreground mt-1">{fieldMeta.description}</p>
)}
</div>
<button
onClick={() => handleFieldChange(fieldKey, !value)}
className={`relative inline-flex h-6 w-11 items-center rounded-full transition-colors ${
value ? "bg-primary" : "bg-muted"
}`}
>
<span
className={`inline-block h-4 w-4 transform rounded-full bg-white transition-transform ${
value ? "translate-x-6" : "translate-x-1"
}`}
/>
</button>
</div>
</div>
);
}
if (fieldMeta.type === "select") {
return (
<div key={fieldKey} className="space-y-2">
<Label htmlFor={fieldKey} className="font-mono">
{fieldKey}
</Label>
{fieldMeta.description && (
<p className="text-xs text-muted-foreground mt-1">{fieldMeta.description}</p>
)}
<Select
value={value?.toString()}
onValueChange={(val) => handleFieldChange(fieldKey, val)}
>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{fieldMeta.options.map((opt: string) => (
<SelectItem key={opt} value={opt}>
{opt}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
);
}
if (fieldMeta.type === "textarea") {
return (
<div key={fieldKey} className="space-y-2">
<Label htmlFor={fieldKey} className="font-mono">
{fieldKey}
</Label>
{fieldMeta.description && (
<p className="text-xs text-muted-foreground mt-1">{fieldMeta.description}</p>
)}
<Textarea
id={fieldKey}
value={value || ""}
onChange={(e) => handleFieldChange(fieldKey, e.target.value || null)}
placeholder={fieldMeta.placeholder}
rows={fieldMeta.rows || 3}
className="font-mono text-sm"
/>
</div>
);
}
// number or text
return (
<div key={fieldKey} className="space-y-2">
<Label htmlFor={fieldKey} className="font-mono">
{fieldKey}
</Label>
{fieldMeta.description && (
<p className="text-xs text-muted-foreground mt-1">{fieldMeta.description}</p>
)}
<Input
id={fieldKey}
type={fieldMeta.type || "text"}
value={value ?? ""}
onChange={(e) =>
handleFieldChange(
fieldKey,
fieldMeta.type === "number" ? parseFloat(e.target.value) : e.target.value
)
}
min={fieldMeta.min}
max={fieldMeta.max}
/>
</div>
);
};
return (
<Dialog open onOpenChange={onClose}>
<DialogContent className="max-w-2xl max-h-[90vh] overflow-y-auto">
<DialogHeader>
<DialogTitle>Edit Configuration</DialogTitle>
<DialogDescription>
Customize behavioral settings for this bank. Changes only affect this bank and override
global defaults.
</DialogDescription>
</DialogHeader>
<div className="space-y-6 py-4">
{error && (
<Alert variant="destructive">
<AlertCircle className="h-4 w-4" />
<AlertDescription>{error}</AlertDescription>
</Alert>
)}
{Object.entries(FIELD_CATEGORIES).map(([catKey, category]) => (
<div key={catKey} className="space-y-4">
<div>
<h3 className="text-sm font-semibold">{category.title}</h3>
<p className="text-xs text-muted-foreground">{category.description}</p>
</div>
<div className="grid gap-4">
{Object.entries(category.fields).map(([fieldKey, fieldMeta]) =>
renderField(fieldKey, fieldMeta)
)}
</div>
</div>
))}
</div>
<DialogFooter>
<Button onClick={onClose} variant="outline" disabled={saving}>
Cancel
</Button>
<Button onClick={handleSave} disabled={saving}>
{saving ? (
<>
<Loader2 className="h-4 w-4 mr-2 animate-spin" />
Saving...
</>
) : (
"Save Changes"
)}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
@@ -0,0 +1,245 @@
"use client";
import { useState, useEffect } from "react";
import { useBank } from "@/lib/bank-context";
import { client } from "@/lib/api";
import { Button } from "@/components/ui/button";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import { RefreshCw, Clock, AlertCircle, CheckCircle, Loader2, X } from "lucide-react";
interface Operation {
id: string;
task_type: string;
items_count: number;
document_id: string | null;
created_at: string;
status: string;
error_message: string | null;
}
export function BankOperationsView() {
const { currentBank } = useBank();
const [operations, setOperations] = useState<Operation[]>([]);
const [totalOperations, setTotalOperations] = useState(0);
const [statusFilter, setStatusFilter] = useState<string | null>(null);
const [limit] = useState(10);
const [offset, setOffset] = useState(0);
const [cancellingOpId, setCancellingOpId] = useState<string | null>(null);
const [loading, setLoading] = useState(false);
const loadOperations = async (
newStatusFilter: string | null = statusFilter,
newOffset: number = offset
) => {
if (!currentBank) return;
setLoading(true);
try {
const opsData = await client.listOperations(currentBank, {
status: newStatusFilter || undefined,
limit,
offset: newOffset,
});
setOperations(opsData.operations || []);
setTotalOperations(opsData.total || 0);
} catch (error) {
console.error("Error loading operations:", error);
} finally {
setLoading(false);
}
};
const handleFilterChange = (newFilter: string | null) => {
setStatusFilter(newFilter);
setOffset(0);
loadOperations(newFilter, 0);
};
const handlePageChange = (newOffset: number) => {
setOffset(newOffset);
loadOperations(statusFilter, newOffset);
};
const handleCancelOperation = async (operationId: string) => {
if (!currentBank) return;
setCancellingOpId(operationId);
try {
await client.cancelOperation(currentBank, operationId);
await loadOperations();
} catch (error) {
console.error("Error cancelling operation:", error);
alert("Error cancelling operation: " + (error as Error).message);
} finally {
setCancellingOpId(null);
}
};
useEffect(() => {
if (currentBank) {
loadOperations();
// Refresh operations every 5 seconds
const interval = setInterval(() => loadOperations(), 5000);
return () => clearInterval(interval);
}
}, [currentBank]);
if (!currentBank) return null;
return (
<div className="space-y-4">
<div className="flex items-center justify-between">
<div>
<div className="flex items-center gap-2">
<h3 className="text-lg font-semibold">Background Operations</h3>
<button
onClick={() => loadOperations()}
className="p-1 rounded hover:bg-muted transition-colors"
title="Refresh operations"
disabled={loading}
>
<RefreshCw
className={`w-4 h-4 text-muted-foreground hover:text-foreground ${loading ? "animate-spin" : ""}`}
/>
</button>
</div>
<p className="text-sm text-muted-foreground">
{totalOperations} operation{totalOperations !== 1 ? "s" : ""}
{statusFilter ? ` (${statusFilter})` : ""}
</p>
</div>
<div className="flex gap-1 bg-muted p-1 rounded-lg">
{[
{ value: null, label: "All" },
{ value: "pending", label: "Pending" },
{ value: "completed", label: "Completed" },
{ value: "failed", label: "Failed" },
].map((filter) => (
<button
key={filter.value ?? "all"}
onClick={() => handleFilterChange(filter.value)}
className={`px-3 py-1.5 text-sm font-medium rounded-md transition-colors ${
statusFilter === filter.value
? "bg-background shadow-sm"
: "text-muted-foreground hover:text-foreground"
}`}
>
{filter.label}
</button>
))}
</div>
</div>
<div>
{operations.length > 0 ? (
<>
<div className="overflow-x-auto">
<Table>
<TableHeader>
<TableRow>
<TableHead className="w-[100px]">ID</TableHead>
<TableHead>Type</TableHead>
<TableHead>Created</TableHead>
<TableHead>Status</TableHead>
<TableHead className="w-[80px]"></TableHead>
</TableRow>
</TableHeader>
<TableBody>
{operations.map((op) => (
<TableRow key={op.id} className={op.status === "failed" ? "bg-red-500/5" : ""}>
<TableCell className="font-mono text-xs text-muted-foreground">
{op.id.substring(0, 8)}
</TableCell>
<TableCell className="font-medium">{op.task_type}</TableCell>
<TableCell className="text-sm text-muted-foreground">
{new Date(op.created_at).toLocaleString()}
</TableCell>
<TableCell>
{op.status === "pending" && (
<span className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-amber-500/10 text-amber-600 dark:text-amber-400 border border-amber-500/20">
<Clock className="w-3 h-3" />
pending
</span>
)}
{op.status === "failed" && (
<span
className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-red-500/10 text-red-600 dark:text-red-400 border border-red-500/20"
title={op.error_message ?? undefined}
>
<AlertCircle className="w-3 h-3" />
failed
</span>
)}
{op.status === "completed" && (
<span className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-emerald-500/10 text-emerald-600 dark:text-emerald-400 border border-emerald-500/20">
<CheckCircle className="w-3 h-3" />
completed
</span>
)}
</TableCell>
<TableCell>
{op.status === "pending" && (
<Button
variant="ghost"
size="sm"
className="h-7 text-xs text-muted-foreground hover:text-red-600 dark:hover:text-red-400"
onClick={() => handleCancelOperation(op.id)}
disabled={cancellingOpId === op.id}
>
{cancellingOpId === op.id ? (
<Loader2 className="w-3 h-3 animate-spin" />
) : (
<X className="w-3 h-3 mr-1" />
)}
{cancellingOpId === op.id ? "" : "Cancel"}
</Button>
)}
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
</div>
{/* Pagination */}
{totalOperations > limit && (
<div className="flex items-center justify-between mt-4 pt-4 border-t">
<p className="text-sm text-muted-foreground">
Showing {offset + 1}-{Math.min(offset + limit, totalOperations)} of{" "}
{totalOperations}
</p>
<div className="flex gap-2">
<Button
variant="outline"
size="sm"
onClick={() => handlePageChange(Math.max(0, offset - limit))}
disabled={offset === 0}
>
Previous
</Button>
<Button
variant="outline"
size="sm"
onClick={() => handlePageChange(offset + limit)}
disabled={offset + limit >= totalOperations}
>
Next
</Button>
</div>
</div>
)}
</>
) : (
<p className="text-muted-foreground text-center py-8 text-sm">
No {statusFilter ? `${statusFilter} ` : ""}operations
</p>
)}
</div>
</div>
);
}
@@ -151,66 +151,6 @@ const TRAIT_LABELS: Record<
},
};
function DispositionEditor({
disposition,
editMode,
editDisposition,
onEditChange,
}: {
disposition: DispositionTraits;
editMode: boolean;
editDisposition: DispositionTraits;
onEditChange: (trait: keyof DispositionTraits, value: number) => void;
}) {
const data = editMode ? editDisposition : disposition;
return (
<div className="space-y-4">
{(Object.keys(TRAIT_LABELS) as Array<keyof DispositionTraits>).map((trait) => (
<div key={trait} className="space-y-2">
<div className="flex justify-between items-center">
<div>
<label className="text-sm font-medium text-foreground">
{TRAIT_LABELS[trait].label}
</label>
<p className="text-xs text-muted-foreground">{TRAIT_LABELS[trait].description}</p>
</div>
<span className="text-sm font-bold text-primary">{data[trait]}/5</span>
</div>
{editMode ? (
<>
<div className="flex justify-between text-[10px] text-muted-foreground">
<span>{TRAIT_LABELS[trait].lowLabel}</span>
<span>{TRAIT_LABELS[trait].highLabel}</span>
</div>
<input
type="range"
min="1"
max="5"
step="1"
value={editDisposition[trait]}
onChange={(e) => onEditChange(trait, parseInt(e.target.value))}
className="w-full h-2 bg-muted rounded-lg appearance-none cursor-pointer accent-primary"
/>
</>
) : (
<div className="flex items-center gap-2">
<span className="text-xs text-muted-foreground">{TRAIT_LABELS[trait].lowLabel}</span>
<div className="flex-1 h-2 bg-muted rounded-full overflow-hidden">
<div
className="h-full bg-primary rounded-full transition-all"
style={{ width: `${((data[trait] - 1) / 4) * 100}%` }}
/>
</div>
<span className="text-xs text-muted-foreground">{TRAIT_LABELS[trait].highLabel}</span>
</div>
)}
</div>
))}
</div>
);
}
export function BankProfileView() {
const router = useRouter();
const { currentBank, setCurrentBank, loadBanks } = useBank();
@@ -223,8 +163,8 @@ export function BankProfileView() {
const [directives, setDirectives] = useState<Directive[]>([]);
const [mentalModelsCount, setMentalModelsCount] = useState(0);
const [loading, setLoading] = useState(false);
const [saving, setSaving] = useState(false);
const [editMode, setEditMode] = useState(false);
const [showDispositionDialog, setShowDispositionDialog] = useState(false);
const [showMissionDialog, setShowMissionDialog] = useState(false);
// Directive state
const [showCreateDirective, setShowCreateDirective] = useState(false);
@@ -235,12 +175,6 @@ export function BankProfileView() {
} | null>(null);
const [deletingDirective, setDeletingDirective] = useState(false);
// Ref to track editMode for polling (avoids stale closure)
const editModeRef = useRef(editMode);
useEffect(() => {
editModeRef.current = editMode;
}, [editMode]);
// Delete state
const [showDeleteDialog, setShowDeleteDialog] = useState(false);
const [isDeleting, setIsDeleting] = useState(false);
@@ -258,14 +192,6 @@ export function BankProfileView() {
const [opsOffset, setOpsOffset] = useState(0);
const [cancellingOpId, setCancellingOpId] = useState<string | null>(null);
// Edit state
const [editMission, setEditMission] = useState("");
const [editDisposition, setEditDisposition] = useState<DispositionTraits>({
skepticism: 3,
literalism: 3,
empathy: 3,
});
const loadOperations = async (
statusFilter: string | null = opsStatusFilter,
offset: number = opsOffset
@@ -319,12 +245,6 @@ export function BankProfileView() {
setDirectives(directivesData.items || []);
setMentalModelsCount(mentalModelsData.items?.length || 0);
await loadOperations();
// Only initialize edit state when not in edit mode
if (!editModeRef.current) {
setEditMission(profileData.mission || "");
setEditDisposition(profileData.disposition);
}
} catch (error) {
console.error("Error loading bank profile:", error);
alert("Error loading bank profile: " + (error as Error).message);
@@ -333,33 +253,6 @@ export function BankProfileView() {
}
};
const handleSave = async () => {
if (!currentBank) return;
setSaving(true);
try {
await client.updateBankProfile(currentBank, {
mission: editMission,
disposition: editDisposition,
});
await loadData();
setEditMode(false);
} catch (error) {
console.error("Error saving bank profile:", error);
alert("Error saving bank profile: " + (error as Error).message);
} finally {
setSaving(false);
}
};
const handleCancel = () => {
if (profile) {
setEditMission(profile.mission || "");
setEditDisposition(profile.disposition);
}
setEditMode(false);
};
const handleDeleteBank = async () => {
if (!currentBank) return;
@@ -501,238 +394,58 @@ export function BankProfileView() {
return (
<div className="space-y-6">
{/* Header with actions */}
<div className="flex justify-between items-center">
<div>
<h2 className="text-2xl font-bold text-foreground">{profile?.name || currentBank}</h2>
<p className="text-sm text-muted-foreground font-mono">{currentBank}</p>
</div>
<div className="flex gap-2">
{editMode ? (
<>
<Button onClick={handleCancel} variant="secondary" disabled={saving}>
Cancel
</Button>
<Button onClick={handleSave} disabled={saving}>
{saving ? (
<>
<RefreshCw className="w-4 h-4 mr-2 animate-spin" />
Saving...
</>
) : (
<>
<Save className="w-4 h-4 mr-2" />
Save Changes
</>
)}
</Button>
</>
) : (
<DropdownMenu>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm">
Actions
<MoreVertical className="w-4 h-4 ml-2" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent align="end" className="w-48">
<DropdownMenuItem onClick={() => setEditMode(true)}>
<Pencil className="w-4 h-4 mr-2" />
Edit Profile
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem
onClick={handleTriggerConsolidation}
disabled={isConsolidating || !observationsEnabled}
title={!observationsEnabled ? "Observations feature is not enabled" : undefined}
>
{isConsolidating ? (
<Loader2 className="w-4 h-4 mr-2 animate-spin" />
) : (
<Brain className="w-4 h-4 mr-2" />
)}
{isConsolidating ? "Consolidating..." : "Run Consolidation"}
{!observationsEnabled && (
<span className="ml-auto text-xs text-muted-foreground">Off</span>
)}
</DropdownMenuItem>
<DropdownMenuItem
onClick={() => setShowClearObservationsDialog(true)}
disabled={!observationsEnabled}
className="text-amber-600 dark:text-amber-400 focus:text-amber-700 dark:focus:text-amber-300"
title={!observationsEnabled ? "Observations feature is not enabled" : undefined}
>
<Trash2 className="w-4 h-4 mr-2" />
Clear Observations
{!observationsEnabled && (
<span className="ml-auto text-xs text-muted-foreground">Off</span>
)}
</DropdownMenuItem>
<DropdownMenuSeparator />
<DropdownMenuItem
onClick={() => setShowDeleteDialog(true)}
className="text-red-600 dark:text-red-400 focus:text-red-700 dark:focus:text-red-300"
>
<Trash2 className="w-4 h-4 mr-2" />
Delete Bank
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
)}
</div>
</div>
{/* Stats Overview - Compact cards */}
{stats && (
<div className="grid grid-cols-2 md:grid-cols-4 gap-4">
<Card className="bg-gradient-to-br from-blue-500/10 to-blue-600/5 border-blue-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-blue-500/20">
<Database className="w-5 h-5 text-blue-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Memories</p>
<p className="text-2xl font-bold text-foreground">{stats.total_nodes}</p>
</div>
</div>
</CardContent>
</Card>
<Card className="bg-gradient-to-br from-purple-500/10 to-purple-600/5 border-purple-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-purple-500/20">
<Link2 className="w-5 h-5 text-purple-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Links</p>
<p className="text-2xl font-bold text-foreground">{stats.total_links}</p>
</div>
</div>
</CardContent>
</Card>
<Card className="bg-gradient-to-br from-emerald-500/10 to-emerald-600/5 border-emerald-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-emerald-500/20">
<FolderOpen className="w-5 h-5 text-emerald-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Documents</p>
<p className="text-2xl font-bold text-foreground">{stats.total_documents}</p>
</div>
</div>
</CardContent>
</Card>
<Card
className={`bg-gradient-to-br ${stats.pending_operations > 0 ? "from-amber-500/10 to-amber-600/5 border-amber-500/20" : "from-slate-500/10 to-slate-600/5 border-slate-500/20"}`}
>
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div
className={`p-2 rounded-lg ${stats.pending_operations > 0 ? "bg-amber-500/20" : "bg-slate-500/20"}`}
>
<Activity
className={`w-5 h-5 ${stats.pending_operations > 0 ? "text-amber-500 animate-pulse" : "text-slate-500"}`}
/>
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Pending</p>
<p className="text-2xl font-bold text-foreground">{stats.pending_operations}</p>
</div>
</div>
</CardContent>
</Card>
</div>
)}
{/* Memory Type Breakdown */}
{stats && (
<div className="grid grid-cols-5 gap-3">
<div className="bg-blue-500/10 border border-blue-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-blue-600 dark:text-blue-400 font-semibold uppercase tracking-wide">
World Facts
</p>
<p className="text-2xl font-bold text-blue-600 dark:text-blue-400 mt-1">
{stats.nodes_by_fact_type?.world || 0}
</p>
</div>
<div className="bg-purple-500/10 border border-purple-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-purple-600 dark:text-purple-400 font-semibold uppercase tracking-wide">
Experience
</p>
<p className="text-2xl font-bold text-purple-600 dark:text-purple-400 mt-1">
{stats.nodes_by_fact_type?.experience || 0}
</p>
</div>
<div
className={`rounded-xl p-4 text-center ${
observationsEnabled
? "bg-amber-500/10 border border-amber-500/20"
: "bg-muted/50 border border-muted"
}`}
title={!observationsEnabled ? "Observations feature is not enabled" : undefined}
>
<p
className={`text-xs font-semibold uppercase tracking-wide ${
observationsEnabled ? "text-amber-600 dark:text-amber-400" : "text-muted-foreground"
}`}
>
Observations
{!observationsEnabled && <span className="ml-1 normal-case">(Off)</span>}
</p>
<p
className={`text-2xl font-bold mt-1 ${
observationsEnabled ? "text-amber-600 dark:text-amber-400" : "text-muted-foreground"
}`}
>
{observationsEnabled ? stats.total_mental_models || 0 : "—"}
</p>
</div>
<div className="bg-cyan-500/10 border border-cyan-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-cyan-600 dark:text-cyan-400 font-semibold uppercase tracking-wide">
Mental Models
</p>
<p className="text-2xl font-bold text-cyan-600 dark:text-cyan-400 mt-1">
{mentalModelsCount}
</p>
</div>
<div className="bg-rose-500/10 border border-rose-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-rose-600 dark:text-rose-400 font-semibold uppercase tracking-wide">
Directives
</p>
<p className="text-2xl font-bold text-rose-600 dark:text-rose-400 mt-1">
{directives.length}
</p>
</div>
</div>
)}
<div className="grid grid-cols-1 lg:grid-cols-2 gap-6">
{/* Disposition Chart */}
<Card>
<CardHeader className="pb-2">
<CardTitle className="flex items-center gap-2 text-lg">
<Brain className="w-5 h-5 text-primary" />
Disposition Profile
</CardTitle>
<CardDescription>
Traits that shape how observations are formed via Reflect
</CardDescription>
<div className="flex items-start justify-between">
<div>
<CardTitle className="flex items-center gap-2 text-lg">
<Brain className="w-5 h-5 text-primary" />
Disposition Profile
</CardTitle>
<CardDescription>Traits that shape the reasoning and perspective</CardDescription>
</div>
<Button onClick={() => setShowDispositionDialog(true)} variant="ghost" size="sm">
<Pencil className="h-4 w-4" />
</Button>
</div>
</CardHeader>
<CardContent>
{profile && (
<DispositionEditor
disposition={profile.disposition}
editMode={editMode}
editDisposition={editDisposition}
onEditChange={(trait, value) =>
setEditDisposition((prev) => ({ ...prev, [trait]: value }))
}
/>
<div className="space-y-4">
{(Object.keys(TRAIT_LABELS) as Array<keyof DispositionTraits>).map((trait) => (
<div key={trait} className="space-y-2">
<div className="flex justify-between items-center">
<div>
<label className="text-sm font-medium text-foreground">
{TRAIT_LABELS[trait].label}
</label>
<p className="text-xs text-muted-foreground">
{TRAIT_LABELS[trait].description}
</p>
</div>
<span className="text-sm font-bold text-primary">
{profile.disposition[trait]}/5
</span>
</div>
<div className="flex items-center gap-2">
<span className="text-xs text-muted-foreground">
{TRAIT_LABELS[trait].lowLabel}
</span>
<div className="flex-1 h-2 bg-muted rounded-full overflow-hidden">
<div
className="h-full bg-primary rounded-full transition-all"
style={{ width: `${((profile.disposition[trait] - 1) / 4) * 100}%` }}
/>
</div>
<span className="text-xs text-muted-foreground">
{TRAIT_LABELS[trait].highLabel}
</span>
</div>
</div>
))}
</div>
)}
</CardContent>
</Card>
@@ -740,30 +453,26 @@ export function BankProfileView() {
{/* Mission */}
<Card>
<CardHeader className="pb-2">
<CardTitle className="flex items-center gap-2 text-lg">
<Target className="w-5 h-5 text-primary" />
Mission
</CardTitle>
<CardDescription>
Who the agent is and what they&apos;re trying to accomplish. Used for mental models
and reflect.
</CardDescription>
<div className="flex items-start justify-between">
<div>
<CardTitle className="flex items-center gap-2 text-lg">
<Target className="w-5 h-5 text-primary" />
Mission
</CardTitle>
<CardDescription>
Affects how observations, reflect, and mental models are created
</CardDescription>
</div>
<Button onClick={() => setShowMissionDialog(true)} variant="ghost" size="sm">
<Pencil className="h-4 w-4" />
</Button>
</div>
</CardHeader>
<CardContent>
{editMode ? (
<Textarea
value={editMission}
onChange={(e) => setEditMission(e.target.value)}
placeholder="e.g., I am a PM for the engineering team. I help coordinate sprints and track project progress..."
rows={6}
className="resize-none"
/>
) : (
<p className="text-sm text-foreground whitespace-pre-wrap leading-relaxed">
{profile?.mission ||
"No mission set. Set a mission to derive structural mental models and personalize reflect responses."}
</p>
)}
<p className="text-sm text-foreground whitespace-pre-wrap leading-relaxed">
{profile?.mission ||
"No mission set. Set a mission to derive structural mental models and personalize reflect responses."}
</p>
</CardContent>
</Card>
</div>
@@ -841,158 +550,6 @@ export function BankProfileView() {
</CardContent>
</Card>
{/* Operations Section */}
<Card>
<CardHeader className="pb-2">
<div className="flex items-center justify-between">
<div>
<CardTitle className="flex items-center gap-2 text-lg">
<Activity className="w-5 h-5 text-primary" />
Background Operations
<button
onClick={() => loadOperations()}
className="p-1 rounded hover:bg-muted transition-colors"
title="Refresh operations"
>
<RefreshCw className="w-4 h-4 text-muted-foreground hover:text-foreground" />
</button>
</CardTitle>
<CardDescription>
{totalOperations} operation{totalOperations !== 1 ? "s" : ""}
{opsStatusFilter ? ` (${opsStatusFilter})` : ""}
</CardDescription>
</div>
<div className="flex gap-1 bg-muted p-1 rounded-lg">
{[
{ value: null, label: "All" },
{ value: "pending", label: "Pending" },
{ value: "completed", label: "Completed" },
{ value: "failed", label: "Failed" },
].map((filter) => (
<button
key={filter.value ?? "all"}
onClick={() => handleOpsFilterChange(filter.value)}
className={`px-3 py-1.5 text-sm font-medium rounded-md transition-colors ${
opsStatusFilter === filter.value
? "bg-background shadow-sm"
: "text-muted-foreground hover:text-foreground"
}`}
>
{filter.label}
</button>
))}
</div>
</div>
</CardHeader>
<CardContent>
{operations.length > 0 ? (
<>
<div className="overflow-x-auto">
<Table>
<TableHeader>
<TableRow>
<TableHead className="w-[100px]">ID</TableHead>
<TableHead>Type</TableHead>
<TableHead>Created</TableHead>
<TableHead>Status</TableHead>
<TableHead className="w-[80px]"></TableHead>
</TableRow>
</TableHeader>
<TableBody>
{operations.map((op) => (
<TableRow
key={op.id}
className={op.status === "failed" ? "bg-red-500/5" : ""}
>
<TableCell className="font-mono text-xs text-muted-foreground">
{op.id.substring(0, 8)}
</TableCell>
<TableCell className="font-medium">{op.task_type}</TableCell>
<TableCell className="text-sm text-muted-foreground">
{new Date(op.created_at).toLocaleString()}
</TableCell>
<TableCell>
{op.status === "pending" && (
<span className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-amber-500/10 text-amber-600 dark:text-amber-400 border border-amber-500/20">
<Clock className="w-3 h-3" />
pending
</span>
)}
{op.status === "failed" && (
<span
className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-red-500/10 text-red-600 dark:text-red-400 border border-red-500/20"
title={op.error_message ?? undefined}
>
<AlertCircle className="w-3 h-3" />
failed
</span>
)}
{op.status === "completed" && (
<span className="inline-flex items-center gap-1.5 px-2 py-0.5 rounded-full text-xs font-medium bg-emerald-500/10 text-emerald-600 dark:text-emerald-400 border border-emerald-500/20">
<CheckCircle className="w-3 h-3" />
completed
</span>
)}
</TableCell>
<TableCell>
{op.status === "pending" && (
<Button
variant="ghost"
size="sm"
className="h-7 text-xs text-muted-foreground hover:text-red-600 dark:hover:text-red-400"
onClick={() => handleCancelOperation(op.id)}
disabled={cancellingOpId === op.id}
>
{cancellingOpId === op.id ? (
<Loader2 className="w-3 h-3 animate-spin" />
) : (
<X className="w-3 h-3 mr-1" />
)}
{cancellingOpId === op.id ? "" : "Cancel"}
</Button>
)}
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
</div>
{/* Pagination */}
{totalOperations > opsLimit && (
<div className="flex items-center justify-between mt-4 pt-4 border-t">
<p className="text-sm text-muted-foreground">
Showing {opsOffset + 1}-{Math.min(opsOffset + opsLimit, totalOperations)} of{" "}
{totalOperations}
</p>
<div className="flex gap-2">
<Button
variant="outline"
size="sm"
onClick={() => handleOpsPageChange(Math.max(0, opsOffset - opsLimit))}
disabled={opsOffset === 0}
>
Previous
</Button>
<Button
variant="outline"
size="sm"
onClick={() => handleOpsPageChange(opsOffset + opsLimit)}
disabled={opsOffset + opsLimit >= totalOperations}
>
Next
</Button>
</div>
</div>
)}
</>
) : (
<p className="text-muted-foreground text-center py-8 text-sm">
No {opsStatusFilter ? `${opsStatusFilter} ` : ""}operations
</p>
)}
</CardContent>
</Card>
{/* Delete Confirmation Dialog */}
<AlertDialog open={showDeleteDialog} onOpenChange={setShowDeleteDialog}>
<AlertDialogContent>
@@ -1142,10 +699,197 @@ export function BankProfileView() {
}}
/>
)}
{/* Disposition Edit Dialog */}
{showDispositionDialog && profile && (
<DispositionEditDialog
disposition={profile.disposition}
onClose={() => setShowDispositionDialog(false)}
onSaved={async () => {
await loadData();
setShowDispositionDialog(false);
}}
/>
)}
{/* Mission Edit Dialog */}
{showMissionDialog && profile && (
<MissionEditDialog
mission={profile.mission || ""}
onClose={() => setShowMissionDialog(false)}
onSaved={async () => {
await loadData();
setShowMissionDialog(false);
}}
/>
)}
</div>
);
}
// ============= DISPOSITION EDIT DIALOG =============
function DispositionEditDialog({
disposition,
onClose,
onSaved,
}: {
disposition: DispositionTraits;
onClose: () => void;
onSaved: () => void;
}) {
const { currentBank } = useBank();
const [saving, setSaving] = useState(false);
const [editDisposition, setEditDisposition] = useState<DispositionTraits>(disposition);
const handleSave = async () => {
if (!currentBank) return;
setSaving(true);
try {
await client.updateBankProfile(currentBank, {
disposition: editDisposition,
});
onSaved();
} catch (error) {
console.error("Error saving disposition:", error);
alert("Error saving disposition: " + (error as Error).message);
} finally {
setSaving(false);
}
};
return (
<Dialog open onOpenChange={onClose}>
<DialogContent className="max-w-xl">
<DialogHeader>
<DialogTitle>Edit Disposition Traits</DialogTitle>
<DialogDescription>Traits that shape the reasoning and perspective</DialogDescription>
</DialogHeader>
<div className="space-y-4 py-4">
{(Object.keys(TRAIT_LABELS) as Array<keyof DispositionTraits>).map((trait) => (
<div key={trait} className="space-y-2">
<div className="flex justify-between items-center">
<div>
<label className="text-sm font-medium text-foreground">
{TRAIT_LABELS[trait].label}
</label>
<p className="text-xs text-muted-foreground">{TRAIT_LABELS[trait].description}</p>
</div>
<span className="text-sm font-bold text-primary">{editDisposition[trait]}/5</span>
</div>
<div className="flex justify-between text-[10px] text-muted-foreground">
<span>{TRAIT_LABELS[trait].lowLabel}</span>
<span>{TRAIT_LABELS[trait].highLabel}</span>
</div>
<input
type="range"
min="1"
max="5"
step="1"
value={editDisposition[trait]}
onChange={(e) =>
setEditDisposition((prev) => ({ ...prev, [trait]: parseInt(e.target.value) }))
}
className="w-full h-2 bg-muted rounded-lg appearance-none cursor-pointer accent-primary"
/>
</div>
))}
</div>
<DialogFooter>
<Button onClick={onClose} variant="outline" disabled={saving}>
Cancel
</Button>
<Button onClick={handleSave} disabled={saving}>
{saving ? (
<>
<RefreshCw className="w-4 h-4 mr-2 animate-spin" />
Saving...
</>
) : (
"Save Changes"
)}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
// ============= MISSION EDIT DIALOG =============
function MissionEditDialog({
mission,
onClose,
onSaved,
}: {
mission: string;
onClose: () => void;
onSaved: () => void;
}) {
const { currentBank } = useBank();
const [saving, setSaving] = useState(false);
const [editMission, setEditMission] = useState(mission);
const handleSave = async () => {
if (!currentBank) return;
setSaving(true);
try {
await client.updateBankProfile(currentBank, {
mission: editMission,
});
onSaved();
} catch (error) {
console.error("Error saving mission:", error);
alert("Error saving mission: " + (error as Error).message);
} finally {
setSaving(false);
}
};
return (
<Dialog open onOpenChange={onClose}>
<DialogContent className="max-w-xl">
<DialogHeader>
<DialogTitle>Edit Mission</DialogTitle>
<DialogDescription>
Affects how observations, reflect, and mental models are created
</DialogDescription>
</DialogHeader>
<div className="space-y-2 py-4">
<Textarea
value={editMission}
onChange={(e) => setEditMission(e.target.value)}
placeholder="e.g., I am a PM for the engineering team. I help coordinate sprints and track project progress..."
rows={8}
className="resize-none"
/>
</div>
<DialogFooter>
<Button onClick={onClose} variant="outline" disabled={saving}>
Cancel
</Button>
<Button onClick={handleSave} disabled={saving}>
{saving ? (
<>
<RefreshCw className="w-4 h-4 mr-2 animate-spin" />
Saving...
</>
) : (
"Save Changes"
)}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
// ============= DIRECTIVE FORM DIALOG (CREATE/EDIT) =============
function DirectiveFormDialog({
@@ -0,0 +1,208 @@
"use client";
import { useState, useEffect } from "react";
import { useBank } from "@/lib/bank-context";
import { useFeatures } from "@/lib/features-context";
import { client } from "@/lib/api";
import { Card, CardContent } from "@/components/ui/card";
import { Database, Link2, FolderOpen, Activity, Clock } from "lucide-react";
interface BankStats {
bank_id: string;
total_nodes: number;
total_links: number;
total_documents: number;
nodes_by_fact_type: {
world?: number;
experience?: number;
opinion?: number;
};
links_by_link_type: {
temporal?: number;
semantic?: number;
entity?: number;
};
pending_operations: number;
failed_operations: number;
last_consolidated_at: string | null;
pending_consolidation: number;
total_mental_models: number;
}
export function BankStatsView() {
const { currentBank } = useBank();
const { features } = useFeatures();
const observationsEnabled = features?.observations ?? false;
const [stats, setStats] = useState<BankStats | null>(null);
const [mentalModelsCount, setMentalModelsCount] = useState(0);
const [directivesCount, setDirectivesCount] = useState(0);
const [loading, setLoading] = useState(false);
const loadData = async () => {
if (!currentBank) return;
setLoading(true);
try {
const [statsData, mentalModelsData, directivesData] = await Promise.all([
client.getBankStats(currentBank),
client.listMentalModels(currentBank),
client.listDirectives(currentBank),
]);
setStats(statsData as BankStats);
setMentalModelsCount(mentalModelsData.items?.length || 0);
setDirectivesCount(directivesData.items?.length || 0);
} catch (error) {
console.error("Error loading bank stats:", error);
} finally {
setLoading(false);
}
};
useEffect(() => {
if (currentBank) {
loadData();
// Refresh stats every 5 seconds
const interval = setInterval(loadData, 5000);
return () => clearInterval(interval);
}
}, [currentBank]);
if (loading && !stats) {
return (
<div className="flex items-center justify-center py-12">
<Clock className="w-12 h-12 mx-auto mb-3 text-muted-foreground animate-pulse" />
</div>
);
}
if (!stats) return null;
return (
<div className="space-y-6">
{/* Stats Overview - Compact cards */}
<div className="grid grid-cols-2 md:grid-cols-4 gap-4">
<Card className="bg-gradient-to-br from-blue-500/10 to-blue-600/5 border-blue-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-blue-500/20">
<Database className="w-5 h-5 text-blue-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Memories</p>
<p className="text-2xl font-bold text-foreground">{stats.total_nodes}</p>
</div>
</div>
</CardContent>
</Card>
<Card className="bg-gradient-to-br from-purple-500/10 to-purple-600/5 border-purple-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-purple-500/20">
<Link2 className="w-5 h-5 text-purple-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Links</p>
<p className="text-2xl font-bold text-foreground">{stats.total_links}</p>
</div>
</div>
</CardContent>
</Card>
<Card className="bg-gradient-to-br from-emerald-500/10 to-emerald-600/5 border-emerald-500/20">
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div className="p-2 rounded-lg bg-emerald-500/20">
<FolderOpen className="w-5 h-5 text-emerald-500" />
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Documents</p>
<p className="text-2xl font-bold text-foreground">{stats.total_documents}</p>
</div>
</div>
</CardContent>
</Card>
<Card
className={`bg-gradient-to-br ${stats.pending_operations > 0 ? "from-amber-500/10 to-amber-600/5 border-amber-500/20" : "from-slate-500/10 to-slate-600/5 border-slate-500/20"}`}
>
<CardContent className="p-4">
<div className="flex items-center gap-3">
<div
className={`p-2 rounded-lg ${stats.pending_operations > 0 ? "bg-amber-500/20" : "bg-slate-500/20"}`}
>
<Activity
className={`w-5 h-5 ${stats.pending_operations > 0 ? "text-amber-500 animate-pulse" : "text-slate-500"}`}
/>
</div>
<div>
<p className="text-xs text-muted-foreground font-medium">Pending</p>
<p className="text-2xl font-bold text-foreground">{stats.pending_operations}</p>
</div>
</div>
</CardContent>
</Card>
</div>
{/* Memory Type Breakdown */}
<div className="grid grid-cols-5 gap-3">
<div className="bg-blue-500/10 border border-blue-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-blue-600 dark:text-blue-400 font-semibold uppercase tracking-wide">
World Facts
</p>
<p className="text-2xl font-bold text-blue-600 dark:text-blue-400 mt-1">
{stats.nodes_by_fact_type?.world || 0}
</p>
</div>
<div className="bg-purple-500/10 border border-purple-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-purple-600 dark:text-purple-400 font-semibold uppercase tracking-wide">
Experience
</p>
<p className="text-2xl font-bold text-purple-600 dark:text-purple-400 mt-1">
{stats.nodes_by_fact_type?.experience || 0}
</p>
</div>
<div
className={`rounded-xl p-4 text-center ${
observationsEnabled
? "bg-amber-500/10 border border-amber-500/20"
: "bg-muted/50 border border-muted"
}`}
title={!observationsEnabled ? "Observations feature is not enabled" : undefined}
>
<p
className={`text-xs font-semibold uppercase tracking-wide ${
observationsEnabled ? "text-amber-600 dark:text-amber-400" : "text-muted-foreground"
}`}
>
Observations
{!observationsEnabled && <span className="ml-1 normal-case">(Off)</span>}
</p>
<p
className={`text-2xl font-bold mt-1 ${
observationsEnabled ? "text-amber-600 dark:text-amber-400" : "text-muted-foreground"
}`}
>
{observationsEnabled ? stats.total_mental_models || 0 : "—"}
</p>
</div>
<div className="bg-cyan-500/10 border border-cyan-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-cyan-600 dark:text-cyan-400 font-semibold uppercase tracking-wide">
Mental Models
</p>
<p className="text-2xl font-bold text-cyan-600 dark:text-cyan-400 mt-1">
{mentalModelsCount}
</p>
</div>
<div className="bg-rose-500/10 border border-rose-500/20 rounded-xl p-4 text-center">
<p className="text-xs text-rose-600 dark:text-rose-400 font-semibold uppercase tracking-wide">
Directives
</p>
<p className="text-2xl font-bold text-rose-600 dark:text-rose-400 mt-1">
{directivesCount}
</p>
</div>
</div>
</div>
);
}
@@ -2,6 +2,7 @@
import { useState } from "react";
import { useBank } from "@/lib/bank-context";
import { useFeatures } from "@/lib/features-context";
import {
Search,
Sparkles,
@@ -11,6 +12,7 @@ import {
ChevronLeft,
ChevronRight,
Box,
Settings,
} from "lucide-react";
import { cn } from "@/lib/utils";
import Link from "next/link";
@@ -24,6 +26,7 @@ interface SidebarProps {
export function Sidebar({ currentTab, onTabChange }: SidebarProps) {
const { currentBank } = useBank();
const { features } = useFeatures();
const [isCollapsed, setIsCollapsed] = useState(true);
if (!currentBank) {
@@ -36,7 +39,7 @@ export function Sidebar({ currentTab, onTabChange }: SidebarProps) {
{ id: "reflect" as NavItem, label: "Reflect", icon: Sparkles },
{ id: "documents" as NavItem, label: "Documents", icon: FileText },
{ id: "entities" as NavItem, label: "Entities", icon: Users },
{ id: "profile" as NavItem, label: "Memory Bank", icon: Box },
{ id: "profile" as NavItem, label: "Bank Configuration", icon: Settings },
];
return (
@@ -0,0 +1,49 @@
import * as React from "react";
import { cva, type VariantProps } from "class-variance-authority";
import { cn } from "@/lib/utils";
const alertVariants = cva(
"relative w-full rounded-lg border px-4 py-3 text-sm [&>svg+div]:translate-y-[-3px] [&>svg]:absolute [&>svg]:left-4 [&>svg]:top-4 [&>svg]:text-foreground [&>svg~*]:pl-7",
{
variants: {
variant: {
default: "bg-background text-foreground",
destructive:
"border-destructive/50 text-destructive dark:border-destructive [&>svg]:text-destructive",
},
},
defaultVariants: {
variant: "default",
},
}
);
const Alert = React.forwardRef<
HTMLDivElement,
React.HTMLAttributes<HTMLDivElement> & VariantProps<typeof alertVariants>
>(({ className, variant, ...props }, ref) => (
<div ref={ref} role="alert" className={cn(alertVariants({ variant }), className)} {...props} />
));
Alert.displayName = "Alert";
const AlertTitle = React.forwardRef<HTMLParagraphElement, React.HTMLAttributes<HTMLHeadingElement>>(
({ className, ...props }, ref) => (
<h5
ref={ref}
className={cn("mb-1 font-medium leading-none tracking-tight", className)}
{...props}
/>
)
);
AlertTitle.displayName = "AlertTitle";
const AlertDescription = React.forwardRef<
HTMLParagraphElement,
React.HTMLAttributes<HTMLParagraphElement>
>(({ className, ...props }, ref) => (
<div ref={ref} className={cn("text-sm [&_p]:leading-relaxed", className)} {...props} />
));
AlertDescription.displayName = "AlertDescription";
export { Alert, AlertTitle, AlertDescription };
+39
View File
@@ -684,9 +684,48 @@ export class ControlPlaneClient {
observations: boolean;
mcp: boolean;
worker: boolean;
bank_config_api: boolean;
};
}>("/api/version");
}
/**
* Get bank configuration (resolved with hierarchy)
*/
async getBankConfig(bankId: string) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`);
}
/**
* Update bank configuration overrides
*/
async updateBankConfig(bankId: string, updates: Record<string, any>) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`, {
method: "PATCH",
body: JSON.stringify({ updates }),
});
}
/**
* Reset bank configuration to defaults
*/
async resetBankConfig(bankId: string) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`, {
method: "DELETE",
});
}
}
// Export singleton instance
@@ -7,6 +7,7 @@ interface Features {
observations: boolean;
mcp: boolean;
worker: boolean;
bank_config_api: boolean;
}
interface FeaturesContextType {
@@ -19,6 +20,7 @@ const defaultFeatures: Features = {
observations: false,
mcp: false,
worker: false,
bank_config_api: false,
};
const FeaturesContext = createContext<FeaturesContextType | undefined>(undefined);
+179 -1
View File
@@ -57,6 +57,64 @@ hindsight-admin run-db-migration
hindsight-admin run-db-migration --schema tenant_acme
```
### Vector Extension
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_API_VECTOR_EXTENSION` | Vector extension to use: `auto`, `pgvector`, or `vchord` | `auto` |
Hindsight supports two PostgreSQL vector extensions:
- **pgvector**: Standard extension, works well for most embeddings (up to ~2000 dimensions)
- **vchord**: Optimized for high-dimensional embeddings (3000+ dimensions), includes BM25 search
When set to `auto` (default), Hindsight automatically detects which extension is installed, preferring vchord if both are available.
**When to use vchord:**
- Using high-dimensional embeddings (e.g., `text-embedding-3-large` with 3072 dimensions)
- Need better performance with large embedding dimensions
- Want to use vchord's BM25 search capabilities
**When to use pgvector:**
- Using standard embedding dimensions (384-1536)
- Prefer the widely-adopted pgvector extension
- Simpler deployment (pgvector is more commonly available)
**Switching extensions:**
If you need to switch from one extension to another:
1. Set `HINDSIGHT_API_VECTOR_EXTENSION` to your desired extension (`pgvector` or `vchord`)
2. If your database has existing data, you'll get an error with migration instructions
3. For empty databases, indexes will be automatically recreated on startup
### Text Search Extension
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native` or `vchord` | `native` |
Hindsight supports two text search backends for BM25 keyword retrieval:
- **native**: PostgreSQL's built-in full-text search (`tsvector` + GIN indexes)
- **vchord**: VectorChord BM25 (`bm25vector` + BM25 indexes) - requires `vchord_bm25` extension
**When to use vchord:**
- Already using vchord for vector search (good integration)
- Want better BM25 ranking performance
- Need advanced tokenization (uses `llmlingua2` tokenizer)
**When to use native:**
- Standard PostgreSQL deployment (no extra extensions)
- Simpler setup and wider compatibility
- Works well for most use cases
**Switching backends:**
To switch from native to vchord (or vice versa):
1. Set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION=vchord` (or `native`)
2. If your database has existing data, you'll get an error with migration instructions
3. For empty databases, the columns/indexes will be automatically recreated on startup
**Note:** VectorChord text search uses the `llmlingua2` tokenizer for multilingual support, while native uses PostgreSQL's English tokenizer.
### LLM Provider
| Variable | Description | Default |
@@ -657,6 +715,127 @@ The Control Plane is the web UI for managing memory banks.
export HINDSIGHT_CP_DATAPLANE_API_URL=http://api.example.com:8888
```
### Hierarchical Configuration
Hindsight supports per-bank configuration overrides through a hierarchical system: **Global (env vars) → Tenant → Bank**.
#### Type-Safe Config Access
To prevent accidentally using global defaults when bank-specific overrides exist, Hindsight enforces type-safe config access:
**In Application Code:**
```python
from hindsight_api.config import get_config
# ✅ Access static (infrastructure) fields
config = get_config()
host = config.host # OK - static field
port = config.port # OK - static field
# ❌ Attempting to access bank-configurable fields raises an error
chunk_size = config.retain_chunk_size # ConfigFieldAccessError!
```
**Error Message:**
```
ConfigFieldAccessError: Field 'retain_chunk_size' is bank-configurable and cannot
be accessed from global config. Use ConfigResolver.resolve_full_config(bank_id, context)
to get bank-specific config.
```
**For Bank-Specific Config:**
```python
# Internal code that needs bank-specific settings
from hindsight_api.config_resolver import ConfigResolver
# Resolve full config for a specific bank
config = await config_resolver.resolve_full_config(bank_id, request_context)
chunk_size = config.retain_chunk_size # ✅ Uses bank-specific value
```
This design prevents bugs where global defaults are used instead of bank overrides, making it impossible to make this mistake at compile/development time.
#### Security Model
Configuration fields are categorized for security:
1. **Configurable Fields** - Safe behavioral settings that can be customized per-bank:
- Retention: `retain_chunk_size`, `retain_extraction_mode`, `retain_custom_instructions`
- Consolidation: `enable_observations`
2. **Credential Fields** - NEVER exposed or configurable via API:
- API keys: `*_api_key` (all LLM API keys)
- Infrastructure: `*_base_url` (all base URLs)
3. **Static Fields** - Server-level only, cannot be overridden:
- Infrastructure: `database_url`, `port`, `host`, `worker_count`
- Provider/Model selection: `llm_provider`, `llm_model` (requires presets - not yet implemented)
- Performance tuning: `llm_max_concurrent`, `llm_timeout`, retrieval settings, optimization flags
#### Enabling the API
| Variable | Description | Default |
|----------|-------------|---------|
| `HINDSIGHT_API_ENABLE_BANK_CONFIG_API` | Enable per-bank config API | `false` |
**Important:** The bank config API is **disabled by default** for security. Enable it explicitly:
```bash
export HINDSIGHT_API_ENABLE_BANK_CONFIG_API=true
```
#### API Endpoints
- `GET /v1/default/banks/{bank_id}/config` - View resolved config (filtered by permissions)
- `PATCH /v1/default/banks/{bank_id}/config` - Update bank overrides (only allowed fields)
- `DELETE /v1/default/banks/{bank_id}/config` - Reset to defaults
#### Permission System
Tenant extensions can control which fields banks are allowed to modify via `get_allowed_config_fields()`:
```python
class CustomTenantExtension(TenantExtension):
async def get_allowed_config_fields(self, context, bank_id):
# Option 1: Allow all configurable fields
return None
# Option 2: Allow specific fields only
return {"retain_chunk_size", "retain_custom_instructions"}
# Option 3: Read-only (no modifications)
return set()
```
#### Examples
```bash
# Update retention settings for a bank
curl -X PATCH http://localhost:8888/v1/default/banks/my-bank/config \
-H "Content-Type: application/json" \
-d '{
"updates": {
"retain_chunk_size": 4000,
"retain_extraction_mode": "custom",
"retain_custom_instructions": "Focus on technical details and implementation specifics"
}
}'
# Note: retain_extraction_mode must be "custom" to use retain_custom_instructions
# View resolved config (respects permissions)
curl http://localhost:8888/v1/default/banks/my-bank/config
# Reset to defaults
curl -X DELETE http://localhost:8888/v1/default/banks/my-bank/config
```
**Security Notes:**
- Credentials (API keys, base URLs) are never returned in responses
- Only configurable fields can be modified
- Responses are filtered by tenant permissions
- Attempting to set credentials returns 400 error
### Reverse Proxy / Subpath Deployment
To deploy Hindsight under a subpath (e.g., `example.com/hindsight/`):
@@ -721,7 +900,6 @@ See `docker/compose-examples/` directory for:
- Docker Compose setups (`docker-compose.yml`, `reverse-proxy-only.yml`)
- Traefik and other reverse proxy examples
- Full deployment documentation
---
## Example .env File
+281 -14
View File
@@ -2749,6 +2749,189 @@
}
}
},
"/v1/default/banks/{bank_id}/config": {
"get": {
"tags": [
"Banks"
],
"summary": "Get bank configuration",
"description": "Get fully resolved configuration for a bank including all hierarchical overrides (global \u2192 tenant \u2192 bank). The 'config' field contains all resolved config values. The 'overrides' field shows only bank-specific overrides.",
"operationId": "get_bank_config",
"parameters": [
{
"name": "bank_id",
"in": "path",
"required": true,
"schema": {
"type": "string",
"title": "Bank Id"
}
},
{
"name": "authorization",
"in": "header",
"required": false,
"schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Authorization"
}
}
],
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/BankConfigResponse"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
},
"patch": {
"tags": [
"Banks"
],
"summary": "Update bank configuration",
"description": "Update configuration overrides for a bank. Only hierarchical fields can be overridden (LLM settings, retention parameters, etc.). Keys can be provided in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER).",
"operationId": "update_bank_config",
"parameters": [
{
"name": "bank_id",
"in": "path",
"required": true,
"schema": {
"type": "string",
"title": "Bank Id"
}
},
{
"name": "authorization",
"in": "header",
"required": false,
"schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Authorization"
}
}
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/BankConfigUpdate"
}
}
}
},
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/BankConfigResponse"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
},
"delete": {
"tags": [
"Banks"
],
"summary": "Reset bank configuration",
"description": "Reset bank configuration to defaults by removing all bank-specific overrides. The bank will then use global and tenant-level configuration only.",
"operationId": "reset_bank_config",
"parameters": [
{
"name": "bank_id",
"in": "path",
"required": true,
"schema": {
"type": "string",
"title": "Bank Id"
}
},
{
"name": "authorization",
"in": "header",
"required": false,
"schema": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Authorization"
}
}
],
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/BankConfigResponse"
}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
},
"/v1/default/banks/{bank_id}/consolidate": {
"post": {
"tags": [
@@ -3042,6 +3225,70 @@
"mission": "I was born in Texas. I am a software engineer with 10 years of experience."
}
},
"BankConfigResponse": {
"properties": {
"bank_id": {
"type": "string",
"title": "Bank Id",
"description": "Bank identifier"
},
"config": {
"additionalProperties": true,
"type": "object",
"title": "Config",
"description": "Fully resolved configuration with all hierarchical overrides applied (Python field names)"
},
"overrides": {
"additionalProperties": true,
"type": "object",
"title": "Overrides",
"description": "Bank-specific configuration overrides only (Python field names)"
}
},
"type": "object",
"required": [
"bank_id",
"config",
"overrides"
],
"title": "BankConfigResponse",
"description": "Response model for bank configuration.",
"example": {
"bank_id": "my-bank",
"config": {
"llm_model": "gpt-4",
"llm_provider": "openai",
"retain_extraction_mode": "verbose"
},
"overrides": {
"llm_model": "gpt-4",
"retain_extraction_mode": "verbose"
}
}
},
"BankConfigUpdate": {
"properties": {
"updates": {
"additionalProperties": true,
"type": "object",
"title": "Updates",
"description": "Configuration overrides. Keys can be in Python field format (llm_provider) or environment variable format (HINDSIGHT_API_LLM_PROVIDER). Only hierarchical fields can be overridden per-bank."
}
},
"type": "object",
"required": [
"updates"
],
"title": "BankConfigUpdate",
"description": "Request model for updating bank configuration.",
"example": {
"updates": {
"llm_model": "claude-sonnet-4-5",
"retain_custom_instructions": "Extract technical details carefully",
"retain_extraction_mode": "verbose"
}
}
},
"BankListItem": {
"properties": {
"bank_id": {
@@ -3560,7 +3807,8 @@
},
"type": "array",
"title": "Tags",
"description": "Tags for filtering"
"description": "Tags for filtering",
"default": []
}
},
"type": "object",
@@ -3601,7 +3849,8 @@
},
"type": "array",
"title": "Tags",
"description": "Tags for scoped visibility"
"description": "Tags for scoped visibility",
"default": []
},
"max_tokens": {
"type": "integer",
@@ -3613,7 +3862,8 @@
},
"trigger": {
"$ref": "#/components/schemas/MentalModelTrigger",
"description": "Trigger settings"
"description": "Trigger settings",
"default": {}
}
},
"type": "object",
@@ -3789,7 +4039,8 @@
"type": "string"
},
"type": "array",
"title": "Tags"
"title": "Tags",
"default": []
},
"created_at": {
"anyOf": [
@@ -3905,7 +4156,8 @@
},
"type": "array",
"title": "Tags",
"description": "Tags associated with this document"
"description": "Tags associated with this document",
"default": []
}
},
"type": "object",
@@ -4238,13 +4490,19 @@
"type": "boolean",
"title": "Worker",
"description": "Whether the background worker is enabled"
},
"bank_config_api": {
"type": "boolean",
"title": "Bank Config Api",
"description": "Whether per-bank configuration API is enabled"
}
},
"type": "object",
"required": [
"observations",
"mcp",
"worker"
"worker",
"bank_config_api"
],
"title": "FeaturesInfo",
"description": "Feature flags indicating which capabilities are enabled."
@@ -4686,7 +4944,8 @@
"type": "string"
},
"type": "array",
"title": "Tags"
"title": "Tags",
"default": []
},
"max_tokens": {
"type": "integer",
@@ -4694,7 +4953,8 @@
"default": 2048
},
"trigger": {
"$ref": "#/components/schemas/MentalModelTrigger"
"$ref": "#/components/schemas/MentalModelTrigger",
"default": {}
},
"last_refreshed_at": {
"anyOf": [
@@ -5008,7 +5268,8 @@
},
"include": {
"$ref": "#/components/schemas/IncludeOptions",
"description": "Options for including additional data (entities are included by default)"
"description": "Options for including additional data (entities are included by default)",
"default": {}
},
"tags": {
"anyOf": [
@@ -5333,7 +5594,8 @@
},
"type": "array",
"title": "Memories",
"description": "Memory facts used to generate the response"
"description": "Memory facts used to generate the response",
"default": []
},
"mental_models": {
"items": {
@@ -5341,7 +5603,8 @@
},
"type": "array",
"title": "Mental Models",
"description": "Mental models used during reflection"
"description": "Mental models used during reflection",
"default": []
},
"directives": {
"items": {
@@ -5349,7 +5612,8 @@
},
"type": "array",
"title": "Directives",
"description": "Directives applied during reflection"
"description": "Directives applied during reflection",
"default": []
}
},
"type": "object",
@@ -5825,7 +6089,8 @@
},
"type": "array",
"title": "Tool Calls",
"description": "Tool calls made during reflection"
"description": "Tool calls made during reflection",
"default": []
},
"llm_calls": {
"items": {
@@ -5833,7 +6098,8 @@
},
"type": "array",
"title": "Llm Calls",
"description": "LLM calls made during reflection"
"description": "LLM calls made during reflection",
"default": []
}
},
"type": "object",
@@ -6246,6 +6512,7 @@
"example": {
"api_version": "0.4.0",
"features": {
"bank_config_api": false,
"mcp": true,
"observations": false,
"worker": true