Compare commits

...
12 Commits
Author SHA1 Message Date
Nicolò Boschi eed2225f5f docs: remove Advanced Configuration section from embed.md 2026-02-04 14:35:54 +01:00
Nicolò Boschi 625cbae409 fix: remove leftover async cleanup code from test_list_banks 2026-02-04 14:23:19 +01:00
Nicolò Boschi a7a0b53a56 fix: add list() method to BanksAPI namespace 2026-02-04 14:13:32 +01:00
Nicolò Boschi efc961fe06 fix: update tests to use Pydantic model attributes instead of dict access
- Fixed test_server_integration.py to access Pydantic model attributes directly
- Changed dict-style access (response["field"]) to attribute access (response.field)
- Fixed .get() calls on Pydantic models
- Updated recall() calls to access .results attribute
- Updated reflect() calls to access .text attribute
- Fixed test_list_banks to use namespace API instead of deleted default_api
- Fixed attribute shadowing in HindsightClient wrapper (renamed _*_api to _*_namespace)
2026-02-04 14:03:17 +01:00
Nicolò Boschi 747f5552b6 fix test 2026-02-04 13:40:20 +01:00
Nicolò Boschi a84c50f599 fix test 2026-02-04 13:25:04 +01:00
Nicolò Boschi dc516af780 improvemnts 2026-02-04 12:55:30 +01:00
Nicolò Boschi 32598810a3 ci 2026-02-04 12:03:05 +01:00
Nicolò Boschi 87318bad38 improve 2026-02-04 11:43:30 +01:00
Nicolò Boschi 6656502af3 fixes 2026-02-04 11:29:19 +01:00
Nicolò Boschi a2095b9450 feat: HindsightEmbedded python SDK 2026-02-04 11:02:00 +01:00
Nicolò Boschi bf371ab63e feat: HindsightEmbedded python SDK 2026-02-04 10:50:42 +01:00
19 changed files with 2083 additions and 607 deletions
+48 -31
View File
@@ -9,37 +9,6 @@ concurrency:
cancel-in-progress: true
jobs:
build-python-packages:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- name: hindsight-all
path: hindsight
- name: hindsight-api
path: hindsight-api
- name: hindsight-client
path: hindsight-clients/python
- name: hindsight-embed
path: hindsight-embed
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: ".python-version"
- name: Build ${{ matrix.name }}
working-directory: ./${{ matrix.path }}
run: uv build
build-api-python-versions:
runs-on: ubuntu-latest
strategy:
@@ -790,6 +759,54 @@ jobs:
working-directory: ./hindsight-embed
run: ./test.sh
test-hindsight-all:
runs-on: ubuntu-latest
env:
HINDSIGHT_API_LLM_PROVIDER: groq
HINDSIGHT_API_LLM_API_KEY: ${{ secrets.GROQ_API_KEY }}
HINDSIGHT_API_LLM_MODEL: openai/gpt-oss-20b
# For test_server_integration.py compatibility
HINDSIGHT_LLM_PROVIDER: groq
HINDSIGHT_LLM_API_KEY: ${{ secrets.GROQ_API_KEY }}
HINDSIGHT_LLM_MODEL: openai/gpt-oss-20b
# Prefer CPU-only PyTorch in CI
UV_INDEX: pytorch=https://download.pytorch.org/whl/cpu
steps:
- uses: actions/checkout@v4
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
prune-cache: false
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version-file: ".python-version"
- name: Build hindsight-all
working-directory: ./hindsight
run: uv build
- name: Install dependencies
working-directory: ./hindsight
run: uv sync --frozen --extra test --index-strategy unsafe-best-match
- name: Cache HuggingFace models
uses: actions/cache@v4
with:
path: ~/.cache/huggingface
key: ${{ runner.os }}-huggingface-all-${{ hashFiles('hindsight/pyproject.toml') }}
restore-keys: |
${{ runner.os }}-huggingface-all-
${{ runner.os }}-huggingface-
- name: Run unit tests
working-directory: ./hindsight
run: uv run pytest tests/ -v
test-doc-examples:
runs-on: ubuntu-latest
needs: test-rust-cli
-15
View File
@@ -230,21 +230,6 @@ rm ~/.hindsight/embed
hindsight-embed configure
```
## Advanced Configuration
While `hindsight-embed` aims to be zero-config, you can customize the underlying API behavior by setting `HINDSIGHT_API_*` variables in `~/.hindsight/embed`:
```bash
# Example: Custom embedding model
HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-large
# Example: Verbose extraction
HINDSIGHT_API_RETAIN_EXTRACTION_MODE=verbose
```
See [Configuration](/developer/configuration) for all available `HINDSIGHT_API_*` options.
## When to Use
**Perfect for:**
+104 -2
View File
@@ -85,17 +85,119 @@ print(answer.text)
</TabItem>
</Tabs>
## Embedded Client (Easiest Option)
`HindsightEmbedded` provides the simplest way to use Hindsight in Python. It automatically manages a background server for you - no manual setup required:
```python
from hindsight import HindsightEmbedded
import os
# Server starts automatically on first use
client = HindsightEmbedded(
profile="myapp", # Profile for data isolation
llm_provider="openai",
llm_model="gpt-4o-mini",
llm_api_key=os.environ["OPENAI_API_KEY"],
)
# Use immediately - no manual server management needed
client.retain(bank_id="my-bank", content="Alice works at Google")
results = client.recall(bank_id="my-bank", query="What does Alice do?")
# Server continues running (auto-stops after idle timeout)
# Or explicitly stop it:
client.close(stop_daemon=True)
```
**What's a Profile?**
A profile is an isolated Hindsight environment. Each profile gets its own PostgreSQL database (stored in `~/.pg0/instances/hindsight-embed-{profile}/`) and its own API server. Use different profiles to separate environments (dev/prod), applications, or users.
**When to Use HindsightEmbedded**
Use `HindsightEmbedded` when you want the server to start automatically and manage itself. Use `HindsightServer` when you need explicit control over server lifecycle (e.g., testing where you want immediate startup/shutdown).
**Advanced Operations**
`HindsightEmbedded` provides organized API namespaces for advanced operations. Each method call automatically ensures the daemon is running:
```python
from hindsight import HindsightEmbedded
import os
embedded = HindsightEmbedded(
profile="myapp",
llm_provider="openai",
llm_api_key=os.environ["OPENAI_API_KEY"],
)
# Core operations (automatically proxied)
embedded.retain(bank_id="test", content="Hello")
results = embedded.recall(bank_id="test", query="Hello")
# Bank management
embedded.banks.create(bank_id="test", name="Test Bank", mission="Help users")
embedded.banks.set_mission(bank_id="test", mission="Updated mission")
embedded.banks.delete(bank_id="test")
# Mental models
embedded.mental_models.create(
bank_id="test",
name="User Preferences",
content="User prefers dark mode"
)
models = embedded.mental_models.list(bank_id="test")
embedded.mental_models.update(bank_id="test", mental_model_id="...", content="New content")
# Directives
embedded.directives.create(
bank_id="test",
name="Response Style",
content="Be concise and friendly"
)
directives = embedded.directives.list(bank_id="test")
# List memories
memories = embedded.memories.list(bank_id="test", type="world", limit=50)
```
**Why Use API Namespaces?**
API namespaces (`banks`, `mental_models`, `directives`, `memories`) ensure the daemon is running before each call. This handles daemon crashes gracefully:
```python
# ✅ GOOD - Uses API namespace (daemon restarts handled)
embedded.banks.create(bank_id="test", name="Test")
# ❌ BAD - Direct client access (daemon crashes NOT handled)
client = embedded.client
client.create_bank(bank_id="test", name="Test") # Fails if daemon crashed
```
## Client Initialization
```python
from hindsight_client import Hindsight
from hindsight import HindsightClient
client = Hindsight(
client = HindsightClient(
base_url="http://localhost:8888", # Hindsight API URL
timeout=30.0, # Request timeout in seconds
)
# Core operations
client.retain(bank_id="test", content="Hello world")
results = client.recall(bank_id="test", query="Hello")
# Organized API access (same as HindsightEmbedded)
client.banks.create(bank_id="test", name="Test Bank")
models = client.mental_models.list(bank_id="test")
directives = client.directives.list(bank_id="test")
memories = client.memories.list(bank_id="test")
```
Both `HindsightClient` and `HindsightEmbedded` provide the same organized API namespaces (`banks`, `mental_models`, `directives`, `memories`) for consistent developer experience.
## Core Operations
### Retain (Store Memory)
@@ -1,3 +1,16 @@
"""Hindsight embedded CLI - local memory operations without a server."""
from .daemon_embed_manager import DaemonEmbedManager
from .embed_manager import EmbedManager
__version__ = "0.4.8"
__all__ = [
"EmbedManager",
"DaemonEmbedManager",
]
def get_embed_manager() -> EmbedManager:
"""Get the default embed manager instance."""
return DaemonEmbedManager()
+8 -8
View File
@@ -30,6 +30,8 @@ import os
import sys
from pathlib import Path
from . import get_embed_manager
CONFIG_DIR = Path.home() / ".hindsight"
CONFIG_FILE = CONFIG_DIR / "embed"
CONFIG_FILE_ALT = CONFIG_DIR / "config.env" # Alternative config file location
@@ -424,7 +426,7 @@ def _do_configure_interactive(profile_name: str | None = None, port: int | None
from . import daemon_client
daemon_profile = profile_name if profile_name else None
if daemon_client._is_daemon_running(daemon_profile):
if daemon_client.is_daemon_running(daemon_profile):
print("\n \033[2mRestarting daemon with new configuration...\033[0m")
daemon_client.stop_daemon(daemon_profile)
@@ -479,7 +481,7 @@ def do_daemon(args, config: dict, logger):
console = Console()
if daemon_client._is_daemon_running(profile):
if daemon_client.is_daemon_running(profile):
# Build title with profile and port
if profile:
already_running_title = (
@@ -516,7 +518,7 @@ def do_daemon(args, config: dict, logger):
console = Console()
if not daemon_client._is_daemon_running(profile):
if not daemon_client.is_daemon_running(profile):
# Build title for not running status
if profile:
not_running_title = f"[bold]Daemon Status[/bold] [dim]({profile})[/dim]"
@@ -559,7 +561,6 @@ def do_daemon(args, config: dict, logger):
elif args.daemon_command == "status":
import os
import re
from pathlib import Path
from rich.console import Console
@@ -568,7 +569,7 @@ def do_daemon(args, config: dict, logger):
console = Console()
if daemon_client._is_daemon_running(profile):
if daemon_client.is_daemon_running(profile):
status_text = Text()
status_text.append("Daemon is running\n\n", style="green bold")
status_text.append(" URL: ", style="dim")
@@ -579,9 +580,8 @@ def do_daemon(args, config: dict, logger):
# Check if using pg0 and show database location
database_url = os.getenv("HINDSIGHT_EMBED_API_DATABASE_URL")
if not database_url:
# Default: use profile-specific pg0
safe_profile = re.sub(r"[^a-zA-Z0-9_-]", "-", profile or "default")
database_url = f"pg0://hindsight-embed-{safe_profile}"
# Default: use profile-specific pg0 (shared utility ensures consistency)
database_url = get_embed_manager().get_database_url(profile)
if database_url.startswith("pg0://"):
pg0_name = database_url.replace("pg0://", "")
+27 -391
View File
@@ -1,38 +1,28 @@
"""
Client for communicating with the Hindsight daemon.
CLI utilities for daemon and CLI management.
Handles daemon lifecycle (start if needed) and API requests via the Python client.
This module provides CLI-specific functions for managing the daemon
and the hindsight Rust CLI binary.
"""
import logging
import os
import re
import shlex
import subprocess
import time
from pathlib import Path
import httpx # Used only for health check
from rich.console import Console
from rich.live import Live
from rich.panel import Panel
from rich.text import Text
from .daemon_embed_manager import DaemonEmbedManager
from .profile_manager import ProfileManager, resolve_active_profile
console = Console(stderr=True)
logger = logging.getLogger(__name__)
# Suppress noisy httpx logs
logging.getLogger("httpx").setLevel(logging.WARNING)
# Singleton manager instance
_manager = DaemonEmbedManager()
# Default port for default profile
DEFAULT_DAEMON_PORT = 8888
DAEMON_PORT = DEFAULT_DAEMON_PORT # Backward compatibility
DAEMON_STARTUP_TIMEOUT = 180 # seconds - needs to be long for first run (downloads dependencies)
# Default idle timeout: 5 minutes - users can override with HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT env var
DEFAULT_DAEMON_IDLE_TIMEOUT = 300
# CLI paths - check multiple locations
CLI_INSTALL_DIRS = [
Path.home() / ".local" / "bin", # Standard location from get-cli installer
Path.home() / ".hindsight" / "bin", # Alternative location
]
CLI_INSTALLER_URL = "https://hindsight.vectorize.io/get-cli"
def get_daemon_port(profile: str | None = None) -> int:
@@ -61,330 +51,9 @@ def get_daemon_url(profile: str | None = None) -> str:
Returns:
URL for daemon.
"""
port = get_daemon_port(profile)
return f"http://127.0.0.1:{port}"
# CLI paths - check multiple locations
CLI_INSTALL_DIRS = [
Path.home() / ".local" / "bin", # Standard location from get-cli installer
Path.home() / ".hindsight" / "bin", # Alternative location
]
CLI_INSTALLER_URL = "https://hindsight.vectorize.io/get-cli"
def _find_hindsight_api_command() -> list[str]:
"""Find the command to run hindsight-api."""
# Check if we're in development mode (local hindsight-api available)
# Path: daemon_client.py -> hindsight_embed/ -> hindsight-embed/ -> memory-poc/
dev_api_path = Path(__file__).parent.parent.parent / "hindsight-api"
if dev_api_path.exists() and (dev_api_path / "pyproject.toml").exists():
# Use uv run with the local project
return ["uv", "run", "--project", str(dev_api_path), "hindsight-api"]
# Fall back to uvx for installed version
# Allow version override via environment variable (defaults to matching embed version)
from . import __version__
api_version = os.getenv("HINDSIGHT_EMBED_API_VERSION", __version__)
return ["uvx", f"hindsight-api@{api_version}"]
def _is_daemon_running(profile: str | None = None) -> bool:
"""Check if daemon is running and responsive.
Args:
profile: Profile name (None = resolve from priority).
Returns:
True if daemon is running and responsive.
"""
daemon_url = get_daemon_url(profile)
try:
with httpx.Client(timeout=2) as client:
response = client.get(f"{daemon_url}/health")
return response.status_code == 200
except Exception:
return False
def _start_daemon(config: dict, profile: str | None = None) -> bool:
"""
Start the daemon in background.
Args:
config: Configuration dict with LLM settings.
profile: Profile name (None = resolve from priority).
Returns:
True if daemon started successfully.
"""
import sys
if profile is None:
profile = resolve_active_profile()
# Get profile-specific paths
pm = ProfileManager()
paths = pm.resolve_profile_paths(profile)
profile_label = f"profile '{profile}'" if profile else "default profile"
daemon_log = paths.log
port = paths.port
# Build environment with LLM config
env = os.environ.copy()
if config.get("llm_api_key"):
env["HINDSIGHT_API_LLM_API_KEY"] = config["llm_api_key"]
if config.get("llm_provider"):
env["HINDSIGHT_API_LLM_PROVIDER"] = config["llm_provider"]
if config.get("llm_model"):
env["HINDSIGHT_API_LLM_MODEL"] = config["llm_model"]
# Use profile-specific pg0 database for isolation
# Allow override via HINDSIGHT_EMBED_API_DATABASE_URL for external PostgreSQL
# (e.g. when running as root where embedded pg0 cannot use initdb)
if "HINDSIGHT_EMBED_API_DATABASE_URL" not in env:
# Sanitize profile name for use in database name (allow only alphanumeric, dash, underscore)
safe_profile = re.sub(r"[^a-zA-Z0-9_-]", "-", profile or "default")
env["HINDSIGHT_API_DATABASE_URL"] = f"pg0://hindsight-embed-{safe_profile}"
else:
# Pass through the embed-specific env var to the daemon as the standard API env var
env["HINDSIGHT_API_DATABASE_URL"] = env["HINDSIGHT_EMBED_API_DATABASE_URL"]
# Store database URL for display later
database_url = env["HINDSIGHT_API_DATABASE_URL"]
is_pg0 = database_url.startswith("pg0://")
env["HINDSIGHT_API_LOG_LEVEL"] = "info"
# On macOS, force CPU for embeddings/reranker to avoid MPS/Metal/XPC issues in daemon mode
# Only set if not already configured by user
import platform
if platform.system() == "Darwin":
if "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU" not in env:
env["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1"
if "HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU" not in env:
env["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1"
# Get idle timeout from environment or use default
idle_timeout = int(os.getenv("HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT", str(DEFAULT_DAEMON_IDLE_TIMEOUT)))
# Use profile-specific log file
daemon_log.parent.mkdir(parents=True, exist_ok=True)
# Tell hindsight-api daemon where to write its logs
env["HINDSIGHT_API_DAEMON_LOG"] = str(daemon_log)
# Pass profile-specific port (no lockfile - we use port-based discovery)
cmd = _find_hindsight_api_command() + [
"--daemon",
"--idle-timeout",
str(idle_timeout),
"--port",
str(paths.port),
]
try:
# Start daemon directly (hindsight-api handles its own log redirection via HINDSIGHT_API_DAEMON_LOG)
subprocess.Popen(
cmd,
env=env,
start_new_session=True,
)
# Wait for daemon to be ready
# Note: With --daemon flag, the parent process forks and exits immediately (code 0).
# The child process (actual daemon) continues running. So we can't rely on process.poll()
# to detect failures - we must use the health check.
start_time = time.time()
last_check_time = start_time
last_log_position = 0 # Track position in log file for tailing
log_lines = [f"Starting daemon for {profile_label}...", ""] # Accumulate log lines for display
# Build title with profile and port info
if profile:
title = f"[bold cyan]Starting Daemon[/bold cyan] [dim]({profile} @ :{port})[/dim]"
else:
title = f"[bold cyan]Starting Daemon[/bold cyan] [dim](:{port})[/dim]"
# Use Rich Live display for beautiful real-time updates
with Live(console=console, auto_refresh=False) as live:
# Show initial panel
content = Text("\n".join(log_lines), style="dim")
panel = Panel(
content,
title=title,
border_style="cyan",
padding=(1, 2),
)
live.update(panel)
live.refresh()
while time.time() - start_time < DAEMON_STARTUP_TIMEOUT:
# Tail daemon logs if available
if daemon_log.exists():
try:
with open(daemon_log, "r") as f:
f.seek(last_log_position)
new_lines = f.readlines()
last_log_position = f.tell()
# Add new log lines (keep last 4 for display)
for line in new_lines:
line = line.rstrip()
if line:
log_lines.append(line)
# Keep only last 4 lines
log_lines = log_lines[-4:]
except Exception:
pass # Silently ignore log read errors
if _is_daemon_running(profile):
# Health check passed - but daemon might crash during initialization
# Add status message to logs
log_lines.append("")
log_lines.append("✓ Daemon responding, verifying stability...")
# Update display with success status
content = Text("\n".join(log_lines), style="dim")
panel = Panel(
content,
title=title,
border_style="cyan",
padding=(1, 2),
)
live.update(panel)
live.refresh()
time.sleep(2)
if _is_daemon_running(profile):
log_lines.append("✓ Daemon started successfully!")
log_lines.append("")
log_lines.append(f"Logs: {daemon_log}")
# Show pg0 location if using pg0
if is_pg0:
# pg0 stores data in ~/.pg0/instances/<database_name>
pg0_name = database_url.replace("pg0://", "")
pg0_path = Path.home() / ".pg0" / "instances" / pg0_name
log_lines.append(f"Database: {pg0_path}")
content = Text("\n".join(log_lines), style="dim")
# Build success title with profile and port
if profile:
success_title = (
f"[bold green]✓ Daemon Started[/bold green] [dim]({profile} @ :{port})[/dim]"
)
else:
success_title = f"[bold green]✓ Daemon Started[/bold green] [dim](:{port})[/dim]"
panel = Panel(
content,
title=success_title,
border_style="green",
padding=(1, 2),
)
live.update(panel)
live.refresh()
console.print() # Add newline after panel
return True
else:
# Daemon crashed after initial health check
log_lines.append("")
log_lines.append("✗ Daemon crashed during initialization")
content = Text("\n".join(log_lines), style="dim")
# Build failure title with profile and port
if profile:
fail_title = f"[bold red]✗ Daemon Failed[/bold red] [dim]({profile} @ :{port})[/dim]"
else:
fail_title = f"[bold red]✗ Daemon Failed[/bold red] [dim](:{port})[/dim]"
panel = Panel(
content,
title=fail_title,
border_style="red",
padding=(1, 2),
)
live.update(panel)
live.refresh()
console.print()
break
# Periodically log progress
if time.time() - last_check_time > 3:
elapsed = int(time.time() - start_time)
# Update last status line or add new one
status_msg = f"⏳ Waiting for daemon... ({elapsed}s elapsed)"
if log_lines and log_lines[-1].startswith(""):
log_lines[-1] = status_msg
else:
log_lines.append(status_msg)
last_check_time = time.time()
# Update the live display
content = Text("\n".join(log_lines), style="dim")
panel = Panel(
content,
title=title,
border_style="cyan",
padding=(1, 2),
)
live.update(panel)
live.refresh()
time.sleep(0.5)
# Timeout - show failure
log_lines.append("")
log_lines.append("✗ Daemon failed to start (timeout)")
log_lines.append("")
log_lines.append(f"See full log: {daemon_log}")
content = Text("\n".join(log_lines), style="dim")
# Build timeout title with profile and port
if profile:
timeout_title = f"[bold red]✗ Daemon Failed (Timeout)[/bold red] [dim]({profile} @ :{port})[/dim]"
else:
timeout_title = f"[bold red]✗ Daemon Failed (Timeout)[/bold red] [dim](:{port})[/dim]"
panel = Panel(
content,
title=timeout_title,
border_style="red",
padding=(1, 2),
)
console.print(panel)
console.print()
return False
except FileNotFoundError as e:
error_msg = f"Command not found: {cmd[0]}\nFull command: {' '.join(cmd)}\n\nInstall hindsight-api with: pip install hindsight-api"
error_panel = Panel(
Text(error_msg, style="red"),
title="[bold red]✗ Command Not Found[/bold red]",
border_style="red",
padding=(1, 2),
)
console.print(error_panel)
console.print()
return False
except Exception as e:
error_msg = f"Failed to start daemon: {e}\n\nCommand: {' '.join(cmd)}\nLog file: {daemon_log}"
error_panel = Panel(
Text(error_msg, style="red"),
title="[bold red]✗ Startup Error[/bold red]",
border_style="red",
padding=(1, 2),
)
console.print(error_panel)
console.print()
return False
return _manager.get_url(profile)
def ensure_daemon_running(config: dict, profile: str | None = None) -> bool:
@@ -392,7 +61,8 @@ def ensure_daemon_running(config: dict, profile: str | None = None) -> bool:
Ensure daemon is running, starting it if needed.
Args:
config: Configuration dict with LLM settings.
config: Configuration dict with LLM settings (accepts both simple keys
like "llm_api_key" and env var format like "HINDSIGHT_API_LLM_API_KEY").
profile: Profile name (None = resolve from priority).
Returns:
@@ -401,11 +71,7 @@ def ensure_daemon_running(config: dict, profile: str | None = None) -> bool:
if profile is None:
profile = resolve_active_profile()
if _is_daemon_running(profile):
logger.debug(f"Daemon already running for profile '{profile or 'default'}'")
return True
return _start_daemon(config, profile)
return _manager.ensure_running(config, profile)
def stop_daemon(profile: str | None = None) -> bool:
@@ -417,55 +83,25 @@ def stop_daemon(profile: str | None = None) -> bool:
Returns:
True if daemon stopped successfully.
"""
import subprocess
if profile is None:
profile = resolve_active_profile()
# Check if daemon is actually running via health check
if not _is_daemon_running(profile):
logger.debug(f"Daemon not running for profile '{profile or 'default'}'")
return True
return _manager.stop(profile)
# Get profile-specific port
pm = ProfileManager()
paths = pm.resolve_profile_paths(profile)
port = paths.port
# Find PID by port using lsof (works on macOS/Linux, handles stale lockfiles)
try:
result = subprocess.run(
["lsof", "-ti", f":{port}", "-sTCP:LISTEN"],
capture_output=True,
text=True,
timeout=5,
)
if result.returncode == 0 and result.stdout.strip():
pid = int(result.stdout.strip().split()[0])
logger.debug(f"Found daemon PID {pid} on port {port}")
def is_daemon_running(profile: str | None = None) -> bool:
"""Check if daemon is running for a profile.
# Send SIGTERM
os.kill(pid, 15)
Args:
profile: Profile name (None = resolve from priority).
# Wait for process to exit
for _ in range(50):
time.sleep(0.1)
try:
os.kill(pid, 0)
except OSError:
break # Process exited
else:
logger.warning(f"Could not find PID for port {port}")
except (subprocess.TimeoutExpired, ValueError, OSError, FileNotFoundError) as e:
logger.warning(f"Could not find/kill daemon by port: {e}")
Returns:
True if daemon is running and responsive.
"""
if profile is None:
profile = resolve_active_profile()
# Wait for health check to fail (daemon fully stopped)
for _ in range(30): # Wait up to 3 seconds
if not _is_daemon_running(profile):
return True
time.sleep(0.1)
return not _is_daemon_running(profile)
return _manager.is_running(profile)
def find_cli_binary() -> Path | None:
@@ -0,0 +1,370 @@
"""
Concrete implementation of EmbedManager using daemon-based architecture.
This module provides the production implementation of the embed management interface,
consolidating daemon lifecycle, profile management, and database URL resolution.
"""
import logging
import os
import re
import subprocess
import time
from pathlib import Path
from typing import Optional
import httpx
from rich.console import Console
from rich.live import Live
from rich.panel import Panel
from rich.text import Text
from .embed_manager import EmbedManager
from .profile_manager import ProfileManager, resolve_active_profile
logger = logging.getLogger(__name__)
console = Console(stderr=True)
# Suppress noisy httpx logs
logging.getLogger("httpx").setLevel(logging.WARNING)
# Constants
DAEMON_STARTUP_TIMEOUT = 180 # seconds
DEFAULT_DAEMON_IDLE_TIMEOUT = 300 # 5 minutes
class DaemonEmbedManager(EmbedManager):
"""Production embed manager using daemon-based architecture with profile isolation."""
def __init__(self):
"""Initialize the daemon embed manager."""
self._profile_manager = ProfileManager()
def _sanitize_profile_name(self, profile: str | None) -> str:
"""Sanitize profile name for use in database names and file paths."""
if profile is None:
return "default"
return re.sub(r"[^a-zA-Z0-9_-]", "-", profile)
def get_database_url(self, profile: str, db_url: Optional[str] = None) -> str:
"""
Get the database URL for this profile.
Args:
profile: Profile name
db_url: Optional override database URL
Returns:
Database connection string
"""
if db_url and db_url != "pg0":
return db_url
safe_profile = self._sanitize_profile_name(profile)
return f"pg0://hindsight-embed-{safe_profile}"
def get_url(self, profile: str) -> str:
"""
Get the URL for the daemon serving this profile.
Args:
profile: Profile name
Returns:
URL string (e.g., "http://127.0.0.1:54321")
Raises:
RuntimeError: If daemon is not running
"""
paths = self._profile_manager.resolve_profile_paths(profile)
return f"http://127.0.0.1:{paths.port}"
def is_running(self, profile: str) -> bool:
"""Check if daemon is running and responsive."""
daemon_url = self.get_url(profile)
try:
with httpx.Client(timeout=2) as client:
response = client.get(f"{daemon_url}/health")
return response.status_code == 200
except Exception:
return False
def _find_api_command(self) -> list[str]:
"""Find the command to run hindsight-api."""
# Check if we're in development mode
dev_api_path = Path(__file__).parent.parent.parent / "hindsight-api"
if dev_api_path.exists() and (dev_api_path / "pyproject.toml").exists():
return ["uv", "run", "--project", str(dev_api_path), "hindsight-api"]
# Fall back to uvx for installed version
from . import __version__
api_version = os.getenv("HINDSIGHT_EMBED_API_VERSION", __version__)
return ["uvx", f"hindsight-api@{api_version}"]
def _start_daemon(self, config: dict, profile: str) -> bool:
"""Start the daemon in background."""
paths = self._profile_manager.resolve_profile_paths(profile)
profile_label = f"profile '{profile}'" if profile else "default profile"
daemon_log = paths.log
port = paths.port
# Build environment with LLM config
# Support both formats: simple keys ("llm_api_key") and env var format ("HINDSIGHT_API_LLM_API_KEY")
env = os.environ.copy()
# Map of simple key -> env var key
key_mapping = {
"llm_api_key": "HINDSIGHT_API_LLM_API_KEY",
"llm_provider": "HINDSIGHT_API_LLM_PROVIDER",
"llm_model": "HINDSIGHT_API_LLM_MODEL",
"llm_base_url": "HINDSIGHT_API_LLM_BASE_URL",
"log_level": "HINDSIGHT_API_LOG_LEVEL",
"idle_timeout": "HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT",
}
for simple_key, env_key in key_mapping.items():
# Check both simple format and env var format
value = config.get(simple_key) or config.get(env_key)
if value:
env[env_key] = str(value)
# Use profile-specific database (check config for override)
db_override = config.get("HINDSIGHT_EMBED_API_DATABASE_URL") or env.get("HINDSIGHT_EMBED_API_DATABASE_URL")
if db_override:
env["HINDSIGHT_API_DATABASE_URL"] = db_override
else:
env["HINDSIGHT_API_DATABASE_URL"] = self.get_database_url(profile)
database_url = env["HINDSIGHT_API_DATABASE_URL"]
is_pg0 = database_url.startswith("pg0://")
# Set defaults if not provided
if "HINDSIGHT_API_LOG_LEVEL" not in env:
env["HINDSIGHT_API_LOG_LEVEL"] = "info"
if "HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT" not in env:
env["HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT"] = str(DEFAULT_DAEMON_IDLE_TIMEOUT)
# On macOS, force CPU for embeddings/reranker to avoid MPS issues
import platform
if platform.system() == "Darwin":
if "HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU" not in env:
env["HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU"] = "1"
if "HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU" not in env:
env["HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU"] = "1"
# Get idle timeout from env
idle_timeout = int(env.get("HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT", str(DEFAULT_DAEMON_IDLE_TIMEOUT)))
# Create log directory
daemon_log.parent.mkdir(parents=True, exist_ok=True)
env["HINDSIGHT_API_DAEMON_LOG"] = str(daemon_log)
# Build command
cmd = self._find_api_command() + [
"--daemon",
"--idle-timeout",
str(idle_timeout),
"--port",
str(port),
]
try:
# Start daemon
subprocess.Popen(
cmd,
env=env,
start_new_session=True,
)
# Wait for daemon to be ready with rich UI
start_time = time.time()
last_check_time = start_time
last_log_position = 0
log_lines = [f"Starting daemon for {profile_label}...", ""]
title = f"[bold cyan]Starting Daemon[/bold cyan] [dim]({profile} @ :{port})[/dim]"
with Live(console=console, auto_refresh=False) as live:
content = Text("\n".join(log_lines), style="dim")
panel = Panel(content, title=title, border_style="cyan", padding=(1, 2))
live.update(panel)
live.refresh()
while time.time() - start_time < DAEMON_STARTUP_TIMEOUT:
# Tail daemon logs
if daemon_log.exists():
try:
with open(daemon_log, "r") as f:
f.seek(last_log_position)
new_lines = f.readlines()
last_log_position = f.tell()
for line in new_lines:
line = line.rstrip()
if line:
log_lines.append(line)
log_lines = log_lines[-4:]
except Exception:
pass
if self.is_running(profile):
log_lines.append("")
log_lines.append("✓ Daemon responding, verifying stability...")
content = Text("\n".join(log_lines), style="dim")
panel = Panel(content, title=title, border_style="cyan", padding=(1, 2))
live.update(panel)
live.refresh()
time.sleep(2)
if self.is_running(profile):
log_lines.append("✓ Daemon started successfully!")
log_lines.append("")
log_lines.append(f"Logs: {daemon_log}")
if is_pg0:
pg0_name = database_url.replace("pg0://", "")
pg0_path = Path.home() / ".pg0" / "instances" / pg0_name
log_lines.append(f"Database: {pg0_path}")
content = Text("\n".join(log_lines), style="dim")
success_title = (
f"[bold green]✓ Daemon Started[/bold green] [dim]({profile} @ :{port})[/dim]"
)
panel = Panel(content, title=success_title, border_style="green", padding=(1, 2))
live.update(panel)
live.refresh()
console.print()
return True
else:
log_lines.append("")
log_lines.append("✗ Daemon crashed during initialization")
content = Text("\n".join(log_lines), style="dim")
fail_title = f"[bold red]✗ Daemon Failed[/bold red] [dim]({profile} @ :{port})[/dim]"
panel = Panel(content, title=fail_title, border_style="red", padding=(1, 2))
live.update(panel)
live.refresh()
console.print()
break
# Periodic progress
if time.time() - last_check_time > 3:
elapsed = int(time.time() - start_time)
status_msg = f"⏳ Waiting for daemon... ({elapsed}s elapsed)"
if log_lines and log_lines[-1].startswith(""):
log_lines[-1] = status_msg
else:
log_lines.append(status_msg)
last_check_time = time.time()
content = Text("\n".join(log_lines), style="dim")
panel = Panel(content, title=title, border_style="cyan", padding=(1, 2))
live.update(panel)
live.refresh()
time.sleep(0.5)
# Timeout
log_lines.append("")
log_lines.append("✗ Daemon failed to start (timeout)")
log_lines.append("")
log_lines.append(f"See full log: {daemon_log}")
content = Text("\n".join(log_lines), style="dim")
timeout_title = f"[bold red]✗ Daemon Failed (Timeout)[/bold red] [dim]({profile} @ :{port})[/dim]"
panel = Panel(content, title=timeout_title, border_style="red", padding=(1, 2))
console.print(panel)
console.print()
return False
except FileNotFoundError as e:
error_msg = (
f"Command not found: {cmd[0]}\nFull command: {' '.join(cmd)}\n\n"
"Install hindsight-api with: pip install hindsight-api"
)
error_panel = Panel(
Text(error_msg, style="red"),
title="[bold red]✗ Command Not Found[/bold red]",
border_style="red",
padding=(1, 2),
)
console.print(error_panel)
console.print()
return False
except Exception as e:
error_msg = f"Failed to start daemon: {e}\n\nCommand: {' '.join(cmd)}\nLog file: {daemon_log}"
error_panel = Panel(
Text(error_msg, style="red"),
title="[bold red]✗ Startup Error[/bold red]",
border_style="red",
padding=(1, 2),
)
console.print(error_panel)
console.print()
return False
def ensure_running(self, config: dict, profile: str) -> bool:
"""
Ensure daemon is running, starting it if needed.
Args:
config: Environment configuration dict (HINDSIGHT_API_* vars)
profile: Profile name for isolation
Returns:
True if daemon is running (started or already running), False on failure
"""
if self.is_running(profile):
logger.debug(f"Daemon already running for profile '{profile}'")
return True
return self._start_daemon(config, profile)
def stop(self, profile: str) -> bool:
"""
Stop the daemon for this profile.
Args:
profile: Profile name
Returns:
True if stopped successfully, False otherwise
"""
if not self.is_running(profile):
logger.debug(f"Daemon not running for profile '{profile}'")
return True
# Get port
paths = self._profile_manager.resolve_profile_paths(profile)
port = paths.port
# Find PID by port
try:
result = subprocess.run(
["lsof", "-ti", f":{port}", "-sTCP:LISTEN"],
capture_output=True,
text=True,
timeout=5,
)
if result.returncode == 0 and result.stdout.strip():
pid = int(result.stdout.strip().split()[0])
logger.debug(f"Found daemon PID {pid} on port {port}")
# Send SIGTERM
os.kill(pid, 15)
# Wait for process to exit
for _ in range(50):
time.sleep(0.1)
try:
os.kill(pid, 0)
except OSError:
break
else:
logger.warning(f"Could not find PID for port {port}")
except (subprocess.TimeoutExpired, ValueError, OSError, FileNotFoundError) as e:
logger.warning(f"Could not find/kill daemon by port: {e}")
# Wait for health check to fail
for _ in range(30):
if not self.is_running(profile):
return True
time.sleep(0.1)
return not self.is_running(profile)
@@ -0,0 +1,83 @@
"""
Abstract interface for managing Hindsight embedded servers and profiles.
This module provides a clean interface for daemon lifecycle and profile management,
abstracting away the implementation details.
"""
from abc import ABC, abstractmethod
from typing import Optional
class EmbedManager(ABC):
"""Abstract interface for managing Hindsight embedded servers and profiles."""
@abstractmethod
def ensure_running(self, config: dict, profile: str) -> bool:
"""
Ensure daemon is running for the given profile with config.
Args:
config: Environment configuration dict (HINDSIGHT_API_* vars)
profile: Profile name for isolation
Returns:
True if daemon is running (started or already running), False on failure
"""
pass
@abstractmethod
def get_url(self, profile: str) -> str:
"""
Get the URL for the daemon serving this profile.
Args:
profile: Profile name
Returns:
URL string (e.g., "http://127.0.0.1:54321")
Raises:
RuntimeError: If daemon is not running
"""
pass
@abstractmethod
def stop(self, profile: str) -> bool:
"""
Stop the daemon for this profile.
Args:
profile: Profile name
Returns:
True if stopped successfully, False otherwise
"""
pass
@abstractmethod
def is_running(self, profile: str) -> bool:
"""
Check if daemon is running for this profile.
Args:
profile: Profile name
Returns:
True if daemon is running and responsive
"""
pass
@abstractmethod
def get_database_url(self, profile: str, db_url: Optional[str] = None) -> str:
"""
Get the database URL for this profile.
Args:
profile: Profile name
db_url: Optional override database URL
Returns:
Database connection string
"""
pass
-104
View File
@@ -9,7 +9,6 @@ import pytest
from hindsight_embed import daemon_client
@pytest.fixture
def config():
"""Default config for tests."""
@@ -20,7 +19,6 @@ def config():
"bank_id": "test-bank",
}
@pytest.fixture
def mock_cli_binary(tmp_path):
"""Create a mock CLI binary."""
@@ -29,7 +27,6 @@ def mock_cli_binary(tmp_path):
cli_path.chmod(0o755)
return cli_path
class TestRunCli:
"""Tests for run_cli function."""
@@ -181,104 +178,3 @@ class TestRunCli:
# Verify exit code
assert exit_code == 0
class TestStartDaemon:
"""Tests for _start_daemon function."""
def test_start_daemon_respects_database_url_env(self, config, monkeypatch):
"""Test that HINDSIGHT_EMBED_API_DATABASE_URL is respected if already set."""
custom_db_url = "postgresql://custom:password@localhost:5432/custom_db"
monkeypatch.setenv("HINDSIGHT_EMBED_API_DATABASE_URL", custom_db_url)
# Mock subprocess.Popen to capture the env
captured_env = {}
def mock_popen(*args, **kwargs):
captured_env.update(kwargs.get("env", {}))
# Return a mock process
mock_proc = MagicMock()
mock_proc.poll.return_value = None
return mock_proc
# Reduce timeout to 0.1s to avoid waiting
monkeypatch.setattr(daemon_client, "DAEMON_STARTUP_TIMEOUT", 0.1)
# Mock daemon health check to fail immediately (so we don't wait for startup)
mock_is_running = Mock(return_value=False)
with (
patch("subprocess.Popen", side_effect=mock_popen),
patch.object(daemon_client, "_is_daemon_running", mock_is_running),
patch.object(daemon_client, "_find_hindsight_api_command", return_value=["fake-cmd"]),
):
# Start daemon (will fail health check, but we just want to verify env)
daemon_client._start_daemon(config)
# Verify the custom database URL was NOT overwritten
assert captured_env.get("HINDSIGHT_API_DATABASE_URL") == custom_db_url
def test_start_daemon_sets_default_database_url(self, config, monkeypatch):
"""Test that default database URL is set if not already in env."""
# Ensure HINDSIGHT_EMBED_API_DATABASE_URL is not set
monkeypatch.delenv("HINDSIGHT_EMBED_API_DATABASE_URL", raising=False)
# Mock subprocess.Popen to capture the env
captured_env = {}
def mock_popen(*args, **kwargs):
captured_env.update(kwargs.get("env", {}))
# Return a mock process
mock_proc = MagicMock()
mock_proc.poll.return_value = None
return mock_proc
# Reduce timeout to 0.1s to avoid waiting
monkeypatch.setattr(daemon_client, "DAEMON_STARTUP_TIMEOUT", 0.1)
# Mock daemon health check to fail immediately (so we don't wait for startup)
mock_is_running = Mock(return_value=False)
with (
patch("subprocess.Popen", side_effect=mock_popen),
patch.object(daemon_client, "_is_daemon_running", mock_is_running),
patch.object(daemon_client, "_find_hindsight_api_command", return_value=["fake-cmd"]),
):
# Start daemon (will fail health check, but we just want to verify env)
daemon_client._start_daemon(config)
# Verify the default database URL was set (profile-specific)
assert captured_env.get("HINDSIGHT_API_DATABASE_URL") == "pg0://hindsight-embed-default"
class TestIsDaemonRunning:
"""Tests for _is_daemon_running function."""
def test_daemon_running_returns_true_on_200(self):
"""Test that daemon is considered running when health check returns 200."""
mock_response = Mock()
mock_response.status_code = 200
mock_client = MagicMock()
mock_client.__enter__.return_value.get.return_value = mock_response
with patch("httpx.Client", return_value=mock_client):
assert daemon_client._is_daemon_running() is True
def test_daemon_not_running_returns_false_on_error(self):
"""Test that daemon is considered not running when health check fails."""
mock_client = MagicMock()
mock_client.__enter__.return_value.get.side_effect = Exception("Connection refused")
with patch("httpx.Client", return_value=mock_client):
assert daemon_client._is_daemon_running() is False
def test_daemon_not_running_returns_false_on_non_200(self):
"""Test that daemon is considered not running when health check returns non-200."""
mock_response = Mock()
mock_response.status_code = 500
mock_client = MagicMock()
mock_client.__enter__.return_value.get.return_value = mock_response
with patch("httpx.Client", return_value=mock_client):
assert daemon_client._is_daemon_running() is False
@@ -0,0 +1,53 @@
"""Tests for EmbedManager interface."""
from hindsight_embed import get_embed_manager
def test_sanitize_profile_name_via_db_url():
"""Test profile name sanitization through database URL generation."""
manager = get_embed_manager()
# Test None defaults to "default"
assert manager.get_database_url(None) == "pg0://hindsight-embed-default"
# Test simple alphanumeric names
assert manager.get_database_url("myapp") == "pg0://hindsight-embed-myapp"
assert manager.get_database_url("my-app") == "pg0://hindsight-embed-my-app"
assert manager.get_database_url("my_app") == "pg0://hindsight-embed-my_app"
assert manager.get_database_url("app123") == "pg0://hindsight-embed-app123"
# Test special characters get replaced with dashes
assert manager.get_database_url("my app") == "pg0://hindsight-embed-my-app"
assert manager.get_database_url("my.app") == "pg0://hindsight-embed-my-app"
assert manager.get_database_url("my@app!") == "pg0://hindsight-embed-my-app-"
assert manager.get_database_url("My App 2.0!") == "pg0://hindsight-embed-My-App-2-0-"
def test_get_database_url_default():
"""Test database URL generation with default pg0."""
manager = get_embed_manager()
assert manager.get_database_url("myapp") == "pg0://hindsight-embed-myapp"
assert manager.get_database_url("myapp", None) == "pg0://hindsight-embed-myapp"
assert manager.get_database_url("myapp", "pg0") == "pg0://hindsight-embed-myapp"
def test_get_database_url_custom():
"""Test database URL generation with custom database."""
manager = get_embed_manager()
custom_url = "postgresql://user:pass@localhost/db"
assert manager.get_database_url("myapp", custom_url) == custom_url
assert manager.get_database_url("any-profile", custom_url) == custom_url
def test_manager_singleton():
"""Test that get_embed_manager returns functional instances."""
manager1 = get_embed_manager()
manager2 = get_embed_manager()
# They should be independent instances but same type
assert type(manager1) == type(manager2)
# They should produce the same results
assert manager1.get_database_url("test") == manager2.get_database_url("test")
+23 -7
View File
@@ -3,7 +3,23 @@ Hindsight - All-in-one semantic memory system for AI agents.
This package provides a simple way to run Hindsight locally with embedded PostgreSQL.
Example:
Easiest way - Embedded client (recommended):
```python
from hindsight import HindsightEmbedded
# Server starts automatically on first use
client = HindsightEmbedded(
profile="myapp",
llm_provider="groq",
llm_api_key="your-api-key",
)
# Use immediately - no manual server management needed
client.retain(bank_id="alice", content="Alice loves AI")
results = client.recall(bank_id="alice", query="What does Alice like?")
```
Manual server management:
```python
from hindsight import start_server, HindsightClient
@@ -18,13 +34,13 @@ Example:
client = HindsightClient(base_url=server.url)
# Store memories
client.put(agent_id="assistant", content="User prefers Python for data analysis")
client.retain(bank_id="assistant", content="User prefers Python for data analysis")
# Search memories
results = client.search(agent_id="assistant", query="programming preferences")
results = client.recall(bank_id="assistant", query="programming preferences")
# Generate contextual response
response = client.think(agent_id="assistant", query="What languages should I recommend?")
response = client.reflect(bank_id="assistant", query="What are my interests?")
# Stop server when done
server.stop()
@@ -41,13 +57,13 @@ Using context manager:
```
"""
from .client_wrapper import HindsightClient
from .embedded import HindsightEmbedded
from .server import Server as HindsightServer, start_server
# Re-export Client from hindsight-client
from hindsight_client import Hindsight as HindsightClient
__all__ = [
"HindsightServer",
"start_server",
"HindsightClient",
"HindsightEmbedded",
]
+193
View File
@@ -0,0 +1,193 @@
"""
API namespace classes for organizing client methods.
These classes provide organized access to different parts of the Hindsight API
while ensuring the daemon is running before each call.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .embedded import HindsightEmbedded
class BanksAPI:
"""Namespace for bank-related operations."""
def __init__(self, embedded: "HindsightEmbedded"):
self._embedded = embedded
def create(
self,
bank_id: str,
name: str | None = None,
mission: str | None = None,
disposition: dict[str, Any] | None = None,
):
"""Create a new bank."""
self._embedded._ensure_started()
return self._embedded._client.create_bank(
bank_id=bank_id,
name=name,
mission=mission,
disposition=disposition,
)
def delete(self, bank_id: str):
"""Delete a bank."""
self._embedded._ensure_started()
return self._embedded._client.delete_bank(bank_id=bank_id)
def set_mission(self, bank_id: str, mission: str):
"""Set or update the mission for a bank."""
self._embedded._ensure_started()
return self._embedded._client.set_mission(bank_id=bank_id, mission=mission)
def set_disposition(self, bank_id: str, disposition: dict[str, Any]):
"""Set or update the disposition for a bank."""
self._embedded._ensure_started()
return self._embedded._client.set_disposition(bank_id=bank_id, disposition=disposition)
class MentalModelsAPI:
"""Namespace for mental model operations."""
def __init__(self, embedded: "HindsightEmbedded"):
self._embedded = embedded
def create(
self,
bank_id: str,
name: str,
content: str,
tags: list[str] | None = None,
):
"""Create a new mental model."""
self._embedded._ensure_started()
return self._embedded._client.create_mental_model(
bank_id=bank_id,
name=name,
content=content,
tags=tags,
)
def list(self, bank_id: str, tags: list[str] | None = None):
"""List all mental models for a bank."""
self._embedded._ensure_started()
return self._embedded._client.list_mental_models(bank_id=bank_id, tags=tags)
def get(self, bank_id: str, mental_model_id: str):
"""Get a specific mental model."""
self._embedded._ensure_started()
return self._embedded._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
def refresh(self, bank_id: str, mental_model_id: str):
"""Refresh a mental model."""
self._embedded._ensure_started()
return self._embedded._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
def update(
self,
bank_id: str,
mental_model_id: str,
name: str | None = None,
content: str | None = None,
tags: list[str] | None = None,
):
"""Update a mental model."""
self._embedded._ensure_started()
return self._embedded._client.update_mental_model(
bank_id=bank_id,
mental_model_id=mental_model_id,
name=name,
content=content,
tags=tags,
)
def delete(self, bank_id: str, mental_model_id: str):
"""Delete a mental model."""
self._embedded._ensure_started()
return self._embedded._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
class DirectivesAPI:
"""Namespace for directive operations."""
def __init__(self, embedded: "HindsightEmbedded"):
self._embedded = embedded
def create(
self,
bank_id: str,
name: str,
content: str,
tags: list[str] | None = None,
):
"""Create a new directive."""
self._embedded._ensure_started()
return self._embedded._client.create_directive(
bank_id=bank_id,
name=name,
content=content,
tags=tags,
)
def list(self, bank_id: str, tags: list[str] | None = None):
"""List all directives for a bank."""
self._embedded._ensure_started()
return self._embedded._client.list_directives(bank_id=bank_id, tags=tags)
def get(self, bank_id: str, directive_id: str):
"""Get a specific directive."""
self._embedded._ensure_started()
return self._embedded._client.get_directive(bank_id=bank_id, directive_id=directive_id)
def update(
self,
bank_id: str,
directive_id: str,
name: str | None = None,
content: str | None = None,
tags: list[str] | None = None,
):
"""Update a directive."""
self._embedded._ensure_started()
return self._embedded._client.update_directive(
bank_id=bank_id,
directive_id=directive_id,
name=name,
content=content,
tags=tags,
)
def delete(self, bank_id: str, directive_id: str):
"""Delete a directive."""
self._embedded._ensure_started()
return self._embedded._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
class MemoriesAPI:
"""Namespace for memory operations."""
def __init__(self, embedded: "HindsightEmbedded"):
self._embedded = embedded
def list(
self,
bank_id: str,
type: str | None = None,
search_query: str | None = None,
limit: int = 100,
offset: int = 0,
):
"""List memories in a bank."""
self._embedded._ensure_started()
return self._embedded._client.list_memories(
bank_id=bank_id,
type=type,
search_query=search_query,
limit=limit,
offset=offset,
)
+243
View File
@@ -0,0 +1,243 @@
"""
Wrapper for Hindsight client that adds API namespaces.
Provides organized access to different parts of the Hindsight API through
namespaces like .banks, .mental_models, etc.
"""
from __future__ import annotations
from typing import Any
from hindsight_client import Hindsight
class BanksAPI:
"""Namespace for bank-related operations."""
def __init__(self, client: Hindsight):
self._client = client
def create(
self,
bank_id: str,
name: str | None = None,
mission: str | None = None,
disposition: dict[str, Any] | None = None,
):
"""Create a new bank."""
return self._client.create_bank(
bank_id=bank_id,
name=name,
mission=mission,
disposition=disposition,
)
def delete(self, bank_id: str):
"""Delete a bank."""
return self._client.delete_bank(bank_id=bank_id)
def set_mission(self, bank_id: str, mission: str):
"""Set or update the mission for a bank."""
return self._client.set_mission(bank_id=bank_id, mission=mission)
def set_disposition(self, bank_id: str, disposition: dict[str, Any]):
"""Set or update the disposition for a bank."""
return self._client.set_disposition(bank_id=bank_id, disposition=disposition)
def list(self):
"""List all banks."""
from hindsight_client.hindsight_client import _run_async
return _run_async(self._client._banks_api.list_banks())
class MentalModelsAPI:
"""Namespace for mental model operations."""
def __init__(self, client: Hindsight):
self._client = client
def create(
self,
bank_id: str,
name: str,
content: str,
tags: list[str] | None = None,
):
"""Create a new mental model."""
return self._client.create_mental_model(
bank_id=bank_id,
name=name,
content=content,
tags=tags,
)
def list(self, bank_id: str, tags: list[str] | None = None):
"""List all mental models for a bank."""
return self._client.list_mental_models(bank_id=bank_id, tags=tags)
def get(self, bank_id: str, mental_model_id: str):
"""Get a specific mental model."""
return self._client.get_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
def refresh(self, bank_id: str, mental_model_id: str):
"""Refresh a mental model."""
return self._client.refresh_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
def update(
self,
bank_id: str,
mental_model_id: str,
name: str | None = None,
content: str | None = None,
tags: list[str] | None = None,
):
"""Update a mental model."""
return self._client.update_mental_model(
bank_id=bank_id,
mental_model_id=mental_model_id,
name=name,
content=content,
tags=tags,
)
def delete(self, bank_id: str, mental_model_id: str):
"""Delete a mental model."""
return self._client.delete_mental_model(bank_id=bank_id, mental_model_id=mental_model_id)
class DirectivesAPI:
"""Namespace for directive operations."""
def __init__(self, client: Hindsight):
self._client = client
def create(
self,
bank_id: str,
name: str,
content: str,
tags: list[str] | None = None,
):
"""Create a new directive."""
return self._client.create_directive(
bank_id=bank_id,
name=name,
content=content,
tags=tags,
)
def list(self, bank_id: str, tags: list[str] | None = None):
"""List all directives for a bank."""
return self._client.list_directives(bank_id=bank_id, tags=tags)
def get(self, bank_id: str, directive_id: str):
"""Get a specific directive."""
return self._client.get_directive(bank_id=bank_id, directive_id=directive_id)
def update(
self,
bank_id: str,
directive_id: str,
name: str | None = None,
content: str | None = None,
tags: list[str] | None = None,
):
"""Update a directive."""
return self._client.update_directive(
bank_id=bank_id,
directive_id=directive_id,
name=name,
content=content,
tags=tags,
)
def delete(self, bank_id: str, directive_id: str):
"""Delete a directive."""
return self._client.delete_directive(bank_id=bank_id, directive_id=directive_id)
class MemoriesAPI:
"""Namespace for memory operations."""
def __init__(self, client: Hindsight):
self._client = client
def list(
self,
bank_id: str,
type: str | None = None,
search_query: str | None = None,
limit: int = 100,
offset: int = 0,
):
"""List memories in a bank."""
return self._client.list_memories(
bank_id=bank_id,
type=type,
search_query=search_query,
limit=limit,
offset=offset,
)
class HindsightClient(Hindsight):
"""
Enhanced Hindsight client with organized API namespaces.
This wrapper extends the auto-generated Hindsight client with organized
access to different parts of the API through namespaces.
Example:
```python
from hindsight import HindsightClient
client = HindsightClient(base_url="http://localhost:8888")
# Core operations (inherited from Hindsight)
client.retain(bank_id="test", content="Hello")
results = client.recall(bank_id="test", query="Hello")
# Organized API access through namespaces
client.banks.create(bank_id="test", name="Test Bank")
models = client.mental_models.list(bank_id="test")
directives = client.directives.list(bank_id="test")
memories = client.memories.list(bank_id="test")
```
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._banks_namespace: BanksAPI | None = None
self._mental_models_namespace: MentalModelsAPI | None = None
self._directives_namespace: DirectivesAPI | None = None
self._memories_namespace: MemoriesAPI | None = None
@property
def banks(self) -> BanksAPI:
"""Access bank management operations."""
if self._banks_namespace is None:
self._banks_namespace = BanksAPI(self)
return self._banks_namespace
@property
def mental_models(self) -> MentalModelsAPI:
"""Access mental model operations."""
if self._mental_models_namespace is None:
self._mental_models_namespace = MentalModelsAPI(self)
return self._mental_models_namespace
@property
def directives(self) -> DirectivesAPI:
"""Access directive operations."""
if self._directives_namespace is None:
self._directives_namespace = DirectivesAPI(self)
return self._directives_namespace
@property
def memories(self) -> MemoriesAPI:
"""Access memory listing operations."""
if self._memories_namespace is None:
self._memories_namespace = MemoriesAPI(self)
return self._memories_namespace
+377
View File
@@ -0,0 +1,377 @@
"""
Embedded Hindsight client with automatic daemon lifecycle management.
This module provides HindsightEmbedded, a client that uses the same daemon
management interface as hindsight-embed CLI, ensuring full compatibility.
Example:
```python
from hindsight import HindsightEmbedded
# Daemon starts automatically on first use
client = HindsightEmbedded(
profile="myapp",
llm_provider="groq",
llm_api_key="your-api-key",
)
# Use just like HindsightClient
client.retain(bank_id="alice", content="Alice loves AI")
results = client.recall(bank_id="alice", query="What does Alice like?")
# Optional cleanup
client.close()
```
Using context manager:
```python
from hindsight import HindsightEmbedded
with HindsightEmbedded(profile="myapp") as client:
client.retain(bank_id="alice", content="Alice loves AI")
# Daemon managed automatically
```
"""
import logging
import os
import threading
from typing import Optional
from hindsight_client import Hindsight
from hindsight_embed import get_embed_manager
from .api_namespaces import BanksAPI, DirectivesAPI, MemoriesAPI, MentalModelsAPI
logger = logging.getLogger(__name__)
class HindsightEmbedded:
"""
Hindsight client with automatic daemon lifecycle management.
This client uses the same daemon management interface as hindsight-embed CLI,
ensuring full compatibility and shared profiles. The daemon is started automatically
on first use and manages profile-specific databases.
Profile data is stored in: ~/.pg0/instances/hindsight-embed-{profile}/
All methods from HindsightClient are available:
- retain(), retain_batch()
- recall()
- reflect()
- create_bank(), set_mission(), delete_bank()
- create_mental_model(), list_mental_models(), etc.
- create_directive(), list_directives(), etc.
- And all async variants (aretain, arecall, areflect, etc.)
Args:
profile: Profile name for data isolation (default: "default")
llm_provider: LLM provider ("groq", "openai", "ollama", "gemini", "anthropic", "lmstudio")
llm_api_key: API key for the LLM provider
llm_model: Model name to use
llm_base_url: Optional custom base URL for LLM API
database_url: Optional database URL override (default: profile-specific pg0)
idle_timeout: Seconds before daemon auto-exits when idle (default: 300)
log_level: Daemon log level (default: "info")
"""
def __init__(
self,
profile: str = "default",
llm_provider: str = "groq",
llm_api_key: str = "",
llm_model: str = "openai/gpt-oss-120b",
llm_base_url: Optional[str] = None,
database_url: Optional[str] = None,
idle_timeout: int = 300,
log_level: str = "info",
):
"""
Initialize the embedded client (daemon starts on first use).
Args:
profile: Profile name for data isolation
llm_provider: LLM provider
llm_api_key: API key for the LLM provider
llm_model: Model name to use
llm_base_url: Optional custom base URL for LLM API
database_url: Optional database URL override
idle_timeout: Seconds before daemon auto-exits when idle
log_level: Daemon log level
"""
self.profile = profile
# Build config dict for daemon (matches CLI format)
self.config = {
"HINDSIGHT_API_LLM_PROVIDER": llm_provider,
"HINDSIGHT_API_LLM_API_KEY": llm_api_key,
"HINDSIGHT_API_LLM_MODEL": llm_model,
"HINDSIGHT_API_LOG_LEVEL": log_level,
"HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT": str(idle_timeout),
}
if llm_base_url:
self.config["HINDSIGHT_API_LLM_BASE_URL"] = llm_base_url
if database_url:
self.config["HINDSIGHT_EMBED_API_DATABASE_URL"] = database_url
self._client: Optional[Hindsight] = None
self._lock = threading.Lock()
self._started = False
self._closed = False
self._manager = get_embed_manager()
# API namespaces (initialized once, lazily)
self._banks_api: Optional[BanksAPI] = None
self._mental_models_api: Optional[MentalModelsAPI] = None
self._directives_api: Optional[DirectivesAPI] = None
self._memories_api: Optional[MemoriesAPI] = None
def _ensure_started(self):
"""Ensure daemon is running (thread-safe)."""
if self._started and self._client is not None:
return
with self._lock:
# Double-check after acquiring lock
if self._started and self._client is not None:
return
if self._closed:
raise RuntimeError("Cannot use HindsightEmbedded after it has been closed")
# Use embed manager interface for daemon management
logger.info(f"Ensuring daemon is running for profile '{self.profile}'...")
success = self._manager.ensure_running(self.config, self.profile)
if not success:
raise RuntimeError(f"Failed to start daemon for profile '{self.profile}'")
# Get daemon URL and create client
daemon_url = self._manager.get_url(self.profile)
self._client = Hindsight(base_url=daemon_url)
self._started = True
logger.info(f"Connected to daemon at {daemon_url}")
def _cleanup(self, stop_daemon_on_close: bool = False):
"""
Cleanup client resources (idempotent).
Args:
stop_daemon_on_close: If True, stops the daemon. Otherwise, daemon continues
running (it will auto-stop after idle timeout).
"""
if self._closed:
return
with self._lock:
if self._closed:
return
if self._client is not None:
self._client.close()
self._client = None
# Optionally stop daemon (daemon has idle timeout, so not required)
if stop_daemon_on_close and self._started:
logger.info(f"Stopping daemon for profile '{self.profile}'...")
self._manager.stop(self.profile)
self._closed = True
def close(self, stop_daemon: bool = False):
"""
Explicitly close the client.
Args:
stop_daemon: If True, stops the daemon. Otherwise, daemon continues running
and will auto-stop after idle timeout (default: False).
Note:
The daemon may be shared with other clients or the CLI, so stopping it
might affect other users. By default, we rely on the daemon's idle timeout.
"""
self._cleanup(stop_daemon_on_close=stop_daemon)
def __getattr__(self, name: str):
"""
Proxy all method calls to the underlying Hindsight client.
This allows HindsightEmbedded to expose all HindsightClient methods
without manually wrapping each one.
"""
# Ensure server is started before proxying
self._ensure_started()
# Get the attribute from the underlying client
attr = getattr(self._client, name)
# If it's a callable, wrap it to ensure server is started
# (shouldn't be needed since _ensure_started already called, but defensive)
if callable(attr):
def wrapper(*args, **kwargs):
self._ensure_started()
return attr(*args, **kwargs)
return wrapper
return attr
def __enter__(self):
"""Context manager entry - ensures server is started."""
self._ensure_started()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Context manager exit - stops the server."""
self.close()
def __del__(self):
"""Cleanup on garbage collection."""
self._cleanup()
@property
def banks(self) -> BanksAPI:
"""
Access bank management operations.
Each method call ensures the daemon is running before executing.
Example:
```python
from hindsight import HindsightEmbedded
embedded = HindsightEmbedded(profile="myapp", ...)
# Create a bank
embedded.banks.create(bank_id="test", name="Test Bank")
# Set mission
embedded.banks.set_mission(bank_id="test", mission="Help users")
```
"""
if self._banks_api is None:
self._banks_api = BanksAPI(self)
return self._banks_api
@property
def mental_models(self) -> MentalModelsAPI:
"""
Access mental model operations.
Each method call ensures the daemon is running before executing.
Example:
```python
from hindsight import HindsightEmbedded
embedded = HindsightEmbedded(profile="myapp", ...)
# Create a mental model
embedded.mental_models.create(
bank_id="test",
name="User Preferences",
content="User prefers dark mode"
)
# List mental models
models = embedded.mental_models.list(bank_id="test")
```
"""
if self._mental_models_api is None:
self._mental_models_api = MentalModelsAPI(self)
return self._mental_models_api
@property
def directives(self) -> DirectivesAPI:
"""
Access directive operations.
Each method call ensures the daemon is running before executing.
Example:
```python
from hindsight import HindsightEmbedded
embedded = HindsightEmbedded(profile="myapp", ...)
# Create a directive
embedded.directives.create(
bank_id="test",
name="Response Style",
content="Always be concise and friendly"
)
# List directives
directives = embedded.directives.list(bank_id="test")
```
"""
if self._directives_api is None:
self._directives_api = DirectivesAPI(self)
return self._directives_api
@property
def memories(self) -> MemoriesAPI:
"""
Access memory listing operations.
Each method call ensures the daemon is running before executing.
Example:
```python
from hindsight import HindsightEmbedded
embedded = HindsightEmbedded(profile="myapp", ...)
# List memories
memories = embedded.memories.list(
bank_id="test",
type="world",
limit=50
)
```
"""
if self._memories_api is None:
self._memories_api = MemoriesAPI(self)
return self._memories_api
@property
def client(self) -> Hindsight:
"""
Get the underlying Hindsight client for direct access.
WARNING: Using this property directly means daemon restarts won't be
handled automatically. Prefer using the API namespaces (banks, mental_models,
directives, memories) or direct method calls on HindsightEmbedded instead.
Ensures daemon is started before returning the client.
Returns:
Hindsight: The underlying client instance
Example:
```python
from hindsight import HindsightEmbedded
embedded = HindsightEmbedded(profile="myapp", ...)
# Direct access (not recommended - daemon crashes won't be handled)
client = embedded.client
banks = client.list_banks() # If daemon crashes, this will fail
```
"""
self._ensure_started()
return self._client
@property
def url(self) -> str:
"""Get the daemon URL (starts daemon if needed)."""
self._ensure_started()
return self._manager.get_url(self.profile)
@property
def is_running(self) -> bool:
"""Check if the client is initialized."""
return self._started and not self._closed and self._client is not None
+2
View File
@@ -11,11 +11,13 @@ requires-python = ">=3.11"
dependencies = [
"hindsight-api>=0.0.7",
"hindsight-client>=0.0.7",
"hindsight-embed>=0.1.0",
]
[tool.uv.sources]
hindsight-api = { workspace = true }
hindsight-client = { workspace = true }
hindsight-embed = { workspace = true }
[project.optional-dependencies]
test = [
+338
View File
@@ -0,0 +1,338 @@
"""
Integration tests for HindsightEmbedded client.
Tests the embedded client with automatic server lifecycle management:
1. Lazy server startup on first use
2. Server reuse across multiple operations
3. Context manager support
4. Method proxying to underlying HindsightClient
5. Proper cleanup
Note: Each test uses random bank_ids to avoid conflicts and allow safe parallel execution.
"""
import os
import uuid
import pytest
from hindsight import HindsightEmbedded
@pytest.fixture(scope="session")
def llm_config():
"""Get LLM configuration from environment (session-scoped)."""
# Try both naming conventions
provider = os.getenv("HINDSIGHT_API_LLM_PROVIDER") or os.getenv("HINDSIGHT_LLM_PROVIDER", "groq")
api_key = os.getenv("HINDSIGHT_API_LLM_API_KEY") or os.getenv("HINDSIGHT_LLM_API_KEY", "")
model = os.getenv("HINDSIGHT_API_LLM_MODEL") or os.getenv("HINDSIGHT_LLM_MODEL", "openai/gpt-oss-120b")
if not api_key:
pytest.skip("LLM API key not configured. Set HINDSIGHT_API_LLM_API_KEY or HINDSIGHT_LLM_API_KEY.")
return {
"llm_provider": provider,
"llm_api_key": api_key,
"llm_model": model,
}
def test_embedded_lazy_start(llm_config):
"""
Test that HindsightEmbedded starts server lazily on first use.
"""
profile = f"test_lazy_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
# Create client - should NOT start server yet
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
assert not client.is_running, "Server should not be running after initialization"
# First call should start server
result = client.retain(bank_id=bank_id, content="Test content for lazy start")
# Verify server is now running
assert client.is_running, "Server should be running after first call"
assert result.success, "Retain should succeed"
assert result.items_count >= 1, "Should have stored at least 1 item"
# Cleanup
client.close()
assert not client.is_running, "Server should stop after close()"
def test_embedded_context_manager(llm_config):
"""
Test HindsightEmbedded with context manager.
"""
profile = f"test_ctx_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
# Use context manager
with HindsightEmbedded(profile=profile, log_level="info", **llm_config) as client:
assert client.is_running, "Server should be running inside context"
# Store memory
result = client.retain(bank_id=bank_id, content="Testing context manager")
assert result.success, "Retain should succeed"
# Recall memory
recall_results = client.recall(bank_id=bank_id, query="context")
assert isinstance(recall_results.results, list), "Recall should return results list"
# Server should be stopped after context exit
# Note: We can't check client.is_running here as client is out of scope
def test_embedded_complete_workflow(llm_config):
"""
Test complete workflow with HindsightEmbedded.
This test:
1. Creates a client with lazy start
2. Creates a memory bank
3. Stores multiple memories
4. Recalls memories
5. Reflects on memories
6. Tests cleanup
"""
profile = f"test_workflow_{uuid.uuid4().hex[:8]}"
bank_id = f"assistant_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
try:
# Step 1: Create a memory bank
print(f"\n1. Creating memory bank: {bank_id}")
bank_response = client.create_bank(
bank_id=bank_id, name="Test Assistant", mission="Help with programming tasks"
)
assert bank_response.bank_id == bank_id
# Step 2: Store memories (single)
print("\n2. Storing single memory...")
retain_response = client.retain(
bank_id=bank_id,
content="User prefers Python for data analysis.",
context="Programming preferences",
)
assert retain_response.success
assert retain_response.items_count >= 1
# Step 3: Store batch memories
print("\n3. Storing batch memories...")
batch_response = client.retain_batch(
bank_id=bank_id,
items=[
{"content": "User works with pandas and numpy."},
{"content": "User likes matplotlib for visualization."},
{"content": "User is interested in machine learning with scikit-learn."},
],
)
assert batch_response.success
assert batch_response.items_count >= 3
# Step 4: Recall memories
print("\n4. Recalling memories...")
recall_response = client.recall(bank_id=bank_id, query="What tools does the user prefer?", max_tokens=2000)
assert isinstance(recall_response.results, list)
assert len(recall_response.results) > 0
print(f" Found {len(recall_response.results)} relevant memories")
# Step 5: Reflect on memories
print("\n5. Reflecting on memories...")
reflect_response = client.reflect(
bank_id=bank_id,
query="What programming tools should I recommend?",
budget="low",
)
assert reflect_response.text
assert len(reflect_response.text) > 0
print(f" Answer: {reflect_response.text[:150]}...")
# Verify answer mentions relevant tools
answer_lower = reflect_response.text.lower()
assert any(term in answer_lower for term in ["python", "pandas", "numpy", "data"])
# Step 6: List memories
print("\n6. Listing memories...")
list_response = client.list_memories(bank_id=bank_id, limit=10)
assert len(list_response.items) > 0
print(f" Listed {len(list_response.items)} memories")
finally:
# Cleanup
client.close()
def test_embedded_server_reuse(llm_config):
"""
Test that the same server is reused across multiple calls.
"""
profile = f"test_reuse_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
try:
# First call starts server
result1 = client.retain(bank_id=bank_id, content="First message")
url1 = client.url
assert client.is_running
# Second call should reuse the same server
result2 = client.retain(bank_id=bank_id, content="Second message")
url2 = client.url
# URLs should be identical (same server)
assert url1 == url2, "Server URL should remain the same across calls"
assert result1.success and result2.success
# Third call should also reuse
recall_result = client.recall(bank_id=bank_id, query="message")
url3 = client.url
assert url3 == url1, "Server URL should remain the same for recall"
assert isinstance(recall_result.results, list)
finally:
client.close()
def test_embedded_method_proxying(llm_config):
"""
Test that all HindsightClient methods are properly proxied.
This ensures __getattr__ proxying works for various method types.
"""
profile = f"test_proxy_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
try:
# Test bank operations
bank = client.create_bank(bank_id=bank_id, name="Proxy Test")
assert bank.bank_id == bank_id
# Test mission setting
mission_response = client.set_mission(bank_id=bank_id, mission="Test mission for proxying")
assert mission_response.bank_id == bank_id
# Test retain
retain_result = client.retain(bank_id=bank_id, content="Test content")
assert retain_result.success
# Test retain_batch
batch_result = client.retain_batch(
bank_id=bank_id, items=[{"content": "Item 1"}, {"content": "Item 2"}]
)
assert batch_result.success
assert batch_result.items_count >= 2
# Test recall
recall_result = client.recall(bank_id=bank_id, query="test")
assert hasattr(recall_result, "results")
# Test reflect
reflect_result = client.reflect(bank_id=bank_id, query="What is stored?")
assert hasattr(reflect_result, "text")
# Test list_memories
list_result = client.list_memories(bank_id=bank_id, limit=5)
assert hasattr(list_result, "items")
print("✓ All methods successfully proxied")
finally:
client.close()
def test_embedded_multiple_banks(llm_config):
"""
Test that HindsightEmbedded can work with multiple banks.
"""
profile = f"test_multibank_{uuid.uuid4().hex[:8]}"
bank1_id = f"bank1_{uuid.uuid4().hex[:8]}"
bank2_id = f"bank2_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
try:
# Create first bank and store data
client.create_bank(bank_id=bank1_id, name="Bank 1")
client.retain(bank_id=bank1_id, content="Alice prefers Python for data science")
# Create second bank and store data
client.create_bank(bank_id=bank2_id, name="Bank 2")
client.retain(bank_id=bank2_id, content="Bob uses JavaScript for web development")
# Recall from both banks
results1 = client.recall(bank_id=bank1_id, query="programming language")
results2 = client.recall(bank_id=bank2_id, query="programming language")
assert len(results1.results) > 0
assert len(results2.results) > 0
# Verify banks are isolated (each should only see their own content)
# This is a basic check - content isolation is tested more thoroughly in other tests
assert results1.results[0].text != results2.results[0].text or len(results1.results) != len(
results2.results
)
finally:
client.close()
def test_embedded_profile_isolation(llm_config):
"""
Test that different profiles create isolated data stores.
"""
profile1 = f"test_iso1_{uuid.uuid4().hex[:8]}"
profile2 = f"test_iso2_{uuid.uuid4().hex[:8]}"
bank_id = "shared_bank_name" # Same bank_id in both profiles
client1 = HindsightEmbedded(profile=profile1, log_level="info", **llm_config)
client2 = HindsightEmbedded(profile=profile2, log_level="info", **llm_config)
try:
# Store data in profile1
client1.retain(bank_id=bank_id, content="User likes TypeScript for frontend development")
# Store different data in profile2
client2.retain(bank_id=bank_id, content="User prefers Rust for systems programming")
# Each profile should only see its own data
results1 = client1.recall(bank_id=bank_id, query="programming preference")
results2 = client2.recall(bank_id=bank_id, query="programming preference")
# Both should have results
assert len(results1.results) > 0
assert len(results2.results) > 0
# Results should be different (basic isolation check)
# Note: This is a basic sanity check. Full isolation is ensured by pg0's data directory separation
finally:
client1.close()
client2.close()
def test_embedded_error_after_close(llm_config):
"""
Test that using HindsightEmbedded after close() raises an error.
"""
profile = f"test_error_{uuid.uuid4().hex[:8]}"
bank_id = f"bank_{uuid.uuid4().hex[:8]}"
client = HindsightEmbedded(profile=profile, log_level="info", **llm_config)
# Use it once to start server
client.retain(bank_id=bank_id, content="Test")
# Close the client
client.close()
assert not client.is_running
# Trying to use it after close should raise an error
with pytest.raises(RuntimeError, match="Cannot use HindsightEmbedded after it has been closed"):
client.retain(bank_id=bank_id, content="This should fail")
+170
View File
@@ -0,0 +1,170 @@
"""Test that API namespaces ensure daemon is started before each call."""
from unittest.mock import Mock, patch
import pytest
from hindsight import HindsightEmbedded
@pytest.fixture
def embedded_client():
"""Create an embedded client for testing."""
return HindsightEmbedded(
profile="test",
llm_provider="openai",
llm_api_key="test-key",
)
def test_banks_create_ensures_daemon_started(embedded_client):
"""Test that banks.create() calls _ensure_started()."""
# Mock _ensure_started to track calls
with patch.object(embedded_client, "_ensure_started") as mock_ensure:
# Mock the underlying client to avoid actual API call
mock_client = Mock()
embedded_client._client = mock_client
# Call namespace method
try:
embedded_client.banks.create(bank_id="test", name="Test Bank")
except Exception:
pass # We don't care if the actual call fails
# Verify _ensure_started was called
mock_ensure.assert_called_once()
def test_mental_models_list_ensures_daemon_started(embedded_client):
"""Test that mental_models.list() calls _ensure_started()."""
with patch.object(embedded_client, "_ensure_started") as mock_ensure:
mock_client = Mock()
embedded_client._client = mock_client
try:
embedded_client.mental_models.list(bank_id="test")
except Exception:
pass
mock_ensure.assert_called_once()
def test_directives_list_ensures_daemon_started(embedded_client):
"""Test that directives.list() calls _ensure_started()."""
with patch.object(embedded_client, "_ensure_started") as mock_ensure:
mock_client = Mock()
embedded_client._client = mock_client
try:
embedded_client.directives.list(bank_id="test")
except Exception:
pass
mock_ensure.assert_called_once()
def test_memories_list_ensures_daemon_started(embedded_client):
"""Test that memories.list() calls _ensure_started()."""
with patch.object(embedded_client, "_ensure_started") as mock_ensure:
mock_client = Mock()
embedded_client._client = mock_client
try:
embedded_client.memories.list(bank_id="test")
except Exception:
pass
mock_ensure.assert_called_once()
def test_multiple_calls_ensure_daemon_each_time(embedded_client):
"""Test that each namespace call ensures daemon is started."""
with patch.object(embedded_client, "_ensure_started") as mock_ensure:
mock_client = Mock()
embedded_client._client = mock_client
# Make multiple calls
try:
embedded_client.banks.create(bank_id="test", name="Test")
except Exception:
pass
try:
embedded_client.mental_models.list(bank_id="test")
except Exception:
pass
try:
embedded_client.directives.list(bank_id="test")
except Exception:
pass
# Should be called 3 times (once per namespace method call)
assert mock_ensure.call_count == 3
def test_daemon_restart_handling(embedded_client):
"""Test that namespace methods can recover from daemon crash."""
call_count = 0
def mock_ensure_started():
"""Mock that simulates daemon restart."""
nonlocal call_count
call_count += 1
# Create a new mock client each time (simulating daemon restart)
embedded_client._client = Mock()
embedded_client._started = True
with patch.object(embedded_client, "_ensure_started", side_effect=mock_ensure_started):
# First call - daemon starts
embedded_client.banks.create(bank_id="test", name="Test")
assert call_count == 1
# Simulate daemon crash by clearing client
embedded_client._client = None
embedded_client._started = False
# Second call - daemon restarts
embedded_client.banks.create(bank_id="test", name="Test")
assert call_count == 2
def test_ensure_started_calls_manager(embedded_client):
"""Test that _ensure_started actually starts the daemon via manager."""
# Mock the manager
mock_manager = Mock()
mock_manager.ensure_running.return_value = True
mock_manager.get_url.return_value = "http://localhost:54321"
embedded_client._manager = mock_manager
# Mock Hindsight client constructor
with patch("hindsight.embedded.Hindsight") as mock_hindsight_class:
mock_client = Mock()
mock_hindsight_class.return_value = mock_client
# Call _ensure_started
embedded_client._ensure_started()
# Verify manager was called
mock_manager.ensure_running.assert_called_once_with(
embedded_client.config, embedded_client.profile
)
mock_manager.get_url.assert_called_once_with(embedded_client.profile)
# Verify Hindsight client was created
mock_hindsight_class.assert_called_once_with(base_url="http://localhost:54321")
def test_namespace_singleton_behavior(embedded_client):
"""Test that namespace properties return the same instance."""
banks1 = embedded_client.banks
banks2 = embedded_client.banks
# Should be the same instance
assert banks1 is banks2
# Same for other namespaces
assert embedded_client.mental_models is embedded_client.mental_models
assert embedded_client.directives is embedded_client.directives
assert embedded_client.memories is embedded_client.memories
+29 -49
View File
@@ -75,10 +75,9 @@ def test_server_context_manager_basic_workflow(client):
bank_response = client.create_bank(
bank_id=bank_id,
name="Test Assistant",
background="An AI assistant that helps with programming and data analysis tasks."
mission="An AI assistant that helps with programming and data analysis tasks."
)
assert "bank_id" in bank_response
assert bank_response["bank_id"] == bank_id
assert bank_response.bank_id == bank_id
# Step 2: Store some memories about user preferences
print("\n2. Storing memories...")
@@ -89,7 +88,7 @@ def test_server_context_manager_basic_workflow(client):
content="User prefers Python over JavaScript for data analysis projects.",
context="User conversation about programming languages"
)
assert retain_response1.get("success") is True
assert retain_response1.success is True
# Store second memory
retain_response2 = client.retain(
@@ -97,7 +96,7 @@ def test_server_context_manager_basic_workflow(client):
content="User is working on a machine learning project using scikit-learn.",
context="Discussion about ML frameworks"
)
assert retain_response2.get("success") is True
assert retain_response2.success is True
# Store third memory
retain_response3 = client.retain(
@@ -105,7 +104,7 @@ def test_server_context_manager_basic_workflow(client):
content="User likes visualizing data with matplotlib and seaborn.",
context="Conversation about data visualization"
)
assert retain_response3.get("success") is True
assert retain_response3.success is True
# Store batch memories
batch_response = client.retain_batch(
@@ -116,7 +115,7 @@ def test_server_context_manager_basic_workflow(client):
]
)
# Check if the batch was submitted successfully (items_count shows how many were submitted)
assert batch_response.get("items_count", 0) >= 2
assert batch_response.items_count >= 2
# Step 3: Recall memories based on a query
print("\n3. Recalling memories about programming preferences...")
@@ -127,14 +126,13 @@ def test_server_context_manager_basic_workflow(client):
)
# Verify recall results
assert isinstance(recall_results, list)
assert len(recall_results) > 0
print(f" Found {len(recall_results)} relevant memories")
assert isinstance(recall_results.results, list)
assert len(recall_results.results) > 0
print(f" Found {len(recall_results.results)} relevant memories")
# Check that results have expected structure
for result in recall_results:
assert "content" in result or "text" in result
print(f" - {result.get('content') or result.get('text', '')[:100]}")
for result in recall_results.results:
print(f" - {result.text[:100]}")
# Step 4: Recall memories about machine learning
print("\n4. Recalling memories about machine learning...")
@@ -145,11 +143,11 @@ def test_server_context_manager_basic_workflow(client):
)
# Verify recall results
assert isinstance(ml_recall_results, list)
assert len(ml_recall_results) > 0
print(f" Found {len(ml_recall_results)} ML-related memories")
for result in ml_recall_results[:3]: # Show first 3
print(f" - {result.get('content') or result.get('text', '')[:100]}")
assert isinstance(ml_recall_results.results, list)
assert len(ml_recall_results.results) > 0
print(f" Found {len(ml_recall_results.results)} ML-related memories")
for result in ml_recall_results.results[:3]: # Show first 3
print(f" - {result.text[:100]}")
# Step 5: Reflect (generate contextual answer based on memories)
print("\n5. Reflecting on query about recommendations...")
@@ -160,10 +158,7 @@ def test_server_context_manager_basic_workflow(client):
)
# Verify reflection response
assert isinstance(reflect_response, dict)
assert "answer" in reflect_response or "text" in reflect_response
answer = reflect_response.get("answer") or reflect_response.get("text", "")
answer = reflect_response.text
assert len(answer) > 0
print(f" Answer: {answer[:200]}...")
@@ -180,8 +175,7 @@ def test_server_context_manager_basic_workflow(client):
context="The user is starting a new deep learning project"
)
assert isinstance(reflect_with_context, dict)
context_answer = reflect_with_context.get("answer") or reflect_with_context.get("text", "")
context_answer = reflect_with_context.text
assert len(context_answer) > 0
print(f" Context-aware answer: {context_answer[:150]}...")
@@ -200,21 +194,21 @@ def test_server_manual_start_stop(client):
bank_id=bank_id,
name="Manual Test"
)
assert bank_response["bank_id"] == bank_id
assert bank_response.bank_id == bank_id
# Store a memory
retain_response = client.retain(
bank_id=bank_id,
content="Testing manual server lifecycle."
)
assert retain_response.get("success") is True
assert retain_response.success is True
# Recall the memory
recall_results = client.recall(
bank_id=bank_id,
query="server testing"
)
assert len(recall_results) >= 0 # May or may not find results immediately
assert len(recall_results.results) >= 0 # May or may not find results immediately
def test_server_with_client_context_manager(client):
@@ -233,11 +227,11 @@ def test_server_with_client_context_manager(client):
bank_id=bank_id,
content="Testing nested context managers."
)
assert response.get("success") is True
assert response.success is True
# Verify we can recall
results = client.recall(bank_id=bank_id, query="context")
assert isinstance(results, list)
assert isinstance(results.results, list)
def test_list_banks(client, shared_server):
@@ -252,21 +246,11 @@ def test_list_banks(client, shared_server):
bank1_id = f"test_bank_1_{test_suffix}"
bank2_id = f"test_bank_2_{test_suffix}"
client.create_bank(bank_id=bank1_id, name="Test Bank 1", background="First test bank")
client.create_bank(bank_id=bank2_id, name="Test Bank 2", background="Second test bank")
client.create_bank(bank_id=bank1_id, name="Test Bank 1", mission="First test bank")
client.create_bank(bank_id=bank2_id, name="Test Bank 2", mission="Second test bank")
# List all banks using the generated client
import hindsight_client_api
from hindsight_client_api.api import default_api
config = hindsight_client_api.Configuration(host=shared_server.url)
api_client = hindsight_client_api.ApiClient(config)
api = default_api.DefaultApi(api_client)
# Call list_banks endpoint
import asyncio
loop = asyncio.get_event_loop()
response = loop.run_until_complete(api.list_banks())
# List all banks using the namespace API
response = client.banks.list()
# Verify response structure
assert hasattr(response, 'banks'), "Response should have 'banks' attribute"
@@ -274,9 +258,8 @@ def test_list_banks(client, shared_server):
# Verify each bank has bank_id (not agent_id)
for bank in response.banks:
bank_dict = bank.to_dict() if hasattr(bank, 'to_dict') else bank
assert 'bank_id' in bank_dict, f"Bank should have 'bank_id' field, got: {bank_dict.keys()}"
assert 'agent_id' not in bank_dict, f"Bank should NOT have 'agent_id' field, got: {bank_dict.keys()}"
assert hasattr(bank, 'bank_id'), f"Bank should have 'bank_id' attribute"
assert bank.bank_id is not None, "Bank ID should not be None"
# Find our test banks
bank_ids = [b.bank_id if hasattr(b, 'bank_id') else b['bank_id'] for b in response.banks]
@@ -284,6 +267,3 @@ def test_list_banks(client, shared_server):
assert bank2_id in bank_ids, f"Should find {bank2_id} in bank list"
print(f"✓ Successfully listed {len(response.banks)} banks with correct bank_id field")
# Cleanup
loop.run_until_complete(api_client.close())
Generated
+2
View File
@@ -1453,6 +1453,7 @@ source = { editable = "hindsight" }
dependencies = [
{ name = "hindsight-api" },
{ name = "hindsight-client" },
{ name = "hindsight-embed" },
]
[package.optional-dependencies]
@@ -1465,6 +1466,7 @@ test = [
requires-dist = [
{ name = "hindsight-api", editable = "hindsight-api" },
{ name = "hindsight-client", editable = "hindsight-clients/python" },
{ name = "hindsight-embed", editable = "hindsight-embed" },
{ name = "pytest", marker = "extra == 'test'", specifier = ">=7.0.0" },
{ name = "pytest-asyncio", marker = "extra == 'test'", specifier = ">=0.21.0" },
]