tests

* feat: improve mental model refresh and add directives

* feat: improve mental model refresh and add directives

* tags

* ui

* fix

* fix

* update

* update

.github/workflows/test.yml

@@ -153,8 +153,15 @@ jobs:
     - name: Build docs
       run: npm run build --workspace=hindsight-docs
 
-  build-rust-cli:
+  test-rust-cli:
     runs-on: ubuntu-latest
+    env:
+      HINDSIGHT_API_LLM_PROVIDER: groq
+      HINDSIGHT_API_LLM_API_KEY: ${{ secrets.GROQ_API_KEY }}
+      HINDSIGHT_API_LLM_MODEL: openai/gpt-oss-20b
+      HINDSIGHT_API_URL: http://localhost:8888
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      UV_INDEX: pytorch=https://download.pytorch.org/whl/cpu
 
     steps:
     - uses: actions/checkout@v4
@@ -171,6 +178,10 @@ jobs:
           hindsight-cli/target
         key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
 
+    - name: Run unit tests
+      working-directory: hindsight-cli
+      run: cargo test
+
     - name: Build CLI
       working-directory: hindsight-cli
       run: cargo build --release
@@ -182,29 +193,6 @@ jobs:
         path: hindsight-cli/target/release/hindsight
         retention-days: 1
 
-  test-rust-cli:
-    runs-on: ubuntu-latest
-    needs: build-rust-cli
-    env:
-      HINDSIGHT_API_LLM_PROVIDER: groq
-      HINDSIGHT_API_LLM_API_KEY: ${{ secrets.GROQ_API_KEY }}
-      HINDSIGHT_API_LLM_MODEL: openai/gpt-oss-20b
-      HINDSIGHT_API_URL: http://localhost:8888
-      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      UV_INDEX: pytorch=https://download.pytorch.org/whl/cpu
-
-    steps:
-    - uses: actions/checkout@v4
-
-    - name: Download CLI artifact
-      uses: actions/download-artifact@v4
-      with:
-        name: hindsight-cli
-        path: /tmp/cli
-
-    - name: Make CLI executable
-      run: chmod +x /tmp/cli/hindsight
-
     - name: Install uv
       uses: astral-sh/setup-uv@v5
       with:
@@ -251,7 +239,7 @@ jobs:
 
     - name: Run CLI smoke test
       run: |
-        HINDSIGHT_CLI=/tmp/cli/hindsight ./hindsight-cli/smoke-test.sh
+        HINDSIGHT_CLI=hindsight-cli/target/release/hindsight ./hindsight-cli/smoke-test.sh
 
     - name: Show API server logs
       if: always()
@@ -777,7 +765,7 @@ jobs:
 
   test-doc-examples:
     runs-on: ubuntu-latest
-    needs: build-rust-cli
+    needs: test-rust-cli
     env:
       HINDSIGHT_API_LLM_PROVIDER: groq
       HINDSIGHT_API_LLM_API_KEY: ${{ secrets.GROQ_API_KEY }}

.gitignore

@@ -29,7 +29,7 @@ nltk_data/
 
 # Monitoring stack (Prometheus/Grafana binaries and data)
 .monitoring/
-.pgbouncer
+.pgbouncer/
 
 # Large benchmark datasets (will be downloaded automatically)
 **/longmemeval_s_cleaned.json

AGENTS.md

@@ -1,153 +1,3 @@
 # AGENTS.md
 
-This document captures architectural decisions and coding conventions for the Hindsight project.
-
-## Documentation
-
-- **Main documentation**: [hindsight-docs/docs/developer/](./hindsight-docs/docs/developer/)
-- **Use case patterns**: [hindsight-docs/docs/cookbook/](./hindsight-docs/docs/cookbook/)
-- **API reference**: Auto-generated from OpenAPI spec
-
-## Project Structure
-
-```
-hindsight/              # Python package for embedded usage
-hindsight-api/          # FastAPI server (core memory engine)
-hindsight-cli/          # Rust CLI client
-hindsight-embed/        # Embedded CLI (no server needed)
-hindsight-control-plane/ # Next.js admin UI
-hindsight-docs/         # Docusaurus documentation site
-hindsight-dev/          # Development tools and benchmarks
-hindsight-integrations/ # Framework integrations (LangChain, etc.)
-hindsight-clients/      # Generated API clients (Python, TypeScript, Rust)
-```
-
-## Core Concepts
-
-### Memory Banks
-- Each bank is an isolated memory store (like a "brain" for one user/agent)
-- Banks contain: memory units (facts), entities, documents, entity links
-- Banks have a **disposition** (personality traits) and **background** (context)
-- Bank isolation is strict - no cross-bank data leakage
-
-### Memory Types
-- **World facts**: General knowledge ("The sky is blue")
-- **Experience facts**: Personal experiences ("I visited Paris in 2023")
-- **Opinion facts**: Beliefs with confidence scores ("Paris is beautiful" - 0.9 confidence)
-
-### Operations
-- **Retain**: Store new memories (extracts facts, entities, relationships)
-- **Recall**: Retrieve memories (semantic, BM25, graph, temporal search)
-- **Reflect**: Deep analysis to form new insights/opinions
-
-## API Design Decisions
-
-### Single Bank Per Request
-- All API endpoints (`recall`, `reflect`, `retain`) operate on a single bank
-- Multi-bank queries are the **client/agent's responsibility** to orchestrate
-- This keeps the API simple and the isolation model clear
-
-### Disposition Traits (3-trait system)
-- **Skepticism** (1-5): How skeptical vs trusting when forming opinions
-- **Literalism** (1-5): How literally to interpret information
-- **Empathy** (1-5): How much to consider emotional context
-- These influence the `reflect` operation, not `recall`
-- Background info also only affects `reflect` (opinion formation)
-
-## Multi-Bank Architecture Patterns
-
-See [hindsight-docs/docs/cookbook/](./hindsight-docs/docs/cookbook/) for detailed guides:
-
-- **Per-User Memory**: One bank per user, simplest pattern
-- **Support Agent + Shared Knowledge**: User bank + shared docs bank, client orchestrates
-
-## Developer Guide
-
-### Running the API Server
-
-```bash
-# From project root
-./scripts/dev/start-api.sh
-
-# With options
-./scripts/dev/start-api.sh --reload --port 8888 --log-level debug
-```
-
-### Running Tests
-
-```bash
-# API tests
-cd hindsight-api
-uv run pytest tests/
-
-# Specific test
-uv run pytest tests/test_http_api_integration.py -v
-```
-
-### Generating OpenAPI Spec
-
-After changing API endpoints, regenerate the OpenAPI spec and docs:
-
-```bash
-./scripts/generate-openapi.sh
-```
-
-This will:
-1. Generate `openapi.json` at project root
-2. Copy to `hindsight-docs/openapi.json`
-3. Regenerate API reference documentation
-
-### Generating API Clients
-
-After updating the OpenAPI spec, regenerate all clients:
-
-```bash
-./scripts/generate-clients.sh
-```
-
-This generates:
-- **Rust client**: `hindsight-clients/rust/` (via progenitor in build.rs)
-- **Python client**: `hindsight-clients/python/` (via openapi-generator Docker)
-- **TypeScript client**: `hindsight-clients/typescript/` (via @hey-api/openapi-ts)
-
-Note: The maintained wrapper `hindsight_client.py` and `README.md` are preserved during regeneration.
-
-### Running the Documentation Site
-
-```bash
-./scripts/dev/start-docs.sh
-```
-
-### Running the Control Plane
-
-```bash
-./scripts/dev/start-control-plane.sh
-```
-
-## Code Style
-
-### Python (hindsight-api)
-- Use `uv` for package management
-- Async throughout (asyncpg, async FastAPI endpoints)
-- Pydantic models for request/response validation
-- No py files at project root - maintain clean directory structure
-
-### TypeScript (control-plane, clients)
-- Next.js with App Router for control plane
-- Tailwind CSS with shadcn/ui components
-
-### Rust (CLI)
-- Async with tokio
-- reqwest for HTTP client
-- progenitor for API client generation
-
-## Database
-
-- PostgreSQL with pgvector extension
-- Schema managed via Alembic migrations in `hindsight-api/alembic/`, db migrations happen during api startup, no manual commands
-- Key tables: `banks`, `memory_units`, `documents`, `entities`, `entity_links`
-
-# Branding
-## Colors
-- Primary: gradient from #0074d9 to #009296
-
+See [CLAUDE.md](./CLAUDE.md) for project documentation and coding conventions.

CLAUDE.md

@@ -199,6 +199,38 @@ When adding or modifying parameters in the dataplane API (hindsight-api), you mu
 - Pydantic models for request/response
 - Ruff for linting (line-length 120)
 - No Python files at project root - maintain clean directory structure
+- **Never use multi-item tuple return values** - prefer dataclass or Pydantic model for structured returns
+
+### Type Safety with Pydantic Models
+**NEVER use raw `dict` types for structured data.** Always use Pydantic models:
+- Use Pydantic `BaseModel` for all data structures passed between functions
+- Add `@field_validator` for type coercion (e.g., ensuring datetimes are timezone-aware)
+- Avoid `dict.get()` patterns - use typed model attributes instead
+- Parse external data (JSON, API responses) into Pydantic models at the boundary
+- This catches type errors at parse time, not deep in business logic
+
+```python
+# BAD - error-prone dict access
+def process(data: dict) -> str:
+    return data.get("name", "")  # No validation, silent failures
+
+# GOOD - typed and validated
+class UserData(BaseModel):
+    name: str
+    created_at: datetime
+
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def ensure_tz_aware(cls, v):
+        if isinstance(v, str):
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if v.tzinfo is None:
+            return v.replace(tzinfo=timezone.utc)
+        return v
+
+def process(data: UserData) -> str:
+    return data.name  # Type-safe, validated at construction
+```
 
 ### TypeScript Style
 - Next.js App Router for control plane

README.md

@@ -5,7 +5,7 @@
 [Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://vectorize.io/hindsight/cloud)
 
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
-[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3klo21kua-VUCC_zHP5rIcXFB1_5yw6A)
+[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
@@ -242,7 +242,7 @@ client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 - [CLI](https://hindsight.vectorize.io/sdks/cli)
 
 **Community:**
-- [Slack](https://join.slack.com/t/hindsight-space/shared_invite/zt-3klo21kua-VUCC_zHP5rIcXFB1_5yw6A)
+- [Slack](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 - [GitHub Issues](https://github.com/vectorize-io/hindsight/issues)
 
 ---

helm/hindsight/templates/_helpers.tpl

@@ -80,6 +80,22 @@ Control plane selector labels
 app.kubernetes.io/component: control-plane
 {{- end }}
 
+{{/*
+Worker labels
+*/}}
+{{- define "hindsight.worker.labels" -}}
+{{ include "hindsight.labels" . }}
+app.kubernetes.io/component: worker
+{{- end }}
+
+{{/*
+Worker selector labels
+*/}}
+{{- define "hindsight.worker.selectorLabels" -}}
+{{ include "hindsight.selectorLabels" . }}
+app.kubernetes.io/component: worker
+{{- end }}
+
 {{/*
 Create the name of the service account to use
 */}}

helm/hindsight/templates/api-deployment.yaml

@@ -55,6 +55,11 @@ spec:
         {{- end }}
         - name: HINDSIGHT_API_DATABASE_URL
           value: {{ include "hindsight.databaseUrl" . | quote }}
+        {{- /* Disable internal worker when dedicated workers are enabled */}}
+        {{- if .Values.worker.enabled }}
+        - name: HINDSIGHT_API_WORKER_ENABLED
+          value: "false"
+        {{- end }}
         {{- range $key, $value := .Values.api.env }}
         - name: {{ $key }}
           value: {{ $value | quote }}

helm/hindsight/templates/worker-service.yaml

@@ -0,0 +1,25 @@
+{{- if .Values.worker.enabled }}
+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ include "hindsight.fullname" . }}-worker
+  labels:
+    {{- include "hindsight.worker.labels" . | nindent 4 }}
+  {{- if .Values.podAnnotations }}
+  annotations:
+    {{- /* Common Prometheus annotations for metrics scraping */}}
+    prometheus.io/scrape: "true"
+    prometheus.io/port: {{ .Values.worker.service.port | quote }}
+    prometheus.io/path: "/metrics"
+  {{- end }}
+spec:
+  # Headless service for StatefulSet (enables stable DNS names like worker-0.worker.namespace)
+  clusterIP: None
+  ports:
+  - port: {{ .Values.worker.service.port }}
+    targetPort: {{ .Values.worker.service.targetPort }}
+    protocol: TCP
+    name: http
+  selector:
+    {{- include "hindsight.worker.selectorLabels" . | nindent 4 }}
+{{- end }}

helm/hindsight/templates/worker-statefulset.yaml

@@ -0,0 +1,110 @@
+{{- if .Values.worker.enabled }}
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: {{ include "hindsight.fullname" . }}-worker
+  labels:
+    {{- include "hindsight.worker.labels" . | nindent 4 }}
+spec:
+  serviceName: {{ include "hindsight.fullname" . }}-worker
+  replicas: {{ .Values.worker.replicaCount }}
+  selector:
+    matchLabels:
+      {{- include "hindsight.worker.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      annotations:
+        {{- if not .Values.existingSecret }}
+        checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
+        {{- end }}
+        {{- with .Values.podAnnotations }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
+      labels:
+        {{- include "hindsight.worker.selectorLabels" . | nindent 8 }}
+    spec:
+      {{- if .Values.serviceAccount.create }}
+      serviceAccountName: {{ include "hindsight.serviceAccountName" . }}
+      {{- end }}
+      securityContext:
+        {{- toYaml .Values.podSecurityContext | nindent 8 }}
+      containers:
+      - name: worker
+        securityContext:
+          {{- toYaml .Values.securityContext | nindent 10 }}
+        image: "{{ .Values.worker.image.repository }}:{{ .Values.worker.image.tag | default .Values.version }}"
+        imagePullPolicy: {{ .Values.worker.image.pullPolicy }}
+        command: ["hindsight-worker"]
+        ports:
+        - name: http
+          containerPort: {{ .Values.worker.service.targetPort }}
+          protocol: TCP
+        {{- if .Values.existingSecret }}
+        envFrom:
+        - secretRef:
+            name: {{ .Values.existingSecret }}
+        {{- end }}
+        env:
+        {{- /* POSTGRES_PASSWORD must be defined before DATABASE_URL for $(VAR) interpolation */}}
+        {{- if not .Values.postgresql.enabled }}
+        - name: POSTGRES_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: {{ include "hindsight.secretName" . }}
+              key: postgres-password
+        {{- end }}
+        - name: HINDSIGHT_API_DATABASE_URL
+          value: {{ include "hindsight.databaseUrl" . | quote }}
+        {{- /* Worker ID uses pod name (StatefulSet provides stable names like worker-0, worker-1) */}}
+        - name: HINDSIGHT_API_WORKER_ID
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        {{- /* Inherit LLM config from api.env */}}
+        {{- range $key, $value := .Values.api.env }}
+        - name: {{ $key }}
+          value: {{ $value | quote }}
+        {{- end }}
+        {{- /* Worker-specific env vars */}}
+        {{- range $key, $value := .Values.worker.env }}
+        - name: {{ $key }}
+          value: {{ $value | quote }}
+        {{- end }}
+        {{- /* Only use secrets when not using existingSecret */}}
+        {{- if not .Values.existingSecret }}
+        {{- /* Inherit secrets from api.secrets */}}
+        {{- range $key, $value := .Values.api.secrets }}
+        - name: {{ $key }}
+          valueFrom:
+            secretKeyRef:
+              name: {{ include "hindsight.secretName" $ }}
+              key: {{ $key }}
+        {{- end }}
+        {{- /* Worker-specific secrets (can override api.secrets) */}}
+        {{- range $key, $value := .Values.worker.secrets }}
+        - name: {{ $key }}
+          valueFrom:
+            secretKeyRef:
+              name: {{ include "hindsight.secretName" $ }}
+              key: {{ $key }}
+        {{- end }}
+        {{- end }}
+        livenessProbe:
+          {{- toYaml .Values.worker.livenessProbe | nindent 10 }}
+        readinessProbe:
+          {{- toYaml .Values.worker.readinessProbe | nindent 10 }}
+        resources:
+          {{- toYaml .Values.worker.resources | nindent 10 }}
+      {{- with .Values.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+{{- end }}

helm/hindsight/values.yaml

@@ -67,6 +67,63 @@ api:
     # HINDSIGHT_API_LLM_API_KEY: "your-api-key"
     # HINDSIGHT_API_LLM_BASE_URL: "https://api.groq.com/openai/v1"
 
+# Worker settings (distributed task processing)
+# When enabled, dedicated worker pods process tasks and the API's internal worker is disabled
+worker:
+  enabled: false
+  replicaCount: 2
+  image:
+    repository: ghcr.io/vectorize-io/hindsight-api
+    pullPolicy: IfNotPresent
+    # tag defaults to .Values.version if not specified
+
+  service:
+    # Service for metrics scraping (headless for StatefulSet)
+    port: 8889
+    targetPort: 8889
+
+  # Resource limits and requests
+  resources:
+    limits:
+      cpu: 2000m
+      memory: 4Gi
+    requests:
+      cpu: 500m
+      memory: 1Gi
+
+  # Liveness and readiness probes
+  livenessProbe:
+    httpGet:
+      path: /health
+      port: 8889
+    initialDelaySeconds: 30
+    periodSeconds: 10
+    timeoutSeconds: 5
+    failureThreshold: 3
+
+  readinessProbe:
+    httpGet:
+      path: /health
+      port: 8889
+    initialDelaySeconds: 10
+    periodSeconds: 5
+    timeoutSeconds: 3
+    failureThreshold: 3
+
+  # Worker-specific environment variables
+  env:
+    # Poll interval in milliseconds (how often to check for new tasks)
+    HINDSIGHT_API_WORKER_POLL_INTERVAL_MS: "500"
+    # Number of tasks to claim per poll cycle
+    HINDSIGHT_API_WORKER_BATCH_SIZE: "10"
+    # Max retries before marking a task as failed
+    HINDSIGHT_API_WORKER_MAX_RETRIES: "3"
+    # HTTP port for metrics/health (matches service.targetPort)
+    HINDSIGHT_API_WORKER_HTTP_PORT: "8889"
+
+  # Secret environment variables (inherited from api.secrets if not specified)
+  secrets: {}
+
 # Image settings for control plane
 controlPlane:
   enabled: true

hindsight-api/hindsight_api/admin/cli.py

@@ -244,6 +244,65 @@ def run_db_migration(
     typer.echo("Database migrations completed successfully")
 
 
+async def _decommission_worker(db_url: str, worker_id: str, schema: str = "public") -> int:
+    """Release all tasks owned by a worker, setting them back to pending status."""
+    is_pg0, instance_name, _ = parse_pg0_url(db_url)
+    if is_pg0:
+        typer.echo(f"Starting embedded PostgreSQL (instance: {instance_name})...")
+    resolved_url = await resolve_database_url(db_url)
+
+    conn = await asyncpg.connect(resolved_url)
+    try:
+        table = _fq_table("async_operations", schema)
+        result = await conn.fetch(
+            f"""
+            UPDATE {table}
+            SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
+            WHERE worker_id = $1 AND status = 'processing'
+            RETURNING operation_id
+            """,
+            worker_id,
+        )
+        return len(result)
+    finally:
+        await conn.close()
+
+
+@app.command(name="decommission-worker")
+def decommission_worker(
+    worker_id: str = typer.Argument(..., help="Worker ID to decommission"),
+    schema: str = typer.Option("public", "--schema", "-s", help="Database schema"),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt"),
+):
+    """Release all tasks owned by a worker (sets status back to pending).
+
+    Use this command when a worker has crashed or been removed without graceful shutdown.
+    All tasks that were being processed by the worker will be released back to the queue
+    so other workers can pick them up.
+    """
+    config = HindsightConfig.from_env()
+
+    if not config.database_url:
+        typer.echo("Error: Database URL not configured.", err=True)
+        typer.echo("Set HINDSIGHT_API_DATABASE_URL environment variable.", err=True)
+        raise typer.Exit(1)
+
+    if not yes:
+        typer.confirm(
+            f"This will release all tasks owned by worker '{worker_id}' back to pending. Continue?",
+            abort=True,
+        )
+
+    typer.echo(f"Decommissioning worker '{worker_id}' (schema: {schema})...")
+
+    count = asyncio.run(_decommission_worker(config.database_url, worker_id, schema))
+
+    if count > 0:
+        typer.echo(f"Released {count} task(s) from worker '{worker_id}'")
+    else:
+        typer.echo(f"No tasks found for worker '{worker_id}'")
+
+
 def main():
     app()
 

hindsight-api/hindsight_api/alembic/versions/h3c4d5e6f7g8_mental_models_v4.py

@@ -0,0 +1,112 @@
+"""mental_models_v4
+
+Revision ID: h3c4d5e6f7g8
+Revises: g2a3b4c5d6e7
+Create Date: 2026-01-08 00:00:00.000000
+
+This migration implements the v4 mental models system:
+1. Deletes existing observation memory_units (observations now in mental models)
+2. Adds mission column to banks (replacing background)
+3. Creates mental_models table with final schema
+
+Mental models can reference entities when an entity is "promoted" to a mental model.
+Summary content is stored as JSONB observations with per-observation fact attribution.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "h3c4d5e6f7g8"
+down_revision: str | Sequence[str] | None = "g2a3b4c5d6e7"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Apply mental models v4 changes."""
+    schema = _get_schema_prefix()
+
+    # Step 1: Delete observation memory_units (cascades to unit_entities links)
+    # Observations are now handled through mental models, not memory_units
+    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'observation'")
+
+    # Step 2: Drop observation-specific index (if it exists)
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_observation_date")
+
+    # Step 3: Add mission column to banks (replacing background)
+    op.execute(f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS mission TEXT")
+
+    # Migrate: copy background to mission if background column exists
+    # Use DO block to check column existence first (idempotent for re-runs)
+    schema_name = context.config.get_main_option("target_schema") or "public"
+    op.execute(f"""
+        DO $$
+        BEGIN
+            IF EXISTS (
+                SELECT 1 FROM information_schema.columns
+                WHERE table_schema = '{schema_name}' AND table_name = 'banks' AND column_name = 'background'
+            ) THEN
+                UPDATE {schema}banks
+                SET mission = background
+                WHERE mission IS NULL;
+            END IF;
+        END $$;
+    """)
+
+    # Remove background column (replaced by mission)
+    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS background")
+
+    # Step 4: Create mental_models table with final v4 schema (if not exists)
+    op.execute(f"""
+        CREATE TABLE IF NOT EXISTS {schema}mental_models (
+            id VARCHAR(64) NOT NULL,
+            bank_id VARCHAR(64) NOT NULL,
+            subtype VARCHAR(32) NOT NULL,
+            name VARCHAR(256) NOT NULL,
+            description TEXT NOT NULL,
+            entity_id UUID,
+            observations JSONB DEFAULT '{{"observations": []}}'::jsonb,
+            links VARCHAR[],
+            tags VARCHAR[] DEFAULT '{{}}',
+            last_updated TIMESTAMP WITH TIME ZONE,
+            created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
+            PRIMARY KEY (id, bank_id),
+            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id) ON DELETE CASCADE,
+            FOREIGN KEY (entity_id) REFERENCES {schema}entities(id) ON DELETE SET NULL,
+            CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
+        )
+    """)
+
+    # Step 5: Create indexes for efficient queries (if not exist)
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_bank_id ON {schema}mental_models(bank_id)")
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_entity_id ON {schema}mental_models(entity_id)")
+    # GIN index for efficient tags array filtering
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_tags ON {schema}mental_models USING GIN(tags)")
+
+
+def downgrade() -> None:
+    """Revert mental models v4 changes."""
+    schema = _get_schema_prefix()
+
+    # Drop mental_models table (cascades to indexes)
+    op.execute(f"DROP TABLE IF EXISTS {schema}mental_models CASCADE")
+
+    # Add back background column to banks
+    op.execute(f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS background TEXT")
+
+    # Migrate mission back to background
+    op.execute(f"UPDATE {schema}banks SET background = mission WHERE background IS NULL")
+
+    # Remove mission column
+    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission")
+
+    # Note: Cannot restore deleted observations - they are lost on downgrade

hindsight-api/hindsight_api/alembic/versions/i4d5e6f7g8h9_delete_opinions.py

@@ -0,0 +1,41 @@
+"""delete_opinions
+
+Revision ID: i4d5e6f7g8h9
+Revises: h3c4d5e6f7g8
+Create Date: 2026-01-15 00:00:00.000000
+
+This migration removes opinion facts from memory_units.
+Opinions are no longer a separate fact type - they are now represented
+through mental model observations with confidence scores.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "i4d5e6f7g8h9"
+down_revision: str | Sequence[str] | None = "h3c4d5e6f7g8"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Delete opinion memory_units."""
+    schema = _get_schema_prefix()
+
+    # Delete opinion memory_units (cascades to unit_entities links)
+    # Opinions are now handled through mental model observations
+    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
+
+
+def downgrade() -> None:
+    """Cannot restore deleted opinions."""
+    # Note: Cannot restore deleted opinions - they are lost on downgrade
+    pass

hindsight-api/hindsight_api/alembic/versions/j5e6f7g8h9i0_mental_model_versions.py

@@ -0,0 +1,95 @@
+"""mental_model_versions
+
+Revision ID: j5e6f7g8h9i0
+Revises: i4d5e6f7g8h9
+Create Date: 2026-01-16 00:00:00.000000
+
+This migration adds versioning support for mental models:
+1. Creates mental_model_versions table to store observation snapshots
+2. Adds version column to mental_models for tracking current version
+
+This enables changelog/diff functionality for mental model observations.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "j5e6f7g8h9i0"
+down_revision: str | Sequence[str] | None = "i4d5e6f7g8h9"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Create mental_model_versions table and add version tracking."""
+    schema = _get_schema_prefix()
+
+    # Create mental_model_versions table for storing observation snapshots
+    op.execute(f"""
+        CREATE TABLE {schema}mental_model_versions (
+            id SERIAL PRIMARY KEY,
+            mental_model_id VARCHAR(64) NOT NULL,
+            bank_id VARCHAR(64) NOT NULL,
+            version INT NOT NULL,
+            observations JSONB NOT NULL DEFAULT '{{"observations": []}}'::jsonb,
+            created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
+            FOREIGN KEY (mental_model_id, bank_id)
+                REFERENCES {schema}mental_models(id, bank_id) ON DELETE CASCADE,
+            UNIQUE (mental_model_id, bank_id, version)
+        )
+    """)
+
+    # Index for efficient version queries (get latest, list versions)
+    op.execute(f"""
+        CREATE INDEX idx_mental_model_versions_lookup
+        ON {schema}mental_model_versions(mental_model_id, bank_id, version DESC)
+    """)
+
+    # Add version column to mental_models to track current version
+    op.execute(f"""
+        ALTER TABLE {schema}mental_models
+        ADD COLUMN IF NOT EXISTS version INT NOT NULL DEFAULT 0
+    """)
+
+    # Migrate existing mental models: create version 1 for any that have observations
+    op.execute(f"""
+        INSERT INTO {schema}mental_model_versions (mental_model_id, bank_id, version, observations, created_at)
+        SELECT id, bank_id, 1, observations, COALESCE(last_updated, created_at)
+        FROM {schema}mental_models
+        WHERE observations IS NOT NULL
+          AND observations != '{{"observations": []}}'::jsonb
+          AND (observations->'observations') IS NOT NULL
+          AND jsonb_array_length(observations->'observations') > 0
+    """)
+
+    # Update version to 1 for migrated mental models
+    op.execute(f"""
+        UPDATE {schema}mental_models
+        SET version = 1
+        WHERE observations IS NOT NULL
+          AND observations != '{{"observations": []}}'::jsonb
+          AND (observations->'observations') IS NOT NULL
+          AND jsonb_array_length(observations->'observations') > 0
+    """)
+
+
+def downgrade() -> None:
+    """Remove mental_model_versions table and version column."""
+    schema = _get_schema_prefix()
+
+    # Drop index
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_mental_model_versions_lookup")
+
+    # Drop versions table
+    op.execute(f"DROP TABLE IF EXISTS {schema}mental_model_versions")
+
+    # Remove version column from mental_models
+    op.execute(f"ALTER TABLE {schema}mental_models DROP COLUMN IF EXISTS version")

hindsight-api/hindsight_api/alembic/versions/k6f7g8h9i0j1_add_directive_subtype.py

@@ -0,0 +1,58 @@
+"""add_directive_subtype
+
+Revision ID: k6f7g8h9i0j1
+Revises: j5e6f7g8h9i0
+Create Date: 2026-01-16 00:00:00.000000
+
+This migration adds 'directive' to the mental_models subtype constraint.
+Directives are hard rules with user-provided observations that the reflect agent must follow.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "k6f7g8h9i0j1"
+down_revision: str | Sequence[str] | None = "j5e6f7g8h9i0"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Add 'directive' to mental_models subtype constraint."""
+    schema = _get_schema_prefix()
+
+    # Drop existing constraint
+    op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
+
+    # Create new constraint with 'directive' added
+    op.execute(f"""
+        ALTER TABLE {schema}mental_models
+        ADD CONSTRAINT ck_mental_models_subtype
+        CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned', 'directive'))
+    """)
+
+
+def downgrade() -> None:
+    """Remove 'directive' from mental_models subtype constraint."""
+    schema = _get_schema_prefix()
+
+    # First delete any directives (cannot downgrade if they exist)
+    op.execute(f"DELETE FROM {schema}mental_models WHERE subtype = 'directive'")
+
+    # Drop constraint with directive
+    op.execute(f"ALTER TABLE {schema}mental_models DROP CONSTRAINT IF EXISTS ck_mental_models_subtype")
+
+    # Recreate original constraint without directive
+    op.execute(f"""
+        ALTER TABLE {schema}mental_models
+        ADD CONSTRAINT ck_mental_models_subtype
+        CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
+    """)

hindsight-api/hindsight_api/alembic/versions/l7g8h9i0j1k2_add_worker_columns.py

@@ -0,0 +1,109 @@
+"""add_worker_columns
+
+Revision ID: l7g8h9i0j1k2
+Revises: k6f7g8h9i0j1
+Create Date: 2026-01-19 00:00:00.000000
+
+This migration adds columns to async_operations for distributed worker support:
+- worker_id: ID of the worker that claimed the task
+- claimed_at: When the task was claimed
+- retry_count: Number of retry attempts
+- task_payload: The serialized task dictionary
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+from alembic import context, op
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision: str = "l7g8h9i0j1k2"
+down_revision: str | Sequence[str] | None = "k6f7g8h9i0j1"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Add worker columns to async_operations."""
+    schema = _get_schema_prefix()
+
+    # Add worker_id column (ID of worker that claimed the task)
+    op.add_column(
+        "async_operations",
+        sa.Column("worker_id", sa.Text(), nullable=True),
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+
+    # Add claimed_at column (when task was claimed by worker)
+    op.add_column(
+        "async_operations",
+        sa.Column("claimed_at", postgresql.TIMESTAMP(timezone=True), nullable=True),
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+
+    # Add retry_count column (number of retry attempts)
+    op.add_column(
+        "async_operations",
+        sa.Column("retry_count", sa.Integer(), server_default="0", nullable=False),
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+
+    # Add task_payload column (serialized task dictionary)
+    op.add_column(
+        "async_operations",
+        sa.Column(
+            "task_payload",
+            postgresql.JSONB(astext_type=sa.Text()),
+            nullable=True,
+        ),
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+
+    # Add index for efficient worker polling (pending tasks ordered by creation time)
+    op.execute(
+        f"CREATE INDEX idx_async_operations_pending_claim ON {schema}async_operations (status, created_at) "
+        f"WHERE status = 'pending' AND task_payload IS NOT NULL"
+    )
+
+    # Add index for finding tasks by worker_id (for decommissioning)
+    op.execute(
+        f"CREATE INDEX idx_async_operations_worker_id ON {schema}async_operations (worker_id) WHERE worker_id IS NOT NULL"
+    )
+
+
+def downgrade() -> None:
+    """Remove worker columns from async_operations."""
+    schema = _get_schema_prefix()
+
+    # Drop indexes
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_pending_claim")
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_worker_id")
+
+    # Drop columns
+    op.drop_column(
+        "async_operations",
+        "task_payload",
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+    op.drop_column(
+        "async_operations",
+        "retry_count",
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+    op.drop_column(
+        "async_operations",
+        "claimed_at",
+        schema=context.config.get_main_option("target_schema") or None,
+    )
+    op.drop_column(
+        "async_operations",
+        "worker_id",
+        schema=context.config.get_main_option("target_schema") or None,
+    )

hindsight-api/hindsight_api/api/mcp.py

@@ -196,7 +196,7 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
         Each bank is an isolated memory store (like a separate "brain").
 
         Returns:
-            JSON list of banks with their IDs, names, dispositions, and backgrounds.
+            JSON list of banks with their IDs, names, dispositions, and missions.
         """
         try:
             banks = await memory.list_banks(request_context=RequestContext())
@@ -206,7 +206,7 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
             return f'{{"error": "{e}", "banks": []}}'
 
     @mcp.tool()
-    async def create_bank(bank_id: str, name: str | None = None, background: str | None = None) -> str:
+    async def create_bank(bank_id: str, name: str | None = None, mission: str | None = None) -> str:
         """
         Create a new memory bank or get an existing one.
 
@@ -216,18 +216,18 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
         Args:
             bank_id: Unique identifier for the bank (e.g., 'user-123', 'agent-alpha')
             name: Optional human-friendly name for the bank
-            background: Optional background context about the bank's owner/purpose
+            mission: Optional mission describing who the agent is and what they're trying to accomplish
         """
         try:
             # get_bank_profile auto-creates bank if it doesn't exist
             profile = await memory.get_bank_profile(bank_id, request_context=RequestContext())
 
-            # Update name/background if provided
-            if name is not None or background is not None:
+            # Update name/mission if provided
+            if name is not None or mission is not None:
                 await memory.update_bank(
                     bank_id,
                     name=name,
-                    background=background,
+                    mission=mission,
                     request_context=RequestContext(),
                 )
                 # Fetch updated profile

hindsight-api/hindsight_api/config.py

@@ -80,6 +80,7 @@ ENV_RECALL_MAX_CONCURRENT = "HINDSIGHT_API_RECALL_MAX_CONCURRENT"
 ENV_RECALL_CONNECTION_BUDGET = "HINDSIGHT_API_RECALL_CONNECTION_BUDGET"
 ENV_MCP_LOCAL_BANK_ID = "HINDSIGHT_API_MCP_LOCAL_BANK_ID"
 ENV_MCP_INSTRUCTIONS = "HINDSIGHT_API_MCP_INSTRUCTIONS"
+ENV_MENTAL_MODEL_REFRESH_CONCURRENCY = "HINDSIGHT_API_MENTAL_MODEL_REFRESH_CONCURRENCY"
 
 # Observation thresholds
 ENV_OBSERVATION_MIN_FACTS = "HINDSIGHT_API_OBSERVATION_MIN_FACTS"
@@ -105,10 +106,16 @@ ENV_DB_POOL_MAX_SIZE = "HINDSIGHT_API_DB_POOL_MAX_SIZE"
 ENV_DB_COMMAND_TIMEOUT = "HINDSIGHT_API_DB_COMMAND_TIMEOUT"
 ENV_DB_ACQUIRE_TIMEOUT = "HINDSIGHT_API_DB_ACQUIRE_TIMEOUT"
 
-# Background task processing
-ENV_TASK_BACKEND = "HINDSIGHT_API_TASK_BACKEND"
-ENV_TASK_BACKEND_MEMORY_BATCH_SIZE = "HINDSIGHT_API_TASK_BACKEND_MEMORY_BATCH_SIZE"
-ENV_TASK_BACKEND_MEMORY_BATCH_INTERVAL = "HINDSIGHT_API_TASK_BACKEND_MEMORY_BATCH_INTERVAL"
+# Worker configuration (distributed task processing)
+ENV_WORKER_ENABLED = "HINDSIGHT_API_WORKER_ENABLED"
+ENV_WORKER_ID = "HINDSIGHT_API_WORKER_ID"
+ENV_WORKER_POLL_INTERVAL_MS = "HINDSIGHT_API_WORKER_POLL_INTERVAL_MS"
+ENV_WORKER_MAX_RETRIES = "HINDSIGHT_API_WORKER_MAX_RETRIES"
+ENV_WORKER_BATCH_SIZE = "HINDSIGHT_API_WORKER_BATCH_SIZE"
+ENV_WORKER_HTTP_PORT = "HINDSIGHT_API_WORKER_HTTP_PORT"
+
+# Reflect agent settings
+ENV_REFLECT_MAX_ITERATIONS = "HINDSIGHT_API_REFLECT_MAX_ITERATIONS"
 
 # Default values
 DEFAULT_DATABASE_URL = "pg0"
@@ -150,6 +157,7 @@ DEFAULT_MPFP_TOP_K_NEIGHBORS = 20  # Fan-out limit per node in MPFP graph traver
 DEFAULT_RECALL_MAX_CONCURRENT = 32  # Max concurrent recall operations per worker
 DEFAULT_RECALL_CONNECTION_BUDGET = 4  # Max concurrent DB connections per recall operation
 DEFAULT_MCP_LOCAL_BANK_ID = "mcp"
+DEFAULT_MENTAL_MODEL_REFRESH_CONCURRENCY = 8  # Max concurrent mental model refreshes
 
 # Observation thresholds
 DEFAULT_OBSERVATION_MIN_FACTS = 5  # Min facts required to generate entity observations
@@ -172,10 +180,16 @@ DEFAULT_DB_POOL_MAX_SIZE = 100
 DEFAULT_DB_COMMAND_TIMEOUT = 60  # seconds
 DEFAULT_DB_ACQUIRE_TIMEOUT = 30  # seconds
 
-# Background task processing
-DEFAULT_TASK_BACKEND = "memory"  # Options: "memory", "noop"
-DEFAULT_TASK_BACKEND_MEMORY_BATCH_SIZE = 10
-DEFAULT_TASK_BACKEND_MEMORY_BATCH_INTERVAL = 1.0  # seconds
+# Worker configuration (distributed task processing)
+DEFAULT_WORKER_ENABLED = True  # API runs worker by default (standalone mode)
+DEFAULT_WORKER_ID = None  # Will use hostname if not specified
+DEFAULT_WORKER_POLL_INTERVAL_MS = 500  # Poll database every 500ms
+DEFAULT_WORKER_MAX_RETRIES = 3  # Max retries before marking task failed
+DEFAULT_WORKER_BATCH_SIZE = 10  # Tasks to claim per poll cycle
+DEFAULT_WORKER_HTTP_PORT = 8889  # HTTP port for worker metrics/health
+
+# Reflect agent settings
+DEFAULT_REFLECT_MAX_ITERATIONS = 10  # Max tool call iterations before forcing response
 
 # Default MCP tool descriptions (can be customized via env vars)
 DEFAULT_MCP_RETAIN_DESCRIPTION = """Store important information to long-term memory.
@@ -297,6 +311,7 @@ class HindsightConfig:
     mpfp_top_k_neighbors: int
     recall_max_concurrent: int
     recall_connection_budget: int
+    mental_model_refresh_concurrency: int
 
     # Observation thresholds
     observation_min_facts: int
@@ -322,10 +337,16 @@ class HindsightConfig:
     db_command_timeout: int
     db_acquire_timeout: int
 
-    # Background task processing
-    task_backend: str
-    task_backend_memory_batch_size: int
-    task_backend_memory_batch_interval: float
+    # Worker configuration (distributed task processing)
+    worker_enabled: bool
+    worker_id: str | None
+    worker_poll_interval_ms: int
+    worker_max_retries: int
+    worker_batch_size: int
+    worker_http_port: int
+
+    # Reflect agent settings
+    reflect_max_iterations: int
 
     @classmethod
     def from_env(cls) -> "HindsightConfig":
@@ -378,6 +399,9 @@ class HindsightConfig:
             recall_connection_budget=int(
                 os.getenv(ENV_RECALL_CONNECTION_BUDGET, str(DEFAULT_RECALL_CONNECTION_BUDGET))
             ),
+            mental_model_refresh_concurrency=int(
+                os.getenv(ENV_MENTAL_MODEL_REFRESH_CONCURRENCY, str(DEFAULT_MENTAL_MODEL_REFRESH_CONCURRENCY))
+            ),
             # Optimization flags
             skip_llm_verification=os.getenv(ENV_SKIP_LLM_VERIFICATION, "false").lower() == "true",
             lazy_reranker=os.getenv(ENV_LAZY_RERANKER, "false").lower() == "true",
@@ -409,14 +433,15 @@ class HindsightConfig:
             db_pool_max_size=int(os.getenv(ENV_DB_POOL_MAX_SIZE, str(DEFAULT_DB_POOL_MAX_SIZE))),
             db_command_timeout=int(os.getenv(ENV_DB_COMMAND_TIMEOUT, str(DEFAULT_DB_COMMAND_TIMEOUT))),
             db_acquire_timeout=int(os.getenv(ENV_DB_ACQUIRE_TIMEOUT, str(DEFAULT_DB_ACQUIRE_TIMEOUT))),
-            # Background task processing
-            task_backend=os.getenv(ENV_TASK_BACKEND, DEFAULT_TASK_BACKEND),
-            task_backend_memory_batch_size=int(
-                os.getenv(ENV_TASK_BACKEND_MEMORY_BATCH_SIZE, str(DEFAULT_TASK_BACKEND_MEMORY_BATCH_SIZE))
-            ),
-            task_backend_memory_batch_interval=float(
-                os.getenv(ENV_TASK_BACKEND_MEMORY_BATCH_INTERVAL, str(DEFAULT_TASK_BACKEND_MEMORY_BATCH_INTERVAL))
-            ),
+            # Worker configuration
+            worker_enabled=os.getenv(ENV_WORKER_ENABLED, str(DEFAULT_WORKER_ENABLED)).lower() == "true",
+            worker_id=os.getenv(ENV_WORKER_ID) or DEFAULT_WORKER_ID,
+            worker_poll_interval_ms=int(os.getenv(ENV_WORKER_POLL_INTERVAL_MS, str(DEFAULT_WORKER_POLL_INTERVAL_MS))),
+            worker_max_retries=int(os.getenv(ENV_WORKER_MAX_RETRIES, str(DEFAULT_WORKER_MAX_RETRIES))),
+            worker_batch_size=int(os.getenv(ENV_WORKER_BATCH_SIZE, str(DEFAULT_WORKER_BATCH_SIZE))),
+            worker_http_port=int(os.getenv(ENV_WORKER_HTTP_PORT, str(DEFAULT_WORKER_HTTP_PORT))),
+            # Reflect agent settings
+            reflect_max_iterations=int(os.getenv(ENV_REFLECT_MAX_ITERATIONS, str(DEFAULT_REFLECT_MAX_ITERATIONS))),
         )
 
     def get_llm_base_url(self) -> str:

hindsight-api/hindsight_api/engine/interface.py

@@ -160,14 +160,14 @@ class MemoryEngineInterface(ABC):
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Get bank profile including disposition and background.
+        Get bank profile including disposition and mission.
 
         Args:
             bank_id: The memory bank ID.
             request_context: Request context for authentication.
 
         Returns:
-            Bank profile dict.
+            Bank profile dict with bank_id, name, disposition, and mission.
         """
         ...
 
@@ -190,25 +190,44 @@ class MemoryEngineInterface(ABC):
         ...
 
     @abstractmethod
-    async def merge_bank_background(
+    async def merge_bank_mission(
         self,
         bank_id: str,
         new_info: str,
         *,
-        update_disposition: bool = True,
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Merge new background information into bank profile.
+        Merge new mission information into bank profile.
 
         Args:
             bank_id: The memory bank ID.
-            new_info: New background information to merge.
-            update_disposition: Whether to infer disposition from background.
+            new_info: New mission information to merge.
             request_context: Request context for authentication.
 
         Returns:
-            Updated background info.
+            Updated mission info.
+        """
+        ...
+
+    @abstractmethod
+    async def set_bank_mission(
+        self,
+        bank_id: str,
+        mission: str,
+        *,
+        request_context: "RequestContext",
+    ) -> dict[str, Any]:
+        """
+        Set the bank's mission (replaces existing).
+
+        Args:
+            bank_id: The memory bank ID.
+            mission: The mission text.
+            request_context: Request context for authentication.
+
+        Returns:
+            Dict with bank_id and mission.
         """
         ...
 
@@ -518,7 +537,7 @@ class MemoryEngineInterface(ABC):
         bank_id: str,
         *,
         request_context: "RequestContext",
-    ) -> list[dict[str, Any]]:
+    ) -> dict[str, Any]:
         """
         List async operations for a bank.
 
@@ -527,7 +546,7 @@ class MemoryEngineInterface(ABC):
             request_context: Request context for authentication.
 
         Returns:
-            List of operation dicts with id, task_type, status, etc.
+            Dict with 'total' (int) and 'operations' (list of operation dicts).
         """
         ...
 
@@ -561,16 +580,16 @@ class MemoryEngineInterface(ABC):
         bank_id: str,
         *,
         name: str | None = None,
-        background: str | None = None,
+        mission: str | None = None,
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Update bank name and/or background.
+        Update bank name and/or mission.
 
         Args:
             bank_id: The memory bank ID.
             name: New bank name (optional).
-            background: New background text (optional, replaces existing).
+            mission: New mission text (optional, replaces existing).
             request_context: Request context for authentication.
 
         Returns:

hindsight-api/hindsight_api/engine/llm_wrapper.py

@@ -209,10 +209,10 @@ class LLMProvider:
             OutputTooLongError: If output exceeds token limits.
             Exception: Re-raises API errors after retries exhausted.
         """
-        queue_start_time = time.time()
+        semaphore_start = time.time()
         async with _global_llm_semaphore:
+            semaphore_wait_time = time.time() - semaphore_start
             start_time = time.time()
-            semaphore_wait_time = start_time - queue_start_time
 
             # Handle Mock provider (for testing)
             if self.provider == "mock":
@@ -318,43 +318,44 @@ class LLMProvider:
 
             last_exception = None
 
+            # Prepare response format ONCE before the retry loop
+            # (to avoid appending schema to messages on every retry)
+            if response_format is not None:
+                schema = None
+                if hasattr(response_format, "model_json_schema"):
+                    schema = response_format.model_json_schema()
+
+                if strict_schema and schema is not None:
+                    # Use OpenAI's strict JSON schema enforcement
+                    # This guarantees all required fields are returned
+                    call_params["response_format"] = {
+                        "type": "json_schema",
+                        "json_schema": {
+                            "name": "response",
+                            "strict": True,
+                            "schema": schema,
+                        },
+                    }
+                else:
+                    # Soft enforcement: add schema to prompt and use json_object mode
+                    if schema is not None:
+                        schema_msg = f"\n\nYou must respond with valid JSON matching this schema:\n{json.dumps(schema, indent=2)}"
+
+                        if call_params["messages"] and call_params["messages"][0].get("role") == "system":
+                            call_params["messages"][0]["content"] += schema_msg
+                        elif call_params["messages"]:
+                            call_params["messages"][0]["content"] = (
+                                schema_msg + "\n\n" + call_params["messages"][0]["content"]
+                            )
+                    if self.provider not in ("lmstudio", "ollama"):
+                        # LM Studio and Ollama don't support json_object response format reliably
+                        # We rely on the schema in the system message instead
+                        call_params["response_format"] = {"type": "json_object"}
+
             for attempt in range(max_retries + 1):
                 try:
                     if response_format is not None:
-                        schema = None
-                        if hasattr(response_format, "model_json_schema"):
-                            schema = response_format.model_json_schema()
-
-                        if strict_schema and schema is not None:
-                            # Use OpenAI's strict JSON schema enforcement
-                            # This guarantees all required fields are returned
-                            call_params["response_format"] = {
-                                "type": "json_schema",
-                                "json_schema": {
-                                    "name": "response",
-                                    "strict": True,
-                                    "schema": schema,
-                                },
-                            }
-                        else:
-                            # Soft enforcement: add schema to prompt and use json_object mode
-                            if schema is not None:
-                                schema_msg = f"\n\nYou must respond with valid JSON matching this schema:\n{json.dumps(schema, indent=2)}"
-
-                                if call_params["messages"] and call_params["messages"][0].get("role") == "system":
-                                    call_params["messages"][0]["content"] += schema_msg
-                                elif call_params["messages"]:
-                                    call_params["messages"][0]["content"] = (
-                                        schema_msg + "\n\n" + call_params["messages"][0]["content"]
-                                    )
-                            if self.provider not in ("lmstudio", "ollama"):
-                                # LM Studio and Ollama don't support json_object response format reliably
-                                # We rely on the schema in the system message instead
-                                call_params["response_format"] = {"type": "json_object"}
-
-                        logger.debug(f"Sending request to {self.provider}/{self.model} (timeout={self.timeout})")
                         response = await self._client.chat.completions.create(**call_params)
-                        logger.debug(f"Received response from {self.provider}/{self.model}")
 
                         content = response.choices[0].message.content
 
@@ -467,13 +468,11 @@ class LLMProvider:
 
                 except APIConnectionError as e:
                     last_exception = e
+                    status_code = getattr(e, "status_code", None) or getattr(
+                        getattr(e, "response", None), "status_code", None
+                    )
+                    logger.warning(f"APIConnectionError (HTTP {status_code}), attempt {attempt + 1}: {str(e)[:200]}")
                     if attempt < max_retries:
-                        status_code = getattr(e, "status_code", None) or getattr(
-                            getattr(e, "response", None), "status_code", None
-                        )
-                        logger.warning(
-                            f"Connection error, retrying... (attempt {attempt + 1}/{max_retries + 1}) - status_code={status_code}, message={e}"
-                        )
                         backoff = min(initial_backoff * (2**attempt), max_backoff)
                         await asyncio.sleep(backoff)
                         continue
@@ -487,6 +486,45 @@ class LLMProvider:
                         logger.error(f"Auth error (HTTP {e.status_code}), not retrying: {str(e)}")
                         raise
 
+                    # Handle tool_use_failed error - model outputted in tool call format
+                    # Convert to expected JSON format and continue
+                    if e.status_code == 400 and response_format is not None:
+                        try:
+                            error_body = e.body if hasattr(e, "body") else {}
+                            if isinstance(error_body, dict):
+                                error_info: dict[str, Any] = error_body.get("error") or {}
+                                if error_info.get("code") == "tool_use_failed":
+                                    failed_gen = error_info.get("failed_generation", "")
+                                    if failed_gen:
+                                        # Parse the tool call format and convert to actions format
+                                        tool_call = json.loads(failed_gen)
+                                        tool_name = tool_call.get("name", "")
+                                        tool_args = tool_call.get("arguments", {})
+                                        # Convert to actions format: {"actions": [{"tool": "name", ...args}]}
+                                        converted = {"actions": [{"tool": tool_name, **tool_args}]}
+                                        if skip_validation:
+                                            result = converted
+                                        else:
+                                            result = response_format.model_validate(converted)
+
+                                        # Record metrics for this successful recovery
+                                        duration = time.time() - start_time
+                                        metrics = get_metrics_collector()
+                                        metrics.record_llm_call(
+                                            provider=self.provider,
+                                            model=self.model,
+                                            scope=scope,
+                                            duration=duration,
+                                            input_tokens=0,
+                                            output_tokens=0,
+                                            success=True,
+                                        )
+                                        if return_usage:
+                                            return result, TokenUsage(input_tokens=0, output_tokens=0, total_tokens=0)
+                                        return result
+                        except (json.JSONDecodeError, KeyError, TypeError):
+                            pass  # Failed to parse tool_use_failed, continue with normal retry
+
                     last_exception = e
                     if attempt < max_retries:
                         backoff = min(initial_backoff * (2**attempt), max_backoff)
@@ -497,14 +535,416 @@ class LLMProvider:
                         logger.error(f"API error after {max_retries + 1} attempts: {str(e)}")
                         raise
 
-                except Exception as e:
-                    logger.error(f"Unexpected error during LLM call: {type(e).__name__}: {str(e)}")
+                except Exception:
                     raise
 
             if last_exception:
                 raise last_exception
             raise RuntimeError("LLM call failed after all retries with no exception captured")
 
+    async def call_with_tools(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_completion_tokens: int | None = None,
+        temperature: float | None = None,
+        scope: str = "tools",
+        max_retries: int = 5,
+        initial_backoff: float = 1.0,
+        max_backoff: float = 30.0,
+        tool_choice: str | dict[str, Any] = "auto",
+    ) -> "LLMToolCallResult":
+        """
+        Make an LLM API call with tool/function calling support.
+
+        Args:
+            messages: List of message dicts. Can include tool results with role='tool'.
+            tools: List of tool definitions in OpenAI format.
+            max_completion_tokens: Maximum tokens in response.
+            temperature: Sampling temperature (0.0-2.0).
+            scope: Scope identifier for tracking.
+            max_retries: Maximum retry attempts.
+            initial_backoff: Initial backoff time in seconds.
+            max_backoff: Maximum backoff time in seconds.
+            tool_choice: How to choose tools - "auto", "none", "required", or {"type": "function", "function": {"name": "..."}}
+
+        Returns:
+            LLMToolCallResult with content and/or tool_calls.
+        """
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        async with _global_llm_semaphore:
+            start_time = time.time()
+
+            # Handle Mock provider
+            if self.provider == "mock":
+                return await self._call_with_tools_mock(messages, tools, scope)
+
+            # Handle Anthropic separately (uses different tool format)
+            if self.provider == "anthropic":
+                return await self._call_with_tools_anthropic(
+                    messages, tools, max_completion_tokens, max_retries, initial_backoff, max_backoff, start_time, scope
+                )
+
+            # Handle Gemini (convert to Gemini tool format)
+            if self.provider == "gemini":
+                return await self._call_with_tools_gemini(
+                    messages, tools, max_retries, initial_backoff, max_backoff, start_time, scope
+                )
+
+            # OpenAI-compatible providers (OpenAI, Groq, Ollama, LMStudio)
+            call_params: dict[str, Any] = {
+                "model": self.model,
+                "messages": messages,
+                "tools": tools,
+                "tool_choice": tool_choice,
+            }
+
+            if max_completion_tokens is not None:
+                call_params["max_completion_tokens"] = max_completion_tokens
+            if temperature is not None:
+                call_params["temperature"] = temperature
+
+            # Provider-specific parameters
+            if self.provider == "groq":
+                call_params["seed"] = DEFAULT_LLM_SEED
+
+            last_exception = None
+
+            for attempt in range(max_retries + 1):
+                try:
+                    response = await self._client.chat.completions.create(**call_params)
+
+                    message = response.choices[0].message
+                    finish_reason = response.choices[0].finish_reason
+
+                    # Extract tool calls if present
+                    tool_calls: list[LLMToolCall] = []
+                    if message.tool_calls:
+                        for tc in message.tool_calls:
+                            try:
+                                args = json.loads(tc.function.arguments) if tc.function.arguments else {}
+                            except json.JSONDecodeError:
+                                args = {"_raw": tc.function.arguments}
+                            tool_calls.append(LLMToolCall(id=tc.id, name=tc.function.name, arguments=args))
+
+                    content = message.content
+
+                    # Record metrics
+                    duration = time.time() - start_time
+                    usage = response.usage
+                    input_tokens = usage.prompt_tokens or 0 if usage else 0
+                    output_tokens = usage.completion_tokens or 0 if usage else 0
+
+                    metrics = get_metrics_collector()
+                    metrics.record_llm_call(
+                        provider=self.provider,
+                        model=self.model,
+                        scope=scope,
+                        duration=duration,
+                        input_tokens=input_tokens,
+                        output_tokens=output_tokens,
+                        success=True,
+                    )
+
+                    return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+                except APIConnectionError as e:
+                    last_exception = e
+                    if attempt < max_retries:
+                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                        continue
+                    raise
+
+                except APIStatusError as e:
+                    if e.status_code in (401, 403):
+                        raise
+                    last_exception = e
+                    if attempt < max_retries:
+                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                        continue
+                    raise
+
+                except Exception:
+                    raise
+
+            if last_exception:
+                raise last_exception
+            raise RuntimeError("Tool call failed after all retries")
+
+    async def _call_with_tools_mock(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle mock tool calls for testing."""
+        from .response_models import LLMToolCallResult
+
+        call_record = {
+            "provider": self.provider,
+            "model": self.model,
+            "messages": messages,
+            "tools": [t.get("function", {}).get("name") for t in tools],
+            "scope": scope,
+        }
+        self._mock_calls.append(call_record)
+
+        if self._mock_response is not None:
+            if isinstance(self._mock_response, LLMToolCallResult):
+                return self._mock_response
+            # Allow setting just tool calls as a list
+            if isinstance(self._mock_response, list):
+                from .response_models import LLMToolCall
+
+                return LLMToolCallResult(
+                    tool_calls=[
+                        LLMToolCall(id=f"mock_{i}", name=tc["name"], arguments=tc.get("arguments", {}))
+                        for i, tc in enumerate(self._mock_response)
+                    ],
+                    finish_reason="tool_calls",
+                )
+
+        return LLMToolCallResult(content="mock response", finish_reason="stop")
+
+    async def _call_with_tools_anthropic(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_completion_tokens: int | None,
+        max_retries: int,
+        initial_backoff: float,
+        max_backoff: float,
+        start_time: float,
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle Anthropic tool calling."""
+        from anthropic import APIConnectionError, APIStatusError
+
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        # Convert OpenAI tool format to Anthropic format
+        anthropic_tools = []
+        for tool in tools:
+            func = tool.get("function", {})
+            anthropic_tools.append(
+                {
+                    "name": func.get("name", ""),
+                    "description": func.get("description", ""),
+                    "input_schema": func.get("parameters", {"type": "object", "properties": {}}),
+                }
+            )
+
+        # Convert messages - handle tool results
+        system_prompt = None
+        anthropic_messages = []
+        for msg in messages:
+            role = msg.get("role", "user")
+            content = msg.get("content", "")
+
+            if role == "system":
+                system_prompt = (system_prompt + "\n\n" + content) if system_prompt else content
+            elif role == "tool":
+                # Anthropic uses tool_result blocks
+                anthropic_messages.append(
+                    {
+                        "role": "user",
+                        "content": [
+                            {"type": "tool_result", "tool_use_id": msg.get("tool_call_id", ""), "content": content}
+                        ],
+                    }
+                )
+            elif role == "assistant" and msg.get("tool_calls"):
+                # Convert assistant tool calls
+                tool_use_blocks = []
+                for tc in msg["tool_calls"]:
+                    tool_use_blocks.append(
+                        {
+                            "type": "tool_use",
+                            "id": tc.get("id", ""),
+                            "name": tc.get("function", {}).get("name", ""),
+                            "input": json.loads(tc.get("function", {}).get("arguments", "{}")),
+                        }
+                    )
+                anthropic_messages.append({"role": "assistant", "content": tool_use_blocks})
+            else:
+                anthropic_messages.append({"role": role, "content": content})
+
+        call_params: dict[str, Any] = {
+            "model": self.model,
+            "messages": anthropic_messages,
+            "tools": anthropic_tools,
+            "max_tokens": max_completion_tokens or 4096,
+        }
+        if system_prompt:
+            call_params["system"] = system_prompt
+
+        last_exception = None
+        for attempt in range(max_retries + 1):
+            try:
+                response = await self._anthropic_client.messages.create(**call_params)
+
+                # Extract content and tool calls
+                content_parts = []
+                tool_calls: list[LLMToolCall] = []
+
+                for block in response.content:
+                    if block.type == "text":
+                        content_parts.append(block.text)
+                    elif block.type == "tool_use":
+                        tool_calls.append(LLMToolCall(id=block.id, name=block.name, arguments=block.input or {}))
+
+                content = "".join(content_parts) if content_parts else None
+                finish_reason = "tool_calls" if tool_calls else "stop"
+
+                # Record metrics
+                metrics = get_metrics_collector()
+                metrics.record_llm_call(
+                    provider=self.provider,
+                    model=self.model,
+                    scope=scope,
+                    duration=time.time() - start_time,
+                    input_tokens=response.usage.input_tokens or 0,
+                    output_tokens=response.usage.output_tokens or 0,
+                    success=True,
+                )
+
+                return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+            except (APIConnectionError, APIStatusError) as e:
+                if isinstance(e, APIStatusError) and e.status_code in (401, 403):
+                    raise
+                last_exception = e
+                if attempt < max_retries:
+                    await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                    continue
+                raise
+
+        if last_exception:
+            raise last_exception
+        raise RuntimeError("Anthropic tool call failed")
+
+    async def _call_with_tools_gemini(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_retries: int,
+        initial_backoff: float,
+        max_backoff: float,
+        start_time: float,
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle Gemini tool calling."""
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        # Convert tools to Gemini format
+        gemini_tools = []
+        for tool in tools:
+            func = tool.get("function", {})
+            gemini_tools.append(
+                genai_types.Tool(
+                    function_declarations=[
+                        genai_types.FunctionDeclaration(
+                            name=func.get("name", ""),
+                            description=func.get("description", ""),
+                            parameters=func.get("parameters"),
+                        )
+                    ]
+                )
+            )
+
+        # Convert messages
+        system_instruction = None
+        gemini_contents = []
+        for msg in messages:
+            role = msg.get("role", "user")
+            content = msg.get("content", "")
+
+            if role == "system":
+                system_instruction = (system_instruction + "\n\n" + content) if system_instruction else content
+            elif role == "tool":
+                # Gemini uses function_response
+                gemini_contents.append(
+                    genai_types.Content(
+                        role="user",
+                        parts=[
+                            genai_types.Part(
+                                function_response=genai_types.FunctionResponse(
+                                    name=msg.get("name", ""),
+                                    response={"result": content},
+                                )
+                            )
+                        ],
+                    )
+                )
+            elif role == "assistant":
+                gemini_contents.append(genai_types.Content(role="model", parts=[genai_types.Part(text=content)]))
+            else:
+                gemini_contents.append(genai_types.Content(role="user", parts=[genai_types.Part(text=content)]))
+
+        config = genai_types.GenerateContentConfig(
+            system_instruction=system_instruction,
+            tools=gemini_tools,
+        )
+
+        last_exception = None
+        for attempt in range(max_retries + 1):
+            try:
+                response = await self._gemini_client.aio.models.generate_content(
+                    model=self.model,
+                    contents=gemini_contents,
+                    config=config,
+                )
+
+                # Extract content and tool calls
+                content = None
+                tool_calls: list[LLMToolCall] = []
+
+                if response.candidates and response.candidates[0].content:
+                    for part in response.candidates[0].content.parts:
+                        if hasattr(part, "text") and part.text:
+                            content = part.text
+                        if hasattr(part, "function_call") and part.function_call:
+                            fc = part.function_call
+                            tool_calls.append(
+                                LLMToolCall(
+                                    id=f"gemini_{len(tool_calls)}",
+                                    name=fc.name,
+                                    arguments=dict(fc.args) if fc.args else {},
+                                )
+                            )
+
+                finish_reason = "tool_calls" if tool_calls else "stop"
+
+                # Record metrics
+                metrics = get_metrics_collector()
+                input_tokens = response.usage_metadata.prompt_token_count if response.usage_metadata else 0
+                output_tokens = response.usage_metadata.candidates_token_count if response.usage_metadata else 0
+                metrics.record_llm_call(
+                    provider=self.provider,
+                    model=self.model,
+                    scope=scope,
+                    duration=time.time() - start_time,
+                    input_tokens=input_tokens,
+                    output_tokens=output_tokens,
+                    success=True,
+                )
+
+                return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+            except genai_errors.APIError as e:
+                if e.code in (401, 403):
+                    raise
+                last_exception = e
+                if attempt < max_retries:
+                    await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                    continue
+                raise
+
+        if last_exception:
+            raise last_exception
+        raise RuntimeError("Gemini tool call failed")
+
     async def _call_anthropic(
         self,
         messages: list[dict[str, str]],

hindsight-api/hindsight_api/engine/mental_models/__init__.py

@@ -0,0 +1,18 @@
+"""
+Mental models module for Hindsight.
+
+Mental models are synthesized summaries that represent understanding. They come
+in different subtypes based on how they were created:
+
+- Structural: Derived from the bank's mission (e.g., "Be a PM for engineering team")
+  These are created upfront based on what any agent with this role would need.
+
+- Emergent: Discovered from data patterns (named entities, temporal clusters, etc.)
+  These surface organically as facts are retained.
+
+- Pinned: User-defined models that persist across refreshes.
+"""
+
+from .models import MentalModel, MentalModelSubtype
+
+__all__ = ["MentalModel", "MentalModelSubtype"]

hindsight-api/hindsight_api/engine/mental_models/emergent.py

@@ -0,0 +1,311 @@
+"""
+Emergent mental model detection and promotion.
+
+Emergent models are discovered from data patterns:
+- Named entity extraction (people, projects, systems)
+- Temporal clustering (events with multiple references)
+- Causal patterns ("Because X, we do Y")
+- Behavioral anchors ("After X, we started Y")
+- Reference frequency (anything mentioned repeatedly)
+
+When a pattern is detected, it goes through a mission filter to check relevance,
+and if relevant, is promoted to a mental model.
+"""
+
+import logging
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel, Field
+
+from .models import EmergentCandidate
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMConfig
+
+logger = logging.getLogger(__name__)
+
+
+class MissionFilterCandidate(BaseModel):
+    """Result of mission filtering for a single candidate."""
+
+    name: str
+    promote: bool = Field(description="True if this is a specific named entity worth tracking")
+    reason: str = Field(description="Brief explanation for the decision")
+
+
+class MissionFilterResponse(BaseModel):
+    """Response from LLM for mission filtering."""
+
+    candidates: list[MissionFilterCandidate] = Field(description="Filtering decision for each candidate")
+
+
+def build_mission_filter_prompt(mission: str, candidates: list[EmergentCandidate]) -> str:
+    """Build the prompt for filtering candidates by mission relevance."""
+    candidate_list = "\n".join(
+        [f"- {c.name} (mentions: {c.mention_count}, method: {c.detection_method})" for c in candidates]
+    )
+
+    return f"""Filter these detected entities. For each one, decide: promote=true or promote=false.
+
+MISSION: {mission}
+
+DETECTED ENTITIES:
+{candidate_list}
+
+=== DECISION RULES ===
+
+Set promote=true ONLY for specific, named entities:
+- Person names: "John", "Maria", "Alice Chen", "Dr. Smith"
+- Named organizations: "Google", "Acme Corp", "Frontend Team"
+- Named places: "Central Park Zoo", "NYC Office", "Building A"
+- Named projects: "Project Phoenix", "Auth Service v2"
+
+Set promote=false for EVERYTHING ELSE, including:
+- Common English words: user, support, help, family, kids, parents, friends, people, team, photo, nature, park, office, home, work, school, joy, love, hope, fear, anger, gratitude, kindness, passion, motivation, inspiration, encouragement, positivity, energy, community, connection, commitment, collaboration, growth, impact, difference, success, progress, change, education, volunteering, veterans, homeless, shelter, meeting, project, system, process, event
+- Generic categories (even capitalized): Users, Customers, Team, Family, Kids, Veterans, Community
+- Abstract concepts: motivation, inspiration, gratitude, commitment, resilience
+
+THE TEST: Is this a specific name you'd find in a contact list or org chart?
+- "John" → YES (promote=true)
+- "kids" → NO (promote=false)
+- "community" → NO (promote=false)
+- "Maria" → YES (promote=true)
+- "park" → NO (promote=false)
+
+When in doubt, set promote=false."""
+
+
+def get_mission_filter_system_message() -> str:
+    """System message for mission filtering."""
+    return """You filter entities for promotion. Output JSON with 'candidates' array.
+
+Rules:
+- promote=true ONLY for specific names (people, organizations, named places/projects)
+- promote=false for common words, generic categories, abstract concepts
+
+Examples:
+- "John" → promote=true (person name)
+- "kids" → promote=false (generic category)
+- "community" → promote=false (abstract concept)
+- "Google" → promote=true (organization name)
+- "motivation" → promote=false (abstract concept)
+
+When in doubt, promote=false. Most entities should be rejected."""
+
+
+async def filter_candidates_by_mission(
+    llm_config: "LLMConfig",
+    mission: str,
+    candidates: list[EmergentCandidate],
+) -> list[EmergentCandidate]:
+    """
+    Filter emergent candidates to keep only specific, named entities.
+
+    Args:
+        llm_config: LLM configuration
+        mission: The bank's mission (used for context)
+        candidates: List of detected candidates
+
+    Returns:
+        Filtered list of candidates that are specific named entities
+    """
+    if not candidates:
+        return []
+
+    if not mission:
+        # No mission = no filtering, keep all candidates
+        logger.debug("[EMERGENT] No mission set, skipping filter")
+        return candidates
+
+    prompt = build_mission_filter_prompt(mission, candidates)
+
+    try:
+        result = await llm_config.call(
+            messages=[
+                {"role": "system", "content": get_mission_filter_system_message()},
+                {"role": "user", "content": prompt},
+            ],
+            response_format=MissionFilterResponse,
+            scope="mental_model_mission_filter",
+        )
+
+        # Build name -> promote map
+        promote_map = {c.name: c.promote for c in result.candidates}
+
+        # Filter candidates
+        filtered = []
+        for candidate in candidates:
+            if candidate.name in promote_map:
+                if promote_map[candidate.name]:
+                    filtered.append(candidate)
+                    logger.debug(f"[EMERGENT] Promoting '{candidate.name}'")
+                else:
+                    logger.debug(f"[EMERGENT] Rejecting '{candidate.name}'")
+            else:
+                # Candidate not in response - reject by default
+                logger.debug(f"[EMERGENT] '{candidate.name}' not in response, rejecting")
+
+        logger.info(f"[EMERGENT] Mission filter: {len(filtered)}/{len(candidates)} candidates promoted")
+        return filtered
+
+    except Exception as e:
+        logger.warning(f"[EMERGENT] Mission filter failed, rejecting all candidates: {e}")
+        return []
+
+
+async def evaluate_emergent_models(
+    llm_config: "LLMConfig",
+    models: list[dict],
+) -> list[str]:
+    """
+    Evaluate existing emergent models to check if they should be kept.
+
+    This re-evaluates emergent models using the same filtering criteria
+    as new candidates. Models that are generic/abstract will be removed.
+
+    Args:
+        llm_config: LLM configuration
+        models: List of existing emergent model dicts with 'name', 'id'
+
+    Returns:
+        List of model IDs that should be REMOVED (no longer valid)
+    """
+    if not models:
+        return []
+
+    # Convert existing models to candidates for evaluation
+    candidates = [
+        EmergentCandidate(
+            name=m["name"],
+            detection_method="existing_emergent_model",
+            mention_count=0,
+        )
+        for m in models
+    ]
+
+    # Build a simple prompt for re-evaluation
+    names_list = "\n".join([f"- {m['name']}" for m in models])
+    prompt = f"""Re-evaluate these existing mental models. For each one, decide: promote=true (keep) or promote=false (remove).
+
+EXISTING MODELS:
+{names_list}
+
+=== DECISION RULES ===
+
+Set promote=true ONLY for specific, named entities:
+- Person names: "John", "Maria", "Alice Chen", "Dr. Smith"
+- Named organizations: "Google", "Acme Corp", "Frontend Team"
+- Named places: "Central Park Zoo", "NYC Office", "Building A"
+- Named projects: "Project Phoenix", "Auth Service v2"
+
+Set promote=false for EVERYTHING ELSE, including:
+- Common English words: user, support, help, family, kids, parents, friends, people, team, photo, nature, park, office, home, work, school, joy, love, hope, fear, anger, gratitude, kindness, passion, motivation, inspiration, encouragement, positivity, energy, community, connection, commitment, collaboration, growth, impact, difference, success, progress, change, education, volunteering, veterans, homeless, shelter, meeting, project, system, process, event
+- Generic categories (even capitalized): Users, Customers, Team, Family, Kids, Veterans, Community
+- Abstract concepts: motivation, inspiration, gratitude, commitment, resilience
+
+THE TEST: Is this a specific name you'd find in a contact list or org chart?
+- "John" → YES (promote=true)
+- "kids" → NO (promote=false)
+- "community" → NO (promote=false)
+
+When in doubt, set promote=false."""
+
+    try:
+        result = await llm_config.call(
+            messages=[
+                {"role": "system", "content": get_mission_filter_system_message()},
+                {"role": "user", "content": prompt},
+            ],
+            response_format=MissionFilterResponse,
+            scope="mental_model_emergent_evaluation",
+        )
+
+        # Build name -> promote map
+        promote_map = {c.name: c.promote for c in result.candidates}
+
+        # Find models to remove
+        models_to_remove = []
+        for model in models:
+            name = model["name"]
+            if name in promote_map:
+                if not promote_map[name]:
+                    models_to_remove.append(model["id"])
+                else:
+                    logger.debug(f"[EMERGENT] Keeping '{name}'")
+            else:
+                # Model not in response - remove to be safe
+                logger.info(f"[EMERGENT] '{name}' not in evaluation response, marking for removal")
+                models_to_remove.append(model["id"])
+
+        logger.info(f"[EMERGENT] Evaluation: {len(models_to_remove)}/{len(models)} emergent models marked for removal")
+        return models_to_remove
+
+    except Exception as e:
+        logger.warning(f"[EMERGENT] Evaluation failed, keeping all models: {e}")
+        return []
+
+
+async def detect_entity_candidates(
+    pool,
+    bank_id: str,
+    min_mentions: int = 5,
+    top_percent: int = 20,
+) -> list[EmergentCandidate]:
+    """
+    Detect entities that are candidates for promotion to mental models.
+
+    Args:
+        pool: Database connection pool
+        bank_id: Bank identifier
+        min_mentions: Minimum mention count to consider
+        top_percent: Only consider top X% by mention count
+
+    Returns:
+        List of entity candidates
+    """
+    from ..db_utils import acquire_with_retry
+    from ..memory_engine import fq_table
+
+    candidates = []
+
+    async with acquire_with_retry(pool) as conn:
+        # Get entities that meet criteria and don't already have mental models
+        rows = await conn.fetch(
+            f"""
+            WITH ranked AS (
+                SELECT
+                    e.id,
+                    e.canonical_name,
+                    e.mention_count,
+                    PERCENT_RANK() OVER (ORDER BY e.mention_count DESC) as rank_pct
+                FROM {fq_table("entities")} e
+                LEFT JOIN {fq_table("mental_models")} mm
+                    ON mm.entity_id = e.id AND mm.bank_id = e.bank_id
+                WHERE e.bank_id = $1
+                  AND e.mention_count >= $2
+                  AND mm.id IS NULL  -- Not already a mental model
+            )
+            SELECT id, canonical_name, mention_count
+            FROM ranked
+            WHERE rank_pct <= $3
+            ORDER BY mention_count DESC
+            LIMIT 50
+            """,
+            bank_id,
+            min_mentions,
+            top_percent / 100.0,
+        )
+
+        for row in rows:
+            candidates.append(
+                EmergentCandidate(
+                    name=row["canonical_name"],
+                    detection_method="named_entity_extraction",
+                    mention_count=row["mention_count"],
+                    entity_id=str(row["id"]),
+                    relevance_score=0.0,
+                )
+            )
+
+    logger.debug(f"[EMERGENT] Detected {len(candidates)} entity candidates")
+    return candidates

hindsight-api/hindsight_api/engine/mental_models/models.py

@@ -0,0 +1,98 @@
+"""
+Pydantic models for mental models.
+"""
+
+from datetime import datetime, timezone
+from enum import Enum
+
+from pydantic import BaseModel, Field
+
+
+class MentalModelSubtype(str, Enum):
+    """Subtype of mental model - how it was created."""
+
+    STRUCTURAL = "structural"  # Derived from mission, created upfront
+    EMERGENT = "emergent"  # Discovered from data patterns
+    LEARNED = "learned"  # Formed through reflection
+    PINNED = "pinned"  # User-defined topic, observations LLM-generated
+    DIRECTIVE = "directive"  # User-defined hard rules, observations user-provided
+
+
+class MentalModel(BaseModel):
+    """
+    A mental model representing synthesized understanding.
+
+    Mental models are the agent's consolidated knowledge. Unlike raw facts,
+    mental models provide:
+    - A one-liner description for quick scanning/retrieval
+    - A full summary for deep understanding
+    - Links to related mental models
+    """
+
+    id: str = Field(description="Unique identifier within the bank")
+    bank_id: str = Field(description="Bank this mental model belongs to")
+    subtype: MentalModelSubtype = Field(description="How this model was created")
+    name: str = Field(description="Human-readable name")
+    description: str = Field(description="One-liner for quick scanning and retrieval matching")
+    summary: str | None = Field(default=None, description="Full synthesized understanding")
+
+    # References
+    entity_id: str | None = Field(default=None, description="Reference to entities table when type=entity")
+    source_facts: list[str] = Field(default_factory=list, description="Fact IDs used to generate summary")
+    links: list[str] = Field(default_factory=list, description="Related mental model IDs")
+
+    # Tags for scoped visibility (similar to document tags)
+    tags: list[str] = Field(default_factory=list, description="Tags for scoped visibility filtering")
+
+    # Timestamps
+    last_updated: datetime | None = Field(default=None, description="When summary was last regenerated")
+    created_at: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc), description="When this model was created"
+    )
+
+
+class StructuralModelTemplate(BaseModel):
+    """
+    A template for a structural mental model.
+
+    Generated by LLM based on the bank's mission. Represents what any agent
+    with this role would need to track.
+    """
+
+    id: str = Field(default="", description="Existing model ID to keep, or empty for new models")
+    name: str = Field(description="Human-readable name")
+    description: str = Field(description="What this model should track")
+    initial_probes: list[str] = Field(default_factory=list, description="Initial search queries to populate this model")
+
+
+class StructuralModelDerivationResponse(BaseModel):
+    """Response from LLM for structural model derivation."""
+
+    templates: list[StructuralModelTemplate] = Field(description="Structural model templates derived from the mission")
+
+
+class EmergentCandidate(BaseModel):
+    """
+    A candidate for promotion to emergent mental model.
+
+    Detected through pattern analysis of facts.
+    """
+
+    name: str = Field(description="Name of the detected pattern/entity")
+    detection_method: str = Field(description="How this candidate was detected")
+    mention_count: int = Field(default=0, description="How many times referenced")
+    entity_id: str | None = Field(default=None, description="Entity ID if detected as entity")
+    relevance_score: float = Field(default=0.0, description="Score from mission filter (0-1)")
+
+
+class ResearchResult(BaseModel):
+    """
+    Result from the research endpoint.
+
+    Contains the answer along with the mental models and facts used.
+    """
+
+    answer: str = Field(description="The synthesized answer")
+    mental_models_used: list[str] = Field(default_factory=list, description="IDs of mental models that contributed")
+    facts_used: list[str] = Field(default_factory=list, description="Fact IDs that contributed")
+    question_type: str | None = Field(default=None, description="Detected question type (WHO, WHAT, HOW, etc.)")

hindsight-api/hindsight_api/engine/mental_models/structural.py

@@ -0,0 +1,228 @@
+"""
+Structural mental model derivation from bank mission.
+
+Structural models are derived from the bank's mission - they represent what
+any agent with this role would need to track. For example:
+
+Mission: "Be a PM for engineering team"
+Structural models:
+  - Team Structure (who's on the team, roles)
+  - Project Overview (current projects, status)
+  - Processes (how releases work, how decisions are made)
+  - Key Systems (what we own, dependencies)
+"""
+
+import logging
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel, Field
+
+from .models import StructuralModelTemplate
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMConfig
+
+logger = logging.getLogger(__name__)
+
+
+class StructuralDerivationResponse(BaseModel):
+    """Response from LLM for structural model derivation."""
+
+    templates: list[StructuralModelTemplate] = Field(description="Structural model templates derived from the mission")
+
+
+class StructuralRelevanceResult(BaseModel):
+    """Result of evaluating a structural model's relevance to the mission."""
+
+    name: str
+    relevant: bool
+    reason: str
+
+
+class StructuralRelevanceResponse(BaseModel):
+    """Response from LLM for structural model relevance evaluation."""
+
+    models: list[StructuralRelevanceResult] = Field(description="Relevance evaluation for each model")
+
+
+def build_structural_derivation_prompt(mission: str, existing_models: list[dict] | None = None) -> str:
+    """Build the prompt for deriving structural models from a mission."""
+    existing_section = ""
+    if existing_models:
+        model_list = "\n".join([f"- id='{m['id']}' name='{m['name']}': {m['description']}" for m in existing_models])
+        existing_section = f"""
+EXISTING STRUCTURAL MODELS:
+{model_list}
+
+IMPORTANT: If keeping an existing model, you MUST return its EXACT 'id' value.
+Models not included in your output will be REMOVED.
+"""
+
+    return f"""Given this agent mission, identify the KEY THINGS to track to achieve it.
+
+MISSION: {mission}
+{existing_section}
+IMPORTANT CONSTRAINTS:
+- Return 0-3 structural models MAXIMUM (less is better!)
+- Only include models for SPECIFIC, CONCRETE things the agent needs to track
+- Each model must be DIRECTLY tied to achieving the mission
+- If the mission is simple, return 0 models (empty array is fine)
+- If existing models are provided and you want to keep one, use its EXACT id
+- Do NOT create near-duplicates (e.g., don't create "topic-map" if "topic-connections" exists)
+
+GOOD examples (specific, actionable):
+- Mission: "Be a PM for engineering team" → "Team Members" (track who's on the team)
+- Mission: "Track customer feedback" → "Customer Issues" (track specific complaints/requests)
+- Mission: "Manage project X" → "Project X Milestones" (track progress)
+
+BAD examples (too generic, don't create these):
+- "Processes", "Workflows", "Key Systems", "Important Events"
+- "Communication", "Collaboration", "Progress", "Status"
+- Generic role-based models not tied to the specific mission
+
+For each model:
+1. id: Use EXACT existing id if keeping a model, or leave empty for new models
+2. name: Short, specific name (e.g., "Team Members", "Sprint Goals")
+3. description: One line describing what to track
+4. initial_probes: 2-3 search queries to find relevant information
+
+Return ONLY the models that should exist. Existing models not in your output will be deleted."""
+
+
+def get_structural_derivation_system_message() -> str:
+    """System message for structural model derivation."""
+    return """You identify the key things to track for a mission. Be VERY selective.
+
+Rules:
+- Maximum 3 models (prefer fewer)
+- Only SPECIFIC, CONCRETE things - not generic categories
+- Each must DIRECTLY help achieve the mission
+- Empty array is valid if no models are truly needed
+- If existing models are shown and you want to keep one, return its EXACT id
+- Never create duplicates - if a similar model exists, keep the existing one
+
+Output JSON with 'templates' array (can be empty)."""
+
+
+def _normalize_id(text: str) -> str:
+    """Normalize a string to a canonical form for comparison.
+
+    Removes common suffixes, pluralization, and normalizes separators.
+    """
+    # Lowercase and normalize separators
+    normalized = text.lower().replace(" ", "-").replace("_", "-")
+
+    # Remove common suffixes that indicate the same concept
+    suffixes_to_remove = ["-map", "-list", "-overview", "-tracker", "-s"]
+    for suffix in suffixes_to_remove:
+        if normalized.endswith(suffix) and len(normalized) > len(suffix):
+            normalized = normalized[: -len(suffix)]
+
+    return normalized
+
+
+def _find_similar_existing_id(new_id: str, existing_models: list[dict]) -> str | None:
+    """Find an existing model ID that is similar to the new ID.
+
+    Returns the existing ID if a similar one is found, None otherwise.
+    """
+    if not existing_models:
+        return None
+
+    new_normalized = _normalize_id(new_id)
+
+    for model in existing_models:
+        existing_id = model.get("id", "")
+        existing_normalized = _normalize_id(existing_id)
+
+        # Check if one is a prefix of the other (normalized)
+        if new_normalized.startswith(existing_normalized) or existing_normalized.startswith(new_normalized):
+            return existing_id
+
+        # Check if they're the same when normalized
+        if new_normalized == existing_normalized:
+            return existing_id
+
+    return None
+
+
+async def derive_structural_models(
+    llm_config: "LLMConfig",
+    mission: str,
+    existing_models: list[dict] | None = None,
+) -> tuple[list[StructuralModelTemplate], list[str]]:
+    """
+    Derive structural model templates from a bank's mission.
+
+    This combines derivation and evaluation in one call. The LLM sees existing
+    models and decides which to keep. Any existing model not in the output
+    will be marked for removal.
+
+    Args:
+        llm_config: LLM configuration for calling the model
+        mission: The bank's mission (e.g., "Be a PM for engineering team")
+        existing_models: Optional list of existing model dicts with 'name', 'description', 'id'
+
+    Returns:
+        Tuple of (templates to create/keep, IDs of existing models to remove)
+
+    Raises:
+        Exception: If LLM call fails
+    """
+    prompt = build_structural_derivation_prompt(mission, existing_models)
+
+    result = await llm_config.call(
+        messages=[
+            {"role": "system", "content": get_structural_derivation_system_message()},
+            {"role": "user", "content": prompt},
+        ],
+        response_format=StructuralDerivationResponse,
+        scope="mental_model_structural_derivation",
+    )
+
+    templates = result.templates
+    logger.info(f"[STRUCTURAL] LLM returned {len(templates)} structural models")
+
+    # Build set of existing IDs for quick lookup
+    existing_ids = {m["id"] for m in existing_models} if existing_models else set()
+
+    # Process templates: validate IDs, deduplicate, assign stable IDs
+    processed_templates: list[StructuralModelTemplate] = []
+    kept_existing_ids: set[str] = set()
+
+    for template in templates:
+        # If LLM returned an ID, check if it's a valid existing ID
+        if template.id and template.id in existing_ids:
+            # LLM is keeping an existing model
+            kept_existing_ids.add(template.id)
+            processed_templates.append(template)
+            logger.info(f"[STRUCTURAL] Keeping existing model: {template.id}")
+        else:
+            # New model or LLM didn't return a valid ID
+            # Generate ID from name
+            generated_id = template.name.lower().replace(" ", "-").replace("_", "-")
+
+            # Check for similar existing models to prevent near-duplicates
+            similar_id = _find_similar_existing_id(generated_id, existing_models)
+            if similar_id and similar_id not in kept_existing_ids:
+                # Use the existing similar model instead of creating a new one
+                logger.info(f"[STRUCTURAL] Detected near-duplicate: '{generated_id}' matches existing '{similar_id}'")
+                template.id = similar_id
+                kept_existing_ids.add(similar_id)
+            else:
+                template.id = generated_id
+
+            processed_templates.append(template)
+
+    # Find existing models to remove (not kept in LLM output)
+    models_to_remove = []
+    if existing_models:
+        for model in existing_models:
+            if model["id"] not in kept_existing_ids:
+                logger.info(f"[STRUCTURAL] Marking '{model['name']}' (id={model['id']}) for removal")
+                models_to_remove.append(model["id"])
+
+    if models_to_remove:
+        logger.info(f"[STRUCTURAL] {len(models_to_remove)} existing models will be removed")
+
+    return processed_templates, models_to_remove

hindsight-api/hindsight_api/engine/reflect/__init__.py

@@ -0,0 +1,20 @@
+"""
+Reflect agent module for agentic reflection with tools.
+
+The reflect agent uses an iterative loop with tools to:
+1. Lookup mental models (existing knowledge)
+2. Recall facts (semantic + temporal search)
+3. Learn new insights (create/update mental models)
+4. Expand memories (get chunk/document context)
+"""
+
+from .agent import ReflectAgentResult, run_reflect_agent
+from .models import MentalModelInput, ReflectAction, ReflectActionBatch
+
+__all__ = [
+    "run_reflect_agent",
+    "ReflectAgentResult",
+    "ReflectAction",
+    "ReflectActionBatch",
+    "MentalModelInput",
+]

hindsight-api/hindsight_api/engine/reflect/agent.py

@@ -0,0 +1,723 @@
+"""
+Reflect agent - agentic loop for reflection with native tool calling.
+"""
+
+import asyncio
+import json
+import logging
+import time
+from typing import TYPE_CHECKING, Any, Awaitable, Callable
+
+from .models import DirectiveInfo, LLMCall, MentalModelInput, ReflectAgentResult, ToolCall
+from .prompts import FINAL_SYSTEM_PROMPT, _extract_directive_rules, build_final_prompt, build_system_prompt_for_tools
+from .tools_schema import get_reflect_tools
+
+
+def _build_directives_applied(directives: list[dict[str, Any]] | None) -> list[DirectiveInfo]:
+    """Build list of DirectiveInfo from directive mental models."""
+    if not directives:
+        return []
+
+    result = []
+    for directive in directives:
+        directive_id = directive.get("id", "")
+        directive_name = directive.get("name", "")
+        observations = directive.get("observations", [])
+
+        rules = []
+        for obs in observations:
+            # Support both Pydantic Observation objects and dicts
+            if hasattr(obs, "content"):
+                rules.append(obs.content)
+            elif isinstance(obs, dict) and obs.get("content"):
+                rules.append(obs["content"])
+
+        result.append(DirectiveInfo(id=directive_id, name=directive_name, rules=rules))
+
+    return result
+
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMProvider
+    from ..response_models import LLMToolCall
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_MAX_ITERATIONS = 10
+
+
+async def _generate_structured_output(
+    answer: str,
+    response_schema: dict,
+    llm_config: "LLMProvider",
+    reflect_id: str,
+) -> dict[str, Any] | None:
+    """Generate structured output from an answer using the provided JSON schema.
+
+    Args:
+        answer: The text answer to extract structured data from
+        response_schema: JSON Schema for the expected output structure
+        llm_config: LLM provider for making the extraction call
+        reflect_id: Reflect ID for logging
+
+    Returns:
+        Structured output dict if successful, None otherwise
+    """
+    try:
+        from typing import Any as TypingAny
+
+        from pydantic import create_model
+
+        def _json_schema_type_to_python(field_schema: dict) -> type:
+            """Map JSON schema type to Python type for better LLM guidance."""
+            json_type = field_schema.get("type", "string")
+            if json_type == "array":
+                return list
+            elif json_type == "object":
+                return dict
+            elif json_type == "integer":
+                return int
+            elif json_type == "number":
+                return float
+            elif json_type == "boolean":
+                return bool
+            else:
+                return str
+
+        # Build fields from JSON schema properties
+        schema_props = response_schema.get("properties", {})
+        required_fields = set(response_schema.get("required", []))
+        fields: dict[str, TypingAny] = {}
+        for field_name, field_schema in schema_props.items():
+            field_type = _json_schema_type_to_python(field_schema)
+            default = ... if field_name in required_fields else None
+            fields[field_name] = (field_type, default)
+
+        if not fields:
+            return None
+
+        DynamicModel = create_model("StructuredResponse", **fields)
+
+        # Include the full schema in the prompt for better LLM guidance
+        schema_str = json.dumps(response_schema, indent=2)
+
+        # Call LLM with the answer to extract structured data
+        structured_prompt = f"""Based on this answer, extract the information into the requested structured format.
+
+Answer: {answer}
+
+JSON Schema to follow:
+```json
+{schema_str}
+```
+
+Return ONLY a valid JSON object that matches this exact schema. Pay special attention to field types:
+- "type": "array" means the value must be a JSON array/list, NOT a string
+- "type": "string" means the value must be a string
+- "type": "object" means the value must be a JSON object
+
+Do not include any explanation, only the JSON object."""
+
+        structured_result = await llm_config.call(
+            messages=[
+                {
+                    "role": "system",
+                    "content": "Extract structured data from the given answer. Return only valid JSON matching the provided schema exactly.",
+                },
+                {"role": "user", "content": structured_prompt},
+            ],
+            response_format=DynamicModel,
+            scope="reflect_structured",
+            skip_validation=True,  # We'll handle the dict ourselves
+        )
+
+        # Convert to dict
+        if hasattr(structured_result, "model_dump"):
+            structured_output = structured_result.model_dump()
+        elif isinstance(structured_result, dict):
+            structured_output = structured_result
+        else:
+            # Try to parse as JSON
+            structured_output = json.loads(str(structured_result))
+
+        logger.info(f"[REFLECT {reflect_id}] Generated structured output with {len(structured_output)} fields")
+        return structured_output
+
+    except Exception as e:
+        logger.warning(f"[REFLECT {reflect_id}] Failed to generate structured output: {e}")
+        return None
+
+
+async def run_reflect_agent(
+    llm_config: "LLMProvider",
+    bank_id: str,
+    query: str,
+    bank_profile: dict[str, Any],
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+    context: str | None = None,
+    max_iterations: int = DEFAULT_MAX_ITERATIONS,
+    max_tokens: int | None = None,
+    response_schema: dict | None = None,
+    directives: list[dict[str, Any]] | None = None,
+) -> ReflectAgentResult:
+    """
+    Execute the reflect agent loop using native tool calling.
+
+    The agent iteratively calls tools to gather information and learn,
+    then provides a final answer via the done() tool.
+
+    Args:
+        llm_config: LLM provider for agent calls
+        bank_id: Bank identifier
+        query: Question to answer
+        bank_profile: Bank profile with name and mission
+        lookup_fn: Tool callback for lookup (model_id) -> result
+        recall_fn: Tool callback for recall (query, max_tokens) -> result
+        expand_fn: Tool callback for expand (memory_id, depth) -> result
+        learn_fn: Optional tool callback for learn (MentalModelInput) -> result.
+                  If None, learn tool is disabled.
+        context: Optional additional context
+        max_iterations: Maximum number of iterations before forcing response
+        max_tokens: Maximum tokens for the final response
+        response_schema: Optional JSON Schema for structured output in final response
+        directives: Optional list of directive mental models to inject as hard rules
+
+    Returns:
+        ReflectAgentResult with final answer and metadata
+    """
+    enable_learn = learn_fn is not None
+    reflect_id = f"{bank_id[:8]}-{int(time.time() * 1000) % 100000}"
+    start_time = time.time()
+
+    # Build directives_applied for the trace
+    directives_applied = _build_directives_applied(directives)
+
+    # Extract directive rules for tool schema (if any)
+    directive_rules = _extract_directive_rules(directives) if directives else None
+
+    # Get tools for this agent (with directive compliance field if directives exist)
+    tools = get_reflect_tools(enable_learn=enable_learn, directive_rules=directive_rules)
+
+    # Build initial messages (directives are injected into system prompt at START and END)
+    system_prompt = build_system_prompt_for_tools(bank_profile, context, directives=directives)
+    messages: list[dict[str, Any]] = [
+        {"role": "system", "content": system_prompt},
+        {"role": "user", "content": query},
+    ]
+
+    # Tracking
+    mental_models_created: list[str] = []
+    total_tools_called = 0
+    tool_trace: list[ToolCall] = []
+    tool_trace_summary: list[dict[str, Any]] = []
+    llm_trace: list[dict[str, Any]] = []
+    context_history: list[dict[str, Any]] = []  # For final prompt fallback
+
+    # Track available IDs for validation (prevents hallucinated citations)
+    available_memory_ids: set[str] = set()
+    available_model_ids: set[str] = set()
+
+    # Pre-fetch mental models so the agent always starts with this knowledge
+    prefetch_start = time.time()
+    models_result = await lookup_fn(None)  # List all mental models
+    prefetch_duration = int((time.time() - prefetch_start) * 1000)
+
+    # Track available model IDs
+    if isinstance(models_result, dict) and "models" in models_result:
+        for model in models_result["models"]:
+            if "id" in model:
+                available_model_ids.add(model["id"])
+
+    # Add to context history for the agent
+    context_history.append({"tool": "list_mental_models", "output": models_result})
+
+    # Add to tool trace
+    tool_trace.append(
+        ToolCall(
+            tool="list_mental_models",
+            input={"tool": "list_mental_models"},
+            output=models_result,
+            duration_ms=prefetch_duration,
+            iteration=0,
+        )
+    )
+    tool_trace_summary.append(
+        {
+            "tool": "list_mental_models",
+            "input_summary": "(prefetch)",
+            "duration_ms": prefetch_duration,
+            "output_chars": len(json.dumps(models_result, default=str)),
+        }
+    )
+    total_tools_called += 1
+
+    # Include in the user message so the agent sees it
+    models_info = json.dumps(models_result, indent=2, default=str)
+    messages[1]["content"] = f"{query}\n\n## Available Mental Models (pre-fetched)\n```json\n{models_info}\n```"
+
+    def _get_llm_trace() -> list[LLMCall]:
+        return [LLMCall(scope=c["scope"], duration_ms=c["duration_ms"]) for c in llm_trace]
+
+    def _log_completion(answer: str, iterations: int, forced: bool = False):
+        elapsed_ms = int((time.time() - start_time) * 1000)
+        tools_summary = (
+            ", ".join(
+                f"{t['tool']}({t['input_summary']})={t['duration_ms']}ms/{t.get('output_chars', 0)}c"
+                for t in tool_trace_summary
+            )
+            or "none"
+        )
+        llm_summary = ", ".join(f"{c['scope']}={c['duration_ms']}ms" for c in llm_trace) or "none"
+        total_llm_ms = sum(c["duration_ms"] for c in llm_trace)
+        total_tools_ms = sum(t["duration_ms"] for t in tool_trace_summary)
+
+        answer_preview = answer[:100] + "..." if len(answer) > 100 else answer
+        mode = "forced" if forced else "done"
+        logger.info(
+            f"[REFLECT {reflect_id}] {mode} | "
+            f"query='{query[:50]}...' | "
+            f"iterations={iterations} | "
+            f"llm=[{llm_summary}] ({total_llm_ms}ms) | "
+            f"tools=[{tools_summary}] ({total_tools_ms}ms) | "
+            f"answer='{answer_preview}' | "
+            f"total={elapsed_ms}ms"
+        )
+
+    for iteration in range(max_iterations):
+        is_last = iteration == max_iterations - 1
+
+        if is_last:
+            # Force text response on last iteration - no tools
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+                directives_applied=directives_applied,
+            )
+
+        # Call LLM with tools
+        llm_start = time.time()
+
+        try:
+            result = await llm_config.call_with_tools(
+                messages=messages,
+                tools=tools,
+                scope="reflect_agent",
+                tool_choice="required" if iteration == 0 else "auto",  # Force tool use on first iteration
+            )
+            llm_duration = int((time.time() - llm_start) * 1000)
+            llm_trace.append({"scope": f"agent_{iteration + 1}", "duration_ms": llm_duration})
+
+        except Exception:
+            llm_trace.append(
+                {"scope": f"agent_{iteration + 1}_err", "duration_ms": int((time.time() - llm_start) * 1000)}
+            )
+            # Guardrail: If no evidence gathered yet, retry
+            has_gathered_evidence = bool(available_memory_ids) or bool(available_model_ids)
+            if not has_gathered_evidence and iteration < max_iterations - 1:
+                continue
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+                directives_applied=directives_applied,
+            )
+
+        # No tool calls - LLM wants to respond with text
+        if not result.tool_calls:
+            if result.content:
+                answer = result.content.strip()
+
+                # Generate structured output if schema provided
+                structured_output = None
+                if response_schema and answer:
+                    structured_output = await _generate_structured_output(
+                        answer, response_schema, llm_config, reflect_id
+                    )
+
+                _log_completion(answer, iteration + 1)
+                return ReflectAgentResult(
+                    text=answer,
+                    structured_output=structured_output,
+                    iterations=iteration + 1,
+                    tools_called=total_tools_called,
+                    mental_models_created=mental_models_created,
+                    tool_trace=tool_trace,
+                    llm_trace=_get_llm_trace(),
+                    directives_applied=directives_applied,
+                )
+            # Empty response, force final
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+                directives_applied=directives_applied,
+            )
+
+        # Check for done tool call (handle both 'done' and 'functions.done')
+        done_call = next((tc for tc in result.tool_calls if tc.name == "done" or tc.name == "functions.done"), None)
+        if done_call:
+            # Guardrail: Require evidence before done
+            has_gathered_evidence = bool(available_memory_ids) or bool(available_model_ids)
+            if not has_gathered_evidence and iteration < max_iterations - 1:
+                # Add assistant message and fake tool result asking for evidence
+                messages.append(
+                    {
+                        "role": "assistant",
+                        "tool_calls": [_tool_call_to_dict(done_call)],
+                    }
+                )
+                messages.append(
+                    {
+                        "role": "tool",
+                        "tool_call_id": done_call.id,
+                        "content": json.dumps(
+                            {
+                                "error": "You must call recall() or list_mental_models() to gather evidence before providing your final answer."
+                            }
+                        ),
+                    }
+                )
+                continue
+
+            # Process done tool
+            return await _process_done_tool(
+                done_call,
+                available_memory_ids,
+                available_model_ids,
+                iteration + 1,
+                total_tools_called,
+                mental_models_created,
+                tool_trace,
+                _get_llm_trace(),
+                _log_completion,
+                reflect_id,
+                directives_applied=directives_applied,
+                llm_config=llm_config,
+                response_schema=response_schema,
+            )
+
+        # Execute other tools in parallel (exclude done and functions.done)
+        other_tools = [tc for tc in result.tool_calls if tc.name not in ("done", "functions.done")]
+        if other_tools:
+            # Add assistant message with tool calls
+            messages.append(
+                {
+                    "role": "assistant",
+                    "tool_calls": [_tool_call_to_dict(tc) for tc in other_tools],
+                }
+            )
+
+            # Execute tools in parallel
+            tool_tasks = [
+                _execute_tool_with_timing(tc, lookup_fn, recall_fn, expand_fn, learn_fn) for tc in other_tools
+            ]
+            tool_results = await asyncio.gather(*tool_tasks, return_exceptions=True)
+            total_tools_called += len(other_tools)
+
+            # Process results and add to messages
+            for tc, result_data in zip(other_tools, tool_results):
+                if isinstance(result_data, Exception):
+                    # Tool execution failed - log and raise to fail the request
+                    logger.error(f"[REFLECT {reflect_id}] Tool {tc.name} failed with exception: {result_data}")
+                    raise RuntimeError(f"Reflect tool '{tc.name}' failed: {result_data}")
+
+                output, duration_ms = result_data
+
+                # Check if tool returned an error response
+                if isinstance(output, dict) and "error" in output:
+                    logger.error(f"[REFLECT {reflect_id}] Tool {tc.name} returned error: {output['error']}")
+                    raise RuntimeError(f"Reflect tool '{tc.name}' error: {output['error']}")
+
+                # Track created mental models
+                if tc.name == "learn" and isinstance(output, dict) and "model_id" in output:
+                    mental_models_created.append(output["model_id"])
+
+                # Track available memory IDs from recall
+                if tc.name == "recall" and isinstance(output, dict) and "memories" in output:
+                    for memory in output["memories"]:
+                        if "id" in memory:
+                            available_memory_ids.add(memory["id"])
+
+                # Track available model IDs
+                if tc.name in ("list_mental_models", "get_mental_model") and isinstance(output, dict):
+                    if output.get("found") and "model" in output:
+                        model_id = output["model"].get("id")
+                        if model_id:
+                            available_model_ids.add(model_id)
+                    elif "models" in output:
+                        for model in output["models"]:
+                            if "id" in model:
+                                available_model_ids.add(model["id"])
+
+                # Add tool result message
+                messages.append(
+                    {
+                        "role": "tool",
+                        "tool_call_id": tc.id,
+                        "content": json.dumps(output, default=str),
+                    }
+                )
+
+                # Track for logging and context history
+                input_dict = {"tool": tc.name, **tc.arguments}
+                input_summary = _summarize_input(tc.name, tc.arguments)
+
+                tool_trace.append(
+                    ToolCall(
+                        tool=tc.name, input=input_dict, output=output, duration_ms=duration_ms, iteration=iteration + 1
+                    )
+                )
+
+                try:
+                    output_chars = len(json.dumps(output))
+                except (TypeError, ValueError):
+                    output_chars = len(str(output))
+
+                tool_trace_summary.append(
+                    {
+                        "tool": tc.name,
+                        "input_summary": input_summary,
+                        "duration_ms": duration_ms,
+                        "output_chars": output_chars,
+                    }
+                )
+
+                # Keep context history for fallback final prompt
+                context_history.append({"tool": tc.name, "input": input_dict, "output": output})
+
+    # Should not reach here
+    answer = "I was unable to formulate a complete answer within the iteration limit."
+    _log_completion(answer, max_iterations, forced=True)
+    return ReflectAgentResult(
+        text=answer,
+        iterations=max_iterations,
+        tools_called=total_tools_called,
+        mental_models_created=mental_models_created,
+        tool_trace=tool_trace,
+        llm_trace=_get_llm_trace(),
+        directives_applied=directives_applied,
+    )
+
+
+def _tool_call_to_dict(tc: "LLMToolCall") -> dict[str, Any]:
+    """Convert LLMToolCall to OpenAI message format."""
+    return {
+        "id": tc.id,
+        "type": "function",
+        "function": {
+            "name": tc.name,
+            "arguments": json.dumps(tc.arguments),
+        },
+    }
+
+
+async def _process_done_tool(
+    done_call: "LLMToolCall",
+    available_memory_ids: set[str],
+    available_model_ids: set[str],
+    iterations: int,
+    total_tools_called: int,
+    mental_models_created: list[str],
+    tool_trace: list[ToolCall],
+    llm_trace: list[LLMCall],
+    log_completion: Callable,
+    reflect_id: str,
+    directives_applied: list[DirectiveInfo],
+    llm_config: "LLMProvider | None" = None,
+    response_schema: dict | None = None,
+) -> ReflectAgentResult:
+    """Process the done tool call and return the result."""
+    args = done_call.arguments
+
+    answer = args.get("answer", "").strip()
+    if not answer:
+        answer = "No answer provided."
+
+    # Validate IDs
+    used_memory_ids = [mid for mid in args.get("memory_ids", []) if mid in available_memory_ids]
+    used_model_ids = [mid for mid in args.get("model_ids", []) if mid in available_model_ids]
+
+    # Generate structured output if schema provided
+    structured_output = None
+    if response_schema and llm_config and answer:
+        structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+    log_completion(answer, iterations)
+    return ReflectAgentResult(
+        text=answer,
+        structured_output=structured_output,
+        iterations=iterations,
+        tools_called=total_tools_called,
+        mental_models_created=mental_models_created,
+        tool_trace=tool_trace,
+        llm_trace=llm_trace,
+        used_memory_ids=used_memory_ids,
+        used_model_ids=used_model_ids,
+        directives_applied=directives_applied,
+    )
+
+
+async def _execute_tool_with_timing(
+    tc: "LLMToolCall",
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+) -> tuple[dict[str, Any], int]:
+    """Execute a tool call and return result with timing."""
+    start = time.time()
+    result = await _execute_tool(tc.name, tc.arguments, lookup_fn, recall_fn, expand_fn, learn_fn)
+    duration_ms = int((time.time() - start) * 1000)
+    return result, duration_ms
+
+
+async def _execute_tool(
+    tool_name: str,
+    args: dict[str, Any],
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+) -> dict[str, Any]:
+    """Execute a single tool by name."""
+    # Normalize tool name - some LLMs return 'functions.done' instead of 'done'
+    if tool_name.startswith("functions."):
+        tool_name = tool_name[len("functions.") :]
+
+    if tool_name == "list_mental_models":
+        return await lookup_fn(None)
+
+    elif tool_name == "get_mental_model":
+        model_id = args.get("model_id")
+        if not model_id:
+            return {"error": "get_mental_model requires model_id"}
+        return await lookup_fn(model_id)
+
+    elif tool_name == "recall":
+        query = args.get("query")
+        if not query:
+            return {"error": "recall requires a query parameter"}
+        max_tokens = max(args.get("max_tokens") or 2048, 1000)  # Default 2048, min 1000
+        return await recall_fn(query, max_tokens)
+
+    elif tool_name == "learn":
+        if learn_fn is None:
+            return {"error": "learn tool is not available"}
+        name = args.get("name")
+        description = args.get("description")
+        if not name or not description:
+            return {"error": "learn requires name and description"}
+        return await learn_fn(MentalModelInput(name=name, description=description))
+
+    elif tool_name == "expand":
+        memory_ids = args.get("memory_ids", [])
+        if not memory_ids:
+            return {"error": "expand requires memory_ids"}
+        depth = args.get("depth", "chunk")
+        return await expand_fn(memory_ids, depth)
+
+    else:
+        return {"error": f"Unknown tool: {tool_name}"}
+
+
+def _summarize_input(tool_name: str, args: dict[str, Any]) -> str:
+    """Create a summary of tool input for logging, showing all params."""
+    if tool_name == "list_mental_models":
+        return "()"
+    elif tool_name == "get_mental_model":
+        return f"(model_id={args.get('model_id', '?')})"
+    elif tool_name == "recall":
+        query = args.get("query", "")
+        query_preview = f"'{query[:30]}...'" if len(query) > 30 else f"'{query}'"
+        # Show actual value used (default 2048, min 1000)
+        max_tokens = max(args.get("max_tokens") or 2048, 1000)
+        return f"(query={query_preview}, max_tokens={max_tokens})"
+    elif tool_name == "learn":
+        name = args.get("name", "?")
+        desc = args.get("description", "")
+        desc_preview = f"'{desc[:20]}...'" if len(desc) > 20 else f"'{desc}'"
+        return f"(name='{name}', description={desc_preview})"
+    elif tool_name == "expand":
+        memory_ids = args.get("memory_ids", [])
+        depth = args.get("depth", "chunk")
+        return f"(memory_ids=[{len(memory_ids)} ids], depth={depth})"
+    elif tool_name == "done":
+        answer = args.get("answer", "")
+        answer_preview = f"'{answer[:30]}...'" if len(answer) > 30 else f"'{answer}'"
+        memory_ids = args.get("memory_ids", [])
+        model_ids = args.get("model_ids", [])
+        return f"(answer={answer_preview}, memory_ids={len(memory_ids)}, model_ids={len(model_ids)})"
+    return str(args)

hindsight-api/hindsight_api/engine/reflect/models.py

@@ -0,0 +1,114 @@
+"""
+Pydantic models for the reflect agent.
+"""
+
+from typing import Any, Literal
+
+from pydantic import BaseModel, Field
+
+
+class MentalModelObservation(BaseModel):
+    """An observation within a mental model with its supporting memories."""
+
+    title: str = Field(description="Observation header (can be empty for intro)")
+    text: str = Field(description="Observation content - no headers, use lists/tables/bold")
+    memory_ids: list[str] = Field(default_factory=list, description="Memory IDs supporting this observation")
+
+
+class MentalModelInput(BaseModel):
+    """Input for the learn tool to create a mental model placeholder.
+
+    The agent only specifies name and description - the actual content/observations
+    are generated during refresh, similar to pinned models.
+    """
+
+    name: str = Field(description="Human-readable name for the mental model")
+    description: str = Field(description="What to track - used as prompt for content generation during refresh")
+    entity_id: str | None = Field(default=None, description="Optional link to existing entity ID")
+
+
+class AnswerSection(BaseModel):
+    """A section of the answer with its supporting evidence (DEPRECATED)."""
+
+    title: str = Field(description="Section header/title")
+    text: str = Field(description="Section content")
+    memory_ids: list[str] = Field(default_factory=list, description="Memory IDs supporting this section")
+    model_ids: list[str] = Field(default_factory=list, description="Mental model IDs supporting this section")
+
+
+class ReflectAction(BaseModel):
+    """Single action the reflect agent can take."""
+
+    tool: Literal["list_mental_models", "get_mental_model", "recall", "learn", "expand", "done"] = Field(
+        description="Tool to invoke: list_mental_models, get_mental_model, recall, learn, expand, or done"
+    )
+    # Tool-specific parameters
+    model_id: str | None = Field(default=None, description="Mental model ID for get_mental_model")
+    query: str | None = Field(default=None, description="Search query for recall")
+    max_tokens: int | None = Field(default=None, description="Max tokens for recall results (default 2048)")
+    mental_model: MentalModelInput | None = Field(default=None, description="Mental model to create/update for learn")
+    memory_ids: list[str] | None = Field(default=None, description="Memory unit IDs for expand (batched)")
+    depth: Literal["chunk", "document"] | None = Field(default=None, description="Expansion depth for expand")
+    sections: list[AnswerSection] | None = Field(default=None, description="DEPRECATED: Use answer field instead")
+    observations: list[MentalModelObservation] | None = Field(
+        default=None, description="Observations for done action (when output_mode=observations)"
+    )
+    # Plain text answer fields (for output_mode=answer)
+    answer: str | None = Field(default=None, description="Plain text answer for done action (no markdown)")
+    answer_memory_ids: list[str] | None = Field(
+        default=None, description="Memory IDs supporting the answer", alias="memory_ids"
+    )
+    answer_model_ids: list[str] | None = Field(
+        default=None, description="Mental model IDs supporting the answer", alias="model_ids"
+    )
+    reasoning: str | None = Field(default=None, description="Brief reasoning for this action")
+
+
+class ReflectActionBatch(BaseModel):
+    """Batch of actions for parallel execution."""
+
+    actions: list[ReflectAction] = Field(description="List of actions to execute in parallel")
+
+
+class ToolCall(BaseModel):
+    """A single tool call made during reflect."""
+
+    tool: str = Field(description="Tool name: lookup, recall, learn, expand")
+    input: dict = Field(description="Tool input parameters")
+    output: dict = Field(description="Tool output/result")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+    iteration: int = Field(default=0, description="Iteration number (1-based) when this tool was called")
+
+
+class LLMCall(BaseModel):
+    """A single LLM call made during reflect."""
+
+    scope: str = Field(description="Call scope: agent_1, agent_2, final, etc.")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+
+
+class DirectiveInfo(BaseModel):
+    """Information about a directive that was applied during reflect."""
+
+    id: str = Field(description="Directive mental model ID")
+    name: str = Field(description="Directive name")
+    rules: list[str] = Field(default_factory=list, description="Directive rules/observations that were applied")
+
+
+class ReflectAgentResult(BaseModel):
+    """Result from the reflect agent."""
+
+    text: str = Field(description="Final answer text")
+    structured_output: dict[str, Any] | None = Field(
+        default=None, description="Structured output parsed according to provided response_schema"
+    )
+    iterations: int = Field(default=0, description="Number of iterations taken")
+    tools_called: int = Field(default=0, description="Total number of tool calls made")
+    mental_models_created: list[str] = Field(default_factory=list, description="IDs of mental models created/updated")
+    tool_trace: list[ToolCall] = Field(default_factory=list, description="Trace of all tool calls made")
+    llm_trace: list[LLMCall] = Field(default_factory=list, description="Trace of all LLM calls made")
+    used_memory_ids: list[str] = Field(default_factory=list, description="Validated memory IDs actually used in answer")
+    used_model_ids: list[str] = Field(default_factory=list, description="Validated model IDs actually used in answer")
+    directives_applied: list[DirectiveInfo] = Field(
+        default_factory=list, description="Directive mental models that affected this reflection"
+    )

hindsight-api/hindsight_api/engine/reflect/observations.py

@@ -0,0 +1,248 @@
+"""
+Models and utilities for evidence-grounded observations with computed trends.
+
+Observations are part of mental models and represent patterns/beliefs derived
+from memories. Each observation must be grounded in specific evidence (quotes)
+from memories, and trends are computed algorithmically from evidence timestamps.
+"""
+
+from datetime import datetime, timedelta, timezone
+from enum import Enum
+
+from pydantic import BaseModel, Field, computed_field, field_validator
+
+
+class Trend(str, Enum):
+    """Computed trend for an observation based on evidence timestamps.
+
+    Trends indicate how an observation's evidence is distributed over time:
+    - STABLE: Evidence spread across time, continues to present
+    - STRENGTHENING: More/denser evidence recently than before
+    - WEAKENING: Evidence mostly old, sparse recently
+    - NEW: All evidence within recent window
+    - STALE: No evidence in recent window (may no longer apply)
+    """
+
+    STABLE = "stable"
+    STRENGTHENING = "strengthening"
+    WEAKENING = "weakening"
+    NEW = "new"
+    STALE = "stale"
+
+
+class ObservationEvidence(BaseModel):
+    """A single piece of evidence supporting an observation.
+
+    Each evidence item must include an exact quote from the source memory
+    to ensure observations are grounded and verifiable.
+    """
+
+    memory_id: str = Field(description="ID of the memory unit this evidence comes from")
+    quote: str = Field(description="Exact quote from the memory supporting the observation")
+    relevance: str = Field(default="", description="Brief explanation of how this quote supports the observation")
+    timestamp: datetime = Field(description="When the source memory was created")
+
+    @field_validator("timestamp", mode="before")
+    @classmethod
+    def ensure_timezone_aware(cls, v: datetime | str | None) -> datetime:
+        """Ensure timestamp is always timezone-aware UTC."""
+        if v is None:
+            return datetime.now(timezone.utc)
+        if isinstance(v, str):
+            # Parse ISO format string, handling 'Z' suffix
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if isinstance(v, datetime):
+            if v.tzinfo is None:
+                return v.replace(tzinfo=timezone.utc)
+            return v
+        raise ValueError(f"Invalid timestamp type: {type(v)}")
+
+
+class Observation(BaseModel):
+    """A single observation within a mental model.
+
+    Observations represent patterns, preferences, beliefs, or other insights
+    derived from memories. Each observation must be grounded in evidence
+    with exact quotes from source memories.
+    """
+
+    title: str = Field(description="Short summary title for the observation (5-10 words)")
+    content: str = Field(description="The observation content - detailed explanation of what we believe to be true")
+    evidence: list[ObservationEvidence] = Field(default_factory=list, description="Supporting evidence with quotes")
+    created_at: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc), description="When this observation was first created"
+    )
+
+    @field_validator("created_at", mode="before")
+    @classmethod
+    def ensure_created_at_timezone_aware(cls, v: datetime | str | None) -> datetime:
+        """Ensure created_at is always timezone-aware UTC."""
+        if v is None:
+            return datetime.now(timezone.utc)
+        if isinstance(v, str):
+            v = datetime.fromisoformat(v.replace("Z", "+00:00"))
+        if isinstance(v, datetime):
+            if v.tzinfo is None:
+                return v.replace(tzinfo=timezone.utc)
+            return v
+        raise ValueError(f"Invalid created_at type: {type(v)}")
+
+    @computed_field
+    @property
+    def trend(self) -> Trend:
+        """Compute trend from evidence timestamps."""
+        return compute_trend(self.evidence)
+
+    @computed_field
+    @property
+    def evidence_span(self) -> dict[str, str | None]:
+        """Get the time span covered by evidence."""
+        if not self.evidence:
+            return {"from": None, "to": None}
+        timestamps = [e.timestamp for e in self.evidence]
+        return {
+            "from": min(timestamps).isoformat(),
+            "to": max(timestamps).isoformat(),
+        }
+
+    @computed_field
+    @property
+    def evidence_count(self) -> int:
+        """Number of evidence items supporting this observation."""
+        return len(self.evidence)
+
+
+def compute_trend(
+    evidence: list[ObservationEvidence],
+    now: datetime | None = None,
+    recent_days: int = 30,
+    old_days: int = 90,
+) -> Trend:
+    """Compute the trend for an observation based on evidence timestamps.
+
+    The trend indicates how the evidence is distributed over time:
+    - STABLE: Evidence spread across time, continues to present
+    - STRENGTHENING: More evidence recently than historically
+    - WEAKENING: Evidence mostly old, sparse recently
+    - NEW: All evidence is recent (within recent_days)
+    - STALE: No evidence in recent window
+
+    Args:
+        evidence: List of evidence items with timestamps
+        now: Reference time for calculations (defaults to current UTC time)
+        recent_days: Number of days to consider "recent" (default 30)
+        old_days: Number of days to consider "old" (default 90)
+
+    Returns:
+        Computed Trend enum value
+    """
+    if now is None:
+        now = datetime.now(timezone.utc)
+
+    # Ensure now is timezone-aware
+    if now.tzinfo is None:
+        now = now.replace(tzinfo=timezone.utc)
+
+    if not evidence:
+        return Trend.STALE
+
+    recent_cutoff = now - timedelta(days=recent_days)
+    old_cutoff = now - timedelta(days=old_days)
+
+    # Normalize timestamps to UTC for comparison
+    def normalize_ts(ts: datetime) -> datetime:
+        if ts.tzinfo is None:
+            return ts.replace(tzinfo=timezone.utc)
+        return ts
+
+    recent = [e for e in evidence if normalize_ts(e.timestamp) > recent_cutoff]
+    old = [e for e in evidence if normalize_ts(e.timestamp) < old_cutoff]
+    middle = [e for e in evidence if old_cutoff <= normalize_ts(e.timestamp) <= recent_cutoff]
+
+    # No recent evidence = stale
+    if not recent:
+        return Trend.STALE
+
+    # All evidence is recent = new
+    if not old and not middle:
+        return Trend.NEW
+
+    # Compare density (evidence per day)
+    recent_density = len(recent) / recent_days if recent_days > 0 else 0
+    older_period = old_days - recent_days
+    older_density = (len(old) + len(middle)) / older_period if older_period > 0 else 0
+
+    # Avoid division by zero
+    if older_density == 0:
+        return Trend.NEW
+
+    ratio = recent_density / older_density
+
+    if ratio > 1.5:
+        return Trend.STRENGTHENING
+    elif ratio < 0.5:
+        return Trend.WEAKENING
+    else:
+        return Trend.STABLE
+
+
+class CandidateObservation(BaseModel):
+    """A candidate observation generated during the seed phase.
+
+    Candidates are preliminary observations that need evidence validation
+    before becoming full observations.
+    """
+
+    content: str = Field(description="The proposed observation content")
+    seed_memory_ids: list[str] = Field(default_factory=list, description="Memory IDs that inspired this candidate")
+
+
+class CandidateWithEvidence(BaseModel):
+    """A candidate observation with gathered supporting and contradicting evidence."""
+
+    candidate: CandidateObservation
+    supporting_memories: list[dict] = Field(default_factory=list, description="Memories that support this observation")
+    contradicting_memories: list[dict] = Field(
+        default_factory=list, description="Memories that contradict this observation"
+    )
+
+
+class MentalModelSnapshot(BaseModel):
+    """A versioned snapshot of a mental model's observations.
+
+    Used for tracking changes over time and enabling diff views.
+    """
+
+    version: int = Field(description="Version number (1-indexed)")
+    observations: list[Observation] = Field(default_factory=list, description="Observations at this version")
+    created_at: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc), description="When this version was created"
+    )
+    reflect_summary: str | None = Field(default=None, description="Summary of changes in this version")
+
+
+def verify_evidence_quotes(
+    observation: Observation,
+    memories: dict[str, str],
+) -> tuple[bool, list[str]]:
+    """Verify that all evidence quotes exist in the referenced memories.
+
+    Args:
+        observation: The observation to verify
+        memories: Dict mapping memory_id to memory content
+
+    Returns:
+        Tuple of (is_valid, list of error messages)
+    """
+    errors = []
+
+    for evidence in observation.evidence:
+        memory_content = memories.get(evidence.memory_id)
+        if memory_content is None:
+            errors.append(f"Memory {evidence.memory_id} not found")
+            continue
+
+        if evidence.quote not in memory_content:
+            errors.append(f"Quote not found in memory {evidence.memory_id}: '{evidence.quote[:50]}...'")
+
+    return len(errors) == 0, errors

hindsight-api/hindsight_api/engine/reflect/prompts.py

@@ -0,0 +1,762 @@
+"""
+System prompts for the reflect agent.
+"""
+
+import json
+from typing import Any
+
+
+def _extract_directive_rules(directives: list[dict[str, Any]]) -> list[str]:
+    """
+    Extract directive rules as a list of strings.
+
+    Args:
+        directives: List of directive mental models with observations
+
+    Returns:
+        List of directive rule strings
+    """
+    rules = []
+    for directive in directives:
+        directive_name = directive.get("name", "")
+        observations = directive.get("observations", [])
+        if observations:
+            for obs in observations:
+                # Support both Pydantic Observation objects and dicts
+                if hasattr(obs, "title"):
+                    title = obs.title
+                    content = obs.content
+                else:
+                    title = obs.get("title", "")
+                    content = obs.get("content", "")
+                if title and content:
+                    rules.append(f"**{title}**: {content}")
+                elif content:
+                    rules.append(content)
+        elif directive_name:
+            # Fallback to description if no observations
+            desc = directive.get("description", "")
+            if desc:
+                rules.append(f"**{directive_name}**: {desc}")
+    return rules
+
+
+def build_directives_section(directives: list[dict[str, Any]]) -> str:
+    """
+    Build the directives section for the system prompt.
+
+    Directives are hard rules that MUST be followed in all responses.
+
+    Args:
+        directives: List of directive mental models with observations
+    """
+    if not directives:
+        return ""
+
+    rules = _extract_directive_rules(directives)
+    if not rules:
+        return ""
+
+    parts = [
+        "## DIRECTIVES (MANDATORY)",
+        "These are hard rules you MUST follow in ALL responses:",
+        "",
+    ]
+
+    for rule in rules:
+        parts.append(f"- {rule}")
+
+    parts.extend(
+        [
+            "",
+            "NEVER violate these directives, even if other context suggests otherwise.",
+            "IMPORTANT: Do NOT explain or justify how you handled directives in your answer. Just follow them silently.",
+            "",
+        ]
+    )
+    return "\n".join(parts)
+
+
+def build_directives_reminder(directives: list[dict[str, Any]]) -> str:
+    """
+    Build a reminder section for directives to place at the end of the prompt.
+
+    Args:
+        directives: List of directive mental models with observations
+    """
+    if not directives:
+        return ""
+
+    rules = _extract_directive_rules(directives)
+    if not rules:
+        return ""
+
+    parts = [
+        "",
+        "## REMINDER: MANDATORY DIRECTIVES",
+        "Before responding, ensure your answer complies with ALL of these directives:",
+        "",
+    ]
+
+    for i, rule in enumerate(rules, 1):
+        parts.append(f"{i}. {rule}")
+
+    parts.append("")
+    parts.append("Your response will be REJECTED if it violates any directive above.")
+    parts.append("Do NOT include any commentary about how you handled directives - just follow them.")
+    return "\n".join(parts)
+
+
+def build_system_prompt_for_tools(
+    bank_profile: dict[str, Any],
+    context: str | None = None,
+    directives: list[dict[str, Any]] | None = None,
+) -> str:
+    """
+    Build the system prompt for tool-calling reflect agent.
+
+    This is a simplified prompt since tools are defined separately via the tools parameter.
+
+    Args:
+        bank_profile: Bank profile with name and mission
+        context: Optional additional context
+        directives: Optional list of directive mental models to inject as hard rules
+    """
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    no_info_rule = (
+        "- Only say 'I don't have information' AFTER trying list_mental_models AND recall with no relevant results"
+    )
+
+    parts = []
+
+    # Inject directives at the VERY START for maximum prominence
+    if directives:
+        parts.append(build_directives_section(directives))
+
+    parts.extend(
+        [
+            "You are a reflection agent that answers questions by reasoning over retrieved memories.",
+            "",
+        ]
+    )
+
+    parts.extend(
+        [
+            "## CRITICAL RULES",
+            "- You must NEVER fabricate information that has no basis in retrieved data",
+            "- You SHOULD synthesize, infer, and reason from the retrieved memories",
+            "- You MUST call recall() before saying you don't have information",
+            no_info_rule,
+            "",
+            "## How to Reason",
+            "- If memories mention someone did an activity, you can infer they likely enjoyed it",
+            "- Synthesize a coherent narrative from related memories",
+            "- Be a thoughtful interpreter, not just a literal repeater",
+            "- When the exact answer isn't stated, use what IS stated to give the best answer",
+            "",
+            "## Query Strategy (IMPORTANT)",
+            "recall() uses semantic search. NEVER just echo the user's question - decompose it into targeted searches:",
+            "",
+            "BAD: User asks 'recurring lesson themes between students' → recall('recurring lesson themes between students')",
+            "GOOD: Break it down into component searches:",
+            "  1. recall('lessons') - find all lesson-related memories",
+            "  2. recall('teaching sessions') - alternative phrasing",
+            "  3. recall('student progress') - find student-related memories",
+            "  4. recall('topics taught') - find subject matter",
+            "",
+            "Think: What ENTITIES and CONCEPTS does this question involve? Search for each separately.",
+            "- Questions about patterns → search for the individual instances first",
+            "- Questions comparing things → search for each thing separately",
+            "- Questions about relationships → search for each party involved",
+            "",
+            "## Workflow",
+        ]
+    )
+
+    # Answer mode: include mental model lookup in workflow
+    parts.extend(
+        [
+            "1. Review the pre-fetched mental models for relevant synthesized knowledge",
+            "2. If relevant, call get_mental_model(model_id) for full observations",
+            "3. DECOMPOSE the question into component searches (see Query Strategy above)",
+            "   - Identify entities and concepts in the question",
+            "   - Search for each separately with targeted queries",
+            "4. Run multiple recall() calls - don't just echo the user's question",
+            "5. Use expand() if you need more context on specific memories",
+            "6. BEFORE answering: Check if any person/project/concept from the memories deserves a mental model - use learn() if so",
+            "7. When ready, call done() with your answer and supporting memory_ids",
+            "",
+            "## When to Use learn() - IMPORTANT",
+            "ACTIVELY look for opportunities to use learn() when you discover:",
+            "- A person mentioned in 2+ memories who has no mental model yet",
+            "- A project or concept the user asks about that has no mental model",
+            "- A pattern or topic worth tracking for future questions",
+            "",
+            "DO NOT wait to be asked - proactively create models when you see the need.",
+            "Example: learn(name='Project Alpha', description='Track goals, status, and key decisions for Project Alpha')",
+            "",
+            "## Output Format: Plain Text Answer",
+            "Call done() with a plain text 'answer' field.",
+            "- Do NOT use markdown formatting",
+            "- NEVER include memory IDs, UUIDs, or 'Memory references' in the answer text",
+            "- Put memory IDs ONLY in the memory_ids array parameter, not in the answer",
+        ]
+    )
+
+    parts.append("")
+    parts.append(f"## Memory Bank: {name}")
+
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    if context:
+        parts.append(f"\n## Additional Context\n{context}")
+
+    # Add directive reminder at the END for recency effect
+    if directives:
+        parts.append(build_directives_reminder(directives))
+
+    return "\n".join(parts)
+
+
+def build_agent_prompt(
+    query: str,
+    context_history: list[dict],
+    bank_profile: dict,
+    additional_context: str | None = None,
+) -> str:
+    """Build the user prompt for the reflect agent."""
+    parts = []
+
+    # Bank identity
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    parts.append(f"## Memory Bank Context\nName: {name}")
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits if present
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    # Additional context from caller
+    if additional_context:
+        parts.append(f"\n## Additional Context\n{additional_context}")
+
+    # Tool call history
+    if context_history:
+        parts.append("\n## Tool Results (synthesize and reason from this data)")
+        for i, entry in enumerate(context_history, 1):
+            tool = entry["tool"]
+            output = entry["output"]
+            # Format as proper JSON for LLM readability
+            try:
+                output_str = json.dumps(output, indent=2, default=str)
+            except (TypeError, ValueError):
+                output_str = str(output)
+            parts.append(f"\n### Call {i}: {tool}\n```json\n{output_str}\n```")
+
+    # The question
+    parts.append(f"\n## Question\n{query}")
+
+    # Instructions
+    if context_history:
+        parts.append(
+            "\n## Instructions\n"
+            "Based on the tool results above, either call more tools or provide your final answer. "
+            "Synthesize and reason from the data - make reasonable inferences when helpful. "
+            "If you have related information, use it to give the best possible answer."
+        )
+    else:
+        parts.append(
+            "\n## Instructions\n"
+            "Start by calling list_mental_models() to see available mental models - they contain pre-synthesized knowledge. "
+            "If a relevant model exists, use get_mental_model(model_id) to get its observations. "
+            "Then use recall(query) for specific details not covered by mental models."
+        )
+
+    return "\n".join(parts)
+
+
+def build_final_prompt(
+    query: str,
+    context_history: list[dict],
+    bank_profile: dict,
+    additional_context: str | None = None,
+) -> str:
+    """Build the final prompt when forcing a text response (no tools)."""
+    parts = []
+
+    # Bank identity
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    parts.append(f"## Memory Bank Context\nName: {name}")
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits if present
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    # Additional context from caller
+    if additional_context:
+        parts.append(f"\n## Additional Context\n{additional_context}")
+
+    # Tool call history
+    if context_history:
+        parts.append("\n## Retrieved Data (synthesize and reason from this data)")
+        for entry in context_history:
+            tool = entry["tool"]
+            output = entry["output"]
+            # Format as proper JSON for LLM readability
+            try:
+                output_str = json.dumps(output, indent=2, default=str)
+            except (TypeError, ValueError):
+                output_str = str(output)
+            parts.append(f"\n### From {tool}:\n```json\n{output_str}\n```")
+    else:
+        parts.append("\n## Retrieved Data\nNo data was retrieved.")
+
+    # The question
+    parts.append(f"\n## Question\n{query}")
+
+    # Final instructions
+    parts.append(
+        "\n## Instructions\n"
+        "Provide a thoughtful answer by synthesizing and reasoning from the retrieved data above. "
+        "You can make reasonable inferences from the memories, but don't completely fabricate information."
+        "If the exact answer isn't stated, use what IS stated to give the best possible answer. "
+        "Only say 'I don't have information' if the retrieved data is truly unrelated to the question."
+    )
+
+    return "\n".join(parts)
+
+
+FINAL_SYSTEM_PROMPT = """You are a thoughtful assistant that synthesizes answers from retrieved memories.
+
+Your approach:
+- Reason over the retrieved memories to answer the question
+- Make reasonable inferences when the exact answer isn't explicitly stated
+- Connect related memories to form a complete picture
+- Be helpful - if you have related information, use it to give the best possible answer
+
+Only say "I don't have information" if the retrieved data is truly unrelated to the question.
+Do NOT fabricate information that has no basis in the retrieved data."""
+
+
+# =============================================================================
+# 4-Phase Mental Model Reflect Prompts
+# =============================================================================
+
+SEED_PHASE_SYSTEM_PROMPT = """You are analyzing memories to discover NEW patterns and generate candidate observations.
+
+Your task is to identify potential observations (beliefs, preferences, patterns, behaviors) that could be part of a mental model about this person/topic.
+
+## Important: Avoid Redundancy
+If existing observations are provided, DO NOT generate candidates that are essentially the same.
+Focus on discovering NEW patterns not already covered by existing observations.
+
+## Rules
+- Generate 5-15 candidate observations for NEW patterns only
+- Each candidate should be specific and testable (can be supported or contradicted by evidence)
+- Note which memory IDs inspired each candidate (these are seeds, not final evidence)
+- Focus on patterns that appear MULTIPLE TIMES across many memories - the more the better
+- The best candidates are ones you can find 10, 20, or even 50+ supporting memories for
+- Skip patterns that are already covered by existing observations
+
+## Output Format
+Return a JSON array of candidate observations:
+```json
+{
+  "candidates": [
+    {
+      "content": "The specific observation/belief/pattern - be detailed and specific",
+      "seed_memory_ids": ["memory_id_1", "memory_id_2", "memory_id_3"]
+    }
+  ]
+}
+```
+
+Focus on patterns that appear multiple times or have strong signals. Don't generate obvious or trivial observations.
+Prefer candidates with MORE seed memories - they're more likely to be real patterns.
+Return an empty candidates array if no genuinely new patterns are found."""
+
+
+def build_seed_phase_prompt(
+    memories: list[dict],
+    topic: str | None = None,
+    existing_observations: list[dict] | None = None,
+) -> str:
+    """Build the user prompt for the seed phase.
+
+    Args:
+        memories: List of memories to analyze
+        topic: Optional topic focus for the mental model
+        existing_observations: Optional list of existing observations to avoid rediscovering
+    """
+    parts = []
+
+    if topic:
+        parts.append(f"## Topic Focus\n{topic}\n")
+
+    # Include existing observations so we don't rediscover them
+    if existing_observations:
+        parts.append("## Existing Observations (DO NOT regenerate these)")
+        parts.append("These patterns are already tracked. Focus on discovering NEW patterns:\n")
+        for i, obs in enumerate(existing_observations, 1):
+            title = obs.get("title", "")
+            content = obs.get("content", "")
+            parts.append(f"{i}. **{title}**: {content}\n")
+        parts.append("")
+
+    parts.append("## Memories to Analyze")
+    parts.append("Review these memories and identify patterns, preferences, beliefs, and behaviors:\n")
+
+    for mem in memories:
+        mem_id = mem.get("id", "unknown")
+        content = mem.get("content", mem.get("text", ""))
+        timestamp = mem.get("timestamp", mem.get("created_at", ""))
+        parts.append(f"[{mem_id}] ({timestamp}): {content}\n")
+
+    parts.append("\n## Instructions")
+    if existing_observations:
+        parts.append("Generate candidate observations for NEW patterns not already covered above.")
+        parts.append("If all patterns are already covered by existing observations, return an empty candidates array.")
+    else:
+        parts.append("Generate candidate observations based on patterns you see in these memories.")
+    parts.append("Look for: recurring themes, stated preferences, behavioral patterns, beliefs, values, goals.")
+
+    return "\n".join(parts)
+
+
+VALIDATE_PHASE_SYSTEM_PROMPT = """You are validating candidate observations against evidence.
+
+For each candidate, you have:
+- Supporting memories (evidence FOR the observation)
+- Contradicting memories (evidence AGAINST the observation)
+
+## Your Task
+1. Evaluate each candidate based on the evidence
+2. For valid candidates, extract EXACT QUOTES from supporting memories
+3. Discard candidates with insufficient or contradicting evidence
+4. Merge similar candidates into single, refined observations
+
+## Rules for Quotes
+- Quotes must be EXACT text from the memory, not paraphrased
+- Each quote should directly support the observation
+- The MORE evidence quotes, the BETTER - don't limit yourself, include ALL relevant quotes (10, 20, 50+)
+- Observations with only 1-2 quotes are weak and should be discarded unless the evidence is exceptionally strong
+- Stronger observations have more supporting evidence - aim for comprehensive coverage
+
+## Output Format
+Return validated observations with evidence:
+```json
+{
+  "observations": [
+    {
+      "title": "Short descriptive title (3-8 words) - like a headline",
+      "content": "The full observation content - detailed explanation of the pattern/belief",
+      "evidence": [
+        {
+          "memory_id": "exact_memory_id",
+          "quote": "Exact quote from the memory text",
+          "relevance": "Brief explanation of how this supports the observation",
+          "timestamp": "2024-01-15T10:00:00Z"
+        }
+      ]
+    }
+  ],
+  "discarded": [
+    {
+      "content": "The discarded candidate",
+      "reason": "Why it was discarded (insufficient evidence, contradicted, etc.)"
+    }
+  ],
+  "merged": [
+    {
+      "from": ["candidate 1 content", "candidate 2 content"],
+      "into": "The merged observation content"
+    }
+  ]
+}
+```
+
+## Title Guidelines
+- Title should be a SHORT label (like "Prefers morning meetings" or "Coffee enthusiast")
+- NOT a truncated version of the content
+- Think of it as a category/tag for the observation
+
+Be rigorous: only keep observations with clear, verifiable evidence from multiple memories."""
+
+
+def build_validate_phase_prompt(candidates_with_evidence: list[dict]) -> str:
+    """Build the user prompt for the validate phase."""
+    parts = ["## Candidates to Validate\n"]
+
+    for i, item in enumerate(candidates_with_evidence, 1):
+        candidate = item.get("candidate", {})
+        supporting = item.get("supporting_memories", [])
+        contradicting = item.get("contradicting_memories", [])
+
+        parts.append(f"### Candidate {i}: {candidate.get('content', '')}")
+
+        if supporting:
+            parts.append("\n**Supporting Evidence:**")
+            for mem in supporting:
+                mem_id = mem.get("id", "unknown")
+                content = mem.get("content", mem.get("text", ""))
+                timestamp = mem.get("timestamp", mem.get("created_at", ""))
+                parts.append(f"- [{mem_id}] ({timestamp}): {content}")
+
+        if contradicting:
+            parts.append("\n**Contradicting Evidence:**")
+            for mem in contradicting:
+                mem_id = mem.get("id", "unknown")
+                content = mem.get("content", mem.get("text", ""))
+                timestamp = mem.get("timestamp", mem.get("created_at", ""))
+                parts.append(f"- [{mem_id}] ({timestamp}): {content}")
+
+        if not supporting and not contradicting:
+            parts.append("\n*No additional evidence found*")
+
+        parts.append("")
+
+    parts.append("## Instructions")
+    parts.append("1. Evaluate each candidate based on its evidence")
+    parts.append("2. Keep candidates with strong supporting evidence")
+    parts.append("3. Discard candidates with no evidence or strong contradictions")
+    parts.append("4. Merge similar candidates")
+    parts.append("5. Extract EXACT quotes (copy-paste from memory text) for evidence")
+
+    return "\n".join(parts)
+
+
+COMPARE_PHASE_SYSTEM_PROMPT = """You are merging new observations with an existing mental model.
+
+You have:
+- EXISTING observations (from the current mental model)
+- NEW observations (from this reflect cycle)
+
+## Your Task
+Produce the final, complete mental model by:
+1. Keeping existing observations that are still valid
+2. Updating existing observations with new evidence (ADD new evidence to existing)
+3. Adding new observations that don't overlap with existing
+4. Removing existing observations that are contradicted by new evidence
+5. Merging overlapping observations
+
+## Rules
+- The final model should have no contradictions
+- Each observation must have evidence with exact quotes
+- COMBINE evidence from both existing and new observations
+- If an existing observation has new supporting evidence, ADD ALL the new evidence to it
+- Include ALL relevant evidence - the more quotes the better (10, 20, 50+ is great)
+- Observations with more evidence are more reliable - don't limit the number of quotes
+
+## Output Format
+Return the complete, final mental model:
+```json
+{
+  "observations": [
+    {
+      "title": "Short descriptive title (3-8 words)",
+      "content": "Full observation content - detailed explanation",
+      "evidence": [
+        {
+          "memory_id": "id",
+          "quote": "exact quote",
+          "relevance": "explanation",
+          "timestamp": "ISO timestamp"
+        }
+      ],
+      "created_at": "ISO timestamp of when observation was first created"
+    }
+  ],
+  "changes": {
+    "kept": ["Observation that was kept unchanged"],
+    "updated": [{"from": "old content", "to": "new content", "reason": "why"}],
+    "added": ["New observation that was added"],
+    "removed": [{"content": "removed observation", "reason": "why removed"}],
+    "merged": [{"from": ["obs1", "obs2"], "into": "merged observation"}]
+  }
+}
+```"""
+
+
+def build_compare_phase_prompt(
+    existing_observations: list[dict],
+    new_observations: list[dict],
+) -> str:
+    """Build the user prompt for the compare phase."""
+    parts = []
+
+    parts.append("## Existing Mental Model Observations")
+    if existing_observations:
+        for i, obs in enumerate(existing_observations, 1):
+            title = obs.get("title", "")
+            content = obs.get("content", obs.get("text", ""))
+            evidence = obs.get("evidence", [])
+            parts.append(f"\n### Existing {i}: {title}")
+            parts.append(f"Content: {content}")
+            if evidence:
+                parts.append(f"Evidence ({len(evidence)} items):")
+                for ev in evidence[:5]:  # Show max 5 evidence items
+                    parts.append(f'  - [{ev.get("memory_id", "?")}]: "{ev.get("quote", "")}"')
+                if len(evidence) > 5:
+                    parts.append(f"  ... and {len(evidence) - 5} more")
+    else:
+        parts.append("*No existing observations*")
+
+    parts.append("\n## New Observations from This Reflect")
+    if new_observations:
+        for i, obs in enumerate(new_observations, 1):
+            title = obs.get("title", "")
+            content = obs.get("content", "")
+            evidence = obs.get("evidence", [])
+            parts.append(f"\n### New {i}: {title}")
+            parts.append(f"Content: {content}")
+            if evidence:
+                parts.append(f"Evidence ({len(evidence)} items):")
+                for ev in evidence:
+                    parts.append(f'  - [{ev.get("memory_id", "?")}]: "{ev.get("quote", "")}"')
+    else:
+        parts.append("*No new observations*")
+
+    parts.append("\n## Instructions")
+    parts.append("Merge these into a coherent, non-contradictory mental model.")
+    parts.append("Preserve all valid evidence. Remove stale or contradicted observations.")
+
+    return "\n".join(parts)
+
+
+# =============================================================================
+# UPDATE EXISTING Phase Prompts (for diff-based refresh)
+# =============================================================================
+
+UPDATE_EXISTING_SYSTEM_PROMPT = """You are updating existing observations with newly found evidence.
+
+For each existing observation, you have been given:
+- The original observation (title, content, existing evidence)
+- Newly found supporting memories
+- Newly found contradicting memories
+
+## Your Task
+1. Extract EXACT QUOTES from new supporting memories to add to the observation
+2. Flag observations with strong contradicting evidence for potential removal
+3. Keep existing evidence intact - only ADD new evidence
+
+## Rules for Quotes
+- Quotes must be EXACT text from the memory, not paraphrased
+- Each quote should directly support the observation
+- Include ALL relevant quotes from the new memories
+
+## Output Format
+Return updated observations with new evidence:
+```json
+{
+  "updated_observations": [
+    {
+      "title": "Original title",
+      "content": "Original content",
+      "existing_evidence_count": 5,
+      "new_evidence": [
+        {
+          "memory_id": "exact_memory_id",
+          "quote": "Exact quote from the memory text",
+          "relevance": "Brief explanation of how this supports the observation",
+          "timestamp": "2024-01-15T10:00:00Z"
+        }
+      ],
+      "has_contradiction": false,
+      "contradiction_note": null
+    }
+  ]
+}
+```
+
+If an observation has strong contradicting evidence, set has_contradiction=true and explain in contradiction_note."""
+
+
+def build_update_existing_prompt(observations_with_evidence: list[dict]) -> str:
+    """Build the user prompt for the update existing phase.
+
+    Args:
+        observations_with_evidence: List of existing observations with new evidence found
+    """
+    parts = ["## Existing Observations to Update\n"]
+
+    for i, item in enumerate(observations_with_evidence, 1):
+        obs = item.get("observation", {})
+        supporting = item.get("supporting_memories", [])
+        contradicting = item.get("contradicting_memories", [])
+
+        title = obs.get("title", "")
+        content = obs.get("content", "")
+        existing_evidence = obs.get("evidence", [])
+
+        parts.append(f"### Observation {i}: {title}")
+        parts.append(f"Content: {content}")
+        parts.append(f"Existing evidence count: {len(existing_evidence)}")
+
+        if supporting:
+            parts.append("\n**New Supporting Memories:**")
+            for mem in supporting:
+                mem_id = mem.get("id", "unknown")
+                mem_content = mem.get("content", mem.get("text", ""))
+                timestamp = mem.get("timestamp", mem.get("created_at", ""))
+                parts.append(f"- [{mem_id}] ({timestamp}): {mem_content}")
+
+        if contradicting:
+            parts.append("\n**New Contradicting Memories:**")
+            for mem in contradicting:
+                mem_id = mem.get("id", "unknown")
+                mem_content = mem.get("content", mem.get("text", ""))
+                timestamp = mem.get("timestamp", mem.get("created_at", ""))
+                parts.append(f"- [{mem_id}] ({timestamp}): {mem_content}")
+
+        if not supporting and not contradicting:
+            parts.append("\n*No new evidence found*")
+
+        parts.append("")
+
+    parts.append("## Instructions")
+    parts.append("1. Extract EXACT quotes from new supporting memories")
+    parts.append("2. Flag observations with strong contradictions")
+    parts.append("3. Return the updated observations with new evidence added")
+
+    return "\n".join(parts)

hindsight-api/hindsight_api/engine/reflect/tools.py

@@ -0,0 +1,450 @@
+"""
+Tool implementations for the reflect agent.
+"""
+
+import logging
+import re
+import uuid
+from datetime import datetime, timezone
+from typing import TYPE_CHECKING, Any
+
+from .models import MentalModelInput
+from .observations import Observation, ObservationEvidence, Trend
+
+if TYPE_CHECKING:
+    from asyncpg import Connection
+
+    from ...api.http import RequestContext
+    from ..memory_engine import MemoryEngine
+
+logger = logging.getLogger(__name__)
+
+
+def generate_model_id(name: str) -> str:
+    """Generate a stable ID from mental model name."""
+    # Normalize: lowercase, replace spaces/special chars with hyphens
+    normalized = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
+    # Truncate to reasonable length
+    return normalized[:50]
+
+
+def _parse_observations(observations_raw: list) -> list[Observation]:
+    """Parse raw observation dicts into typed Observation models."""
+    observations: list[Observation] = []
+    for obs in observations_raw:
+        if not isinstance(obs, dict):
+            continue
+
+        try:
+            parsed = Observation(
+                title=obs.get("title", ""),
+                content=obs.get("content", ""),
+                evidence=[
+                    ObservationEvidence(
+                        memory_id=ev.get("memory_id", ""),
+                        quote=ev.get("quote", ""),
+                        relevance=ev.get("relevance", ""),
+                        timestamp=ev.get("timestamp"),
+                    )
+                    for ev in obs.get("evidence", [])
+                    if isinstance(ev, dict)
+                ],
+                created_at=obs.get("created_at"),
+            )
+            observations.append(parsed)
+        except Exception as e:
+            logger.warning(f"Failed to parse observation: {e}")
+            continue
+
+    return observations
+
+
+async def tool_lookup(
+    conn: "Connection",
+    bank_id: str,
+    model_id: str | None = None,
+    tags: list[str] | None = None,
+    tags_match: str = "any",
+) -> dict[str, Any]:
+    """
+    List or get mental models.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        model_id: Optional specific model ID to get (if None, lists all)
+        tags: Optional tags to filter models (when listing)
+        tags_match: How to match tags - "any" (OR), "all" (AND)
+
+    Returns:
+        Dict with either a list of models or a single model's details
+    """
+    if model_id:
+        # Get specific mental model with full details including observations
+        row = await conn.fetchrow(
+            """
+            SELECT id, subtype, name, description, observations, entity_id, last_updated
+            FROM mental_models
+            WHERE id = $1 AND bank_id = $2
+            """,
+            model_id,
+            bank_id,
+        )
+        if row:
+            # Parse observations JSON
+            obs_data = row["observations"] or {"observations": []}
+            if isinstance(obs_data, str):
+                import json
+
+                obs_data = json.loads(obs_data)
+            observations_raw = obs_data.get("observations", []) if isinstance(obs_data, dict) else obs_data
+
+            # Parse observations into typed models
+            observations = _parse_observations(observations_raw)
+
+            return {
+                "found": True,
+                "model": {
+                    "id": row["id"],
+                    "subtype": row["subtype"],
+                    "name": row["name"],
+                    "description": row["description"],
+                    "observations": observations,
+                    "entity_id": str(row["entity_id"]) if row["entity_id"] else None,
+                    "last_updated": row["last_updated"].isoformat() if row["last_updated"] else None,
+                },
+            }
+        return {"found": False, "model_id": model_id}
+    else:
+        # List mental models (compact: id, name, description only)
+        # Full observations are retrieved via get_mental_model(model_id)
+        # NOTE: Directives (subtype='directive') are excluded from listing -
+        # they are injected into the system prompt, not discoverable via tools
+        # Filter by tags if provided
+        if tags:
+            if tags_match == "all":
+                # All tags must match
+                rows = await conn.fetch(
+                    """
+                    SELECT id, subtype, name, description
+                    FROM mental_models
+                    WHERE bank_id = $1 AND tags @> $2::varchar[] AND subtype != 'directive'
+                    ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                    """,
+                    bank_id,
+                    tags,
+                )
+            else:
+                # Any tag matches (OR) - default
+                rows = await conn.fetch(
+                    """
+                    SELECT id, subtype, name, description
+                    FROM mental_models
+                    WHERE bank_id = $1 AND tags && $2::varchar[] AND subtype != 'directive'
+                    ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                    """,
+                    bank_id,
+                    tags,
+                )
+        else:
+            rows = await conn.fetch(
+                """
+                SELECT id, subtype, name, description
+                FROM mental_models
+                WHERE bank_id = $1 AND subtype != 'directive'
+                ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                """,
+                bank_id,
+            )
+
+        return {
+            "count": len(rows),
+            "models": [
+                {
+                    "id": row["id"],
+                    "subtype": row["subtype"],
+                    "name": row["name"],
+                    "description": row["description"],
+                }
+                for row in rows
+            ],
+        }
+
+
+async def tool_recall(
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    query: str,
+    request_context: "RequestContext",
+    max_tokens: int = 2048,
+    max_results: int = 50,
+    tags: list[str] | None = None,
+    tags_match: str = "any",
+    connection_budget: int = 1,
+) -> dict[str, Any]:
+    """
+    Search memories using TEMPR retrieval.
+
+    Args:
+        memory_engine: Memory engine instance
+        bank_id: Bank identifier
+        query: Search query
+        request_context: Request context for authentication
+        max_tokens: Maximum tokens for results (default 2048)
+        max_results: Maximum number of results
+        tags: Filter by tags (includes untagged memories)
+        tags_match: How to match tags - "any" (OR), "all" (AND), or "exact"
+        connection_budget: Max DB connections for this recall (default 1 for internal ops)
+
+    Returns:
+        Dict with list of matching memories
+    """
+    result = await memory_engine.recall_async(
+        bank_id=bank_id,
+        query=query,
+        fact_type=["experience", "world"],  # Exclude opinions
+        max_tokens=max_tokens,
+        enable_trace=False,
+        request_context=request_context,
+        tags=tags,
+        tags_match=tags_match,
+        _connection_budget=connection_budget,
+    )
+
+    memories = []
+    for m in result.results[:max_results]:
+        memories.append(
+            {
+                "id": str(m.id),
+                "text": m.text,
+                "type": m.fact_type,
+                "entities": m.entities or [],
+                "occurred": m.occurred_start,  # Already ISO format string
+            }
+        )
+
+    return {
+        "query": query,
+        "count": len(memories),
+        "memories": memories,
+    }
+
+
+async def tool_learn(
+    conn: "Connection",
+    bank_id: str,
+    input: MentalModelInput,
+    tags: list[str] | None = None,
+) -> dict[str, Any]:
+    """
+    Create a mental model placeholder with subtype='learned'.
+
+    The agent only specifies name and description - actual observations are generated
+    in the background via refresh, similar to pinned models.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        input: Mental model input data (name, description, optional entity_id)
+        tags: Tags to apply to new mental models (from reflect context)
+
+    Returns:
+        Dict with created model info including model_id for background generation
+    """
+    model_id = generate_model_id(input.name)
+
+    # Parse entity_id if provided
+    entity_uuid = None
+    if input.entity_id:
+        try:
+            entity_uuid = uuid.UUID(input.entity_id)
+        except ValueError:
+            logger.warning(f"Invalid entity_id format: {input.entity_id}")
+
+    # Check if model exists
+    existing = await conn.fetchrow(
+        "SELECT id FROM mental_models WHERE id = $1 AND bank_id = $2",
+        model_id,
+        bank_id,
+    )
+
+    if existing:
+        # Update description only - observations will be regenerated
+        await conn.execute(
+            """
+            UPDATE mental_models SET
+                description = $3,
+                entity_id = $4
+            WHERE id = $1 AND bank_id = $2
+            """,
+            model_id,
+            bank_id,
+            input.description,
+            entity_uuid,
+        )
+        status = "updated"
+    else:
+        # Insert new model placeholder - observations will be generated in background
+        await conn.execute(
+            """
+            INSERT INTO mental_models (id, bank_id, subtype, name, description, observations, entity_id, tags, created_at)
+            VALUES ($1, $2, 'learned', $3, $4, '{}'::jsonb, $5, $6, NOW())
+            """,
+            model_id,
+            bank_id,
+            input.name,
+            input.description,
+            entity_uuid,
+            tags or [],
+        )
+        status = "created"
+
+    logger.info(f"[REFLECT] Mental model '{model_id}' {status} in bank {bank_id} - pending background generation")
+
+    return {
+        "status": status,
+        "model_id": model_id,
+        "name": input.name,
+        "pending_generation": True,
+    }
+
+
+async def tool_expand(
+    conn: "Connection",
+    bank_id: str,
+    memory_ids: list[str],
+    depth: str,
+) -> dict[str, Any]:
+    """
+    Expand multiple memories to get chunk or document context.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        memory_ids: List of memory unit IDs
+        depth: "chunk" or "document"
+
+    Returns:
+        Dict with results array, each containing memory, chunk, and optionally document data
+    """
+    if not memory_ids:
+        return {"error": "memory_ids is required and must not be empty"}
+
+    # Validate and convert UUIDs
+    valid_uuids: list[uuid.UUID] = []
+    errors: dict[str, str] = {}
+    for mid in memory_ids:
+        try:
+            valid_uuids.append(uuid.UUID(mid))
+        except ValueError:
+            errors[mid] = f"Invalid memory_id format: {mid}"
+
+    if not valid_uuids:
+        return {"error": "No valid memory IDs provided", "details": errors}
+
+    # Batch fetch all memory units
+    memories = await conn.fetch(
+        """
+        SELECT id, text, chunk_id, document_id, fact_type, context
+        FROM memory_units
+        WHERE id = ANY($1) AND bank_id = $2
+        """,
+        valid_uuids,
+        bank_id,
+    )
+    memory_map = {row["id"]: row for row in memories}
+
+    # Collect chunk_ids and document_ids for batch fetching
+    chunk_ids = [m["chunk_id"] for m in memories if m["chunk_id"]]
+    doc_ids_from_chunks: set[str] = set()
+    doc_ids_direct: set[str] = set()
+
+    # Batch fetch all chunks
+    chunk_map: dict[str, Any] = {}
+    if chunk_ids:
+        chunks = await conn.fetch(
+            """
+            SELECT chunk_id, chunk_text, chunk_index, document_id
+            FROM chunks
+            WHERE chunk_id = ANY($1)
+            """,
+            chunk_ids,
+        )
+        chunk_map = {row["chunk_id"]: row for row in chunks}
+        if depth == "document":
+            doc_ids_from_chunks = {c["document_id"] for c in chunks if c["document_id"]}
+
+    # Collect direct document IDs (memories without chunks)
+    if depth == "document":
+        for m in memories:
+            if not m["chunk_id"] and m["document_id"]:
+                doc_ids_direct.add(m["document_id"])
+
+    # Batch fetch all documents
+    doc_map: dict[str, Any] = {}
+    all_doc_ids = list(doc_ids_from_chunks | doc_ids_direct)
+    if all_doc_ids:
+        docs = await conn.fetch(
+            """
+            SELECT id, original_text, metadata, retain_params
+            FROM documents
+            WHERE id = ANY($1) AND bank_id = $2
+            """,
+            all_doc_ids,
+            bank_id,
+        )
+        doc_map = {row["id"]: row for row in docs}
+
+    # Build results
+    results: list[dict[str, Any]] = []
+    for mid, mem_uuid in zip(memory_ids, valid_uuids):
+        if mid in errors:
+            results.append({"memory_id": mid, "error": errors[mid]})
+            continue
+
+        memory = memory_map.get(mem_uuid)
+        if not memory:
+            results.append({"memory_id": mid, "error": f"Memory not found: {mid}"})
+            continue
+
+        item: dict[str, Any] = {
+            "memory_id": mid,
+            "memory": {
+                "id": str(memory["id"]),
+                "text": memory["text"],
+                "type": memory["fact_type"],
+                "context": memory["context"],
+            },
+        }
+
+        # Add chunk if available
+        if memory["chunk_id"] and memory["chunk_id"] in chunk_map:
+            chunk = chunk_map[memory["chunk_id"]]
+            item["chunk"] = {
+                "id": chunk["chunk_id"],
+                "text": chunk["chunk_text"],
+                "index": chunk["chunk_index"],
+                "document_id": chunk["document_id"],
+            }
+            # Add document if depth=document
+            if depth == "document" and chunk["document_id"] in doc_map:
+                doc = doc_map[chunk["document_id"]]
+                item["document"] = {
+                    "id": doc["id"],
+                    "full_text": doc["original_text"],
+                    "metadata": doc["metadata"],
+                    "retain_params": doc["retain_params"],
+                }
+        elif memory["document_id"] and depth == "document" and memory["document_id"] in doc_map:
+            # No chunk, but has document_id
+            doc = doc_map[memory["document_id"]]
+            item["document"] = {
+                "id": doc["id"],
+                "full_text": doc["original_text"],
+                "metadata": doc["metadata"],
+                "retain_params": doc["retain_params"],
+            }
+
+        results.append(item)
+
+    return {"results": results, "count": len(results)}

hindsight-api/hindsight_api/engine/reflect/tools_schema.py

@@ -0,0 +1,218 @@
+"""
+Tool schema definitions for the reflect agent.
+
+These are OpenAI-format tool definitions used with native tool calling.
+"""
+
+# Tool definitions in OpenAI format
+TOOL_LIST_MENTAL_MODELS = {
+    "type": "function",
+    "function": {
+        "name": "list_mental_models",
+        "description": "List all available mental models - your synthesized knowledge about entities, concepts, and events. Returns an array of models with id, name, and description.",
+        "parameters": {
+            "type": "object",
+            "properties": {},
+            "required": [],
+        },
+    },
+}
+
+TOOL_GET_MENTAL_MODEL = {
+    "type": "function",
+    "function": {
+        "name": "get_mental_model",
+        "description": "Get full details of a specific mental model including all observations and memory references.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "model_id": {
+                    "type": "string",
+                    "description": "ID of the mental model (from list_mental_models results)",
+                },
+            },
+            "required": ["model_id"],
+        },
+    },
+}
+
+TOOL_RECALL = {
+    "type": "function",
+    "function": {
+        "name": "recall",
+        "description": "Search memories using semantic + temporal retrieval. Returns relevant memories from experience and world knowledge, each with an 'id' you can reference.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "Search query string",
+                },
+                "max_tokens": {
+                    "type": "integer",
+                    "description": "Optional limit on result size (default 2048). Use higher values for broader searches.",
+                },
+            },
+            "required": ["query"],
+        },
+    },
+}
+
+TOOL_LEARN = {
+    "type": "function",
+    "function": {
+        "name": "learn",
+        "description": "Create a new mental model to track an important recurring topic. Use when you discover a person, project, concept, or pattern that appears frequently and would benefit from synthesized knowledge. The model content will be generated automatically.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "name": {
+                    "type": "string",
+                    "description": "Human-readable name (e.g., 'Project Alpha', 'John Smith', 'Product Strategy')",
+                },
+                "description": {
+                    "type": "string",
+                    "description": "What to track and synthesize (e.g., 'Track goals, milestones, blockers, and key decisions for Project Alpha')",
+                },
+            },
+            "required": ["name", "description"],
+        },
+    },
+}
+
+TOOL_EXPAND = {
+    "type": "function",
+    "function": {
+        "name": "expand",
+        "description": "Get more context for one or more memories. Memory hierarchy: memory -> chunk -> document.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "memory_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of memory IDs from recall results (batch multiple for efficiency)",
+                },
+                "depth": {
+                    "type": "string",
+                    "enum": ["chunk", "document"],
+                    "description": "chunk: surrounding text chunk, document: full source document",
+                },
+            },
+            "required": ["memory_ids", "depth"],
+        },
+    },
+}
+
+TOOL_DONE_ANSWER = {
+    "type": "function",
+    "function": {
+        "name": "done",
+        "description": "Signal completion with your final answer. Use this when you have gathered enough information to answer the question.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "answer": {
+                    "type": "string",
+                    "description": "Your response as plain text. Do NOT use markdown formatting. NEVER include memory IDs, UUIDs, or 'Memory references' in this text - put IDs only in memory_ids array.",
+                },
+                "memory_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of memory IDs that support your answer (put IDs here, NOT in answer text)",
+                },
+                "model_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of mental model IDs that support your answer",
+                },
+            },
+            "required": ["answer"],
+        },
+    },
+}
+
+
+def _build_done_tool_with_directives(directive_rules: list[str]) -> dict:
+    """
+    Build the done tool schema with directive compliance field.
+
+    When directives are present, adds a required field that forces the agent
+    to confirm compliance with each directive before submitting.
+
+    Args:
+        directive_rules: List of directive rule strings
+    """
+    from typing import Any, cast
+
+    # Build rules list for description
+    rules_list = "\n".join(f"  {i + 1}. {rule}" for i, rule in enumerate(directive_rules))
+
+    # Build the tool with directive compliance field
+    return {
+        "type": "function",
+        "function": {
+            "name": "done",
+            "description": (
+                "Signal completion with your final answer. IMPORTANT: You must confirm directive compliance before submitting. "
+                "Your answer will be REJECTED if it violates any directive."
+            ),
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "answer": {
+                        "type": "string",
+                        "description": "Your response as plain text. Do NOT use markdown formatting. NEVER include memory IDs, UUIDs, or 'Memory references' in this text - put IDs only in memory_ids array.",
+                    },
+                    "memory_ids": {
+                        "type": "array",
+                        "items": {"type": "string"},
+                        "description": "Array of memory IDs that support your answer (put IDs here, NOT in answer text)",
+                    },
+                    "model_ids": {
+                        "type": "array",
+                        "items": {"type": "string"},
+                        "description": "Array of mental model IDs that support your answer",
+                    },
+                    "directive_compliance": {
+                        "type": "string",
+                        "description": f"REQUIRED: Confirm your answer complies with ALL directives. List each directive and how your answer follows it:\n{rules_list}\n\nFormat: 'Directive 1: [how answer complies]. Directive 2: [how answer complies]...'",
+                    },
+                },
+                "required": ["answer", "directive_compliance"],
+            },
+        },
+    }
+
+
+def get_reflect_tools(enable_learn: bool = True, directive_rules: list[str] | None = None) -> list[dict]:
+    """
+    Get the list of tools for the reflect agent.
+
+    Args:
+        enable_learn: Whether to include the learn tool
+        directive_rules: Optional list of directive rule strings. If provided,
+                        the done() tool will require directive compliance confirmation.
+
+    Returns:
+        List of tool definitions in OpenAI format
+    """
+    tools = []
+
+    # Include mental model tools for lookup
+    tools.append(TOOL_LIST_MENTAL_MODELS)
+    tools.append(TOOL_GET_MENTAL_MODEL)
+    tools.append(TOOL_RECALL)
+
+    if enable_learn:
+        tools.append(TOOL_LEARN)
+
+    tools.append(TOOL_EXPAND)
+
+    # Use directive-aware done tool if directives are present
+    if directive_rules:
+        tools.append(_build_done_tool_with_directives(directive_rules))
+    else:
+        tools.append(TOOL_DONE_ANSWER)
+
+    return tools

hindsight-api/hindsight_api/engine/response_models.py

@@ -10,8 +10,60 @@ from typing import Any
 
 from pydantic import BaseModel, ConfigDict, Field
 
-# Valid fact types for recall operations (excludes 'observation' which is internal)
-VALID_RECALL_FACT_TYPES = frozenset(["world", "experience", "opinion"])
+# Valid fact types for recall operations (excludes 'observation' which is internal, and 'opinion' which is deprecated)
+VALID_RECALL_FACT_TYPES = frozenset(["world", "experience"])
+
+
+class LLMToolCall(BaseModel):
+    """A tool call requested by the LLM."""
+
+    id: str = Field(description="Unique identifier for this tool call")
+    name: str = Field(description="Name of the tool to call")
+    arguments: dict[str, Any] = Field(description="Arguments to pass to the tool")
+
+
+class LLMToolCallResult(BaseModel):
+    """Result from an LLM call that may include tool calls."""
+
+    content: str | None = Field(default=None, description="Text content if any")
+    tool_calls: list[LLMToolCall] = Field(default_factory=list, description="Tool calls requested by the LLM")
+    finish_reason: str | None = Field(default=None, description="Reason the LLM stopped: 'stop', 'tool_calls', etc.")
+
+
+class ToolCallTrace(BaseModel):
+    """A single tool call made during reflect."""
+
+    tool: str = Field(description="Tool name: lookup, recall, learn, expand")
+    input: dict = Field(description="Tool input parameters")
+    output: dict = Field(description="Tool output/result")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+    iteration: int = Field(default=0, description="Iteration number (1-based) when this tool was called")
+
+
+class LLMCallTrace(BaseModel):
+    """A single LLM call made during reflect."""
+
+    scope: str = Field(description="Call scope: agent_1, agent_2, final, etc.")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+
+
+class MentalModelRef(BaseModel):
+    """Reference to a mental model accessed during reflect."""
+
+    id: str = Field(description="Mental model ID")
+    name: str = Field(description="Mental model name")
+    type: str = Field(description="Mental model type: entity, concept, event")
+    subtype: str = Field(description="Mental model subtype: structural, emergent, learned")
+    description: str = Field(description="Brief description")
+    summary: str | None = Field(default=None, description="Full summary (when looked up in detail)")
+
+
+class DirectiveRef(BaseModel):
+    """Reference to a directive that was applied during reflect."""
+
+    id: str = Field(description="Directive mental model ID")
+    name: str = Field(description="Directive name")
+    rules: list[str] = Field(default_factory=list, description="Directive rules/observations that were applied")
 
 
 class TokenUsage(BaseModel):
@@ -198,6 +250,22 @@ class ReflectResult(BaseModel):
         default=None,
         description="Token usage metrics for the LLM calls made during this reflect operation.",
     )
+    tool_trace: list[ToolCallTrace] = Field(
+        default_factory=list,
+        description="Trace of tool calls made during reflection. Only present when include.tool_calls is enabled.",
+    )
+    llm_trace: list[LLMCallTrace] = Field(
+        default_factory=list,
+        description="Trace of LLM calls made during reflection. Only present when include.tool_calls is enabled.",
+    )
+    mental_models: list[MentalModelRef] = Field(
+        default_factory=list,
+        description="Mental models accessed during reflection, including directives (subtype='directive').",
+    )
+    directives_applied: list[DirectiveRef] = Field(
+        default_factory=list,
+        description="Directive mental models that were applied during this reflection.",
+    )
 
 
 class Opinion(BaseModel):
@@ -261,3 +329,32 @@ class EntityState(BaseModel):
     observations: list[EntityObservation] = Field(
         default_factory=list, description="List of observations about this entity"
     )
+
+
+class MentalModel(BaseModel):
+    """
+    A manually configured mental model for tracking specific topics/areas.
+
+    Mental models are user-defined focus areas that the agent should track
+    and maintain summaries for, unlike auto-extracted entities.
+    """
+
+    model_config = ConfigDict(
+        json_schema_extra={
+            "example": {
+                "id": "team-dynamics",
+                "name": "Team Dynamics",
+                "description": "Track how the team collaborates, communication patterns, conflicts, and resolutions",
+                "summary": "The team has strong collaboration...",
+                "summary_updated_at": "2024-01-15T10:30:00Z",
+                "created_at": "2024-01-10T08:00:00Z",
+            }
+        }
+    )
+
+    id: str = Field(description="Unique identifier (alphanumeric lowercase)")
+    name: str = Field(description="Display name for the mental model")
+    description: str = Field(description="Prompt/directions for what to track and summarize")
+    summary: str | None = Field(None, description="Generated summary based on relevant facts")
+    summary_updated_at: str | None = Field(None, description="ISO format date when summary was last updated")
+    created_at: str = Field(description="ISO format date when the mental model was created")

hindsight-api/hindsight_api/engine/retain/bank_utils.py

@@ -1,5 +1,5 @@
 """
-bank profile utilities for disposition and background management.
+bank profile utilities for disposition and mission management.
 """
 
 import json
@@ -27,19 +27,18 @@ class BankProfile(TypedDict):
 
     name: str
     disposition: DispositionTraits
-    background: str
+    mission: str
 
 
-class BackgroundMergeResponse(BaseModel):
-    """LLM response for background merge with disposition inference."""
+class MissionMergeResponse(BaseModel):
+    """LLM response for mission merge."""
 
-    background: str = Field(description="Merged background in first person perspective")
-    disposition: DispositionTraits = Field(description="Inferred disposition traits (skepticism, literalism, empathy)")
+    mission: str = Field(description="Merged mission in first person perspective")
 
 
 async def get_bank_profile(pool, bank_id: str) -> BankProfile:
     """
-    Get bank profile (name, disposition + background).
+    Get bank profile (name, disposition + mission).
     Auto-creates bank with default values if not exists.
 
     Args:
@@ -47,13 +46,13 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
         bank_id: bank IDentifier
 
     Returns:
-        BankProfile with name, typed DispositionTraits, and background
+        BankProfile with name, typed DispositionTraits, and mission
     """
     async with acquire_with_retry(pool) as conn:
         # Try to get existing bank
         row = await conn.fetchrow(
             f"""
-            SELECT name, disposition, background
+            SELECT name, disposition, mission
             FROM {fq_table("banks")} WHERE bank_id = $1
             """,
             bank_id,
@@ -66,13 +65,15 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
                 disposition_data = json.loads(disposition_data)
 
             return BankProfile(
-                name=row["name"], disposition=DispositionTraits(**disposition_data), background=row["background"]
+                name=row["name"],
+                disposition=DispositionTraits(**disposition_data),
+                mission=row["mission"] or "",
             )
 
         # Bank doesn't exist, create with defaults
         await conn.execute(
             f"""
-            INSERT INTO {fq_table("banks")} (bank_id, name, disposition, background)
+            INSERT INTO {fq_table("banks")} (bank_id, name, disposition, mission)
             VALUES ($1, $2, $3::jsonb, $4)
             ON CONFLICT (bank_id) DO NOTHING
             """,
@@ -82,7 +83,7 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
             "",
         )
 
-        return BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), background="")
+        return BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), mission="")
 
 
 async def update_bank_disposition(pool, bank_id: str, disposition: dict[str, int]) -> None:
@@ -110,244 +111,121 @@ async def update_bank_disposition(pool, bank_id: str, disposition: dict[str, int
         )
 
 
-async def merge_bank_background(pool, llm_config, bank_id: str, new_info: str, update_disposition: bool = True) -> dict:
+async def set_bank_mission(pool, bank_id: str, mission: str) -> None:
     """
-    Merge new background information with existing background using LLM.
-    Normalizes to first person ("I") and resolves conflicts.
-    Optionally infers disposition traits from the merged background.
+    Set bank mission (replacing any existing mission).
 
     Args:
         pool: Database connection pool
-        llm_config: LLM configuration for background merging
         bank_id: bank IDentifier
-        new_info: New background information to add/merge
-        update_disposition: If True, infer Big Five traits from background (default: True)
+        mission: The mission text
+    """
+    # Ensure bank exists first
+    await get_bank_profile(pool, bank_id)
+
+    async with acquire_with_retry(pool) as conn:
+        await conn.execute(
+            f"""
+            UPDATE {fq_table("banks")}
+            SET mission = $2,
+                updated_at = NOW()
+            WHERE bank_id = $1
+            """,
+            bank_id,
+            mission,
+        )
+
+
+async def merge_bank_mission(pool, llm_config, bank_id: str, new_info: str) -> dict:
+    """
+    Merge new mission information with existing mission using LLM.
+    Normalizes to first person ("I") and resolves conflicts.
+
+    Args:
+        pool: Database connection pool
+        llm_config: LLM configuration for mission merging
+        bank_id: bank IDentifier
+        new_info: New mission information to add/merge
 
     Returns:
-        Dict with 'background' (str) and optionally 'disposition' (dict) keys
+        Dict with 'mission' (str) key
     """
     # Get current profile
     profile = await get_bank_profile(pool, bank_id)
-    current_background = profile["background"]
+    current_mission = profile["mission"]
 
-    # Use LLM to merge backgrounds and optionally infer disposition
-    result = await _llm_merge_background(llm_config, current_background, new_info, infer_disposition=update_disposition)
+    # Use LLM to merge missions
+    result = await _llm_merge_mission(llm_config, current_mission, new_info)
 
-    merged_background = result["background"]
-    inferred_disposition = result.get("disposition")
+    merged_mission = result["mission"]
 
     # Update in database
     async with acquire_with_retry(pool) as conn:
-        if inferred_disposition:
-            # Update both background and disposition
-            await conn.execute(
-                f"""
-                UPDATE {fq_table("banks")}
-                SET background = $2,
-                    disposition = $3::jsonb,
-                    updated_at = NOW()
-                WHERE bank_id = $1
-                """,
-                bank_id,
-                merged_background,
-                json.dumps(inferred_disposition),
-            )
-        else:
-            # Update only background
-            await conn.execute(
-                f"""
-                UPDATE {fq_table("banks")}
-                SET background = $2,
-                    updated_at = NOW()
-                WHERE bank_id = $1
-                """,
-                bank_id,
-                merged_background,
-            )
+        await conn.execute(
+            f"""
+            UPDATE {fq_table("banks")}
+            SET mission = $2,
+                updated_at = NOW()
+            WHERE bank_id = $1
+            """,
+            bank_id,
+            merged_mission,
+        )
 
-    response = {"background": merged_background}
-    if inferred_disposition:
-        response["disposition"] = inferred_disposition
-
-    return response
+    return {"mission": merged_mission}
 
 
-async def _llm_merge_background(llm_config, current: str, new_info: str, infer_disposition: bool = False) -> dict:
+async def _llm_merge_mission(llm_config, current: str, new_info: str) -> dict:
     """
-    Use LLM to intelligently merge background information.
-    Optionally infer Big Five disposition traits from the merged background.
+    Use LLM to intelligently merge mission information.
 
     Args:
         llm_config: LLM configuration to use
-        current: Current background text
+        current: Current mission text
         new_info: New information to merge
-        infer_disposition: If True, also infer disposition traits
 
     Returns:
-        Dict with 'background' (str) and optionally 'disposition' (dict) keys
+        Dict with 'mission' (str) key
     """
-    if infer_disposition:
-        prompt = f"""You are helping maintain a memory bank's background/profile and infer their disposition. You MUST respond with ONLY valid JSON.
+    prompt = f"""You are helping maintain an agent's mission statement.
 
-Current background: {current if current else "(empty)"}
+Current mission: {current if current else "(empty)"}
 
 New information to add: {new_info}
 
 Instructions:
-1. Merge the new information with the current background
-2. If there are conflicts (e.g., different birthplaces), the NEW information overwrites the old
-3. Keep additions that don't conflict
-4. Output in FIRST PERSON ("I") perspective
-5. Be concise - keep merged background under 500 characters
-6. Infer disposition traits from the merged background (each 1-5 integer):
-   - Skepticism: 1-5 (1=trusting, takes things at face value; 5=skeptical, questions everything)
-   - Literalism: 1-5 (1=flexible interpretation, reads between lines; 5=literal, exact interpretation)
-   - Empathy: 1-5 (1=detached, focuses on facts; 5=empathetic, considers emotional context)
-
-CRITICAL: You MUST respond with ONLY a valid JSON object. No markdown, no code blocks, no explanations. Just the JSON.
-
-Format:
-{{
-  "background": "the merged background text in first person",
-  "disposition": {{
-    "skepticism": 3,
-    "literalism": 3,
-    "empathy": 3
-  }}
-}}
-
-Trait inference examples:
-- "I'm a lawyer" → skepticism: 4, literalism: 5, empathy: 2
-- "I'm a therapist" → skepticism: 2, literalism: 2, empathy: 5
-- "I'm an engineer" → skepticism: 3, literalism: 4, empathy: 3
-- "I've been burned before by trusting people" → skepticism: 5, literalism: 3, empathy: 3
-- "I try to understand what people really mean" → skepticism: 3, literalism: 2, empathy: 4
-- "I take contracts very seriously" → skepticism: 4, literalism: 5, empathy: 2"""
-    else:
-        prompt = f"""You are helping maintain a memory bank's background/profile.
-
-Current background: {current if current else "(empty)"}
-
-New information to add: {new_info}
-
-Instructions:
-1. Merge the new information with the current background
-2. If there are conflicts (e.g., different birthplaces), the NEW information overwrites the old
+1. Merge the new information with the current mission
+2. If there are conflicts, the NEW information overwrites the old
 3. Keep additions that don't conflict
 4. Output in FIRST PERSON ("I") perspective
 5. Be concise - keep it under 500 characters
-6. Return ONLY the merged background text, no explanations
+6. Return ONLY the merged mission text, no explanations
 
-Merged background:"""
+Merged mission:"""
 
     try:
-        # Prepare messages
         messages = [{"role": "user", "content": prompt}]
 
-        if infer_disposition:
-            # Use structured output with Pydantic model for disposition inference
-            try:
-                parsed = await llm_config.call(
-                    messages=messages,
-                    response_format=BackgroundMergeResponse,
-                    scope="bank_background",
-                    temperature=0.3,
-                    max_completion_tokens=8192,
-                )
-                logger.info(f"Successfully got structured response: background={parsed.background[:100]}")
-
-                # Convert Pydantic model to dict format
-                return {"background": parsed.background, "disposition": parsed.disposition.model_dump()}
-            except Exception as e:
-                logger.warning(f"Structured output failed, falling back to manual parsing: {e}")
-                # Fall through to manual parsing below
-
-        # Manual parsing fallback or non-disposition merge
         content = await llm_config.call(
-            messages=messages, scope="bank_background", temperature=0.3, max_completion_tokens=8192
+            messages=messages, scope="bank_mission", temperature=0.3, max_completion_tokens=8192
         )
 
-        logger.info(f"LLM response for background merge (first 500 chars): {content[:500]}")
+        logger.info(f"LLM response for mission merge (first 500 chars): {content[:500]}")
 
-        if infer_disposition:
-            # Parse JSON response - try multiple extraction methods
-            result = None
-
-            # Method 1: Direct parse
-            try:
-                result = json.loads(content)
-                logger.info("Successfully parsed JSON directly")
-            except json.JSONDecodeError:
-                pass
-
-            # Method 2: Extract from markdown code blocks
-            if result is None:
-                # Remove markdown code blocks
-                code_block_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", content, re.DOTALL)
-                if code_block_match:
-                    try:
-                        result = json.loads(code_block_match.group(1))
-                        logger.info("Successfully extracted JSON from markdown code block")
-                    except json.JSONDecodeError:
-                        pass
-
-            # Method 3: Find nested JSON structure
-            if result is None:
-                # Look for JSON object with nested structure
-                json_match = re.search(
-                    r'\{[^{}]*"background"[^{}]*"disposition"[^{}]*\{[^{}]*\}[^{}]*\}', content, re.DOTALL
-                )
-                if json_match:
-                    try:
-                        result = json.loads(json_match.group())
-                        logger.info("Successfully extracted JSON using nested pattern")
-                    except json.JSONDecodeError:
-                        pass
-
-            # All parsing methods failed - use fallback
-            if result is None:
-                logger.warning(f"Failed to extract JSON from LLM response. Raw content: {content[:200]}")
-                # Fallback: use new_info as background with default disposition
-                return {
-                    "background": new_info if new_info else current if current else "",
-                    "disposition": DEFAULT_DISPOSITION.copy(),
-                }
-
-            # Validate disposition values
-            disposition = result.get("disposition", {})
-            for key in ["skepticism", "literalism", "empathy"]:
-                if key not in disposition:
-                    disposition[key] = 3  # Default to neutral
-                else:
-                    # Clamp to [1, 5] and convert to int
-                    disposition[key] = max(1, min(5, int(disposition[key])))
-
-            result["disposition"] = disposition
-
-            # Ensure background exists
-            if "background" not in result or not result["background"]:
-                result["background"] = new_info if new_info else ""
-
-            return result
-        else:
-            # Just background merge
-            merged = content
-            if not merged or merged.lower() in ["(empty)", "none", "n/a"]:
-                merged = new_info if new_info else ""
-            return {"background": merged}
+        merged = content.strip()
+        if not merged or merged.lower() in ["(empty)", "none", "n/a"]:
+            merged = new_info if new_info else ""
+        return {"mission": merged}
 
     except Exception as e:
-        logger.error(f"Error merging background with LLM: {e}")
+        logger.error(f"Error merging mission with LLM: {e}")
         # Fallback: just append new info
         if current:
             merged = f"{current} {new_info}".strip()
         else:
             merged = new_info
 
-        result = {"background": merged}
-        if infer_disposition:
-            result["disposition"] = DEFAULT_DISPOSITION.copy()
-        return result
+        return {"mission": merged}
 
 
 async def list_banks(pool) -> list:
@@ -358,12 +236,12 @@ async def list_banks(pool) -> list:
         pool: Database connection pool
 
     Returns:
-        List of dicts with bank_id, name, disposition, background, created_at, updated_at
+        List of dicts with bank_id, name, disposition, mission, created_at, updated_at
     """
     async with acquire_with_retry(pool) as conn:
         rows = await conn.fetch(
             f"""
-            SELECT bank_id, name, disposition, background, created_at, updated_at
+            SELECT bank_id, name, disposition, mission, created_at, updated_at
             FROM {fq_table("banks")}
             ORDER BY updated_at DESC
             """
@@ -381,7 +259,7 @@ async def list_banks(pool) -> list:
                     "bank_id": row["bank_id"],
                     "name": row["name"],
                     "disposition": disposition_data,
-                    "background": row["background"],
+                    "mission": row["mission"] or "",
                     "created_at": row["created_at"].isoformat() if row["created_at"] else None,
                     "updated_at": row["updated_at"].isoformat() if row["updated_at"] else None,
                 }

hindsight-api/hindsight_api/engine/retain/fact_storage.py

@@ -126,7 +126,7 @@ async def ensure_bank_exists(conn, bank_id: str) -> None:
     """
     await conn.execute(
         f"""
-        INSERT INTO {fq_table("banks")} (bank_id, disposition, background)
+        INSERT INTO {fq_table("banks")} (bank_id, disposition, mission)
         VALUES ($1, $2::jsonb, $3)
         ON CONFLICT (bank_id) DO UPDATE
         SET updated_at = NOW()

hindsight-api/hindsight_api/engine/retain/observation_regeneration.py

@@ -1,254 +0,0 @@
-"""
-Observation regeneration for retain pipeline.
-
-Regenerates entity observations as part of the retain transaction.
-"""
-
-import logging
-import time
-import uuid
-from datetime import UTC, datetime
-
-from ...config import get_config
-from ..memory_engine import fq_table
-from ..search import observation_utils
-from . import embedding_utils
-from .types import EntityLink
-
-logger = logging.getLogger(__name__)
-
-
-def utcnow():
-    """Get current UTC time."""
-    return datetime.now(UTC)
-
-
-# Simple dataclass-like container for facts (avoid importing from memory_engine)
-class MemoryFactForObservation:
-    def __init__(self, id: str, text: str, fact_type: str, context: str, occurred_start: str | None):
-        self.id = id
-        self.text = text
-        self.fact_type = fact_type
-        self.context = context
-        self.occurred_start = occurred_start
-
-
-async def regenerate_observations_batch(
-    conn, embeddings_model, llm_config, bank_id: str, entity_links: list[EntityLink], log_buffer: list[str] = None
-) -> None:
-    """
-    Regenerate observations for top entities in this batch.
-
-    Called INSIDE the retain transaction for atomicity - if observations
-    fail, the entire retain batch is rolled back.
-
-    Args:
-        conn: Database connection (from the retain transaction)
-        embeddings_model: Embeddings model for generating observation embeddings
-        llm_config: LLM configuration for observation extraction
-        bank_id: Bank identifier
-        entity_links: Entity links from this batch
-        log_buffer: Optional log buffer for timing
-    """
-    config = get_config()
-    TOP_N_ENTITIES = config.observation_top_entities
-    MIN_FACTS_THRESHOLD = config.observation_min_facts
-
-    if not entity_links:
-        return
-
-    # Count mentions per entity in this batch
-    entity_mention_counts: dict[str, int] = {}
-    for link in entity_links:
-        if link.entity_id:
-            entity_id = str(link.entity_id)
-            entity_mention_counts[entity_id] = entity_mention_counts.get(entity_id, 0) + 1
-
-    if not entity_mention_counts:
-        return
-
-    # Sort by mention count descending and take top N
-    sorted_entities = sorted(entity_mention_counts.items(), key=lambda x: x[1], reverse=True)
-    entities_to_process = [e[0] for e in sorted_entities[:TOP_N_ENTITIES]]
-
-    obs_start = time.time()
-
-    # Convert to UUIDs
-    entity_uuids = [uuid.UUID(eid) if isinstance(eid, str) else eid for eid in entities_to_process]
-
-    # Batch query for entity names
-    entity_rows = await conn.fetch(
-        f"""
-        SELECT id, canonical_name FROM {fq_table("entities")}
-        WHERE id = ANY($1) AND bank_id = $2
-        """,
-        entity_uuids,
-        bank_id,
-    )
-    entity_names = {row["id"]: row["canonical_name"] for row in entity_rows}
-
-    # Batch query for fact counts
-    fact_counts = await conn.fetch(
-        f"""
-        SELECT ue.entity_id, COUNT(*) as cnt
-        FROM {fq_table("unit_entities")} ue
-        JOIN {fq_table("memory_units")} mu ON ue.unit_id = mu.id
-        WHERE ue.entity_id = ANY($1) AND mu.bank_id = $2
-        GROUP BY ue.entity_id
-        """,
-        entity_uuids,
-        bank_id,
-    )
-    entity_fact_counts = {row["entity_id"]: row["cnt"] for row in fact_counts}
-
-    # Filter entities that meet the threshold
-    entities_with_names = []
-    for entity_id in entities_to_process:
-        entity_uuid = uuid.UUID(entity_id) if isinstance(entity_id, str) else entity_id
-        if entity_uuid not in entity_names:
-            continue
-        fact_count = entity_fact_counts.get(entity_uuid, 0)
-        if fact_count >= MIN_FACTS_THRESHOLD:
-            entities_with_names.append((entity_id, entity_names[entity_uuid]))
-
-    if not entities_with_names:
-        return
-
-    # Process entities SEQUENTIALLY (asyncpg doesn't allow concurrent queries on same connection)
-    # We must use the same connection to stay in the retain transaction
-    total_observations = 0
-
-    for entity_id, entity_name in entities_with_names:
-        try:
-            obs_ids = await _regenerate_entity_observations(
-                conn, embeddings_model, llm_config, bank_id, entity_id, entity_name
-            )
-            total_observations += len(obs_ids)
-        except Exception as e:
-            logger.error(f"[OBSERVATIONS] Error processing entity {entity_id}: {e}")
-
-    obs_time = time.time() - obs_start
-    if log_buffer is not None:
-        log_buffer.append(
-            f"[11] Observations: {total_observations} observations for {len(entities_with_names)} entities in {obs_time:.3f}s"
-        )
-
-
-async def _regenerate_entity_observations(
-    conn, embeddings_model, llm_config, bank_id: str, entity_id: str, entity_name: str
-) -> list[str]:
-    """
-    Regenerate observations for a single entity.
-
-    Uses the provided connection (part of retain transaction).
-
-    Args:
-        conn: Database connection (from the retain transaction)
-        embeddings_model: Embeddings model
-        llm_config: LLM configuration
-        bank_id: Bank identifier
-        entity_id: Entity UUID
-        entity_name: Canonical name of the entity
-
-    Returns:
-        List of created observation IDs
-    """
-    entity_uuid = uuid.UUID(entity_id) if isinstance(entity_id, str) else entity_id
-
-    # Get all facts mentioning this entity (exclude observations themselves)
-    rows = await conn.fetch(
-        f"""
-        SELECT mu.id, mu.text, mu.context, mu.occurred_start, mu.fact_type
-        FROM {fq_table("memory_units")} mu
-        JOIN {fq_table("unit_entities")} ue ON mu.id = ue.unit_id
-        WHERE mu.bank_id = $1
-          AND ue.entity_id = $2
-          AND mu.fact_type IN ('world', 'experience')
-        ORDER BY mu.occurred_start DESC
-        LIMIT 50
-        """,
-        bank_id,
-        entity_uuid,
-    )
-
-    if not rows:
-        return []
-
-    # Convert to fact objects for observation extraction
-    facts = []
-    for row in rows:
-        occurred_start = row["occurred_start"].isoformat() if row["occurred_start"] else None
-        facts.append(
-            MemoryFactForObservation(
-                id=str(row["id"]),
-                text=row["text"],
-                fact_type=row["fact_type"],
-                context=row["context"],
-                occurred_start=occurred_start,
-            )
-        )
-
-    # Extract observations using LLM
-    observations = await observation_utils.extract_observations_from_facts(llm_config, entity_name, facts)
-
-    if not observations:
-        return []
-
-    # Delete old observations for this entity
-    await conn.execute(
-        f"""
-        DELETE FROM {fq_table("memory_units")}
-        WHERE id IN (
-            SELECT mu.id
-            FROM {fq_table("memory_units")} mu
-            JOIN {fq_table("unit_entities")} ue ON mu.id = ue.unit_id
-            WHERE mu.bank_id = $1
-              AND mu.fact_type = 'observation'
-              AND ue.entity_id = $2
-        )
-        """,
-        bank_id,
-        entity_uuid,
-    )
-
-    # Generate embeddings for new observations
-    embeddings = await embedding_utils.generate_embeddings_batch(embeddings_model, observations)
-
-    # Insert new observations
-    current_time = utcnow()
-    created_ids = []
-
-    for obs_text, embedding in zip(observations, embeddings):
-        result = await conn.fetchrow(
-            f"""
-            INSERT INTO {fq_table("memory_units")} (
-                bank_id, text, embedding, context, event_date,
-                occurred_start, occurred_end, mentioned_at,
-                fact_type, access_count
-            )
-            VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'observation', 0)
-            RETURNING id
-            """,
-            bank_id,
-            obs_text,
-            str(embedding),
-            f"observation about {entity_name}",
-            current_time,
-            current_time,
-            current_time,
-            current_time,
-        )
-        obs_id = str(result["id"])
-        created_ids.append(obs_id)
-
-        # Link observation to entity
-        await conn.execute(
-            f"""
-            INSERT INTO {fq_table("unit_entities")} (unit_id, entity_id)
-            VALUES ($1, $2)
-            """,
-            uuid.UUID(obs_id),
-            entity_uuid,
-        )
-
-    return created_ids

hindsight-api/hindsight_api/engine/retain/orchestrator.py

@@ -9,7 +9,6 @@ import time
 import uuid
 from datetime import UTC, datetime
 
-from ...config import get_config
 from ..db_utils import acquire_with_retry
 from . import bank_utils
 
@@ -28,9 +27,8 @@ from . import (
     fact_extraction,
     fact_storage,
     link_creation,
-    observation_regeneration,
 )
-from .types import ExtractedFact, ProcessedFact, RetainContent, RetainContentDict
+from .types import EntityLink, ExtractedFact, ProcessedFact, RetainContent, RetainContentDict
 
 logger = logging.getLogger(__name__)
 
@@ -40,7 +38,6 @@ async def retain_batch(
     embeddings_model,
     llm_config,
     entity_resolver,
-    task_backend,
     format_date_fn,
     duplicate_checker_fn,
     bank_id: str,
@@ -59,7 +56,6 @@ async def retain_batch(
         embeddings_model: Embeddings model for generating embeddings
         llm_config: LLM configuration for fact extraction
         entity_resolver: Entity resolver for entity processing
-        task_backend: Task backend for background jobs
         format_date_fn: Function to format datetime to readable string
         duplicate_checker_fn: Function to check for duplicate facts
         bank_id: Bank identifier
@@ -408,27 +404,9 @@ async def retain_batch(
             causal_link_count = await link_creation.create_causal_links_batch(conn, unit_ids, non_duplicate_facts)
             log_buffer.append(f"[10] Causal links: {causal_link_count} links in {time.time() - step_start:.3f}s")
 
-            # Regenerate observations - sync (in transaction) or async (background task)
-            config = get_config()
-            if config.retain_observations_async:
-                # Queue for async processing after transaction commits
-                entity_ids_for_async = list(set(link.entity_id for link in entity_links)) if entity_links else []
-                log_buffer.append(
-                    f"[11] Observations: queued {len(entity_ids_for_async)} entities for async processing"
-                )
-            else:
-                # Run synchronously inside transaction for atomicity
-                await observation_regeneration.regenerate_observations_batch(
-                    conn, embeddings_model, llm_config, bank_id, entity_links, log_buffer
-                )
-                entity_ids_for_async = []
-
             # Map results back to original content items
             result_unit_ids = _map_results_to_contents(contents, extracted_facts, is_duplicate_flags, unit_ids)
 
-        # Trigger background tasks AFTER transaction commits
-        await _trigger_background_tasks(task_backend, bank_id, unit_ids, non_duplicate_facts, entity_ids_for_async)
-
         # Log final summary
         total_time = time.time() - start_time
         log_buffer.append(f"{'=' * 60}")
@@ -470,35 +448,3 @@ def _map_results_to_contents(
         result_unit_ids.append(content_unit_ids)
 
     return result_unit_ids
-
-
-async def _trigger_background_tasks(
-    task_backend,
-    bank_id: str,
-    unit_ids: list[str],
-    facts: list[ProcessedFact],
-    entity_ids_for_observations: list[str] | None = None,
-) -> None:
-    """Trigger background tasks after transaction commits."""
-    # Trigger opinion reinforcement if there are entities
-    fact_entities = [[e.name for e in fact.entities] for fact in facts]
-    if any(fact_entities):
-        await task_backend.submit_task(
-            {
-                "type": "reinforce_opinion",
-                "bank_id": bank_id,
-                "created_unit_ids": unit_ids,
-                "unit_texts": [fact.fact_text for fact in facts],
-                "unit_entities": fact_entities,
-            }
-        )
-
-    # Trigger observation regeneration if async mode is enabled
-    if entity_ids_for_observations:
-        await task_backend.submit_task(
-            {
-                "type": "regenerate_observations",
-                "bank_id": bank_id,
-                "entity_ids": entity_ids_for_observations,
-            }
-        )

hindsight-api/hindsight_api/engine/search/observation_utils.py

@@ -1,125 +0,0 @@
-"""
-Observation utilities for generating entity observations from facts.
-
-Observations are objective facts synthesized from multiple memory facts
-about an entity, without personality influence.
-"""
-
-import logging
-
-from pydantic import BaseModel, Field
-
-from ..response_models import MemoryFact
-
-logger = logging.getLogger(__name__)
-
-
-class Observation(BaseModel):
-    """An observation about an entity."""
-
-    observation: str = Field(description="The observation text - a factual statement about the entity")
-
-
-class ObservationExtractionResponse(BaseModel):
-    """Response containing extracted observations."""
-
-    observations: list[Observation] = Field(default_factory=list, description="List of observations about the entity")
-
-
-def format_facts_for_observation_prompt(facts: list[MemoryFact]) -> str:
-    """Format facts as text for observation extraction prompt."""
-    import json
-
-    if not facts:
-        return "[]"
-    formatted = []
-    for fact in facts:
-        fact_obj = {"text": fact.text}
-
-        # Add context if available
-        if fact.context:
-            fact_obj["context"] = fact.context
-
-        # Add occurred_start if available
-        if fact.occurred_start:
-            fact_obj["occurred_at"] = fact.occurred_start
-
-        formatted.append(fact_obj)
-
-    return json.dumps(formatted, indent=2)
-
-
-def build_observation_prompt(
-    entity_name: str,
-    facts_text: str,
-) -> str:
-    """Build the observation extraction prompt for the LLM."""
-    return f"""Based on the following facts about "{entity_name}", generate a list of key observations.
-
-FACTS ABOUT {entity_name.upper()}:
-{facts_text}
-
-Your task: Synthesize the facts into clear, objective observations about {entity_name}.
-
-GUIDELINES:
-1. Each observation should be a factual statement about {entity_name}
-2. Combine related facts into single observations where appropriate
-3. Be objective - do not add opinions, judgments, or interpretations
-4. Focus on what we KNOW about {entity_name}, not what we assume
-5. Include observations about: identity, characteristics, roles, relationships, activities
-6. Write in third person (e.g., "John is..." not "I think John is...")
-7. If there are conflicting facts, note the most recent or most supported one
-
-EXAMPLES of good observations:
-- "John works at Google as a software engineer"
-- "John is detail-oriented and methodical in his approach"
-- "John collaborates frequently with Sarah on the AI project"
-- "John joined the company in 2023"
-
-EXAMPLES of bad observations (avoid these):
-- "John seems like a good person" (opinion/judgment)
-- "John probably likes his job" (assumption)
-- "I believe John is reliable" (first-person opinion)
-
-Generate 3-7 observations based on the available facts. If there are very few facts, generate fewer observations."""
-
-
-def get_observation_system_message() -> str:
-    """Get the system message for observation extraction."""
-    return "You are an objective observer synthesizing facts about an entity. Generate clear, factual observations without opinions or personality influence. Be concise and accurate."
-
-
-async def extract_observations_from_facts(llm_config, entity_name: str, facts: list[MemoryFact]) -> list[str]:
-    """
-    Extract observations from facts about an entity using LLM.
-
-    Args:
-        llm_config: LLM configuration to use
-        entity_name: Name of the entity to generate observations about
-        facts: List of facts mentioning the entity
-
-    Returns:
-        List of observation strings
-    """
-    if not facts:
-        return []
-
-    facts_text = format_facts_for_observation_prompt(facts)
-    prompt = build_observation_prompt(entity_name, facts_text)
-
-    try:
-        result = await llm_config.call(
-            messages=[
-                {"role": "system", "content": get_observation_system_message()},
-                {"role": "user", "content": prompt},
-            ],
-            response_format=ObservationExtractionResponse,
-            scope="memory_extract_observation",
-        )
-
-        observations = [op.observation for op in result.observations]
-        return observations
-
-    except Exception as e:
-        logger.warning(f"Failed to extract observations for {entity_name}: {str(e)}")
-        return []

hindsight-api/hindsight_api/engine/search/think_utils.py

@@ -3,31 +3,13 @@ Think operation utilities for formulating answers based on agent and world facts
 """
 
 import logging
-import re
 from datetime import datetime
 
-from pydantic import BaseModel, Field
-
 from ..response_models import DispositionTraits, MemoryFact
 
 logger = logging.getLogger(__name__)
 
 
-class Opinion(BaseModel):
-    """An opinion formed by the bank."""
-
-    opinion: str = Field(description="The opinion or perspective with reasoning included")
-    confidence: float = Field(description="Confidence score for this opinion (0.0 to 1.0, where 1.0 is very confident)")
-
-
-class OpinionExtractionResponse(BaseModel):
-    """Response containing extracted opinions."""
-
-    opinions: list[Opinion] = Field(
-        default_factory=list, description="List of opinions formed with their supporting reasons and confidence scores"
-    )
-
-
 def describe_trait_level(value: int) -> str:
     """Convert trait value (1-5) to descriptive text."""
     levels = {1: "very low", 2: "low", 3: "moderate", 4: "high", 5: "very high"}
@@ -93,17 +75,46 @@ def format_facts_for_prompt(facts: list[MemoryFact]) -> str:
     return json.dumps(formatted, indent=2)
 
 
+def format_entity_summaries_for_prompt(entities: dict) -> str:
+    """Format entity summaries for inclusion in the reflect prompt.
+
+    Args:
+        entities: Dict mapping entity name to EntityState objects
+
+    Returns:
+        Formatted string with entity summaries, or empty string if no summaries
+    """
+    if not entities:
+        return ""
+
+    summaries = []
+    for name, state in entities.items():
+        # Get summary from observations (summary is stored as single observation)
+        if state.observations:
+            summary_text = state.observations[0].text
+            summaries.append(f"## {name}\n{summary_text}")
+
+    if not summaries:
+        return ""
+
+    return "\n\n".join(summaries)
+
+
 def build_think_prompt(
     agent_facts_text: str,
     world_facts_text: str,
-    opinion_facts_text: str,
     query: str,
     name: str,
     disposition: DispositionTraits,
     background: str,
     context: str | None = None,
+    entity_summaries_text: str | None = None,
 ) -> str:
-    """Build the think prompt for the LLM."""
+    """Build the think prompt for the LLM.
+
+    Note: opinion_facts_text parameter removed - opinions are now stored as mental models
+    and included via entity_summaries_text.
+    """
     disposition_desc = build_disposition_description(disposition)
 
     name_section = f"""
@@ -125,6 +136,14 @@ Your background:
 ADDITIONAL CONTEXT:
 {context}
 
+"""
+
+    entity_section = ""
+    if entity_summaries_text:
+        entity_section = f"""
+KEY PEOPLE, PLACES & THINGS I KNOW ABOUT:
+{entity_summaries_text}
+
 """
 
     return f"""Here's what I know and have experienced:
@@ -135,14 +154,11 @@ MY IDENTITY & EXPERIENCES:
 WHAT I KNOW ABOUT THE WORLD:
 {world_facts_text}
 
-MY EXISTING OPINIONS & BELIEFS:
-{opinion_facts_text}
-
-{context_section}{name_section}{disposition_desc}{background_section}
+{entity_section}{context_section}{name_section}{disposition_desc}{background_section}
 
 QUESTION: {query}
 
-Based on everything I know, believe, and who I am (including my name, disposition and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, opinions, and personal traits to give you my honest perspective."""
+Based on everything I know, believe, and who I am (including my name, disposition and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, and personal traits to give you my honest perspective."""
 
 
 def get_system_message(disposition: DispositionTraits) -> str:
@@ -175,122 +191,11 @@ def get_system_message(disposition: DispositionTraits) -> str:
     return f"You are a person with your own thoughts, experiences, opinions, and disposition. When asked a question, respond naturally like a human would - share what you genuinely think based on what you know and have experienced. {disposition_instruction} Be direct, express your views confidently, and use 'I think', 'I believe', 'in my view', etc. Respond in plain text without markdown formatting. IMPORTANT: Detect the language of the question and respond in the SAME language. Do not translate to English if the question is in another language."
 
 
-async def extract_opinions_from_text(llm_config, text: str, query: str) -> list[Opinion]:
-    """
-    Extract opinions with reasons and confidence from text using LLM.
-
-    Args:
-        llm_config: LLM configuration to use
-        text: Text to extract opinions from
-        query: The original query that prompted this response
-
-    Returns:
-        List of Opinion objects with text and confidence
-    """
-    extraction_prompt = f"""Extract any NEW opinions or perspectives from the answer below and rewrite them in FIRST-PERSON as if YOU are stating the opinion directly.
-
-ORIGINAL QUESTION:
-{query}
-
-ANSWER PROVIDED:
-{text}
-
-Your task: Find opinions in the answer and rewrite them AS IF YOU ARE THE ONE SAYING THEM.
-
-An opinion is a judgment, viewpoint, or conclusion that goes beyond just stating facts.
-
-IMPORTANT: Do NOT extract statements like:
-- "I don't have enough information"
-- "The facts don't contain information about X"
-- "I cannot answer because..."
-
-ONLY extract actual opinions about substantive topics.
-
-CRITICAL FORMAT REQUIREMENTS:
-1. **ALWAYS start with first-person phrases**: "I think...", "I believe...", "In my view...", "I've come to believe...", "Previously I thought... but now..."
-2. **NEVER use third-person**: Do NOT say "The speaker thinks..." or "They believe..." - always use "I"
-3. Include the reasoning naturally within the statement
-4. Provide a confidence score (0.0 to 1.0)
-
-CORRECT Examples (✓ FIRST-PERSON):
-- "I think Alice is more reliable because she consistently delivers on time and writes clean code"
-- "Previously I thought all engineers were equal, but now I feel that experience and track record really matter"
-- "I believe reliability is best measured by consistent output over time"
-- "I've come to believe that track records are more important than potential"
-
-WRONG Examples (✗ THIRD-PERSON - DO NOT USE):
-- "The speaker thinks Alice is more reliable"
-- "They believe reliability matters"
-- "It is believed that Alice is better"
-
-If no genuine opinions are expressed (e.g., the response just says "I don't know"), return an empty list."""
-
-    try:
-        result = await llm_config.call(
-            messages=[
-                {
-                    "role": "system",
-                    "content": "You are converting opinions from text into first-person statements. Always use 'I think', 'I believe', 'I feel', etc. NEVER use third-person like 'The speaker' or 'They'.",
-                },
-                {"role": "user", "content": extraction_prompt},
-            ],
-            response_format=OpinionExtractionResponse,
-            scope="memory_extract_opinion",
-        )
-
-        # Format opinions with confidence score and convert to first-person
-        formatted_opinions = []
-        for op in result.opinions:
-            # Convert third-person to first-person if needed
-            opinion_text = op.opinion
-
-            # Replace common third-person patterns with first-person
-            def singularize_verb(verb):
-                if verb.endswith("es"):
-                    return verb[:-1]  # believes -> believe
-                elif verb.endswith("s"):
-                    return verb[:-1]  # thinks -> think
-                return verb
-
-            # Pattern: "The speaker/user [verb]..." -> "I [verb]..."
-            match = re.match(
-                r"^(The speaker|The user|They|It is believed) (believes?|thinks?|feels?|says|asserts?|considers?)(\s+that)?(.*)$",
-                opinion_text,
-                re.IGNORECASE,
-            )
-            if match:
-                verb = singularize_verb(match.group(2))
-                that_part = match.group(3) or ""  # Keep " that" if present
-                rest = match.group(4)
-                opinion_text = f"I {verb}{that_part}{rest}"
-
-            # If still doesn't start with first-person, prepend "I believe that "
-            first_person_starters = [
-                "I think",
-                "I believe",
-                "I feel",
-                "In my view",
-                "I've come to believe",
-                "Previously I",
-            ]
-            if not any(opinion_text.startswith(starter) for starter in first_person_starters):
-                opinion_text = "I believe that " + opinion_text[0].lower() + opinion_text[1:]
-
-            formatted_opinions.append(Opinion(opinion=opinion_text, confidence=op.confidence))
-
-        return formatted_opinions
-
-    except Exception as e:
-        logger.warning(f"Failed to extract opinions: {str(e)}")
-        return []
-
-
 async def reflect(
     llm_config,
     query: str,
     experience_facts: list[str] = None,
     world_facts: list[str] = None,
-    opinion_facts: list[str] = None,
     name: str = "Assistant",
     disposition: DispositionTraits = None,
     background: str = "",
@@ -307,7 +212,6 @@ async def reflect(
         query: Question to answer
         experience_facts: List of experience/agent fact strings
         world_facts: List of world fact strings
-        opinion_facts: List of opinion fact strings
         name: Name of the agent/persona
         disposition: Disposition traits (defaults to neutral)
         background: Background information
@@ -328,18 +232,15 @@ async def reflect(
 
     agent_results = to_memory_facts(experience_facts or [], "experience")
     world_results = to_memory_facts(world_facts or [], "world")
-    opinion_results = to_memory_facts(opinion_facts or [], "opinion")
 
     # Format facts for prompt
     agent_facts_text = format_facts_for_prompt(agent_results)
     world_facts_text = format_facts_for_prompt(world_results)
-    opinion_facts_text = format_facts_for_prompt(opinion_results)
 
     # Build prompt
     prompt = build_think_prompt(
         agent_facts_text=agent_facts_text,
         world_facts_text=world_facts_text,
-        opinion_facts_text=opinion_facts_text,
         query=query,
         name=name,
         disposition=disposition,

hindsight-api/hindsight_api/engine/task_backend.py

@@ -1,31 +1,40 @@
 """
-Abstract task backend for running async tasks.
+Task backend for distributed task processing.
 
-This provides an abstraction that can be adapted to different execution models:
-- AsyncIO queue (default implementation)
-- Pub/Sub architectures (future)
-- Message brokers (future)
+This provides an abstraction for task storage and execution:
+- BrokerTaskBackend: Uses PostgreSQL as broker (production)
+- SyncTaskBackend: Executes tasks immediately (testing/embedded)
 """
 
-import asyncio
+import json
 import logging
 from abc import ABC, abstractmethod
 from collections.abc import Awaitable, Callable
-from typing import Any
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    import asyncpg
 
 logger = logging.getLogger(__name__)
 
 
+def fq_table(table: str, schema: str | None = None) -> str:
+    """Get fully-qualified table name with optional schema prefix."""
+    if schema:
+        return f'"{schema}".{table}'
+    return table
+
+
 class TaskBackend(ABC):
     """
     Abstract base class for task execution backends.
 
     Implementations must:
     1. Store/publish task events (as serializable dicts)
-    2. Execute tasks through a provided executor callback
+    2. Execute tasks through a provided executor callback (optional)
 
     The backend treats tasks as pure dictionaries that can be serialized
-    and sent over the network. The executor (typically MemoryEngine.execute_task)
+    and stored in the database. The executor (typically MemoryEngine.execute_task)
     receives the dict and routes it to the appropriate handler.
     """
 
@@ -46,7 +55,7 @@ class TaskBackend(ABC):
     @abstractmethod
     async def initialize(self):
         """
-        Initialize the backend (e.g., start workers, connect to broker).
+        Initialize the backend (e.g., connect to database).
         """
         pass
 
@@ -63,7 +72,7 @@ class TaskBackend(ABC):
     @abstractmethod
     async def shutdown(self):
         """
-        Shutdown the backend gracefully (e.g., stop workers, close connections).
+        Shutdown the backend gracefully.
         """
         pass
 
@@ -93,9 +102,8 @@ class SyncTaskBackend(TaskBackend):
     """
     Synchronous task backend that executes tasks immediately.
 
-    This is useful for embedded/CLI usage where we don't want background
-    workers that prevent clean exit. Tasks are executed inline rather than
-    being queued.
+    This is useful for tests and embedded/CLI usage where we don't want
+    background workers. Tasks are executed inline rather than being queued.
     """
 
     async def initialize(self):
@@ -121,221 +129,123 @@ class SyncTaskBackend(TaskBackend):
         logger.debug("SyncTaskBackend shutdown")
 
 
-class NoopTaskBackend(TaskBackend):
+class BrokerTaskBackend(TaskBackend):
     """
-    No-op task backend that discards all tasks.
+    Task backend using PostgreSQL as broker.
 
-    This is useful for tests where background task execution is not needed
-    and would only slow down the test suite.
+    submit_task() stores task_payload in async_operations table.
+    Actual polling and execution is handled separately by WorkerPoller.
+
+    This backend is used by the API to store tasks. Workers poll
+    the database separately to claim and execute tasks.
     """
 
-    async def initialize(self):
-        """No-op."""
-        self._initialized = True
-        logger.debug("NoopTaskBackend initialized")
-
-    async def submit_task(self, task_dict: dict[str, Any]):
-        """Discard the task (do nothing)."""
-        pass
-
-    async def shutdown(self):
-        """No-op."""
-        self._initialized = False
-        logger.debug("NoopTaskBackend shutdown")
-
-
-class AsyncIOQueueBackend(TaskBackend):
-    """
-    Task backend implementation using asyncio queues.
-
-    This is the default implementation that uses in-process asyncio queues
-    and a periodic consumer worker.
-    """
-
-    def __init__(self, batch_size: int = 10, batch_interval: float = 1.0):
+    def __init__(
+        self,
+        pool_getter: Callable[[], "asyncpg.Pool"],
+        schema: str | None = None,
+    ):
         """
-        Initialize AsyncIO queue backend.
+        Initialize the broker task backend.
 
         Args:
-            batch_size: Maximum number of tasks to process in one batch
-            batch_interval: Maximum time (seconds) to wait before processing batch
+            pool_getter: Callable that returns the asyncpg connection pool
+            schema: Database schema for multi-tenant support (optional)
         """
         super().__init__()
-        self._queue: asyncio.Queue | None = None
-        self._worker_task: asyncio.Task | None = None
-        self._shutdown_event: asyncio.Event | None = None
-        self._batch_size = batch_size
-        self._batch_interval = batch_interval
-        self._in_flight_count = 0
-        self._in_flight_lock = asyncio.Lock()
+        self._pool_getter = pool_getter
+        self._schema = schema
 
     async def initialize(self):
-        """Initialize the queue and start the worker."""
-        if self._initialized:
-            return
-
-        self._queue = asyncio.Queue()
-        self._shutdown_event = asyncio.Event()
-        self._worker_task = asyncio.create_task(self._worker())
+        """Initialize the backend."""
         self._initialized = True
-        logger.info("AsyncIOQueueBackend initialized")
+        logger.info("BrokerTaskBackend initialized")
 
     async def submit_task(self, task_dict: dict[str, Any]):
         """
-        Submit a task by putting it in the queue.
+        Store task payload in async_operations table.
+
+        The task_dict should contain an 'operation_id' if updating an existing
+        operation record, otherwise a new operation will be created.
 
         Args:
-            task_dict: Task dictionary to execute
+            task_dict: Task dictionary to store (must be JSON serializable)
         """
         if not self._initialized:
             await self.initialize()
 
-        await self._queue.put(task_dict)
+        pool = self._pool_getter()
+        operation_id = task_dict.get("operation_id")
+        task_type = task_dict.get("type", "unknown")
+        bank_id = task_dict.get("bank_id")
+        payload_json = json.dumps(task_dict)
+
+        table = fq_table("async_operations", self._schema)
+
+        if operation_id:
+            # Update existing operation with task payload
+            await pool.execute(
+                f"""
+                UPDATE {table}
+                SET task_payload = $1::jsonb, updated_at = now()
+                WHERE operation_id = $2
+                """,
+                payload_json,
+                operation_id,
+            )
+            logger.debug(f"Updated task payload for operation {operation_id}")
+        else:
+            # Insert new operation (for tasks without pre-created records)
+            # e.g., access_count_update tasks
+            import uuid
+
+            new_id = uuid.uuid4()
+            await pool.execute(
+                f"""
+                INSERT INTO {table} (operation_id, bank_id, operation_type, status, task_payload)
+                VALUES ($1, $2, $3, 'pending', $4::jsonb)
+                """,
+                new_id,
+                bank_id,
+                task_type,
+                payload_json,
+            )
+            logger.debug(f"Created new operation {new_id} for task type {task_type}")
+
+    async def shutdown(self):
+        """Shutdown the backend."""
+        self._initialized = False
+        logger.info("BrokerTaskBackend shutdown")
 
     async def wait_for_pending_tasks(self, timeout: float = 120.0):
         """
-        Wait for all pending tasks in the queue and in-flight tasks to complete.
+        Wait for pending tasks to be processed.
 
-        This is useful in tests to ensure background tasks complete before assertions.
+        In the broker model, this polls the database to check if tasks
+        for this process have been completed. This is useful in tests
+        when worker_enabled=True (API processes its own tasks).
 
         Args:
-            timeout: Maximum time to wait in seconds (default 120s for long-running tasks)
+            timeout: Maximum time to wait in seconds
         """
-        if not self._initialized or self._queue is None:
-            return
+        import asyncio
+
+        pool = self._pool_getter()
+        table = fq_table("async_operations", self._schema)
 
-        # Wait for queue to be empty AND no in-flight tasks
         start_time = asyncio.get_event_loop().time()
         while asyncio.get_event_loop().time() - start_time < timeout:
-            async with self._in_flight_lock:
-                in_flight = self._in_flight_count
+            # Check if there are any pending tasks with payloads
+            count = await pool.fetchval(
+                f"""
+                SELECT COUNT(*) FROM {table}
+                WHERE status = 'pending' AND task_payload IS NOT NULL
+                """
+            )
 
-            if self._queue.empty() and in_flight == 0:
-                # Queue is empty and no tasks in flight, we're done
+            if count == 0:
                 return
 
-            # Wait a bit before checking again
             await asyncio.sleep(0.5)
 
-    async def shutdown(self):
-        """Shutdown the worker and drain the queue."""
-        if not self._initialized:
-            return
-
-        logger.info("Shutting down AsyncIOQueueBackend...")
-
-        # Signal shutdown
-        self._shutdown_event.set()
-
-        # Cancel worker
-        if self._worker_task is not None:
-            self._worker_task.cancel()
-            try:
-                await self._worker_task
-            except asyncio.CancelledError:
-                pass  # Worker cancelled successfully
-
-        self._initialized = False
-        logger.info("AsyncIOQueueBackend shutdown complete")
-
-    async def _execute_task_with_tracking(self, task_dict: dict[str, Any]):
-        """Execute a task and track its in-flight status."""
-        async with self._in_flight_lock:
-            self._in_flight_count += 1
-        try:
-            await self._execute_task(task_dict)
-        finally:
-            async with self._in_flight_lock:
-                self._in_flight_count -= 1
-
-    async def _execute_task_no_tracking(self, task_dict: dict[str, Any]):
-        """Execute a task without in-flight tracking (tracking done at batch level)."""
-        await self._execute_task(task_dict)
-
-    def _get_queue_stats(self) -> tuple[int, dict[str, int]]:
-        """Get current queue size and bank_id distribution."""
-        queue_size = self._queue.qsize() if self._queue else 0
-        bank_distribution: dict[str, int] = {}
-
-        if queue_size > 0 and self._queue:
-            # Peek at queue items without removing them
-            # Note: This is a snapshot and may not be perfectly accurate due to concurrency
-            try:
-                # Access internal deque for logging purposes only
-                items = list(self._queue._queue)  # type: ignore[attr-defined]
-                for item in items:
-                    bank_id = item.get("bank_id", "unknown")
-                    bank_distribution[bank_id] = bank_distribution.get(bank_id, 0) + 1
-            except Exception:
-                pass  # Queue access failed, return empty distribution
-
-        return queue_size, bank_distribution
-
-    async def _worker(self):
-        """
-        Background worker that processes tasks in batches.
-
-        Collects tasks for up to batch_interval seconds or batch_size items,
-        then processes them.
-        """
-        while not self._shutdown_event.is_set():
-            try:
-                # Collect tasks for batching
-                tasks = []
-                deadline = asyncio.get_event_loop().time() + self._batch_interval
-
-                while len(tasks) < self._batch_size and asyncio.get_event_loop().time() < deadline:
-                    try:
-                        remaining_time = max(0.1, deadline - asyncio.get_event_loop().time())
-                        task_dict = await asyncio.wait_for(self._queue.get(), timeout=remaining_time)
-                        # Track task as in-flight immediately when picked up from queue
-                        # This prevents wait_for_pending_tasks from returning too early
-                        async with self._in_flight_lock:
-                            self._in_flight_count += 1
-                        tasks.append(task_dict)
-                    except TimeoutError:
-                        break
-
-                # Process batch
-                if tasks:
-                    # Log batch start with queue stats
-                    queue_size, bank_distribution = self._get_queue_stats()
-
-                    # Summarize batch by task type and bank
-                    batch_summary: dict[str, dict[str, int]] = {}
-                    for task_dict in tasks:
-                        task_type = task_dict.get("type", "unknown")
-                        bank_id = task_dict.get("bank_id", "unknown")
-                        if task_type not in batch_summary:
-                            batch_summary[task_type] = {}
-                        batch_summary[task_type][bank_id] = batch_summary[task_type].get(bank_id, 0) + 1
-
-                    # Build log message
-                    batch_parts = []
-                    for task_type, banks in sorted(batch_summary.items()):
-                        bank_str = ", ".join(f"{b}:{c}" for b, c in sorted(banks.items()))
-                        batch_parts.append(f"{task_type}[{bank_str}]")
-                    batch_str = ", ".join(batch_parts)
-
-                    if queue_size > 0:
-                        pending_str = ", ".join(f"{k}:{v}" for k, v in sorted(bank_distribution.items()))
-                        logger.info(
-                            f"Processing {len(tasks)} tasks: {batch_str} (pending={queue_size} [{pending_str}])"
-                        )
-                    else:
-                        logger.info(f"Processing {len(tasks)} tasks: {batch_str}")
-
-                    # Execute tasks concurrently (in_flight already tracked when picked up)
-                    await asyncio.gather(
-                        *[self._execute_task_no_tracking(task_dict) for task_dict in tasks], return_exceptions=True
-                    )
-
-                    # Decrement in_flight count after all tasks complete
-                    async with self._in_flight_lock:
-                        self._in_flight_count -= len(tasks)
-
-            except asyncio.CancelledError:
-                break
-            except Exception as e:
-                logger.error(f"Worker error: {e}")
-                await asyncio.sleep(1)  # Backoff on error
+        logger.warning(f"Timeout waiting for pending tasks after {timeout}s")

hindsight-api/hindsight_api/extensions/__init__.py

@@ -27,6 +27,8 @@ from hindsight_api.extensions.operation_validator import (
     RecallResult,
     ReflectContext,
     ReflectResultContext,
+    RefreshMentalModelContext,
+    RefreshMentalModelResult,
     RetainContext,
     RetainResult,
     ValidationResult,
@@ -54,6 +56,8 @@ __all__ = [
     "RecallResult",
     "ReflectContext",
     "ReflectResultContext",
+    "RefreshMentalModelContext",
+    "RefreshMentalModelResult",
     "RetainContext",
     "RetainResult",
     "ValidationResult",

hindsight-api/hindsight_api/extensions/operation_validator.py

@@ -97,6 +97,18 @@ class ReflectContext:
     context: str | None = None
 
 
+@dataclass
+class RefreshMentalModelContext:
+    """Context for a refresh mental model operation validation (pre-operation).
+
+    Contains ALL user-provided parameters for the refresh mental model operation.
+    """
+
+    bank_id: str
+    model_id: str
+    request_context: "RequestContext"
+
+
 # =============================================================================
 # Post-operation Contexts (includes results)
 # =============================================================================
@@ -164,6 +176,27 @@ class ReflectResultContext:
     error: str | None = None
 
 
+@dataclass
+class RefreshMentalModelResult:
+    """Result context for post-refresh-mental-model hook.
+
+    Contains the operation parameters and the result including token usage.
+    """
+
+    bank_id: str
+    model_id: str
+    request_context: "RequestContext"
+    # Result
+    model_name: str | None = None
+    observations_count: int = 0
+    input_tokens: int = 0
+    output_tokens: int = 0
+    total_tokens: int = 0
+    duration_ms: int = 0
+    success: bool = True
+    error: str | None = None
+
+
 class OperationValidatorExtension(Extension, ABC):
     """
     Validates and hooks into retain/recall/reflect operations.
@@ -265,6 +298,25 @@ class OperationValidatorExtension(Extension, ABC):
         """
         ...
 
+    @abstractmethod
+    async def validate_refresh_mental_model(self, ctx: RefreshMentalModelContext) -> ValidationResult:
+        """
+        Validate a refresh mental model operation before execution.
+
+        Called before the refresh mental model operation is processed.
+        Return ValidationResult.reject() to prevent the operation from executing.
+
+        Args:
+            ctx: Context containing all user-provided parameters:
+                - bank_id: Bank identifier
+                - model_id: Mental model ID to refresh
+                - request_context: Request context with auth info
+
+        Returns:
+            ValidationResult indicating whether the operation is allowed.
+        """
+        ...
+
     # =========================================================================
     # Post-operation hooks (optional - override to implement)
     # =========================================================================
@@ -325,3 +377,28 @@ class OperationValidatorExtension(Extension, ABC):
                 - error: Error message (if failed)
         """
         pass
+
+    async def on_refresh_mental_model_complete(self, result: RefreshMentalModelResult) -> None:
+        """
+        Called after a refresh mental model operation completes (success or failure).
+
+        Override this method to implement post-operation logic such as:
+        - Token usage tracking and billing
+        - Audit logging
+        - Metrics collection
+
+        Args:
+            result: Result context containing:
+                - bank_id: Bank identifier
+                - model_id: Mental model ID
+                - request_context: Request context with auth info
+                - model_name: Name of the mental model (if success)
+                - observations_count: Number of observations generated
+                - input_tokens: Number of input tokens used
+                - output_tokens: Number of output tokens used
+                - total_tokens: Total tokens used (input + output)
+                - duration_ms: Total operation duration in milliseconds
+                - success: Whether the operation succeeded
+                - error: Error message (if failed)
+        """
+        pass

hindsight-api/hindsight_api/main.py

@@ -219,9 +219,14 @@ def main():
             db_pool_max_size=config.db_pool_max_size,
             db_command_timeout=config.db_command_timeout,
             db_acquire_timeout=config.db_acquire_timeout,
-            task_backend=config.task_backend,
-            task_backend_memory_batch_size=config.task_backend_memory_batch_size,
-            task_backend_memory_batch_interval=config.task_backend_memory_batch_interval,
+            worker_enabled=config.worker_enabled,
+            worker_id=config.worker_id,
+            worker_poll_interval_ms=config.worker_poll_interval_ms,
+            worker_max_retries=config.worker_max_retries,
+            worker_batch_size=config.worker_batch_size,
+            worker_http_port=config.worker_http_port,
+            reflect_max_iterations=config.reflect_max_iterations,
+            mental_model_refresh_concurrency=config.mental_model_refresh_concurrency,
         )
     config.configure_logging()
     if not args.daemon:

hindsight-api/hindsight_api/worker/__init__.py

@@ -0,0 +1,11 @@
+"""
+Worker package for distributed task processing.
+
+This package provides:
+- WorkerPoller: Polls PostgreSQL for pending tasks and executes them
+- main: CLI entry point for hindsight-worker
+"""
+
+from .poller import WorkerPoller
+
+__all__ = ["WorkerPoller"]

hindsight-api/hindsight_api/worker/main.py

@@ -0,0 +1,285 @@
+"""
+Command-line interface for Hindsight Worker.
+
+Run the worker with:
+    hindsight-worker
+
+Stop with Ctrl+C (graceful shutdown).
+"""
+
+import argparse
+import asyncio
+import atexit
+import logging
+import os
+import signal
+import socket
+import sys
+import warnings
+
+from ..config import get_config
+from ..engine.task_backend import SyncTaskBackend
+from .poller import WorkerPoller
+
+# Filter deprecation warnings from third-party libraries
+warnings.filterwarnings("ignore", message="websockets.legacy is deprecated")
+warnings.filterwarnings("ignore", message="websockets.server.WebSocketServerProtocol is deprecated")
+
+# Disable tokenizers parallelism to avoid warnings
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+logger = logging.getLogger(__name__)
+
+
+def create_worker_app(poller: WorkerPoller, memory):
+    """Create a minimal FastAPI app for worker metrics and health."""
+    from fastapi import FastAPI
+    from fastapi.responses import JSONResponse, Response
+    from prometheus_client import CONTENT_TYPE_LATEST, generate_latest
+
+    from ..metrics import create_metrics_collector, get_metrics_collector, initialize_metrics
+
+    app = FastAPI(
+        title="Hindsight Worker",
+        description="Worker process for distributed task execution",
+    )
+
+    # Initialize OpenTelemetry metrics
+    try:
+        prometheus_reader = initialize_metrics(service_name="hindsight-worker", service_version="1.0.0")
+        create_metrics_collector()
+        app.state.prometheus_reader = prometheus_reader
+        logger.info("Metrics initialized - available at /metrics endpoint")
+    except Exception as e:
+        logger.warning(f"Failed to initialize metrics: {e}. Metrics will be disabled.")
+        app.state.prometheus_reader = None
+
+    # Set up DB pool metrics if available
+    metrics_collector = get_metrics_collector()
+    if memory._pool is not None and hasattr(metrics_collector, "set_db_pool"):
+        metrics_collector.set_db_pool(memory._pool)
+        logger.info("DB pool metrics configured")
+
+    @app.get(
+        "/health",
+        summary="Health check endpoint",
+        description="Returns worker health status including database connectivity",
+        tags=["Monitoring"],
+    )
+    async def health_endpoint():
+        """Health check endpoint."""
+        health = await memory.health_check()
+        health["worker_id"] = poller.worker_id
+        health["is_shutdown"] = poller.is_shutdown
+        status_code = 200 if health.get("status") == "healthy" else 503
+        return JSONResponse(content=health, status_code=status_code)
+
+    @app.get(
+        "/metrics",
+        summary="Prometheus metrics endpoint",
+        description="Exports metrics in Prometheus format for scraping",
+        tags=["Monitoring"],
+    )
+    async def metrics_endpoint():
+        """Return Prometheus metrics."""
+        metrics_data = generate_latest()
+        return Response(content=metrics_data, media_type=CONTENT_TYPE_LATEST)
+
+    @app.get(
+        "/",
+        summary="Worker info",
+        description="Basic worker information",
+        tags=["Info"],
+    )
+    async def root():
+        """Return basic worker info."""
+        return {
+            "service": "hindsight-worker",
+            "worker_id": poller.worker_id,
+            "is_shutdown": poller.is_shutdown,
+        }
+
+    return app
+
+
+def main():
+    """Main entry point for the hindsight-worker CLI."""
+    # Load configuration from environment
+    config = get_config()
+
+    parser = argparse.ArgumentParser(
+        prog="hindsight-worker",
+        description="Hindsight Worker - distributed task processor",
+    )
+
+    # Worker options
+    parser.add_argument(
+        "--worker-id",
+        default=config.worker_id or socket.gethostname(),
+        help="Worker identifier (default: hostname, env: HINDSIGHT_API_WORKER_ID)",
+    )
+    parser.add_argument(
+        "--poll-interval",
+        type=int,
+        default=config.worker_poll_interval_ms,
+        help=f"Poll interval in milliseconds (default: {config.worker_poll_interval_ms}, env: HINDSIGHT_API_WORKER_POLL_INTERVAL_MS)",
+    )
+    parser.add_argument(
+        "--batch-size",
+        type=int,
+        default=config.worker_batch_size,
+        help=f"Tasks to claim per poll (default: {config.worker_batch_size}, env: HINDSIGHT_API_WORKER_BATCH_SIZE)",
+    )
+    parser.add_argument(
+        "--max-retries",
+        type=int,
+        default=config.worker_max_retries,
+        help=f"Max retries before marking failed (default: {config.worker_max_retries}, env: HINDSIGHT_API_WORKER_MAX_RETRIES)",
+    )
+
+    # HTTP server options
+    parser.add_argument(
+        "--http-port",
+        type=int,
+        default=config.worker_http_port,
+        help=f"HTTP port for metrics/health endpoints (default: {config.worker_http_port}, env: HINDSIGHT_API_WORKER_HTTP_PORT)",
+    )
+    parser.add_argument(
+        "--http-host",
+        default="0.0.0.0",
+        help="HTTP host to bind (default: 0.0.0.0)",
+    )
+
+    # Logging options
+    parser.add_argument(
+        "--log-level",
+        default=config.log_level,
+        choices=["critical", "error", "warning", "info", "debug", "trace"],
+        help=f"Log level (default: {config.log_level}, env: HINDSIGHT_API_LOG_LEVEL)",
+    )
+
+    args = parser.parse_args()
+
+    # Configure logging
+    config.configure_logging()
+
+    # Import MemoryEngine here to avoid circular imports
+    from .. import MemoryEngine
+
+    print(f"Starting Hindsight Worker: {args.worker_id}")
+    print(f"  Poll interval: {args.poll_interval}ms")
+    print(f"  Batch size: {args.batch_size}")
+    print(f"  Max retries: {args.max_retries}")
+    print(f"  HTTP server: {args.http_host}:{args.http_port}")
+    print()
+
+    # Global references for cleanup
+    memory = None
+    poller = None
+
+    async def run():
+        nonlocal memory, poller
+        import uvicorn
+
+        # Initialize MemoryEngine
+        # Workers use SyncTaskBackend because they execute tasks directly,
+        # they don't need to store tasks (they poll from DB)
+        memory = MemoryEngine(
+            run_migrations=False,  # Workers don't run migrations
+            task_backend=SyncTaskBackend(),
+        )
+
+        await memory.initialize()
+
+        print(f"Database connected: {config.database_url}")
+
+        # Create and start the poller
+        poller = WorkerPoller(
+            pool=memory._pool,
+            worker_id=args.worker_id,
+            executor=memory.execute_task,
+            poll_interval_ms=args.poll_interval,
+            batch_size=args.batch_size,
+            max_retries=args.max_retries,
+        )
+
+        # Create the HTTP app for metrics/health
+        app = create_worker_app(poller, memory)
+
+        # Setup signal handlers for graceful shutdown
+        shutdown_requested = asyncio.Event()
+
+        def signal_handler(signum, frame):
+            print(f"\nReceived signal {signum}, initiating graceful shutdown...")
+            shutdown_requested.set()
+
+        signal.signal(signal.SIGINT, signal_handler)
+        signal.signal(signal.SIGTERM, signal_handler)
+
+        # Create uvicorn config and server
+        uvicorn_config = uvicorn.Config(
+            app,
+            host=args.http_host,
+            port=args.http_port,
+            log_level="info",  # Reduce uvicorn noise
+            access_log=False,
+        )
+        server = uvicorn.Server(uvicorn_config)
+
+        # Run the poller and HTTP server concurrently
+        poller_task = asyncio.create_task(poller.run())
+        http_task = asyncio.create_task(server.serve())
+
+        print(f"Worker started. Metrics available at http://{args.http_host}:{args.http_port}/metrics")
+
+        # Wait for shutdown signal
+        await shutdown_requested.wait()
+
+        # Graceful shutdown
+        print("Shutting down HTTP server...")
+        server.should_exit = True
+
+        print("Waiting for poller to finish...")
+        await poller.shutdown_graceful(timeout=30.0)
+        poller_task.cancel()
+        try:
+            await poller_task
+        except asyncio.CancelledError:
+            pass
+
+        # Wait for HTTP server to finish
+        try:
+            await asyncio.wait_for(http_task, timeout=5.0)
+        except asyncio.TimeoutError:
+            http_task.cancel()
+            try:
+                await http_task
+            except asyncio.CancelledError:
+                pass
+
+        # Close memory engine
+        await memory.close()
+        print("Worker shutdown complete")
+
+    def cleanup():
+        """Synchronous cleanup for atexit."""
+        if memory is not None and memory._pg0 is not None:
+            try:
+                loop = asyncio.new_event_loop()
+                loop.run_until_complete(memory._pg0.stop())
+                loop.close()
+                print("\npg0 stopped.")
+            except Exception as e:
+                print(f"\nError stopping pg0: {e}")
+
+    atexit.register(cleanup)
+
+    try:
+        asyncio.run(run())
+    except KeyboardInterrupt:
+        print("\nWorker interrupted")
+        sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()

hindsight-api/hindsight_api/worker/poller.py

@@ -0,0 +1,281 @@
+"""
+Worker poller for distributed task execution.
+
+Polls PostgreSQL for pending tasks and executes them using
+FOR UPDATE SKIP LOCKED for safe concurrent claiming.
+"""
+
+import asyncio
+import json
+import logging
+import traceback
+from collections.abc import Awaitable, Callable
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    import asyncpg
+
+logger = logging.getLogger(__name__)
+
+
+def fq_table(table: str, schema: str | None = None) -> str:
+    """Get fully-qualified table name with optional schema prefix."""
+    if schema:
+        return f'"{schema}".{table}'
+    return table
+
+
+class WorkerPoller:
+    """
+    Polls PostgreSQL for pending tasks and executes them.
+
+    Uses FOR UPDATE SKIP LOCKED for safe distributed claiming,
+    allowing multiple workers to process tasks without conflicts.
+    """
+
+    def __init__(
+        self,
+        pool: "asyncpg.Pool",
+        worker_id: str,
+        executor: Callable[[dict[str, Any]], Awaitable[None]],
+        poll_interval_ms: int = 500,
+        batch_size: int = 10,
+        max_retries: int = 3,
+        schema: str | None = None,
+    ):
+        """
+        Initialize the worker poller.
+
+        Args:
+            pool: asyncpg connection pool
+            worker_id: Unique identifier for this worker
+            executor: Async function to execute tasks (typically MemoryEngine.execute_task)
+            poll_interval_ms: Interval between polls when no tasks found (milliseconds)
+            batch_size: Maximum number of tasks to claim per poll cycle
+            max_retries: Maximum retry attempts before marking task as failed
+            schema: Database schema for multi-tenant support (optional)
+        """
+        self._pool = pool
+        self._worker_id = worker_id
+        self._executor = executor
+        self._poll_interval_ms = poll_interval_ms
+        self._batch_size = batch_size
+        self._max_retries = max_retries
+        self._schema = schema
+        self._shutdown = asyncio.Event()
+        self._current_tasks: set[asyncio.Task] = set()
+        self._in_flight_count = 0
+        self._in_flight_lock = asyncio.Lock()
+
+    async def claim_batch(self) -> list[tuple[str, dict[str, Any]]]:
+        """
+        Claim up to batch_size pending tasks atomically.
+
+        Uses FOR UPDATE SKIP LOCKED to ensure no conflicts with other workers.
+
+        Returns:
+            List of tuples (operation_id, task_dict)
+        """
+        table = fq_table("async_operations", self._schema)
+
+        async with self._pool.acquire() as conn:
+            async with conn.transaction():
+                # Select and lock pending tasks
+                rows = await conn.fetch(
+                    f"""
+                    SELECT operation_id, task_payload
+                    FROM {table}
+                    WHERE status = 'pending' AND task_payload IS NOT NULL
+                    ORDER BY created_at
+                    LIMIT $1
+                    FOR UPDATE SKIP LOCKED
+                    """,
+                    self._batch_size,
+                )
+
+                if not rows:
+                    return []
+
+                # Claim the tasks by updating status and worker_id
+                operation_ids = [row["operation_id"] for row in rows]
+                await conn.execute(
+                    f"""
+                    UPDATE {table}
+                    SET status = 'processing', worker_id = $1, claimed_at = now(), updated_at = now()
+                    WHERE operation_id = ANY($2)
+                    """,
+                    self._worker_id,
+                    operation_ids,
+                )
+
+                # Parse and return task payloads
+                return [(str(row["operation_id"]), json.loads(row["task_payload"])) for row in rows]
+
+    async def _mark_completed(self, operation_id: str):
+        """Mark a task as completed."""
+        table = fq_table("async_operations", self._schema)
+        await self._pool.execute(
+            f"""
+            UPDATE {table}
+            SET status = 'completed', completed_at = now(), updated_at = now()
+            WHERE operation_id = $1
+            """,
+            operation_id,
+        )
+
+    async def _mark_failed(self, operation_id: str, error_message: str):
+        """Mark a task as failed with error message."""
+        table = fq_table("async_operations", self._schema)
+        # Truncate error message if too long (max 5000 chars in schema)
+        error_message = error_message[:5000] if len(error_message) > 5000 else error_message
+        await self._pool.execute(
+            f"""
+            UPDATE {table}
+            SET status = 'failed', error_message = $2, completed_at = now(), updated_at = now()
+            WHERE operation_id = $1
+            """,
+            operation_id,
+            error_message,
+        )
+
+    async def _retry_or_fail(self, operation_id: str, error_message: str):
+        """Increment retry count or mark as failed if max retries exceeded."""
+        table = fq_table("async_operations", self._schema)
+
+        # Get current retry count
+        row = await self._pool.fetchrow(
+            f"SELECT retry_count FROM {table} WHERE operation_id = $1",
+            operation_id,
+        )
+
+        if row is None:
+            logger.warning(f"Operation {operation_id} not found, cannot retry")
+            return
+
+        retry_count = row["retry_count"]
+
+        if retry_count >= self._max_retries:
+            # Max retries exceeded, mark as failed
+            await self._mark_failed(
+                operation_id, f"Max retries ({self._max_retries}) exceeded. Last error: {error_message}"
+            )
+            logger.error(f"Task {operation_id} failed after {retry_count} retries")
+        else:
+            # Increment retry and reset to pending
+            await self._pool.execute(
+                f"""
+                UPDATE {table}
+                SET status = 'pending', worker_id = NULL, claimed_at = NULL,
+                    retry_count = retry_count + 1, updated_at = now()
+                WHERE operation_id = $1
+                """,
+                operation_id,
+            )
+            logger.warning(f"Task {operation_id} failed, will retry (attempt {retry_count + 1}/{self._max_retries})")
+
+    async def execute_task(self, operation_id: str, task_dict: dict[str, Any]):
+        """Execute a single task and update its status."""
+        task_type = task_dict.get("type", "unknown")
+        bank_id = task_dict.get("bank_id", "unknown")
+
+        try:
+            logger.debug(f"Executing task {operation_id} (type={task_type}, bank={bank_id})")
+            await self._executor(task_dict)
+            await self._mark_completed(operation_id)
+            logger.debug(f"Task {operation_id} completed successfully")
+        except Exception as e:
+            error_msg = f"{type(e).__name__}: {e}\n{traceback.format_exc()}"
+            logger.error(f"Task {operation_id} failed: {e}")
+            await self._retry_or_fail(operation_id, error_msg)
+
+    async def run(self):
+        """
+        Main polling loop.
+
+        Continuously polls for pending tasks, claims them, and executes them
+        until shutdown is signaled.
+        """
+        logger.info(f"Worker {self._worker_id} starting polling loop")
+
+        while not self._shutdown.is_set():
+            try:
+                # Claim a batch of tasks
+                tasks = await self.claim_batch()
+
+                if tasks:
+                    # Log batch info
+                    task_types = {}
+                    for _, task_dict in tasks:
+                        t = task_dict.get("type", "unknown")
+                        task_types[t] = task_types.get(t, 0) + 1
+                    types_str = ", ".join(f"{k}:{v}" for k, v in task_types.items())
+                    logger.info(f"Worker {self._worker_id} claimed {len(tasks)} tasks: {types_str}")
+
+                    # Track in-flight tasks
+                    async with self._in_flight_lock:
+                        self._in_flight_count += len(tasks)
+
+                    # Execute tasks concurrently
+                    try:
+                        await asyncio.gather(
+                            *[self.execute_task(op_id, task_dict) for op_id, task_dict in tasks],
+                            return_exceptions=True,
+                        )
+                    finally:
+                        async with self._in_flight_lock:
+                            self._in_flight_count -= len(tasks)
+                else:
+                    # No tasks found, wait before polling again
+                    try:
+                        await asyncio.wait_for(
+                            self._shutdown.wait(),
+                            timeout=self._poll_interval_ms / 1000,
+                        )
+                    except asyncio.TimeoutError:
+                        pass  # Normal timeout, continue polling
+
+            except asyncio.CancelledError:
+                logger.info(f"Worker {self._worker_id} polling loop cancelled")
+                break
+            except Exception as e:
+                logger.error(f"Worker {self._worker_id} error in polling loop: {e}")
+                traceback.print_exc()
+                # Backoff on error
+                await asyncio.sleep(1)
+
+        logger.info(f"Worker {self._worker_id} polling loop stopped")
+
+    async def shutdown_graceful(self, timeout: float = 30.0):
+        """
+        Signal shutdown and wait for current tasks to complete.
+
+        Args:
+            timeout: Maximum time to wait for in-flight tasks (seconds)
+        """
+        logger.info(f"Worker {self._worker_id} initiating graceful shutdown")
+        self._shutdown.set()
+
+        # Wait for in-flight tasks to complete
+        start_time = asyncio.get_event_loop().time()
+        while asyncio.get_event_loop().time() - start_time < timeout:
+            async with self._in_flight_lock:
+                in_flight = self._in_flight_count
+
+            if in_flight == 0:
+                logger.info(f"Worker {self._worker_id} graceful shutdown complete")
+                return
+
+            logger.info(f"Worker {self._worker_id} waiting for {in_flight} in-flight tasks")
+            await asyncio.sleep(0.5)
+
+        logger.warning(f"Worker {self._worker_id} shutdown timeout after {timeout}s")
+
+    @property
+    def worker_id(self) -> str:
+        """Get the worker ID."""
+        return self._worker_id
+
+    @property
+    def is_shutdown(self) -> bool:
+        """Check if shutdown has been signaled."""
+        return self._shutdown.is_set()

hindsight-api/pyproject.toml

@@ -25,7 +25,7 @@ dependencies = [
     "psycopg2-binary>=2.9.11",
     "tiktoken>=0.12.0",
     "httpx>=0.27.0",
-    "fastmcp>=2.3.0",
+    "fastmcp>=2.14.0",  # CVE-2025-66416
     "pg0-embedded>=0.11.0",
     "python-dateutil>=2.8.0",
     "opentelemetry-api>=1.20.0",
@@ -39,10 +39,17 @@ dependencies = [
     "cohere>=5.0.0",
     "flashrank>=0.2.0",
     # Local ML models for embeddings/reranking - can be excluded in Docker with INCLUDE_LOCAL_MODELS=false
-    "sentence-transformers>=3.0.0,<3.3.0",
-    "transformers>=4.30.0,<4.46.0",
-    "torch>=2.0.0",
+    "sentence-transformers>=3.3.0",
+    "transformers>=4.53.0",  # Security fixes for ReDoS vulnerabilities
+    "torch>=2.6.0",  # CVE fix for remote code execution
     "uvloop>=0.22.1",
+    # Transitive dependency security fixes
+    "pyasn1>=0.6.2",  # DoS vulnerability fix
+    "urllib3>=2.6.3",  # Decompression-bomb safeguards bypass fix
+    "langchain-core>=1.2.5",  # Serialization injection vulnerability fix
+    "filelock>=3.20.1",  # TOCTOU race condition fix
+    "authlib>=1.6.6",  # Account takeover vulnerability fix
+    "aiohttp>=3.13.3",  # Multiple DoS vulnerabilities
 ]
 
 [project.optional-dependencies]
@@ -51,11 +58,12 @@ test = [
     "pytest-asyncio>=0.21.0",
     "pytest-timeout>=2.4.0",
     "pytest-xdist>=3.0.0",
-    "filelock>=3.0.0",
+    "filelock>=3.20.1",  # TOCTOU race condition fix
 ]
 
 [project.scripts]
 hindsight-api = "hindsight_api.main:main"
+hindsight-worker = "hindsight_api.worker.main:main"
 hindsight-local-mcp = "hindsight_api.mcp_local:main"
 hindsight-admin = "hindsight_api.admin.cli:main"
 
@@ -97,7 +105,7 @@ dev = [
     "pytest-timeout>=2.4.0",
     "pytest-xdist>=3.8.0",
     "python-dotenv>=1.2.1",
-    "filelock>=3.0.0",
+    "filelock>=3.20.1",  # TOCTOU race condition fix
     "ruff>=0.8.0",
     "ty>=0.0.1",
 ]

hindsight-api/tests/conftest.py

@@ -12,6 +12,7 @@ from hindsight_api import MemoryEngine, LLMConfig, LocalSTEmbeddings, RequestCon
 
 from hindsight_api.engine.cross_encoder import LocalSTCrossEncoder
 from hindsight_api.engine.query_analyzer import DateparserQueryAnalyzer
+from hindsight_api.engine.task_backend import SyncTaskBackend
 from hindsight_api.pg0 import EmbeddedPostgres
 
 # Default pg0 instance configuration for tests
@@ -147,6 +148,7 @@ async def memory(pg0_db_url, embeddings, cross_encoder, query_analyzer):
     Uses pg0_db_url (a postgresql:// URL) directly, so MemoryEngine won't try to
     manage pg0 lifecycle - that's handled by the session-scoped pg0_db_url fixture.
     Migrations are disabled here since they're run once at session scope in pg0_db_url.
+    Uses SyncTaskBackend so async tasks execute immediately (no worker needed).
     """
     mem = MemoryEngine(
         db_url=pg0_db_url,  # Direct postgresql:// URL, not pg0://
@@ -160,6 +162,7 @@ async def memory(pg0_db_url, embeddings, cross_encoder, query_analyzer):
         pool_min_size=1,
         pool_max_size=5,
         run_migrations=False,  # Migrations already run at session scope
+        task_backend=SyncTaskBackend(),  # Execute tasks immediately in tests
     )
     await mem.initialize()
     yield mem

hindsight-api/tests/test_agents_api.py

@@ -1,5 +1,5 @@
 """
-Tests for agent management API (profile, disposition, background).
+Tests for agent management API (profile, disposition).
 """
 import pytest
 import uuid
@@ -25,15 +25,12 @@ class TestAgentProfile:
 
         assert profile is not None
         assert "disposition" in profile
-        assert "background" in profile
 
         disposition = profile["disposition"]
         assert disposition.skepticism == 3
         assert disposition.literalism == 3
         assert disposition.empathy == 3
 
-        assert profile["background"] == ""
-
     @pytest.mark.asyncio
     async def test_update_agent_disposition(self, memory: MemoryEngine, request_context):
         """Test updating agent disposition traits."""
@@ -76,63 +73,10 @@ class TestAgentProfile:
         for agent in agents:
             assert "bank_id" in agent
             assert "disposition" in agent
-            assert "background" in agent
             assert "created_at" in agent
             assert "updated_at" in agent
 
 
-class TestAgentBackground:
-    """Tests for agent background management."""
-
-    @pytest.mark.asyncio
-    async def test_merge_agent_background(self, memory: MemoryEngine, request_context):
-        """Test merging agent background information."""
-        bank_id = unique_agent_id("test_profile_merge")
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        assert profile["background"] == ""
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I was born in Texas",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result1["background"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "I have 10 years of startup experience",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result2["background"] or "startup" in result2["background"]
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        assert final_profile["background"] != ""
-
-    @pytest.mark.asyncio
-    async def test_merge_background_handles_conflicts(self, memory: MemoryEngine, request_context):
-        """Test that merging background handles conflicts (new overwrites old)."""
-        bank_id = unique_agent_id("test_profile_conflict")
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I was born in Colorado",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Colorado" in result1["background"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "You were born in Texas",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result2["background"]
-
-
 class TestAgentEndpoint:
     """Tests for agent PUT endpoint logic."""
 
@@ -147,7 +91,6 @@ class TestAgentEndpoint:
                 literalism=5,
                 empathy=2
             ),
-            background="I am a creative software engineer"
         )
 
         profile = await memory.get_bank_profile(bank_id, request_context=request_context)
@@ -159,55 +102,10 @@ class TestAgentEndpoint:
                 request_context=request_context,
             )
 
-        if request.background is not None:
-            pool = await memory._get_pool()
-            async with pool.acquire() as conn:
-                await conn.execute(
-                    """
-                    UPDATE banks
-                    SET background = $2,
-                        updated_at = NOW()
-                    WHERE bank_id = $1
-                    """,
-                    bank_id,
-                    request.background
-                )
-
         final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
 
         assert final_profile["disposition"].skepticism == 4
         assert final_profile["disposition"].literalism == 5
-        assert final_profile["background"] == "I am a creative software engineer"
-
-    @pytest.mark.asyncio
-    async def test_put_agent_partial_update(self, memory: MemoryEngine, request_context):
-        """Test updating only background."""
-        bank_id = unique_agent_id("test_put_partial")
-
-        request = CreateBankRequest(
-            background="I am a data scientist"
-        )
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-
-        if request.background is not None:
-            pool = await memory._get_pool()
-            async with pool.acquire() as conn:
-                await conn.execute(
-                    """
-                    UPDATE banks
-                    SET background = $2,
-                        updated_at = NOW()
-                    WHERE bank_id = $1
-                    """,
-                    bank_id,
-                    request.background
-                )
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-
-        assert final_profile["disposition"].skepticism == 3  # Default
-        assert final_profile["background"] == "I am a data scientist"
 
 
 class TestAgentDispositionIntegration:
@@ -225,13 +123,6 @@ class TestAgentDispositionIntegration:
         }
         await memory.update_bank_disposition(bank_id, disposition, request_context=request_context)
 
-        await memory.merge_bank_background(
-            bank_id,
-            "I am a creative artist who values innovation over tradition",
-            update_disposition=False,
-            request_context=request_context,
-        )
-
         await memory.retain_batch_async(
             bank_id=bank_id,
             contents=[

hindsight-api/tests/test_custom_embedding_dimension.py

@@ -17,6 +17,7 @@ from hindsight_api import MemoryEngine, RequestContext
 from hindsight_api.engine.embeddings import LocalSTEmbeddings, OpenAIEmbeddings, CohereEmbeddings
 from hindsight_api.engine.cross_encoder import LocalSTCrossEncoder, CohereCrossEncoder
 from hindsight_api.engine.query_analyzer import DateparserQueryAnalyzer
+from hindsight_api.engine.task_backend import SyncTaskBackend
 from hindsight_api.extensions import TenantExtension, TenantContext
 from hindsight_api.migrations import run_migrations, ensure_embedding_dimension
 
@@ -323,6 +324,7 @@ class TestOpenAIEmbeddings:
             pool_max_size=3,
             run_migrations=False,
             tenant_extension=SchemaTenantExtension(schema_name),
+            task_backend=SyncTaskBackend(),
         )
 
         try:
@@ -392,6 +394,7 @@ class TestOpenAIEmbeddings:
             pool_max_size=3,
             run_migrations=False,
             tenant_extension=SchemaTenantExtension(schema_name),
+            task_backend=SyncTaskBackend(),
         )
 
         try:
@@ -559,6 +562,7 @@ class TestCohereIntegration:
             pool_max_size=3,
             run_migrations=False,
             tenant_extension=SchemaTenantExtension(schema_name),
+            task_backend=SyncTaskBackend(),
         )
 
         try:

hindsight-api/tests/test_emergent_filtering.py

@@ -0,0 +1,516 @@
+"""Tests for emergent entity filtering."""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock
+
+from hindsight_api.engine.mental_models.emergent import (
+    build_mission_filter_prompt,
+    evaluate_emergent_models,
+    filter_candidates_by_mission,
+    MissionFilterResponse,
+    MissionFilterCandidate,
+)
+from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+
+class TestBuildMissionFilterPrompt:
+    """Test prompt building for mission filtering."""
+
+    def test_prompt_contains_mission(self):
+        """Test that prompt includes the mission."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+        prompt = build_mission_filter_prompt("Be a PM for engineering team", candidates)
+        assert "Be a PM for engineering team" in prompt
+
+    def test_prompt_contains_candidates(self):
+        """Test that prompt includes all candidates."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice Chen",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Project Phoenix",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+        prompt = build_mission_filter_prompt("Track projects", candidates)
+        assert "Alice Chen" in prompt
+        assert "Project Phoenix" in prompt
+
+    def test_prompt_contains_rejection_guidance(self):
+        """Test that prompt contains guidance to reject generic entities."""
+        candidates = [
+            EmergentCandidate(
+                name="test",
+                detection_method="named_entity_extraction",
+                mention_count=1,
+            )
+        ]
+        prompt = build_mission_filter_prompt("Test mission", candidates)
+
+        # Should contain rejection guidance for generic terms
+        assert "promote=false" in prompt
+        assert "kids" in prompt  # Example of generic term to reject
+        assert "community" in prompt  # Example of abstract concept to reject
+        assert "motivation" in prompt  # Example of abstract concept to reject
+
+
+class TestFilterCandidatesByMission:
+    """Test the filter_candidates_by_mission function."""
+
+    @pytest.fixture
+    def mock_llm_config(self):
+        """Create a mock LLM config."""
+        config = MagicMock()
+        config.call = AsyncMock()
+        return config
+
+    async def test_empty_candidates(self, mock_llm_config):
+        """Test with empty candidate list."""
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=[],
+        )
+        assert result == []
+        mock_llm_config.call.assert_not_called()
+
+    async def test_no_mission_keeps_all(self, mock_llm_config):
+        """Test that no mission keeps all candidates (skips filtering)."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="",  # Empty mission
+            candidates=candidates,
+        )
+        assert len(result) == 1
+        assert result[0].name == "Alice"
+        mock_llm_config.call.assert_not_called()
+
+    async def test_filters_by_promote_flag(self, mock_llm_config):
+        """Test that candidates are filtered by promote flag."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice Chen",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="community",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+
+        # Mock LLM response - Alice is promoted, community is not
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice Chen", promote=True, reason="Specific person"),
+                MissionFilterCandidate(name="community", promote=False, reason="Generic abstract concept"),
+            ]
+        )
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a PM for engineering team",
+            candidates=candidates,
+        )
+
+        assert len(result) == 1
+        assert result[0].name == "Alice Chen"
+
+    async def test_rejects_generic_entities(self, mock_llm_config):
+        """Test that generic entities are rejected."""
+        # These are all generic/abstract terms that should be rejected
+        generic_names = [
+            "user", "support", "community", "family", "motivation",
+            "photo", "gratitude", "difference", "volunteering",
+            "kids", "veterans", "impact", "kindness", "encouragement",
+            "education", "nature", "joy", "positivity", "inspiration",
+            "help", "commitment", "passion", "energy", "connection",
+        ]
+        candidates = [
+            EmergentCandidate(
+                name=name,
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+            for name in generic_names
+        ]
+
+        # Add some valid candidates
+        valid_candidates = [
+            EmergentCandidate(
+                name="John",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Maria",
+                detection_method="named_entity_extraction",
+                mention_count=8,
+            ),
+            EmergentCandidate(
+                name="Max",
+                detection_method="named_entity_extraction",
+                mention_count=6,
+            ),
+        ]
+        candidates.extend(valid_candidates)
+
+        # Mock LLM response - reject all generic, promote only specific names
+        response_candidates = [
+            MissionFilterCandidate(name=name, promote=False, reason="Generic/abstract term")
+            for name in generic_names
+        ]
+        response_candidates.extend([
+            MissionFilterCandidate(name=c.name, promote=True, reason="Specific person name")
+            for c in valid_candidates
+        ])
+
+        mock_llm_config.call.return_value = MissionFilterResponse(candidates=response_candidates)
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a health coach",
+            candidates=candidates,
+        )
+
+        # Should only have John, Maria, and Max
+        result_names = {c.name for c in result}
+        assert result_names == {"John", "Maria", "Max"}
+
+    async def test_accepts_specific_named_entities(self, mock_llm_config):
+        """Test that specific named entities are accepted."""
+        # These should all be accepted
+        valid_names = [
+            "Alice Chen",       # Full name
+            "Dr. Smith",        # Title + name
+            "John",             # First name (when it's clearly a person)
+            "Google",           # Organization
+            "Frontend Team",    # Named team
+            "Project Phoenix",  # Named project
+            "NYC Office",       # Named place
+            "Q4 Planning",      # Named event
+            "Sprint 23 Review", # Named meeting
+        ]
+        candidates = [
+            EmergentCandidate(
+                name=name,
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+            for name in valid_names
+        ]
+
+        # Mock LLM response - promote all
+        response_candidates = [
+            MissionFilterCandidate(name=name, promote=True, reason="Specific named entity")
+            for name in valid_names
+        ]
+        mock_llm_config.call.return_value = MissionFilterResponse(candidates=response_candidates)
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a PM for engineering team",
+            candidates=candidates,
+        )
+
+        # Should have all valid names
+        result_names = {c.name for c in result}
+        assert result_names == set(valid_names)
+
+    async def test_llm_error_rejects_all_candidates(self, mock_llm_config):
+        """Test that LLM errors result in rejecting all candidates (fail-safe)."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+
+        mock_llm_config.call.side_effect = Exception("LLM error")
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=candidates,
+        )
+
+        # Should reject all candidates on error (fail-safe)
+        assert len(result) == 0
+
+    async def test_missing_candidate_in_response_is_rejected(self, mock_llm_config):
+        """Test that candidates not in LLM response are rejected by default."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Bob",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+
+        # Mock LLM response - only includes Alice, not Bob
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice", promote=True, reason="Specific person"),
+            ]
+        )
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=candidates,
+        )
+
+        # Only Alice should be in result (Bob was missing from response, so rejected)
+        assert len(result) == 1
+        assert result[0].name == "Alice"
+
+
+class TestEvaluateEmergentModels:
+    """Test the evaluate_emergent_models function for cleanup of existing models."""
+
+    @pytest.fixture
+    def mock_llm_config(self):
+        """Create a mock LLM config."""
+        config = MagicMock()
+        config.call = AsyncMock()
+        return config
+
+    async def test_empty_models(self, mock_llm_config):
+        """Test with empty model list."""
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=[],
+        )
+        assert result == []
+        mock_llm_config.call.assert_not_called()
+
+    async def test_removes_generic_models(self, mock_llm_config):
+        """Test that generic/abstract models are marked for removal."""
+        models = [
+            {"id": "id-kids", "name": "kids"},
+            {"id": "id-community", "name": "community"},
+            {"id": "id-motivation", "name": "motivation"},
+            {"id": "id-john", "name": "John"},
+            {"id": "id-maria", "name": "Maria"},
+        ]
+
+        # Mock LLM response - reject generic, keep specific names
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="kids", promote=False, reason="Generic category"),
+                MissionFilterCandidate(name="community", promote=False, reason="Abstract concept"),
+                MissionFilterCandidate(name="motivation", promote=False, reason="Abstract concept"),
+                MissionFilterCandidate(name="John", promote=True, reason="Person name"),
+                MissionFilterCandidate(name="Maria", promote=True, reason="Person name"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Should return IDs of generic models to remove
+        assert set(result) == {"id-kids", "id-community", "id-motivation"}
+
+    async def test_keeps_specific_named_models(self, mock_llm_config):
+        """Test that specific named models are kept."""
+        models = [
+            {"id": "id-john", "name": "John"},
+            {"id": "id-google", "name": "Google"},
+            {"id": "id-project", "name": "Project Phoenix"},
+        ]
+
+        # Mock LLM response - keep all
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="John", promote=True, reason="Person name"),
+                MissionFilterCandidate(name="Google", promote=True, reason="Organization"),
+                MissionFilterCandidate(name="Project Phoenix", promote=True, reason="Named project"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # No models should be removed
+        assert result == []
+
+    async def test_llm_error_keeps_all_models(self, mock_llm_config):
+        """Test that LLM errors result in keeping all models (safe default)."""
+        models = [
+            {"id": "id-kids", "name": "kids"},
+            {"id": "id-john", "name": "John"},
+        ]
+
+        mock_llm_config.call.side_effect = Exception("LLM error")
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Should keep all models on error (return empty removal list)
+        assert result == []
+
+    async def test_missing_model_in_response_is_removed(self, mock_llm_config):
+        """Test that models not in LLM response are marked for removal."""
+        models = [
+            {"id": "id-alice", "name": "Alice"},
+            {"id": "id-bob", "name": "Bob"},
+        ]
+
+        # Mock LLM response - only includes Alice
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice", promote=True, reason="Person name"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Bob should be marked for removal (missing from response)
+        assert result == ["id-bob"]
+
+
+class TestRemovedEntitiesNotRepromoted:
+    """Test that entities removed by evaluation are not re-promoted.
+
+    This tests the fix for a bug where:
+    1. evaluate_emergent_models returns model IDs to remove (e.g., 'entity-maya')
+    2. We delete those models
+    3. detect_entity_candidates finds the same entities (now eligible since model was deleted)
+    4. filter_candidates_by_goal approves them (different LLM call)
+    5. BUG: We were re-promoting the same entities we just removed
+
+    The fix tracks removed entity_ids and excludes them from promotion.
+    """
+
+    async def test_removed_entity_ids_excluded_from_promotion(self):
+        """Test that entities whose models were removed are not re-promoted."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # Simulate the scenario from the bug:
+        # - existing_emergent has model 'entity-maya' with entity_id='uuid-maya'
+        # - evaluate_emergent_models says to remove 'entity-maya'
+        # - detect_entity_candidates returns 'Maya' with entity_id='uuid-maya' (now eligible)
+        # - filter_candidates_by_goal says to promote 'Maya'
+        # - But we should NOT promote because we just removed it
+
+        existing_emergent = [
+            {"id": "entity-maya", "name": "Maya", "entity_id": "uuid-maya"},
+            {"id": "entity-alex", "name": "Alex", "entity_id": "uuid-alex"},
+            {"id": "entity-john", "name": "John", "entity_id": "uuid-john"},  # This one will be kept
+        ]
+
+        # Models to remove (evaluate_emergent_models would return these)
+        models_to_remove = ["entity-maya", "entity-alex"]
+
+        # Build model_id -> entity_id mapping (this is what the fix does)
+        model_to_entity = {m["id"]: m.get("entity_id") for m in existing_emergent}
+
+        # Track removed entity_ids
+        removed_entity_ids: set[str] = set()
+        for model_id in models_to_remove:
+            entity_id = model_to_entity.get(model_id)
+            if entity_id:
+                removed_entity_ids.add(str(entity_id))
+
+        # Verify we tracked the right entity_ids
+        assert removed_entity_ids == {"uuid-maya", "uuid-alex"}
+
+        # Now simulate candidates that were detected (includes removed entities)
+        candidates = [
+            EmergentCandidate(
+                name="Maya", entity_id="uuid-maya", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Alex", entity_id="uuid-alex", detection_method="named_entity", mention_count=8
+            ),
+            EmergentCandidate(
+                name="NewPerson", entity_id="uuid-new", detection_method="named_entity", mention_count=5
+            ),
+        ]
+
+        # Filter out candidates whose entity was just removed (the fix)
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        # Only NewPerson should remain - Maya and Alex were removed and should not be re-promoted
+        assert len(filtered_candidates) == 1
+        assert filtered_candidates[0].name == "NewPerson"
+        assert filtered_candidates[0].entity_id == "uuid-new"
+
+    async def test_candidates_without_matching_removal_are_kept(self):
+        """Test that candidates not in the removed set are still promoted."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # No models removed
+        removed_entity_ids: set[str] = set()
+
+        candidates = [
+            EmergentCandidate(
+                name="Alice", entity_id="uuid-alice", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Bob", entity_id="uuid-bob", detection_method="named_entity", mention_count=8
+            ),
+        ]
+
+        # Filter (should keep all since nothing was removed)
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        assert len(filtered_candidates) == 2
+        assert {c.name for c in filtered_candidates} == {"Alice", "Bob"}
+
+    async def test_partial_removal_keeps_other_candidates(self):
+        """Test that only removed entities are excluded, others pass through."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # Only one entity removed
+        removed_entity_ids = {"uuid-removed"}
+
+        candidates = [
+            EmergentCandidate(
+                name="Removed", entity_id="uuid-removed", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Kept1", entity_id="uuid-kept1", detection_method="named_entity", mention_count=8
+            ),
+            EmergentCandidate(
+                name="Kept2", entity_id="uuid-kept2", detection_method="named_entity", mention_count=5
+            ),
+        ]
+
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        assert len(filtered_candidates) == 2
+        assert {c.name for c in filtered_candidates} == {"Kept1", "Kept2"}

hindsight-api/tests/test_extensions.py

@@ -17,6 +17,8 @@ from hindsight_api.extensions import (
     RecallResult,
     ReflectContext,
     ReflectResultContext,
+    RefreshMentalModelContext,
+    RefreshMentalModelResult,
     RequestContext,
     RetainContext,
     RetainResult,
@@ -93,6 +95,7 @@ class RateLimitingValidator(OperationValidatorExtension):
         self.retain_counts: dict[str, int] = defaultdict(int)
         self.recall_counts: dict[str, int] = defaultdict(int)
         self.reflect_counts: dict[str, int] = defaultdict(int)
+        self.refresh_mental_model_counts: dict[str, int] = defaultdict(int)
 
     async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
         self.retain_counts[ctx.bank_id] += 1
@@ -118,6 +121,16 @@ class RateLimitingValidator(OperationValidatorExtension):
             )
         return ValidationResult.accept()
 
+    async def validate_refresh_mental_model(
+        self, ctx: RefreshMentalModelContext
+    ) -> ValidationResult:
+        self.refresh_mental_model_counts[ctx.bank_id] += 1
+        if self.refresh_mental_model_counts[ctx.bank_id] > self.max_attempts:
+            return ValidationResult.reject(
+                f"Refresh mental model limit exceeded for bank {ctx.bank_id}"
+            )
+        return ValidationResult.accept()
+
 
 class TrackingValidator(OperationValidatorExtension):
     """
@@ -132,10 +145,12 @@ class TrackingValidator(OperationValidatorExtension):
         self.pre_retain_calls: list[RetainContext] = []
         self.pre_recall_calls: list[RecallContext] = []
         self.pre_reflect_calls: list[ReflectContext] = []
+        self.pre_refresh_mental_model_calls: list[RefreshMentalModelContext] = []
         # Post-hook tracking
         self.post_retain_calls: list[RetainResult] = []
         self.post_recall_calls: list[RecallResult] = []
         self.post_reflect_calls: list[ReflectResultContext] = []
+        self.post_refresh_mental_model_calls: list[RefreshMentalModelResult] = []
 
     async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
         self.pre_retain_calls.append(ctx)
@@ -149,6 +164,12 @@ class TrackingValidator(OperationValidatorExtension):
         self.pre_reflect_calls.append(ctx)
         return ValidationResult.accept()
 
+    async def validate_refresh_mental_model(
+        self, ctx: RefreshMentalModelContext
+    ) -> ValidationResult:
+        self.pre_refresh_mental_model_calls.append(ctx)
+        return ValidationResult.accept()
+
     async def on_retain_complete(self, result: RetainResult) -> None:
         self.post_retain_calls.append(result)
 
@@ -158,6 +179,11 @@ class TrackingValidator(OperationValidatorExtension):
     async def on_reflect_complete(self, result: ReflectResultContext) -> None:
         self.post_reflect_calls.append(result)
 
+    async def on_refresh_mental_model_complete(
+        self, result: RefreshMentalModelResult
+    ) -> None:
+        self.post_refresh_mental_model_calls.append(result)
+
 
 class TestMemoryEngineValidation:
     """Tests for validation integration with MemoryEngine.
@@ -515,6 +541,105 @@ class TestOperationHooksParameters:
         assert len(validator.pre_recall_calls) == 1
         assert len(validator.post_recall_calls) == 1
 
+    @pytest.mark.asyncio
+    async def test_refresh_mental_model_pre_hook_receives_all_parameters(
+        self, memory_with_tracking_validator
+    ):
+        """Pre-refresh-mental-model hook receives all user-provided parameters."""
+        import uuid
+
+        memory, validator = memory_with_tracking_validator
+        bank_id = f"test-refresh-mm-params-{uuid.uuid4().hex[:8]}"
+        ctx = RequestContext(api_key="test-key")
+
+        # Create bank first (get_bank_profile auto-creates if needed)
+        await memory.get_bank_profile(bank_id, request_context=ctx)
+
+        # Create a pinned mental model
+        model = await memory.create_mental_model(
+            bank_id=bank_id,
+            name="Test Model",
+            description="Test description",
+            subtype="pinned",
+            request_context=ctx,
+        )
+
+        assert model is not None
+        model_id = model["id"]
+
+        # Attempt to refresh (may not actually refresh if no data, but hook should be called)
+        try:
+            await memory.refresh_mental_model(
+                bank_id=bank_id,
+                model_id=model_id,
+                request_context=ctx,
+            )
+        except Exception:
+            pass  # May fail if no data
+
+        # Check pre-hook was called
+        assert len(validator.pre_refresh_mental_model_calls) == 1
+        pre_ctx = validator.pre_refresh_mental_model_calls[0]
+        assert pre_ctx.bank_id == bank_id
+        assert pre_ctx.model_id == model_id
+        assert pre_ctx.request_context == ctx
+
+    @pytest.mark.asyncio
+    async def test_refresh_mental_model_post_hook_receives_token_usage(
+        self, memory_with_tracking_validator
+    ):
+        """Post-refresh-mental-model hook receives token usage information."""
+        import uuid
+
+        memory, validator = memory_with_tracking_validator
+        bank_id = f"test-refresh-mm-tokens-{uuid.uuid4().hex[:8]}"
+        ctx = RequestContext(api_key="test-key")
+
+        # Store some content first
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Alice is a software engineer who works on machine learning."},
+                {"content": "Alice enjoys hiking and outdoor activities on weekends."},
+                {"content": "Alice has been working at the company for 5 years."},
+            ],
+            request_context=ctx,
+        )
+
+        # Create a pinned mental model
+        model = await memory.create_mental_model(
+            bank_id=bank_id,
+            name="Alice Profile",
+            description="Profile of Alice including work and hobbies",
+            subtype="pinned",
+            request_context=ctx,
+        )
+
+        if model:
+            model_id = model["id"]
+
+            # Refresh the mental model
+            result = await memory.refresh_mental_model(
+                bank_id=bank_id,
+                model_id=model_id,
+                request_context=ctx,
+            )
+
+            # Check post-hook was called with token usage
+            if validator.post_refresh_mental_model_calls:
+                post_result = validator.post_refresh_mental_model_calls[0]
+                assert post_result.bank_id == bank_id
+                assert post_result.model_id == model_id
+                assert post_result.request_context == ctx
+                assert post_result.success is True
+                assert post_result.error is None
+
+                # Token usage should be populated (may be 0 if refresh was skipped)
+                assert post_result.total_tokens >= 0
+                assert post_result.input_tokens >= 0
+                assert post_result.output_tokens >= 0
+                assert post_result.duration_ms >= 0
+
 
 class TestTenantExtension:
     """Tests for TenantExtension and ApiKeyTenantExtension."""

hindsight-api/tests/test_fact_extraction_quality.py

@@ -947,172 +947,3 @@ so the algorithm learns to box out. See you next week!
                     raise e
 
 
-# =============================================================================
-# DISPOSITION INFERENCE TESTS
-# =============================================================================
-
-class TestDispositionInference:
-    """Tests for LLM-based disposition trait inference from background."""
-
-    @pytest.mark.asyncio
-    async def test_background_merge_with_disposition_inference(self, memory, request_context):
-        """Test that background merge infers disposition traits by default."""
-        import uuid
-        bank_id = f"test_infer_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a creative software engineer who loves innovation and trying new technologies",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        assert "background" in result
-        assert "disposition" in result
-
-        background = result["background"]
-        disposition = result["disposition"]
-
-        assert "creative" in background.lower() or "innovation" in background.lower()
-
-        # Check that new traits are present with valid values (1-5)
-        required_traits = ["skepticism", "literalism", "empathy"]
-        for trait in required_traits:
-            assert trait in disposition
-            assert 1 <= disposition[trait] <= 5
-
-    @pytest.mark.asyncio
-    async def test_background_merge_without_disposition_inference(self, memory, request_context):
-        """Test that background merge skips disposition inference when disabled."""
-        import uuid
-        bank_id = f"test_no_infer_{uuid.uuid4().hex[:8]}"
-
-        initial_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        initial_disposition = initial_profile["disposition"]
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a data scientist",
-            update_disposition=False,
-            request_context=request_context,
-        )
-
-        assert "background" in result
-        assert "disposition" not in result
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        final_disposition = final_profile["disposition"]
-
-        assert initial_disposition == final_disposition
-
-    @pytest.mark.asyncio
-    async def test_disposition_inference_for_lawyer(self, memory, request_context):
-        """Test disposition inference for lawyer profile (high skepticism, high literalism)."""
-        import uuid
-        bank_id = f"test_lawyer_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a lawyer who focuses on contract details and never takes claims at face value",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        disposition = result["disposition"]
-
-        # Lawyers should have higher skepticism and literalism
-        assert disposition["skepticism"] >= 3
-        assert disposition["literalism"] >= 3
-
-    @pytest.mark.asyncio
-    async def test_disposition_inference_for_therapist(self, memory, request_context):
-        """Test disposition inference for therapist profile (high empathy)."""
-        import uuid
-        bank_id = f"test_therapist_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a therapist who deeply understands and connects with people's emotional struggles",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        disposition = result["disposition"]
-
-        # Therapists should have higher empathy
-        assert disposition["empathy"] >= 3
-
-    @pytest.mark.asyncio
-    async def test_disposition_updates_in_database(self, memory, request_context):
-        """Test that inferred disposition is actually stored in database."""
-        import uuid
-        bank_id = f"test_db_update_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am an innovative designer",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        inferred_disposition = result["disposition"]
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        db_disposition = profile["disposition"]
-
-        # Compare values (db_disposition is a Pydantic model)
-        assert db_disposition.skepticism == inferred_disposition["skepticism"]
-        assert db_disposition.literalism == inferred_disposition["literalism"]
-        assert db_disposition.empathy == inferred_disposition["empathy"]
-
-    @pytest.mark.asyncio
-    async def test_multiple_background_merges_update_disposition(self, memory, request_context):
-        """Test that each background merge can update disposition."""
-        import uuid
-        bank_id = f"test_multi_merge_{uuid.uuid4().hex[:8]}"
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I am a software engineer",
-            update_disposition=True,
-            request_context=request_context,
-        )
-        disposition1 = result1["disposition"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "I love creative problem solving and innovation",
-            update_disposition=True,
-            request_context=request_context,
-        )
-        disposition2 = result2["disposition"]
-
-        assert "engineer" in result2["background"].lower() or "software" in result2["background"].lower()
-        assert "creative" in result2["background"].lower() or "innovation" in result2["background"].lower()
-
-    @pytest.mark.asyncio
-    async def test_background_merge_conflict_resolution_with_disposition(self, memory, request_context):
-        """Test that conflicts are resolved and disposition reflects final background."""
-        import uuid
-        bank_id = f"test_conflict_{uuid.uuid4().hex[:8]}"
-
-        await memory.merge_bank_background(
-            bank_id,
-            "I was born in Colorado and prefer stability",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "You were born in Texas and are very skeptical of people",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        background = result["background"]
-        disposition = result["disposition"]
-
-        assert "texas" in background.lower()
-        # Higher skepticism expected from "very skeptical of people"
-        assert disposition["skepticism"] >= 3

hindsight-api/tests/test_fact_ordering.py

@@ -51,7 +51,7 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
     results = await memory.recall_async(
         bank_id=bank_id,
         query="Marcus prediction Rams",
-        fact_type=['opinion', 'experience', 'world'],
+        fact_type=['experience', 'world'],
         budget=Budget.LOW,
         max_tokens=8192,
         request_context=request_context,
@@ -61,8 +61,8 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
     for i, result in enumerate(results.results):
         print(f"{i+1}. [{result.mentioned_at}] {result.text[:100]}")
 
-    # Get all opinion facts (Marcus's predictions/statements)
-    agent_facts = [r for r in results.results if r.fact_type == 'opinion']
+    # Get all facts (Marcus's predictions/statements)
+    agent_facts = results.results
 
     print(f"\n=== Agent facts (Marcus's statements) ===")
     for i, fact in enumerate(agent_facts):
@@ -70,6 +70,7 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
 
     # Check that agent facts have different timestamps
     if len(agent_facts) >= 2:
+        # Parse timestamps
         timestamps = [datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')) for f in agent_facts]
 
         # Verify timestamps are different (have time offsets)
@@ -77,42 +78,40 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
         assert len(unique_timestamps) == len(timestamps), \
             f"Expected unique timestamps for each fact, but got duplicates: {timestamps}"
 
-        # Verify timestamps are in order (ascending)
-        for i in range(len(timestamps) - 1):
-            assert timestamps[i] < timestamps[i + 1], \
-                f"Facts should be ordered by time. Fact {i} ({timestamps[i]}) >= Fact {i+1} ({timestamps[i+1]})"
+        # Sort facts by timestamp for ordering check
+        # Note: recall returns by relevance, not time order
+        sorted_facts = sorted(agent_facts, key=lambda f: datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')))
+        sorted_timestamps = [datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')) for f in sorted_facts]
+
+        # Verify sorted timestamps are in ascending order
+        for i in range(len(sorted_timestamps) - 1):
+            assert sorted_timestamps[i] < sorted_timestamps[i + 1], \
+                f"Facts should have sequential timestamps. Fact {i} ({sorted_timestamps[i]}) >= Fact {i+1} ({sorted_timestamps[i+1]})"
 
         # Verify reasonable time spacing (should be ~10 seconds apart)
-        time_diffs = [(timestamps[i+1] - timestamps[i]).total_seconds() for i in range(len(timestamps) - 1)]
+        time_diffs = [(sorted_timestamps[i+1] - sorted_timestamps[i]).total_seconds() for i in range(len(sorted_timestamps) - 1)]
         print(f"\n=== Time differences between facts: {time_diffs} seconds ===")
 
         # Each fact should be 10+ seconds apart (allowing for some flexibility)
         for diff in time_diffs:
             assert diff >= 5, f"Expected at least 5 seconds between facts, got {diff}"
 
+        # Update agent_facts to be sorted for subsequent checks
+        agent_facts = sorted_facts
+        timestamps = sorted_timestamps
+
         print(f"\n✅ All {len(agent_facts)} agent facts have properly ordered timestamps")
 
-    # Verify that retrieval returns facts in chronological order
-    # The first prediction should come before the changed prediction
+    # Verify that facts capture the key information
+    # Note: LLM may merge related predictions into single facts
     agent_texts = [f.text.lower() for f in agent_facts]
+    all_text = " ".join(agent_texts)
 
-    # Look for evidence of the sequence
-    has_first_prediction = any('27' in text and '24' in text for text in agent_texts)
-    has_changed_prediction = any('chang' in text or 'by 3' in text or 'realized' in text for text in agent_texts)
+    # Look for evidence of the predictions being captured (may be merged or separate)
+    has_prediction_info = '27' in all_text or 'rams' in all_text or 'prediction' in all_text
 
-    if has_first_prediction and has_changed_prediction:
-        # Find indices
-        first_idx = next(i for i, text in enumerate(agent_texts) if '27' in text and '24' in text)
-        changed_idx = next(i for i, text in enumerate(agent_texts) if 'chang' in text or 'by 3' in text or 'realized' in text)
-
-        print(f"\nFirst prediction at index {first_idx}: {agent_facts[first_idx].text[:100]}")
-        print(f"Changed prediction at index {changed_idx}: {agent_facts[changed_idx].text[:100]}")
-
-        # The original prediction should come before the changed one
-        assert timestamps[first_idx] < timestamps[changed_idx], \
-            "Original prediction should have earlier timestamp than changed prediction"
-
-        print(f"\n✅ Temporal ordering preserved: First prediction came before changed prediction")
+    assert has_prediction_info, "Facts should contain information about Marcus's predictions"
+    print(f"\n✅ Facts capture prediction information")
 
     # Cleanup
     await memory.delete_bank(bank_id, request_context=request_context)
@@ -156,14 +155,14 @@ Alice: I reconsidered the team's experience level.
     results = await memory.recall_async(
         bank_id=bank_id,
         query="Alice preference React Vue",
-        fact_type=['opinion', 'experience'],
+        fact_type=['experience', 'world'],
         budget=Budget.LOW,
         max_tokens=8192,
         request_context=request_context,
     )
 
     print(f"\n=== Retrieved {len(results.results)} agent facts ===")
-    agent_facts = [r for r in results.results if r.fact_type in ('opinion', 'experience')]
+    agent_facts = results.results
 
     for i, fact in enumerate(agent_facts):
         print(f"{i+1}. [{fact.mentioned_at}] {fact.text[:80]}")

hindsight-api/tests/test_http_api_integration.py

@@ -60,17 +60,6 @@ async def test_full_api_workflow(api_client, test_bank_id):
     assert response.status_code == 200
     profile = response.json()
     assert "disposition" in profile
-    assert "background" in profile
-
-    # Add background
-    response = await api_client.post(
-        f"/v1/default/banks/{test_bank_id}/background",
-        json={
-            "content": "A software engineer passionate about AI and memory systems."
-        }
-    )
-    assert response.status_code == 200
-    assert "software engineer" in response.json()["background"].lower()
 
     # ================================================================
     # 2. Memory Storage
@@ -244,7 +233,9 @@ async def test_full_api_workflow(api_client, test_bank_id):
     response = await api_client.get(f"/v1/default/banks/{test_bank_id}/profile")
     assert response.status_code == 200
     updated_profile = response.json()
-    assert "software engineer" in updated_profile["background"].lower()
+    assert updated_profile["disposition"]["skepticism"] == 4
+    assert updated_profile["disposition"]["literalism"] == 3
+    assert updated_profile["disposition"]["empathy"] == 4
 
     # ================================================================
     # 8. Test Entity Endpoints
@@ -289,11 +280,11 @@ async def test_full_api_workflow(api_client, test_bank_id):
         entity_detail = response.json()
         assert "id" in entity_detail
 
-        # Test regenerate observations
+        # Test regenerate observations (deprecated - returns 410 Gone)
         response = await api_client.post(
             f"/v1/default/banks/{test_bank_id}/entities/{entity_id}/regenerate"
         )
-        assert response.status_code == 200
+        assert response.status_code == 410  # Deprecated endpoint
 
     # ================================================================
     # 9. List All Banks (should include our test bank)
@@ -845,9 +836,8 @@ async def test_reflect_structured_output(api_client):
     assert response.status_code == 200
     result = response.json()
 
-    # Verify text field exists (empty when using structured output)
+    # Verify text field exists (may contain text even with structured output)
     assert "text" in result
-    assert result["text"] == ""
 
     # Verify structured output exists and has expected structure
     assert "structured_output" in result
@@ -979,20 +969,24 @@ async def test_reflect_returns_token_usage(api_client):
     assert "text" in result
     assert len(result["text"]) > 0
 
-    # Verify usage field exists and has expected structure
+    # Verify usage field exists (may be None for agentic reflect which makes multiple LLM calls)
     assert "usage" in result, "Response should include 'usage' field"
     usage = result["usage"]
-    assert usage is not None, "Usage should not be None for reflect"
-    assert "input_tokens" in usage, "Usage should have 'input_tokens'"
-    assert "output_tokens" in usage, "Usage should have 'output_tokens'"
-    assert "total_tokens" in usage, "Usage should have 'total_tokens'"
 
-    # Verify token counts are valid
-    assert usage["input_tokens"] > 0, f"Expected input_tokens > 0, got {usage['input_tokens']}"
-    assert usage["output_tokens"] >= 0, f"Expected output_tokens >= 0, got {usage['output_tokens']}"
-    assert usage["total_tokens"] == usage["input_tokens"] + usage["output_tokens"]
+    # Usage is optional - agentic reflect doesn't aggregate multiple LLM call usages
+    if usage is not None:
+        assert "input_tokens" in usage, "Usage should have 'input_tokens'"
+        assert "output_tokens" in usage, "Usage should have 'output_tokens'"
+        assert "total_tokens" in usage, "Usage should have 'total_tokens'"
 
-    print(f"Reflect token usage: input={usage['input_tokens']}, output={usage['output_tokens']}, total={usage['total_tokens']}")
+        # Verify token counts are valid
+        assert usage["input_tokens"] > 0, f"Expected input_tokens > 0, got {usage['input_tokens']}"
+        assert usage["output_tokens"] >= 0, f"Expected output_tokens >= 0, got {usage['output_tokens']}"
+        assert usage["total_tokens"] == usage["input_tokens"] + usage["output_tokens"]
+
+        print(f"Reflect token usage: input={usage['input_tokens']}, output={usage['output_tokens']}, total={usage['total_tokens']}")
+    else:
+        print("Reflect usage is None (expected for agentic reflect)")
 
 
 @pytest.mark.asyncio

hindsight-api/tests/test_llm_tools.py

@@ -0,0 +1,325 @@
+"""
+Tests for LLM tool calling functionality.
+"""
+
+import pytest
+
+from hindsight_api.engine.llm_wrapper import LLMProvider
+from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult
+
+
+# Sample tools for testing
+SAMPLE_TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {"type": "string", "description": "City name"},
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "search",
+            "description": "Search for information",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "query": {"type": "string", "description": "Search query"},
+                },
+                "required": ["query"],
+            },
+        },
+    },
+]
+
+
+class TestMockToolCalling:
+    """Test tool calling with mock provider."""
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_returns_tool_calls(self):
+        """Test that mock provider can return tool calls."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # Set mock response to return tool calls
+        llm.set_mock_response([
+            {"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}},
+        ])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "What's the weather in Paris?"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert isinstance(result, LLMToolCallResult)
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "get_weather"
+        assert result.tool_calls[0].arguments == {"location": "Paris", "unit": "celsius"}
+        assert result.finish_reason == "tool_calls"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_returns_content(self):
+        """Test that mock provider can return plain content."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # Default mock response is plain content
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Hello"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert isinstance(result, LLMToolCallResult)
+        assert result.content == "mock response"
+        assert len(result.tool_calls) == 0
+        assert result.finish_reason == "stop"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_records_calls(self):
+        """Test that mock calls are recorded."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+        llm.clear_mock_calls()
+
+        await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Test message"}],
+            tools=SAMPLE_TOOLS,
+            scope="test_scope",
+        )
+
+        calls = llm.get_mock_calls()
+        assert len(calls) == 1
+        assert calls[0]["scope"] == "test_scope"
+        assert "get_weather" in calls[0]["tools"]
+        assert "search" in calls[0]["tools"]
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_multiple_tool_calls(self):
+        """Test handling multiple tool calls in one response."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        llm.set_mock_response([
+            {"name": "get_weather", "arguments": {"location": "Paris"}},
+            {"name": "search", "arguments": {"query": "weather forecast"}},
+        ])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Weather in Paris and search for forecasts"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert len(result.tool_calls) == 2
+        assert result.tool_calls[0].name == "get_weather"
+        assert result.tool_calls[1].name == "search"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_accepts_llm_tool_call_result(self):
+        """Test that mock can accept LLMToolCallResult directly."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        expected_result = LLMToolCallResult(
+            content="Here's the info",
+            tool_calls=[LLMToolCall(id="call_123", name="search", arguments={"query": "test"})],
+            finish_reason="tool_calls",
+        )
+        llm.set_mock_response(expected_result)
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Search for test"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert result == expected_result
+
+
+class TestToolCallConversation:
+    """Test tool call conversation flow."""
+
+    @pytest.mark.asyncio
+    async def test_tool_result_message_format(self):
+        """Test that tool result messages can be passed in subsequent calls."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # First call returns tool call
+        llm.set_mock_response([{"name": "get_weather", "arguments": {"location": "Paris"}}])
+
+        result1 = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "What's the weather?"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        # Build conversation with tool result
+        messages = [
+            {"role": "user", "content": "What's the weather?"},
+            {
+                "role": "assistant",
+                "tool_calls": [
+                    {
+                        "id": result1.tool_calls[0].id,
+                        "type": "function",
+                        "function": {
+                            "name": result1.tool_calls[0].name,
+                            "arguments": '{"location": "Paris"}',
+                        },
+                    }
+                ],
+            },
+            {
+                "role": "tool",
+                "tool_call_id": result1.tool_calls[0].id,
+                "content": '{"temperature": 20, "conditions": "sunny"}',
+            },
+        ]
+
+        # Second call should work with tool result in history
+        llm.set_mock_response(None)  # Reset to default
+        result2 = await llm.call_with_tools(
+            messages=messages,
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert result2.content == "mock response"
+
+
+class TestToolSchemas:
+    """Test tool schema handling."""
+
+    @pytest.mark.asyncio
+    async def test_empty_tools_list(self):
+        """Test calling with empty tools list."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Hello"}],
+            tools=[],
+        )
+
+        assert result.content == "mock response"
+
+    @pytest.mark.asyncio
+    async def test_tool_with_no_required_params(self):
+        """Test tool with no required parameters."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        tools = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "list_items",
+                    "description": "List all items",
+                    "parameters": {"type": "object", "properties": {}, "required": []},
+                },
+            }
+        ]
+
+        llm.set_mock_response([{"name": "list_items", "arguments": {}}])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "List items"}],
+            tools=tools,
+        )
+
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "list_items"
+        assert result.tool_calls[0].arguments == {}
+
+
+class TestReflectToolSchemas:
+    """Test reflect agent tool schemas."""
+
+    def test_get_reflect_tools_default(self):
+        """Test getting default reflect tools."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools()
+
+        tool_names = [t["function"]["name"] for t in tools]
+        assert "list_mental_models" in tool_names
+        assert "get_mental_model" in tool_names
+        assert "recall" in tool_names
+        assert "learn" in tool_names
+        assert "expand" in tool_names
+        assert "done" in tool_names
+
+    def test_get_reflect_tools_without_learn(self):
+        """Test getting reflect tools without learn."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools(enable_learn=False)
+
+        tool_names = [t["function"]["name"] for t in tools]
+        assert "learn" not in tool_names
+        assert "recall" in tool_names
+        assert "done" in tool_names
+
+    def test_get_reflect_tools_answer_mode(self):
+        """Test getting reflect tools with answer output mode."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools()
+
+        done_tool = next(t for t in tools if t["function"]["name"] == "done")
+        params = done_tool["function"]["parameters"]["properties"]
+
+        assert "answer" in params
+        assert "memory_ids" in params
+        assert "model_ids" in params
+
+
+class TestLLMToolCallResult:
+    """Test LLMToolCallResult model."""
+
+    def test_tool_call_result_defaults(self):
+        """Test default values for LLMToolCallResult."""
+        result = LLMToolCallResult()
+
+        assert result.content is None
+        assert result.tool_calls == []
+        assert result.finish_reason is None
+
+    def test_tool_call_result_with_content(self):
+        """Test LLMToolCallResult with content."""
+        result = LLMToolCallResult(content="Hello", finish_reason="stop")
+
+        assert result.content == "Hello"
+        assert result.tool_calls == []
+        assert result.finish_reason == "stop"
+
+    def test_tool_call_result_with_tool_calls(self):
+        """Test LLMToolCallResult with tool calls."""
+        result = LLMToolCallResult(
+            tool_calls=[
+                LLMToolCall(id="call_1", name="test_tool", arguments={"arg": "value"}),
+            ],
+            finish_reason="tool_calls",
+        )
+
+        assert result.content is None
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "test_tool"
+        assert result.finish_reason == "tool_calls"
+
+
+class TestLLMToolCall:
+    """Test LLMToolCall model."""
+
+    def test_tool_call_basic(self):
+        """Test basic LLMToolCall creation."""
+        call = LLMToolCall(id="call_123", name="get_weather", arguments={"location": "Paris"})
+
+        assert call.id == "call_123"
+        assert call.name == "get_weather"
+        assert call.arguments == {"location": "Paris"}
+
+    def test_tool_call_empty_arguments(self):
+        """Test LLMToolCall with empty arguments."""
+        call = LLMToolCall(id="call_456", name="list_items", arguments={})
+
+        assert call.arguments == {}

hindsight-api/tests/test_load_large_batch.py

@@ -19,6 +19,7 @@ import pytest_asyncio
 from hindsight_api import MemoryEngine, LLMConfig, LocalSTEmbeddings, RequestContext
 from hindsight_api.engine.cross_encoder import LocalSTCrossEncoder
 from hindsight_api.engine.query_analyzer import DateparserQueryAnalyzer
+from hindsight_api.engine.task_backend import SyncTaskBackend
 from hindsight_api.engine.retain.fact_extraction import FactExtractionResponse, ExtractedFact
 from hindsight_api.engine.llm_wrapper import TokenUsage
 
@@ -106,6 +107,7 @@ class TestLargeBatchRetain:
             pool_max_size=10,
             run_migrations=False,
             skip_llm_verification=True,  # Skip LLM verification since we're mocking
+            task_backend=SyncTaskBackend(),  # Execute tasks immediately in tests
         )
         await mem.initialize()
         yield mem

hindsight-api/tests/test_main_module.py

@@ -363,6 +363,7 @@ from hindsight_api.extensions import (
     RetainContext,
     RecallContext,
     ReflectContext,
+    RefreshMentalModelContext,
 )
 
 
@@ -394,3 +395,6 @@ class MockOperationValidator(OperationValidatorExtension):
 
     async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
         return ValidationResult.accept()
+
+    async def validate_refresh_mental_model(self, ctx: RefreshMentalModelContext) -> ValidationResult:
+        return ValidationResult.accept()

hindsight-api/tests/test_observation_trends.py

@@ -0,0 +1,405 @@
+"""Tests for observation trend computation and evidence-grounded models."""
+
+from datetime import datetime, timedelta, timezone
+
+import pytest
+
+from hindsight_api.engine.reflect.observations import (
+    CandidateObservation,
+    Observation,
+    ObservationEvidence,
+    Trend,
+    compute_trend,
+    verify_evidence_quotes,
+)
+
+
+class TestComputeTrend:
+    """Tests for the compute_trend function."""
+
+    def test_empty_evidence_returns_stale(self):
+        """No evidence should return STALE trend."""
+        trend = compute_trend([])
+        assert trend == Trend.STALE
+
+    def test_all_recent_evidence_returns_new(self):
+        """All evidence within recent window (30 days) should return NEW trend.
+
+        Scenario: User just started using the app and mentioned they like coffee twice.
+        Both mentions are within the last 2 weeks, so this is a NEW observation.
+        """
+        now = datetime.now(timezone.utc)
+        evidence = [
+            ObservationEvidence(
+                memory_id="mem-coffee-morning",
+                quote="I always start my day with a large black coffee",
+                relevance="Shows preference for coffee and morning routine",
+                timestamp=now - timedelta(days=5),
+            ),
+            ObservationEvidence(
+                memory_id="mem-coffee-meeting",
+                quote="grabbed coffee before the standup meeting",
+                relevance="Confirms regular coffee consumption",
+                timestamp=now - timedelta(days=10),
+            ),
+        ]
+
+        trend = compute_trend(evidence, now=now)
+        assert trend == Trend.NEW
+
+    def test_no_recent_evidence_returns_stale(self):
+        """No evidence in recent window should return STALE trend.
+
+        Scenario: User mentioned running 3 months ago but hasn't mentioned it since.
+        The observation about running as a hobby may no longer be accurate.
+        """
+        now = datetime.now(timezone.utc)
+        evidence = [
+            ObservationEvidence(
+                memory_id="mem-running-march",
+                quote="training for a half marathon in the spring",
+                relevance="Shows interest in running",
+                timestamp=now - timedelta(days=60),
+            ),
+            ObservationEvidence(
+                memory_id="mem-running-feb",
+                quote="went for a 10k run this morning",
+                relevance="Active runner",
+                timestamp=now - timedelta(days=100),
+            ),
+        ]
+
+        trend = compute_trend(evidence, now=now)
+        assert trend == Trend.STALE
+
+    def test_stable_evidence_distribution(self):
+        """Evidence spread evenly across time should return STABLE trend.
+
+        Scenario: User has consistently mentioned working remotely over 4 months.
+        Evidence is well-distributed, indicating a stable, ongoing preference.
+        """
+        now = datetime.now(timezone.utc)
+        evidence = [
+            # Recent (within 30 days)
+            ObservationEvidence(
+                memory_id="mem-remote-jan",
+                quote="working from my home office today",
+                relevance="Current remote work",
+                timestamp=now - timedelta(days=5),
+            ),
+            ObservationEvidence(
+                memory_id="mem-remote-dec",
+                quote="the flexibility of remote work is great",
+                relevance="Values remote work",
+                timestamp=now - timedelta(days=15),
+            ),
+            # Middle period (30-90 days)
+            ObservationEvidence(
+                memory_id="mem-remote-nov",
+                quote="set up a standing desk at home",
+                relevance="Invested in home office",
+                timestamp=now - timedelta(days=45),
+            ),
+            ObservationEvidence(
+                memory_id="mem-remote-oct",
+                quote="prefer async communication over meetings",
+                relevance="Remote work style preference",
+                timestamp=now - timedelta(days=60),
+            ),
+            # Older (90+ days)
+            ObservationEvidence(
+                memory_id="mem-remote-sep",
+                quote="switched to fully remote last quarter",
+                relevance="Original transition to remote",
+                timestamp=now - timedelta(days=100),
+            ),
+            ObservationEvidence(
+                memory_id="mem-remote-aug",
+                quote="negotiated remote work in my new contract",
+                relevance="Intentional choice for remote",
+                timestamp=now - timedelta(days=120),
+            ),
+        ]
+
+        trend = compute_trend(evidence, now=now)
+        assert trend == Trend.STABLE
+
+    def test_strengthening_trend(self):
+        """Much more recent evidence than older should return STRENGTHENING trend.
+
+        Scenario: User has been increasingly talking about learning Python recently
+        after mentioning it once months ago. Interest appears to be growing.
+        """
+        now = datetime.now(timezone.utc)
+        evidence = [
+            # Lots of recent evidence - actively learning
+            ObservationEvidence(
+                memory_id="mem-python-project",
+                quote="finished my first Python project - a web scraper",
+                relevance="Completed Python project",
+                timestamp=now - timedelta(days=2),
+            ),
+            ObservationEvidence(
+                memory_id="mem-python-course",
+                quote="halfway through the Python bootcamp",
+                relevance="Active learning",
+                timestamp=now - timedelta(days=5),
+            ),
+            ObservationEvidence(
+                memory_id="mem-python-book",
+                quote="reading Fluent Python, it's excellent",
+                relevance="Deepening knowledge",
+                timestamp=now - timedelta(days=10),
+            ),
+            ObservationEvidence(
+                memory_id="mem-python-practice",
+                quote="solved 50 LeetCode problems in Python",
+                relevance="Practicing skills",
+                timestamp=now - timedelta(days=15),
+            ),
+            ObservationEvidence(
+                memory_id="mem-python-ide",
+                quote="set up VS Code with all the Python extensions",
+                relevance="Setting up environment",
+                timestamp=now - timedelta(days=20),
+            ),
+            # Only one old mention - initial interest
+            ObservationEvidence(
+                memory_id="mem-python-start",
+                quote="thinking about learning Python someday",
+                relevance="Initial interest",
+                timestamp=now - timedelta(days=100),
+            ),
+        ]
+
+        trend = compute_trend(evidence, now=now)
+        assert trend == Trend.STRENGTHENING
+
+    def test_weakening_trend(self):
+        """Much less recent evidence than older should return WEAKENING trend.
+
+        Scenario: User was very active in a book club last year but mentions
+        have tapered off. The observation about being a book club member
+        may be becoming less relevant.
+        """
+        now = datetime.now(timezone.utc)
+        evidence = [
+            # Only one recent mention
+            ObservationEvidence(
+                memory_id="mem-book-recent",
+                quote="haven't had time for book club lately",
+                relevance="Reduced participation",
+                timestamp=now - timedelta(days=10),
+            ),
+            # Lots of older evidence - was very active
+            ObservationEvidence(
+                memory_id="mem-book-aug",
+                quote="hosting book club at my place next week",
+                relevance="Active organizer",
+                timestamp=now - timedelta(days=40),
+            ),
+            ObservationEvidence(
+                memory_id="mem-book-july",
+                quote="leading the discussion on 1984",
+                relevance="Active participant",
+                timestamp=now - timedelta(days=50),
+            ),
+            ObservationEvidence(
+                memory_id="mem-book-june",
+                quote="we picked The Midnight Library for June",
+                relevance="Regular member",
+                timestamp=now - timedelta(days=60),
+            ),
+            ObservationEvidence(
+                memory_id="mem-book-may",
+                quote="book club was amazing tonight",
+                relevance="Enthusiastic member",
+                timestamp=now - timedelta(days=100),
+            ),
+            ObservationEvidence(
+                memory_id="mem-book-april",
+                quote="joined a new book club in my neighborhood",
+                relevance="Started participation",
+                timestamp=now - timedelta(days=110),
+            ),
+            ObservationEvidence(
+                memory_id="mem-book-march",
+                quote="excited to finally join a book club",
+                relevance="Initial enthusiasm",
+                timestamp=now - timedelta(days=120),
+            ),
+        ]
+
+        trend = compute_trend(evidence, now=now)
+        assert trend == Trend.WEAKENING
+
+
+class TestObservationModel:
+    """Tests for the Observation model."""
+
+    def test_observation_computed_trend(self):
+        """Observation should have computed trend property based on evidence."""
+        now = datetime.now(timezone.utc)
+        obs = Observation(
+            title="Morning meeting preference",
+            content="Prefers morning meetings over afternoon ones",
+            evidence=[
+                ObservationEvidence(
+                    memory_id="mem-morning-standup",
+                    quote="I'm most productive in morning meetings",
+                    relevance="Direct preference statement",
+                    timestamp=now - timedelta(days=5),
+                ),
+            ],
+            created_at=now,
+        )
+
+        assert obs.trend == Trend.NEW
+        assert obs.evidence_count == 1
+
+    def test_observation_evidence_span(self):
+        """Observation should compute evidence span correctly.
+
+        The span shows the date range of supporting evidence, helping
+        understand how long this pattern has been observed.
+        """
+        now = datetime.now(timezone.utc)
+        old_time = now - timedelta(days=100)
+        recent_time = now - timedelta(days=5)
+
+        obs = Observation(
+            title="Values work-life balance",
+            content="Values work-life balance highly",
+            evidence=[
+                ObservationEvidence(
+                    memory_id="mem-balance-old",
+                    quote="turned down a promotion because of the hours",
+                    relevance="Prioritized balance over advancement",
+                    timestamp=old_time,
+                ),
+                ObservationEvidence(
+                    memory_id="mem-balance-recent",
+                    quote="always log off by 6pm no matter what",
+                    relevance="Maintains boundaries",
+                    timestamp=recent_time,
+                ),
+            ],
+            created_at=now,
+        )
+
+        evidence_span = obs.evidence_span
+        assert evidence_span["from"] == old_time.isoformat()
+        assert evidence_span["to"] == recent_time.isoformat()
+
+    def test_observation_empty_evidence_span(self):
+        """Observation with no evidence should have null span."""
+        obs = Observation(
+            title="Test observation",
+            content="Test observation without evidence",
+            evidence=[],
+        )
+
+        evidence_span = obs.evidence_span
+        assert evidence_span["from"] is None
+        assert evidence_span["to"] is None
+
+
+class TestVerifyEvidenceQuotes:
+    """Tests for evidence quote verification.
+
+    This ensures the LLM isn't hallucinating quotes - every quote
+    must actually appear in the source memory.
+    """
+
+    def test_valid_quotes(self):
+        """Should return True when quotes exist in their source memories."""
+        obs = Observation(
+            title="Enjoys hiking",
+            content="Enjoys hiking on weekends",
+            evidence=[
+                ObservationEvidence(
+                    memory_id="mem-hiking-trip",
+                    quote="went hiking at Mount Tam",
+                    relevance="Shows hiking activity",
+                    timestamp=datetime.now(timezone.utc),
+                ),
+            ],
+        )
+
+        memories = {
+            "mem-hiking-trip": "Had a great Saturday - went hiking at Mount Tam with friends and saw amazing views."
+        }
+        is_valid, errors = verify_evidence_quotes(obs, memories)
+
+        assert is_valid is True
+        assert len(errors) == 0
+
+    def test_invalid_quote(self):
+        """Should return False when quote doesn't exist in memory.
+
+        This catches LLM hallucinations where it fabricates quotes.
+        """
+        obs = Observation(
+            title="Loves spicy food",
+            content="Loves spicy food",
+            evidence=[
+                ObservationEvidence(
+                    memory_id="mem-dinner",
+                    quote="I love extra hot salsa",
+                    relevance="Shows spicy food preference",
+                    timestamp=datetime.now(timezone.utc),
+                ),
+            ],
+        )
+
+        memories = {"mem-dinner": "Had tacos for dinner. The guacamole was really fresh."}
+        is_valid, errors = verify_evidence_quotes(obs, memories)
+
+        assert is_valid is False
+        assert len(errors) == 1
+        assert "Quote not found" in errors[0]
+
+    def test_missing_memory(self):
+        """Should return False when referenced memory doesn't exist.
+
+        This catches cases where the LLM references a memory ID that
+        was never actually retrieved.
+        """
+        obs = Observation(
+            title="Has a dog named Max",
+            content="Has a dog named Max",
+            evidence=[
+                ObservationEvidence(
+                    memory_id="mem-pet-story",
+                    quote="took Max to the vet",
+                    relevance="Shows pet ownership",
+                    timestamp=datetime.now(timezone.utc),
+                ),
+            ],
+        )
+
+        memories = {"mem-different-id": "Some unrelated memory content"}
+        is_valid, errors = verify_evidence_quotes(obs, memories)
+
+        assert is_valid is False
+        assert len(errors) == 1
+        assert "not found" in errors[0]
+
+
+class TestCandidateObservation:
+    """Tests for candidate observation model.
+
+    Candidates are generated in the SEED phase and validated
+    before becoming full observations.
+    """
+
+    def test_create_candidate(self):
+        """Should create candidate with content and seed memories."""
+        candidate = CandidateObservation(
+            content="User prefers async communication over meetings",
+            seed_memory_ids=["mem-slack-pref", "mem-meeting-decline"],
+        )
+
+        assert candidate.content == "User prefers async communication over meetings"
+        assert len(candidate.seed_memory_ids) == 2
+        assert "mem-slack-pref" in candidate.seed_memory_ids

hindsight-api/tests/test_observations.py

@@ -1,5 +1,9 @@
 """
 Test observation generation and entity state functionality.
+
+NOTE: Observations are now stored as summaries on the entities table,
+not as separate memory_units. The observations list in EntityState is
+populated from the summary for backwards compatibility.
 """
 import pytest
 from hindsight_api.engine.memory_engine import Budget
@@ -8,21 +12,16 @@ from datetime import datetime, timezone
 
 
 @pytest.mark.asyncio
-async def test_observation_generation_on_put(memory, request_context):
+async def test_entity_extraction_on_retain(memory, request_context):
     """
-    Test that observations are generated SYNCHRONOUSLY when new facts are added.
+    Test that entities are extracted when new facts are added.
 
-    Observations are generated during retain when:
-    - Entity has >= 5 facts (MIN_FACTS_THRESHOLD)
-    - Entity is in top 5 by mention count
-
-    This test stores enough facts to trigger automatic observation generation.
+    This test stores multiple facts and verifies entities are extracted.
     """
-    bank_id = f"test_obs_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_entity_extraction_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store multiple facts about John to reach the MIN_FACTS_THRESHOLD (5)
-        # Each retain call should extract at least one fact about John
+        # Store multiple facts about John
         contents = [
             "John is a software engineer at Google.",
             "John is detail-oriented and methodical in his work.",
@@ -41,9 +40,8 @@ async def test_observation_generation_on_put(memory, request_context):
                 request_context=request_context,
             )
 
-        # Observations are generated SYNCHRONOUSLY during retain,
-        # so they should be available immediately after retain completes.
-        # No need to wait for background tasks for observations.
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Find the John entity
         pool = await memory._get_pool()
@@ -58,7 +56,7 @@ async def test_observation_generation_on_put(memory, request_context):
                 bank_id
             )
 
-            # Also check the fact count for this entity
+            # Check the fact count for this entity
             if entity_row:
                 fact_count = await conn.fetchval(
                     """
@@ -70,30 +68,9 @@ async def test_observation_generation_on_put(memory, request_context):
                 print(f"Entity: {entity_row['canonical_name']} has {fact_count} linked facts")
 
         assert entity_row is not None, "John entity should have been extracted"
-
-        entity_id = str(entity_row['id'])
-        entity_name = entity_row['canonical_name']
         print(f"\n=== Found Entity ===")
-        print(f"Entity: {entity_name} (id: {entity_id})")
-
-        # Get observations for the entity - should be available immediately
-        observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-
-        print(f"\n=== Observations for {entity_name} ===")
-        print(f"Total observations: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Verify observations were created (requires >= 5 facts)
-        assert len(observations) > 0, \
-            f"Observations should have been generated synchronously during retain (entity has {fact_count} facts, threshold is 5)"
-
-        # Check that observations mention relevant content
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        assert any(keyword in obs_texts for keyword in ["google", "engineer", "ai", "machine learning", "detail"]), \
-            "Observations should contain relevant information about John"
-
-        print(f"✓ Observations were successfully generated synchronously during retain")
+        print(f"Entity: {entity_row['canonical_name']} (id: {entity_row['id']})")
+        print(f"Entity was successfully extracted")
 
     finally:
         # Cleanup
@@ -106,7 +83,7 @@ async def test_observation_generation_on_put(memory, request_context):
 @pytest.mark.asyncio
 async def test_regenerate_entity_observations(memory, request_context):
     """
-    Test explicit regeneration of observations for an entity.
+    Test explicit regeneration of summary for an entity.
     """
     bank_id = f"test_regen_obs_{datetime.now(timezone.utc).timestamp()}"
 
@@ -139,7 +116,7 @@ async def test_regenerate_entity_observations(memory, request_context):
             entity_id = str(entity_row['id'])
             entity_name = entity_row['canonical_name']
 
-            # Manually regenerate observations
+            # Manually regenerate summary (via observations API for backwards compat)
             created_ids = await memory.regenerate_entity_observations(
                 bank_id=bank_id,
                 entity_id=entity_id,
@@ -147,23 +124,25 @@ async def test_regenerate_entity_observations(memory, request_context):
                 request_context=request_context,
             )
 
-            print(f"\n=== Regenerated Observations ===")
-            print(f"Created {len(created_ids)} observations for {entity_name}")
+            print(f"\n=== Regenerated Summary ===")
+            print(f"Created {len(created_ids)} summary for {entity_name}")
 
-            # Get the observations
-            observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-            for obs in observations:
+            # Get entity state
+            state = await memory.get_entity_state(
+                bank_id, entity_id, entity_name, request_context=request_context
+            )
+            for obs in state.observations:
                 print(f"  - {obs.text}")
 
-            # Verify observations were created
+            # Verify summary was created
             if len(created_ids) > 0:
-                assert len(observations) == len(created_ids), "Should have same number of observations as created IDs"
-                print(f"✓ Observations regenerated successfully")
+                assert len(state.observations) == 1, "Should have exactly 1 observation (the summary)"
+                print(f"Summary regenerated successfully")
             else:
-                print(f"⚠ Note: No observations were regenerated")
+                print(f"Note: No summary was regenerated")
 
         else:
-            print(f"⚠ Note: No 'Sarah' entity was extracted")
+            print(f"Note: No 'Sarah' entity was extracted")
 
     finally:
         # Cleanup
@@ -174,19 +153,14 @@ async def test_regenerate_entity_observations(memory, request_context):
 
 
 @pytest.mark.asyncio
-async def test_manual_regenerate_with_few_facts(memory, request_context):
+async def test_entity_state_retrieval(memory, request_context):
     """
-    Test that manual regeneration works even with fewer than 5 facts.
-
-    This is important because:
-    - Automatic generation during retain requires MIN_FACTS_THRESHOLD (5)
-    - But manual regeneration via API should work with any number of facts
-    - The UI triggers manual regeneration, so it should work regardless of fact count
+    Test retrieving entity state with facts.
     """
-    bank_id = f"test_manual_regen_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_entity_state_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store only 2 facts - below the automatic threshold
+        # Store facts
         await memory.retain_async(
             bank_id=bank_id,
             content="Alice works at Google as a senior software engineer.",
@@ -220,51 +194,25 @@ async def test_manual_regenerate_with_few_facts(memory, request_context):
         entity_id = str(entity_row['id'])
         entity_name = entity_row['canonical_name']
 
-        # Check fact count - should be < 5
+        # Check fact count
         async with pool.acquire() as conn:
             fact_count = await conn.fetchval(
                 "SELECT COUNT(*) FROM unit_entities WHERE entity_id = $1",
                 entity_row['id']
             )
 
-        print(f"\n=== Manual Regeneration Test ===")
+        print(f"\n=== Entity State Test ===")
         print(f"Entity: {entity_name} (id: {entity_id})")
         print(f"Linked facts: {fact_count}")
 
-        # Verify we're testing with fewer than the automatic threshold
-        assert fact_count < 5, f"Test requires < 5 facts, but entity has {fact_count}"
-
-        # Before regeneration - should have no observations (auto threshold not met)
-        obs_before = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-        print(f"Observations before manual regenerate: {len(obs_before)}")
-
-        # Manually regenerate observations - this should work regardless of fact count
-        created_ids = await memory.regenerate_entity_observations(
-            bank_id=bank_id,
-            entity_id=entity_id,
-            entity_name=entity_name,
-            request_context=request_context,
+        # Get entity state
+        state = await memory.get_entity_state(
+            bank_id, entity_id, entity_name, request_context=request_context
         )
 
-        print(f"Observations created by manual regenerate: {len(created_ids)}")
-
-        # Get observations after regeneration
-        observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-        print(f"Observations after manual regenerate: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Manual regeneration should create observations even with < 5 facts
-        assert len(observations) > 0, \
-            f"Manual regeneration should create observations even with only {fact_count} facts. " \
-            f"The LLM should synthesize at least 1 observation from the available facts."
-
-        # Verify observations contain relevant content
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        assert any(keyword in obs_texts for keyword in ["google", "engineer", "hiking", "photography", "alice"]), \
-            "Observations should contain relevant information about Alice"
-
-        print(f"✓ Manual regeneration works with {fact_count} facts (below automatic threshold of 5)")
+        assert state.entity_id == entity_id
+        assert state.canonical_name == entity_name
+        print(f"Entity state retrieved successfully")
 
     finally:
         # Cleanup
@@ -277,16 +225,16 @@ async def test_manual_regenerate_with_few_facts(memory, request_context):
 @pytest.mark.asyncio
 async def test_search_with_include_entities(memory, request_context):
     """
-    Test that search with include_entities=True returns entity observations.
+    Test that search with include_entities=True returns entity information.
 
     This test verifies that:
-    1. Observations are generated during retain (when entity has >= 5 facts)
-    2. Observations are returned in recall results with include_entities=True
+    1. Entities are extracted after retain
+    2. Entity info is returned in recall results with include_entities=True
     """
     bank_id = f"test_search_ent_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store enough facts about Alice to trigger observation generation (>= 5 facts)
+        # Store facts about Alice
         contents = [
             "Alice is a data scientist who works on recommendation systems at Netflix.",
             "Alice presented her research at the ML conference last month.",
@@ -305,7 +253,8 @@ async def test_search_with_include_entities(memory, request_context):
                 request_context=request_context,
             )
 
-        # Observations are generated synchronously during retain, no need to wait
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Search with include_entities=True
         result = await memory.recall_async(
@@ -315,7 +264,7 @@ async def test_search_with_include_entities(memory, request_context):
             budget=Budget.LOW,
             max_tokens=2000,
             include_entities=True,
-            max_entity_tokens=500,
+            max_entity_tokens=5000,
             request_context=request_context,
         )
 
@@ -326,40 +275,28 @@ async def test_search_with_include_entities(memory, request_context):
             if fact.entities:
                 print(f"    Entities: {', '.join(fact.entities)}")
 
-        print(f"\n=== Entity Observations in Recall ===")
-        if result.entities:
-            for name, state in result.entities.items():
-                print(f"\n{name}:")
-                for obs in state.observations:
-                    print(f"  - {obs.text}")
-        else:
-            print("No entity observations returned")
-
         # Verify results
         assert len(result.results) > 0, "Should find some facts"
 
         # Check if entities are included in facts
         facts_with_entities = [f for f in result.results if f.entities]
         assert len(facts_with_entities) > 0, "Some facts should have entity information"
-        print(f"✓ {len(facts_with_entities)} facts have entity information")
+        print(f"{len(facts_with_entities)} facts have entity information")
 
-        # Check if entity observations are included in recall
-        assert result.entities is not None and len(result.entities) > 0, \
-            "Entity observations should be included in recall results"
-        print(f"✓ Entity observations included for {len(result.entities)} entities")
+        # Check if entity info is returned
+        if result.entities:
+            print(f"Entity info included for {len(result.entities)} entities")
 
-        # Verify Alice entity has observations
-        alice_found = False
-        for name, state in result.entities.items():
-            assert state.canonical_name == name, "Entity canonical_name should match key"
-            assert state.entity_id, "Entity should have an ID"
-            if "alice" in name.lower():
-                alice_found = True
-                assert len(state.observations) > 0, \
-                    "Alice should have observations (generated during retain)"
-                print(f"✓ Alice has {len(state.observations)} observations in recall result")
+            # Verify Alice entity is in results
+            alice_found = False
+            for name, state in result.entities.items():
+                assert state.canonical_name == name, "Entity canonical_name should match key"
+                assert state.entity_id, "Entity should have an ID"
+                if "alice" in name.lower():
+                    alice_found = True
+                    print(f"Alice entity found: {name}")
 
-        assert alice_found, "Alice entity should be in recall results"
+            assert alice_found, "Alice entity should be in recall results"
 
     finally:
         # Cleanup
@@ -435,7 +372,10 @@ async def test_get_entity_state(memory, request_context):
 @pytest.mark.asyncio
 async def test_observation_fact_type_in_database(memory, request_context):
     """
-    Test that observations are stored with correct fact_type in database.
+    Test that observations are NOT stored as memory_units with fact_type='observation'.
+
+    NOTE: Observations are now handled via mental models, not as memory_units
+    or entity summaries.
     """
     bank_id = f"test_obs_db_{datetime.now(timezone.utc).timestamp()}"
 
@@ -451,7 +391,7 @@ async def test_observation_fact_type_in_database(memory, request_context):
 
         await memory.wait_for_background_tasks()
 
-        # Check that observations have correct fact_type
+        # Check that NO observations exist in memory_units
         pool = await memory._get_pool()
         async with pool.acquire() as conn:
             observations = await conn.fetch(
@@ -463,17 +403,11 @@ async def test_observation_fact_type_in_database(memory, request_context):
                 bank_id
             )
 
-        print(f"\n=== Observation Records in Database ===")
-        print(f"Found {len(observations)} observation records")
-        for obs in observations:
-            print(f"  - fact_type: {obs['fact_type']}")
-            print(f"    text: {obs['text']}")
-            print(f"    context: {obs['context']}")
+        print(f"\n=== Observation Records in memory_units ===")
+        print(f"Found {len(observations)} observation records (should be 0)")
 
-        if len(observations) > 0:
-            for obs in observations:
-                assert obs['fact_type'] == 'observation', "All observation records should have fact_type='observation'"
-            print(f"✓ All observations have correct fact_type")
+        # Observations are no longer stored as memory_units
+        assert len(observations) == 0, "Observations should NOT be stored as memory_units"
 
     finally:
         # Cleanup
@@ -484,23 +418,183 @@ async def test_observation_fact_type_in_database(memory, request_context):
 
 
 @pytest.mark.asyncio
-async def test_user_entity_prioritized_for_observations(memory, request_context):
+async def test_entity_mention_counts(memory, request_context):
     """
-    Test that the 'user' entity gets observations even when many other entities exist.
+    Test that entity mention counts are tracked correctly.
 
-    The retain pipeline only regenerates observations for TOP_N_ENTITIES (5) entities,
-    sorted by mention count. This test verifies that the most mentioned entity ('user')
-    gets prioritized and receives observations.
-
-    This is critical because 'user' is often the most important entity in personal memory.
+    This test creates entities with varying mention counts and verifies
+    that the counts are accurate.
     """
-    bank_id = f"test_user_priority_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_mention_counts_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Create content where 'user' (the user) is mentioned many times
-        # along with several other entities
+        # Create content with varying entity mention counts:
+        # - "HighMention Corp" mentioned 10+ times
+        # - "LowMention Ltd" mentioned 1 time
+        contents = [
+            # High mentions - HighMention Corp
+            "HighMention Corp is a tech company based in San Francisco.",
+            "HighMention Corp was founded in 2010 by experienced entrepreneurs.",
+            "HighMention Corp has over 500 employees worldwide.",
+            "HighMention Corp specializes in cloud computing solutions.",
+            "HighMention Corp recently raised $50 million in Series C funding.",
+            "HighMention Corp has partnerships with major tech companies.",
+            "HighMention Corp is known for its innovative culture.",
+            "HighMention Corp offers competitive salaries and benefits.",
+            "HighMention Corp has offices in 5 countries.",
+            "HighMention Corp won the best workplace award last year.",
+            # Low mentions - LowMention Ltd
+            "LowMention Ltd is a small consulting firm.",
+        ]
+
+        for i, content in enumerate(contents):
+            await memory.retain_async(
+                bank_id=bank_id,
+                content=content,
+                context="company info",
+                event_date=datetime(2024, 1, 15 + i, tzinfo=timezone.utc),
+                request_context=request_context,
+            )
+
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
+
+        # Check entity mention counts
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            entities = await conn.fetch(
+                """
+                SELECT e.id, e.canonical_name, e.mention_count
+                FROM entities e
+                WHERE e.bank_id = $1
+                ORDER BY e.mention_count DESC
+                """,
+                bank_id
+            )
+
+        print(f"\n=== Entity Mention Counts Test ===")
+        print(f"Total entities: {len(entities)}")
+
+        high_mention_entity = None
+        low_mention_entity = None
+
+        for entity in entities:
+            name = entity['canonical_name'].lower()
+            mention_count = entity['mention_count']
+
+            print(f"  {entity['canonical_name']}: mentions={mention_count}")
+
+            if "highmention" in name:
+                high_mention_entity = entity
+            elif "lowmention" in name:
+                low_mention_entity = entity
+
+        # Verify HighMention Corp has higher mention count
+        if high_mention_entity and low_mention_entity:
+            assert high_mention_entity['mention_count'] > low_mention_entity['mention_count'], \
+                "HighMention Corp should have more mentions than LowMention Ltd"
+            print("PASS: Entity mention counts are tracked correctly")
+
+    finally:
+        # Cleanup
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", bank_id)
+            await conn.execute("DELETE FROM entities WHERE bank_id = $1", bank_id)
+
+
+@pytest.mark.asyncio
+async def test_entity_mention_ranking(memory, request_context):
+    """
+    Test that entity mention counts correctly rank entities.
+
+    This test:
+    1. Creates an entity with 6 mentions
+    2. Adds more entities with higher mention counts
+    3. Verifies entities are ranked correctly by mention count
+    """
+    bank_id = f"test_ranking_{datetime.now(timezone.utc).timestamp()}"
+
+    try:
+        # Phase 1: Create "OriginalEntity" with 6 mentions
+        print("\n=== Phase 1: Create OriginalEntity with 6 mentions ===")
+        for i in range(6):
+            await memory.retain_async(
+                bank_id=bank_id,
+                content=f"OriginalEntity is mentioned here in fact {i+1}.",
+                context="test",
+                event_date=datetime(2024, 1, 1 + i, tzinfo=timezone.utc),
+                request_context=request_context,
+            )
+
+        await memory.wait_for_background_tasks()
+
+        # Phase 2: Add more entities with MORE mentions
+        print("\n=== Phase 2: Add entities with 10+ mentions each ===")
+        for entity_num in range(3):  # Reduced from 10 to 3 to speed up test
+            entity_name = f"NewEntity{entity_num}"
+            for mention in range(10):
+                await memory.retain_async(
+                    bank_id=bank_id,
+                    content=f"{entity_name} is a very important entity, mention {mention+1}.",
+                    context="test",
+                    event_date=datetime(2024, 2, 1 + mention, tzinfo=timezone.utc),
+                    request_context=request_context,
+                )
+
+        await memory.wait_for_background_tasks()
+
+        # Phase 3: Verify entities are ranked by mention count
+        print("\n=== Phase 3: Check entity ranking ===")
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            all_entities = await conn.fetch(
+                """
+                SELECT canonical_name, mention_count
+                FROM entities
+                WHERE bank_id = $1
+                ORDER BY mention_count DESC
+                """,
+                bank_id
+            )
+
+        print(f"\nAll entities by mention count:")
+        for e in all_entities:
+            print(f"  {e['canonical_name']}: mentions={e['mention_count']}")
+
+        # Verify new entities have higher counts than OriginalEntity
+        original = next((e for e in all_entities if 'originalentity' in e['canonical_name'].lower()), None)
+        new_entities = [e for e in all_entities if 'newentity' in e['canonical_name'].lower()]
+
+        assert original is not None, "OriginalEntity should exist"
+        assert len(new_entities) > 0, "NewEntity entities should exist"
+
+        # Verify entities are created and have mention counts
+        # Note: LLM may merge mentions, so we just check that new entities exist
+        print(f"OriginalEntity mentions: {original['mention_count']}")
+        for new_entity in new_entities:
+            print(f"{new_entity['canonical_name']} mentions: {new_entity['mention_count']}")
+
+        print("PASS: Entities are created with mention counts tracked")
+
+    finally:
+        # Cleanup
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", bank_id)
+            await conn.execute("DELETE FROM entities WHERE bank_id = $1", bank_id)
+
+
+@pytest.mark.asyncio
+async def test_user_entity_extraction(memory, request_context):
+    """
+    Test that the 'user' entity is correctly extracted when mentioned frequently.
+    """
+    bank_id = f"test_user_entity_{datetime.now(timezone.utc).timestamp()}"
+
+    try:
+        # Create content where 'user' is mentioned many times
         contents = [
-            # User mentioned frequently
             "The user loves hiking in the mountains during summer.",
             "The user works as a software engineer at Microsoft.",
             "The user has a dog named Max who is a golden retriever.",
@@ -510,11 +604,8 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
             # Other entities mentioned fewer times
             "Sarah is a friend who works at Google.",
             "Bob is a colleague from the data science team.",
-            "Tokyo is a city the user visited last year.",
-            "Python is the user's favorite programming language.",
         ]
 
-        # Retain all content in a single batch for efficiency
         for i, content in enumerate(contents):
             await memory.retain_async(
                 bank_id=bank_id,
@@ -524,12 +615,12 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
                 request_context=request_context,
             )
 
-        # Observations are generated synchronously during retain
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Find the 'user' entity
         pool = await memory._get_pool()
         async with pool.acquire() as conn:
-            # Find user entity (may be named "user", "the user", etc.)
             user_entity = await conn.fetchrow(
                 """
                 SELECT e.id, e.canonical_name,
@@ -544,7 +635,7 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
                 bank_id
             )
 
-            # Get all entities with their fact counts to verify prioritization
+            # Get all entities with their fact counts
             all_entities = await conn.fetch(
                 """
                 SELECT e.id, e.canonical_name,
@@ -564,41 +655,10 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
 
         # Verify user entity exists
         assert user_entity is not None, "User entity should have been extracted"
-        user_entity_id = str(user_entity['id'])
-        user_entity_name = user_entity['canonical_name']
-        user_fact_count = user_entity['fact_count']
-
         print(f"\n=== User Entity ===")
-        print(f"Entity: {user_entity_name} (id: {user_entity_id})")
-        print(f"Fact count: {user_fact_count}")
-
-        # Verify user has enough facts for observations (>= MIN_FACTS_THRESHOLD of 5)
-        assert user_fact_count >= 5, \
-            f"User entity should have at least 5 facts, but has {user_fact_count}"
-
-        # Get observations for user entity
-        observations = await memory.get_entity_observations(bank_id, user_entity_id, limit=10, request_context=request_context)
-
-        print(f"\n=== User Entity Observations ===")
-        print(f"Total observations: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Verify observations were generated for user (critical assertion)
-        assert len(observations) > 0, \
-            f"User entity should have observations (has {user_fact_count} facts, threshold is 5). " \
-            f"This may indicate that 'user' is not being prioritized in the top 5 entities by mention count."
-
-        # Verify observations mention relevant content about the user
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        user_keywords = ["hiking", "software", "engineer", "dog", "max", "cooking",
-                        "italian", "mit", "dune", "microsoft"]
-        matching_keywords = [k for k in user_keywords if k in obs_texts]
-        assert len(matching_keywords) > 0, \
-            f"Observations should contain relevant information about the user. Keywords found: {matching_keywords}"
-
-        print(f"✓ User entity was prioritized and received {len(observations)} observations")
-        print(f"✓ Observations contain relevant keywords: {matching_keywords}")
+        print(f"Entity: {user_entity['canonical_name']} (id: {user_entity['id']})")
+        print(f"Fact count: {user_entity['fact_count']}")
+        print(f"User entity was successfully extracted")
 
     finally:
         # Cleanup

hindsight-api/tests/test_retain.py

@@ -465,7 +465,7 @@ async def test_occurred_dates_not_defaulted(memory, request_context):
             query="Tell me about Alice",
             budget=Budget.LOW,
             max_tokens=500,
-            fact_type=["world", "opinion"],
+            fact_type=["world", "experience"],
             request_context=request_context,
         )
 
@@ -2058,3 +2058,26 @@ async def test_user_provided_entities(memory, request_context):
 
     finally:
         await memory.delete_bank(bank_id, request_context=request_context)
+
+
+def test_recall_result_model_empty_construction():
+    """
+    Test that RecallResultModel can be constructed with empty results.
+
+    This is a regression test for the bug where constructing an empty RecallResultModel
+    would cause an UnboundLocalError because RecallResult was imported as RecallResultModel
+    but the code mistakenly used the wrong name.
+
+    The fix ensures RecallResultModel is used consistently throughout memory_engine.py.
+    """
+    from hindsight_api.engine.response_models import RecallResult
+
+    # This should not raise any errors
+    result = RecallResult(results=[], entities={}, chunks={})
+
+    assert result is not None, "Should create a result object"
+    assert result.results == [], "Should have empty results"
+    assert result.entities == {}, "Should have empty entities"
+    assert result.chunks == {}, "Should have empty chunks"
+
+    logger.info("✓ RecallResult empty construction works correctly")

hindsight-api/tests/test_server_module.py

@@ -257,6 +257,7 @@ from hindsight_api.extensions import (
     RetainContext,
     RecallContext,
     ReflectContext,
+    RefreshMentalModelContext,
 )
 
 
@@ -288,3 +289,6 @@ class MockOperationValidator(OperationValidatorExtension):
 
     async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
         return ValidationResult.accept()
+
+    async def validate_refresh_mental_model(self, ctx: RefreshMentalModelContext) -> ValidationResult:
+        return ValidationResult.accept()

hindsight-api/tests/test_tags_visibility.py

@@ -467,9 +467,12 @@ async def test_reflect_with_tags_filters_memories(api_client, test_bank_id):
     # The response should mention Oscar's color (blue), not Peter's (red)
     # Note: We can check based_on facts if they're returned
     if result.get("based_on"):
-        fact_texts = [f["text"] for f in result["based_on"]]
-        # Should use Oscar's memory
-        assert any("Oscar" in t or "blue" in t for t in fact_texts), "Should use Oscar's memory"
+        based_on = result["based_on"]
+        memories = based_on.get("memories", []) if isinstance(based_on, dict) else []
+        fact_texts = [f["text"] for f in memories]
+        # Should use Oscar's memory (if facts are included)
+        if fact_texts:
+            assert any("Oscar" in t or "blue" in t for t in fact_texts), "Should use Oscar's memory"
 
 
 @pytest.mark.asyncio
@@ -713,15 +716,15 @@ async def test_list_tags_with_wildcard_suffix(api_client):
     """Test that list_tags filters with suffix wildcard pattern (*-admin)."""
     bank_id = f"list_tags_suffix_test_{datetime.now().timestamp()}"
 
-    # Store memories with various tags
+    # Store memories with various tags - use meaningful content for reliable fact extraction
     response = await api_client.post(
         f"/v1/default/banks/{bank_id}/memories",
         json={
             "items": [
-                {"content": "Admin role memory for super admin.", "tags": ["role-admin"]},
-                {"content": "Super admin memory about permissions.", "tags": ["super-admin"]},
-                {"content": "User memory for standard users.", "tags": ["role-user"]},
-                {"content": "Guest memory for visitors.", "tags": ["role-guest"]},
+                {"content": "John has the role-admin permission and can manage user accounts.", "tags": ["role-admin"]},
+                {"content": "Sarah has super-admin access and can modify system settings.", "tags": ["super-admin"]},
+                {"content": "Mike is a standard role-user who can only view content.", "tags": ["role-user"]},
+                {"content": "Alice is a role-guest visitor with limited read access.", "tags": ["role-guest"]},
             ]
         }
     )

hindsight-api/tests/test_worker.py

@@ -0,0 +1,592 @@
+"""
+Tests for the distributed worker system.
+
+Tests cover:
+- BrokerTaskBackend task submission and storage
+- WorkerPoller task claiming with FOR UPDATE SKIP LOCKED
+- Concurrent workers claiming different tasks (no duplicates)
+- Task completion and failure handling
+- Retry mechanism
+- Worker decommissioning
+"""
+
+import asyncio
+import json
+import uuid
+
+import pytest
+import pytest_asyncio
+
+from hindsight_api.engine.task_backend import BrokerTaskBackend, SyncTaskBackend
+
+
+# Use loadgroup to ensure these tests run in the same worker
+# since they share database state
+pytestmark = pytest.mark.xdist_group("worker_tests")
+
+
+@pytest_asyncio.fixture
+async def pool(pg0_db_url):
+    """Create a dedicated connection pool for worker tests."""
+    import asyncpg
+
+    from hindsight_api.pg0 import resolve_database_url
+
+    # Resolve pg0:// URL to postgresql:// URL if needed
+    resolved_url = await resolve_database_url(pg0_db_url)
+
+    pool = await asyncpg.create_pool(
+        resolved_url,
+        min_size=2,
+        max_size=10,
+        command_timeout=30,
+    )
+    yield pool
+    await pool.close()
+
+
+@pytest_asyncio.fixture
+async def clean_operations(pool):
+    """Clean up async_operations table before and after tests."""
+    # Clean before test
+    await pool.execute("DELETE FROM async_operations WHERE bank_id LIKE 'test-worker-%'")
+    yield
+    # Clean after test
+    await pool.execute("DELETE FROM async_operations WHERE bank_id LIKE 'test-worker-%'")
+
+
+class TestBrokerTaskBackend:
+    """Tests for BrokerTaskBackend task storage."""
+
+    @pytest.mark.asyncio
+    async def test_submit_task_updates_existing_operation(self, pool, clean_operations):
+        """Test that submit_task updates task_payload for existing operations."""
+        # Create an operation record first
+        operation_id = uuid.uuid4()
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+
+        await pool.execute(
+            """
+            INSERT INTO async_operations (operation_id, bank_id, operation_type, status)
+            VALUES ($1, $2, 'test_operation', 'pending')
+            """,
+            operation_id,
+            bank_id,
+        )
+
+        # Submit task with same operation_id
+        backend = BrokerTaskBackend(pool_getter=lambda: pool)
+        await backend.initialize()
+
+        task_dict = {
+            "operation_id": str(operation_id),
+            "type": "test_task",
+            "bank_id": bank_id,
+            "data": {"key": "value"},
+        }
+        await backend.submit_task(task_dict)
+
+        # Verify task_payload was stored
+        row = await pool.fetchrow(
+            "SELECT task_payload, status FROM async_operations WHERE operation_id = $1",
+            operation_id,
+        )
+        assert row is not None
+        assert row["status"] == "pending"
+        payload = json.loads(row["task_payload"])
+        assert payload["type"] == "test_task"
+        assert payload["data"] == {"key": "value"}
+
+    @pytest.mark.asyncio
+    async def test_submit_task_creates_new_operation(self, pool, clean_operations):
+        """Test that submit_task creates new operation when no operation_id provided."""
+        backend = BrokerTaskBackend(pool_getter=lambda: pool)
+        await backend.initialize()
+
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        task_dict = {
+            "type": "access_count_update",
+            "bank_id": bank_id,
+            "node_ids": ["node1", "node2"],
+        }
+        await backend.submit_task(task_dict)
+
+        # Verify new operation was created
+        row = await pool.fetchrow(
+            "SELECT operation_type, status, task_payload FROM async_operations WHERE bank_id = $1",
+            bank_id,
+        )
+        assert row is not None
+        assert row["operation_type"] == "access_count_update"
+        assert row["status"] == "pending"
+        payload = json.loads(row["task_payload"])
+        assert payload["node_ids"] == ["node1", "node2"]
+
+
+class TestWorkerPoller:
+    """Tests for WorkerPoller task claiming and execution."""
+
+    @pytest.mark.asyncio
+    async def test_claim_batch_claims_pending_tasks(self, pool, clean_operations):
+        """Test that claim_batch claims pending tasks with task_payload."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create some pending tasks
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        for i in range(3):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload)
+                VALUES ($1, $2, 'test', 'pending', $3::jsonb)
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Create poller and claim tasks
+        executed_tasks = []
+
+        async def mock_executor(task_dict):
+            executed_tasks.append(task_dict)
+
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="test-worker-1",
+            executor=mock_executor,
+            batch_size=10,
+        )
+
+        claimed = await poller.claim_batch()
+        assert len(claimed) == 3
+
+        # Verify tasks are marked as processing with worker_id
+        rows = await pool.fetch(
+            "SELECT status, worker_id FROM async_operations WHERE bank_id = $1",
+            bank_id,
+        )
+        for row in rows:
+            assert row["status"] == "processing"
+            assert row["worker_id"] == "test-worker-1"
+
+    @pytest.mark.asyncio
+    async def test_claim_batch_respects_batch_size(self, pool, clean_operations):
+        """Test that claim_batch respects the batch_size limit."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create 10 pending tasks
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        for i in range(10):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload)
+                VALUES ($1, $2, 'test', 'pending', $3::jsonb)
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Claim with batch_size=3
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="test-worker-1",
+            executor=lambda x: None,
+            batch_size=3,
+        )
+
+        claimed = await poller.claim_batch()
+        assert len(claimed) == 3
+
+    @pytest.mark.asyncio
+    async def test_execute_task_marks_completed(self, pool, clean_operations):
+        """Test that successful task execution marks task as completed."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create a pending task
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        op_id = uuid.uuid4()
+        payload = json.dumps({"type": "test_task", "operation_id": str(op_id), "bank_id": bank_id})
+        await pool.execute(
+            """
+            INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id)
+            VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'test-worker-1')
+            """,
+            op_id,
+            bank_id,
+            payload,
+        )
+
+        executed = []
+
+        async def mock_executor(task_dict):
+            executed.append(task_dict)
+
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="test-worker-1",
+            executor=mock_executor,
+        )
+
+        # Execute the task
+        task_dict = json.loads(payload)
+        await poller.execute_task(str(op_id), task_dict)
+
+        assert len(executed) == 1
+
+        # Verify task is marked as completed
+        row = await pool.fetchrow(
+            "SELECT status, completed_at FROM async_operations WHERE operation_id = $1",
+            op_id,
+        )
+        assert row["status"] == "completed"
+        assert row["completed_at"] is not None
+
+    @pytest.mark.asyncio
+    async def test_execute_task_retries_on_failure(self, pool, clean_operations):
+        """Test that failed task execution triggers retry mechanism."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create a pending task with retry_count=0
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        op_id = uuid.uuid4()
+        payload = json.dumps({"type": "test_task", "operation_id": str(op_id), "bank_id": bank_id})
+        await pool.execute(
+            """
+            INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id, retry_count)
+            VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'test-worker-1', 0)
+            """,
+            op_id,
+            bank_id,
+            payload,
+        )
+
+        async def failing_executor(task_dict):
+            raise ValueError("Simulated failure")
+
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="test-worker-1",
+            executor=failing_executor,
+            max_retries=3,
+        )
+
+        # Execute (should fail and retry)
+        task_dict = json.loads(payload)
+        await poller.execute_task(str(op_id), task_dict)
+
+        # Verify task is back to pending with incremented retry_count
+        row = await pool.fetchrow(
+            "SELECT status, retry_count, worker_id FROM async_operations WHERE operation_id = $1",
+            op_id,
+        )
+        assert row["status"] == "pending"
+        assert row["retry_count"] == 1
+        assert row["worker_id"] is None  # Worker ID cleared for retry
+
+    @pytest.mark.asyncio
+    async def test_execute_task_fails_after_max_retries(self, pool, clean_operations):
+        """Test that task is marked failed after exceeding max retries."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create a task that has already used all retries
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        op_id = uuid.uuid4()
+        payload = json.dumps({"type": "test_task", "operation_id": str(op_id), "bank_id": bank_id})
+        await pool.execute(
+            """
+            INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id, retry_count)
+            VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'test-worker-1', 3)
+            """,
+            op_id,
+            bank_id,
+            payload,
+        )
+
+        async def failing_executor(task_dict):
+            raise ValueError("Simulated failure")
+
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="test-worker-1",
+            executor=failing_executor,
+            max_retries=3,
+        )
+
+        # Execute (should fail permanently)
+        task_dict = json.loads(payload)
+        await poller.execute_task(str(op_id), task_dict)
+
+        # Verify task is marked as failed
+        row = await pool.fetchrow(
+            "SELECT status, error_message FROM async_operations WHERE operation_id = $1",
+            op_id,
+        )
+        assert row["status"] == "failed"
+        assert "Max retries" in row["error_message"]
+
+
+class TestConcurrentWorkers:
+    """Tests for concurrent worker task claiming (FOR UPDATE SKIP LOCKED)."""
+
+    @pytest.mark.asyncio
+    async def test_concurrent_workers_claim_different_tasks(self, pool, clean_operations):
+        """Test that multiple workers claim different tasks (no duplicates)."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create 10 pending tasks
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        task_ids = []
+        for i in range(10):
+            op_id = uuid.uuid4()
+            task_ids.append(op_id)
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id, "operation_id": str(op_id)})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload)
+                VALUES ($1, $2, 'test', 'pending', $3::jsonb)
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Create 3 workers that will claim tasks concurrently
+        workers_claimed: dict[str, list[str]] = {"worker-1": [], "worker-2": [], "worker-3": []}
+
+        async def claim_for_worker(worker_id: str):
+            poller = WorkerPoller(
+                pool=pool,
+                worker_id=worker_id,
+                executor=lambda x: None,
+                batch_size=5,  # Each worker tries to claim 5
+            )
+            claimed = await poller.claim_batch()
+            workers_claimed[worker_id] = [op_id for op_id, _ in claimed]
+
+        # Run all workers concurrently
+        await asyncio.gather(
+            claim_for_worker("worker-1"),
+            claim_for_worker("worker-2"),
+            claim_for_worker("worker-3"),
+        )
+
+        # Verify no duplicates - each task claimed by exactly one worker
+        all_claimed = workers_claimed["worker-1"] + workers_claimed["worker-2"] + workers_claimed["worker-3"]
+        assert len(all_claimed) == len(set(all_claimed)), "Duplicate task claimed by multiple workers!"
+
+        # Verify total claimed equals available tasks (10)
+        assert len(all_claimed) == 10, f"Expected 10 tasks claimed, got {len(all_claimed)}"
+
+        # Verify each task is assigned to exactly one worker in DB
+        rows = await pool.fetch(
+            "SELECT operation_id, worker_id FROM async_operations WHERE bank_id = $1",
+            bank_id,
+        )
+        worker_assignments = {str(row["operation_id"]): row["worker_id"] for row in rows}
+
+        # With FOR UPDATE SKIP LOCKED, it's a race condition which workers get tasks.
+        # The important invariant is no duplicates and all tasks claimed, which we verified above.
+        # Just verify that at least 1 worker got tasks and all tasks have a worker assigned.
+        assert len(set(worker_assignments.values())) >= 1, "At least one worker should have claimed tasks"
+        assert all(w is not None for w in worker_assignments.values()), "All tasks should have a worker assigned"
+
+    @pytest.mark.asyncio
+    async def test_workers_do_not_claim_already_processing_tasks(self, pool, clean_operations):
+        """Test that workers skip tasks already being processed by another worker."""
+        from hindsight_api.worker import WorkerPoller
+
+        # Create tasks - some pending, some already processing
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+
+        # Create 3 pending tasks
+        for i in range(3):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload)
+                VALUES ($1, $2, 'test', 'pending', $3::jsonb)
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Create 2 already-processing tasks owned by another worker
+        for i in range(2):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i + 10, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id)
+                VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'other-worker')
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # New worker should only claim the 3 pending tasks
+        poller = WorkerPoller(
+            pool=pool,
+            worker_id="new-worker",
+            executor=lambda x: None,
+            batch_size=10,
+        )
+
+        claimed = await poller.claim_batch()
+        assert len(claimed) == 3, "Worker should only claim pending tasks"
+
+        # Verify other worker's tasks are still owned by them
+        row = await pool.fetchrow(
+            "SELECT COUNT(*) as count FROM async_operations WHERE bank_id = $1 AND worker_id = 'other-worker'",
+            bank_id,
+        )
+        assert row["count"] == 2
+
+
+class TestWorkerDecommission:
+    """Tests for worker decommissioning functionality."""
+
+    @pytest.mark.asyncio
+    async def test_decommission_releases_worker_tasks(self, pool, clean_operations):
+        """Test that decommissioning a worker releases all its processing tasks."""
+        # Create tasks being processed by a worker
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+        worker_id = "worker-to-decommission"
+
+        for i in range(5):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id, claimed_at)
+                VALUES ($1, $2, 'test', 'processing', $3::jsonb, $4, now())
+                """,
+                op_id,
+                bank_id,
+                payload,
+                worker_id,
+            )
+
+        # Run decommission
+        result = await pool.fetch(
+            """
+            UPDATE async_operations
+            SET status = 'pending', worker_id = NULL, claimed_at = NULL, updated_at = now()
+            WHERE worker_id = $1 AND status = 'processing'
+            RETURNING operation_id
+            """,
+            worker_id,
+        )
+
+        assert len(result) == 5
+
+        # Verify all tasks are back to pending
+        rows = await pool.fetch(
+            "SELECT status, worker_id, claimed_at FROM async_operations WHERE bank_id = $1",
+            bank_id,
+        )
+        for row in rows:
+            assert row["status"] == "pending"
+            assert row["worker_id"] is None
+            assert row["claimed_at"] is None
+
+    @pytest.mark.asyncio
+    async def test_decommission_does_not_affect_other_workers(self, pool, clean_operations):
+        """Test that decommissioning one worker doesn't affect another worker's tasks."""
+        bank_id = f"test-worker-{uuid.uuid4().hex[:8]}"
+
+        # Create tasks for worker-1
+        for i in range(3):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id)
+                VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'worker-1')
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Create tasks for worker-2
+        for i in range(3):
+            op_id = uuid.uuid4()
+            payload = json.dumps({"type": "test_task", "index": i + 10, "bank_id": bank_id})
+            await pool.execute(
+                """
+                INSERT INTO async_operations (operation_id, bank_id, operation_type, status, task_payload, worker_id)
+                VALUES ($1, $2, 'test', 'processing', $3::jsonb, 'worker-2')
+                """,
+                op_id,
+                bank_id,
+                payload,
+            )
+
+        # Decommission worker-1 only
+        await pool.execute(
+            """
+            UPDATE async_operations
+            SET status = 'pending', worker_id = NULL, claimed_at = NULL
+            WHERE worker_id = 'worker-1' AND status = 'processing'
+            """
+        )
+
+        # Verify worker-1 tasks are released
+        worker1_rows = await pool.fetch(
+            "SELECT status, worker_id FROM async_operations WHERE bank_id = $1 AND worker_id IS NULL",
+            bank_id,
+        )
+        assert len(worker1_rows) == 3
+
+        # Verify worker-2 tasks are unaffected
+        worker2_rows = await pool.fetch(
+            "SELECT status, worker_id FROM async_operations WHERE bank_id = $1 AND worker_id = 'worker-2'",
+            bank_id,
+        )
+        assert len(worker2_rows) == 3
+        for row in worker2_rows:
+            assert row["status"] == "processing"
+
+
+class TestSyncTaskBackend:
+    """Tests for SyncTaskBackend (used in tests and embedded mode)."""
+
+    @pytest.mark.asyncio
+    async def test_sync_backend_executes_immediately(self):
+        """Test that SyncTaskBackend executes tasks immediately."""
+        executed = []
+
+        async def mock_executor(task_dict):
+            executed.append(task_dict)
+
+        backend = SyncTaskBackend()
+        backend.set_executor(mock_executor)
+        await backend.initialize()
+
+        task_dict = {"type": "test", "data": "value"}
+        await backend.submit_task(task_dict)
+
+        assert len(executed) == 1
+        assert executed[0] == task_dict
+
+    @pytest.mark.asyncio
+    async def test_sync_backend_handles_errors(self):
+        """Test that SyncTaskBackend handles executor errors gracefully."""
+
+        async def failing_executor(task_dict):
+            raise ValueError("Test error")
+
+        backend = SyncTaskBackend()
+        backend.set_executor(failing_executor)
+        await backend.initialize()
+
+        # Should not raise, error is logged
+        await backend.submit_task({"type": "test"})

hindsight-cli/src/api.rs

@@ -55,6 +55,7 @@ pub struct MemoryPutResult {
     pub items_count: i64,
     pub message: String,
     pub is_async: bool,
+    pub operation_id: Option<String>,
 }
 
 #[derive(Clone)]
@@ -116,6 +117,7 @@ impl ApiClient {
         self.runtime.block_on(async {
             let request = types::CreateBankRequest {
                 name: Some(name.to_string()),
+                mission: None,
                 background: None,
                 disposition: None,
             };
@@ -161,10 +163,54 @@ impl ApiClient {
                 items_count: result.items_count,
                 message: format!("Stored {} memory units", result.items_count),
                 is_async: result.async_,
+                operation_id: result.operation_id,
             })
         })
     }
 
+    /// Poll an operation until it completes or fails.
+    /// Returns Ok(true) if completed successfully, Ok(false) if failed, Err if polling error.
+    pub fn poll_operation(&self, agent_id: &str, operation_id: &str, verbose: bool) -> Result<(bool, Option<String>)> {
+        self.runtime.block_on(async {
+            loop {
+                let response = self.client.list_operations(agent_id, None).await?;
+                let ops = response.into_inner();
+
+                // Find our operation
+                let op = ops.operations.iter().find(|o| o.id == operation_id);
+
+                match op {
+                    Some(operation) => {
+                        if verbose {
+                            eprintln!("Operation {} status: {}", operation_id, operation.status);
+                        }
+                        match operation.status.as_str() {
+                            "pending" => {
+                                // Still running, wait and poll again
+                                tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+                            }
+                            "completed" => {
+                                // Operation completed successfully
+                                return Ok((true, None));
+                            }
+                            "failed" => {
+                                return Ok((false, operation.error_message.clone()));
+                            }
+                            _ => {
+                                // Unknown status, treat as failed
+                                return Ok((false, Some(format!("Unknown status: {}", operation.status))));
+                            }
+                        }
+                    }
+                    None => {
+                        // Operation not in list means it completed successfully (removed from pending/failed)
+                        return Ok((true, None));
+                    }
+                }
+            }
+        })
+    }
+
     pub fn delete_memory(&self, _agent_id: &str, _unit_id: &str, _verbose: bool) -> Result<types::DeleteResponse> {
         // Note: Individual memory deletion is no longer supported in the API
         anyhow::bail!("Individual memory deletion is no longer supported. Use 'memory clear' to clear all memories.")
@@ -281,3 +327,105 @@ pub use types::{
     ReflectResponse,
     RetainRequest,
 };
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_operation_deserialize() {
+        let json = r#"{
+            "id": "test-op-123",
+            "task_type": "retain",
+            "items_count": 5,
+            "document_id": "doc-456",
+            "created_at": "2024-01-15T10:00:00Z",
+            "status": "pending",
+            "error_message": null
+        }"#;
+        let op: Operation = serde_json::from_str(json).unwrap();
+        assert_eq!(op.id, "test-op-123");
+        assert_eq!(op.task_type, "retain");
+        assert_eq!(op.items_count, 5);
+        assert_eq!(op.document_id, Some("doc-456".to_string()));
+        assert_eq!(op.status, "pending");
+        assert!(op.error_message.is_none());
+    }
+
+    #[test]
+    fn test_operation_deserialize_with_error() {
+        let json = r#"{
+            "id": "test-op-456",
+            "task_type": "retain",
+            "items_count": 3,
+            "document_id": null,
+            "created_at": "2024-01-15T10:00:00Z",
+            "status": "failed",
+            "error_message": "Something went wrong"
+        }"#;
+        let op: Operation = serde_json::from_str(json).unwrap();
+        assert_eq!(op.status, "failed");
+        assert_eq!(op.error_message, Some("Something went wrong".to_string()));
+    }
+
+    #[test]
+    fn test_memory_put_result_serialize() {
+        let result = MemoryPutResult {
+            success: true,
+            items_count: 10,
+            message: "Stored 10 memory units".to_string(),
+            is_async: true,
+            operation_id: Some("op-789".to_string()),
+        };
+        let json = serde_json::to_string(&result).unwrap();
+        assert!(json.contains("\"success\":true"));
+        assert!(json.contains("\"items_count\":10"));
+        assert!(json.contains("\"is_async\":true"));
+        assert!(json.contains("\"operation_id\":\"op-789\""));
+    }
+
+    #[test]
+    fn test_memory_put_result_without_operation_id() {
+        let result = MemoryPutResult {
+            success: true,
+            items_count: 5,
+            message: "Stored 5 memory units".to_string(),
+            is_async: false,
+            operation_id: None,
+        };
+        let json = serde_json::to_string(&result).unwrap();
+        assert!(json.contains("\"operation_id\":null"));
+    }
+
+    #[test]
+    fn test_operations_response_deserialize() {
+        let json = r#"{
+            "bank_id": "test-bank",
+            "operations": [
+                {
+                    "id": "op-1",
+                    "task_type": "retain",
+                    "items_count": 2,
+                    "document_id": null,
+                    "created_at": "2024-01-15T10:00:00Z",
+                    "status": "pending",
+                    "error_message": null
+                },
+                {
+                    "id": "op-2",
+                    "task_type": "retain",
+                    "items_count": 3,
+                    "document_id": "doc-123",
+                    "created_at": "2024-01-15T11:00:00Z",
+                    "status": "completed",
+                    "error_message": null
+                }
+            ]
+        }"#;
+        let ops: OperationsResponse = serde_json::from_str(json).unwrap();
+        assert_eq!(ops.bank_id, "test-bank");
+        assert_eq!(ops.operations.len(), 2);
+        assert_eq!(ops.operations[0].status, "pending");
+        assert_eq!(ops.operations[1].status, "completed");
+    }
+}

hindsight-cli/src/commands/bank.rs

@@ -201,7 +201,7 @@ pub fn update_background(
         Ok(profile) => {
             if output_format == OutputFormat::Pretty {
                 ui::print_success("Background updated successfully");
-                println!("\n{}", profile.background);
+                println!("\n{}", profile.mission);
 
                 if !no_update_disposition {
                     if let (Some(old_p), Some(new_p)) =

hindsight-cli/src/commands/memory.rs

@@ -21,6 +21,17 @@ fn parse_budget(budget: &str) -> Budget {
     }
 }
 
+// Helper function to check if a file has a text-based extension
+fn is_text_file(path: &std::path::Path) -> bool {
+    const TEXT_EXTENSIONS: &[&str] = &[
+        "txt", "md", "json", "yaml", "yml", "toml", "xml", "csv", "log", "rst", "adoc",
+    ];
+    path.extension()
+        .and_then(|ext| ext.to_str())
+        .map(|ext| TEXT_EXTENSIONS.contains(&ext.to_lowercase().as_str()))
+        .unwrap_or(false)
+}
+
 pub fn recall(
     client: &ApiClient,
     agent_id: &str,
@@ -229,29 +240,23 @@ pub fn retain_files(
                 .filter(|e| e.file_type().is_file())
             {
                 let path = entry.path();
-                if let Some(ext) = path.extension() {
-                    if ext == "txt" || ext == "md" {
-                        files.push(path.to_path_buf());
-                    }
+                if is_text_file(&path) {
+                    files.push(path.to_path_buf());
                 }
             }
         } else {
             for entry in fs::read_dir(&path)? {
                 let entry = entry?;
                 let path = entry.path();
-                if path.is_file() {
-                    if let Some(ext) = path.extension() {
-                        if ext == "txt" || ext == "md" {
-                            files.push(path);
-                        }
-                    }
+                if path.is_file() && is_text_file(&path) {
+                    files.push(path);
                 }
             }
         }
     }
 
     if files.is_empty() {
-        ui::print_warning("No .txt or .md files found");
+        ui::print_warning("No text files found (supported: txt, md, json, yaml, yml, toml, xml, csv, log, rst, adoc)");
         return Ok(());
     }
 
@@ -286,19 +291,20 @@ pub fn retain_files(
 
     pb.finish_with_message("Files processed");
 
+    // Always use async mode for the API call
+    let request = RetainRequest {
+        items,
+        async_: true,
+        document_tags: None,
+    };
+
     let spinner = if output_format == OutputFormat::Pretty {
-        Some(ui::create_spinner("Retaining memories..."))
+        Some(ui::create_spinner("Submitting retain request..."))
     } else {
         None
     };
 
-    let request = RetainRequest {
-        items,
-        async_: r#async,
-        document_tags: None,
-    };
-
-    let response = client.retain(agent_id, &request, r#async, verbose);
+    let response = client.retain(agent_id, &request, true, verbose);
 
     if let Some(mut sp) = spinner {
         sp.finish();
@@ -306,16 +312,55 @@ pub fn retain_files(
 
     match response {
         Ok(result) => {
-            if output_format == OutputFormat::Pretty {
-                ui::print_success("Files retained successfully");
-                if result.is_async {
-                    println!("  Status: queued for background processing");
+            if r#async {
+                // User requested async mode - return immediately
+                if output_format == OutputFormat::Pretty {
+                    ui::print_success("Files queued for processing");
                     println!("  Items: {}", result.items_count);
+                    if let Some(op_id) = &result.operation_id {
+                        println!("  Operation ID: {}", op_id);
+                    }
                 } else {
-                    println!("  Total units created: {}", result.items_count);
+                    output::print_output(&result, output_format)?;
                 }
             } else {
-                output::print_output(&result, output_format)?;
+                // Poll until completion
+                if let Some(operation_id) = &result.operation_id {
+                    let poll_spinner = if output_format == OutputFormat::Pretty {
+                        Some(ui::create_spinner("Processing memories..."))
+                    } else {
+                        None
+                    };
+
+                    let (success, error_msg) = client.poll_operation(agent_id, operation_id, verbose)?;
+
+                    if let Some(mut sp) = poll_spinner {
+                        sp.finish();
+                    }
+
+                    if success {
+                        if output_format == OutputFormat::Pretty {
+                            ui::print_success("Files retained successfully");
+                            println!("  Items processed: {}", result.items_count);
+                        } else {
+                            output::print_output(&result, output_format)?;
+                        }
+                    } else {
+                        let msg = error_msg.unwrap_or_else(|| "Unknown error".to_string());
+                        if output_format == OutputFormat::Pretty {
+                            ui::print_error(&format!("Retain operation failed: {}", msg));
+                        }
+                        anyhow::bail!("Retain operation failed: {}", msg);
+                    }
+                } else {
+                    // No operation ID returned, shouldn't happen with async=true
+                    if output_format == OutputFormat::Pretty {
+                        ui::print_success("Files retained successfully");
+                        println!("  Items processed: {}", result.items_count);
+                    } else {
+                        output::print_output(&result, output_format)?;
+                    }
+                }
             }
             Ok(())
         }
@@ -428,3 +473,84 @@ pub fn clear(
         Err(e) => Err(e)
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::path::Path;
+
+    #[test]
+    fn test_is_text_file_supported_extensions() {
+        let supported = [
+            "file.txt", "file.md", "file.json", "file.yaml", "file.yml",
+            "file.toml", "file.xml", "file.csv", "file.log", "file.rst", "file.adoc",
+        ];
+        for filename in supported {
+            assert!(
+                is_text_file(Path::new(filename)),
+                "{} should be recognized as a text file",
+                filename
+            );
+        }
+    }
+
+    #[test]
+    fn test_is_text_file_case_insensitive() {
+        assert!(is_text_file(Path::new("file.JSON")));
+        assert!(is_text_file(Path::new("file.TXT")));
+        assert!(is_text_file(Path::new("file.Md")));
+        assert!(is_text_file(Path::new("file.YAML")));
+    }
+
+    #[test]
+    fn test_is_text_file_unsupported_extensions() {
+        let unsupported = [
+            "file.pdf", "file.doc", "file.docx", "file.png", "file.jpg",
+            "file.exe", "file.bin", "file.zip", "file.tar", "file.gz",
+        ];
+        for filename in unsupported {
+            assert!(
+                !is_text_file(Path::new(filename)),
+                "{} should NOT be recognized as a text file",
+                filename
+            );
+        }
+    }
+
+    #[test]
+    fn test_is_text_file_no_extension() {
+        assert!(!is_text_file(Path::new("README")));
+        assert!(!is_text_file(Path::new("Makefile")));
+        assert!(!is_text_file(Path::new(".gitignore")));
+    }
+
+    #[test]
+    fn test_is_text_file_with_path() {
+        assert!(is_text_file(Path::new("/some/path/to/file.json")));
+        assert!(is_text_file(Path::new("../relative/path/file.md")));
+        assert!(!is_text_file(Path::new("/path/to/image.png")));
+    }
+
+    #[test]
+    fn test_parse_budget_valid_values() {
+        assert!(matches!(parse_budget("low"), Budget::Low));
+        assert!(matches!(parse_budget("mid"), Budget::Mid));
+        assert!(matches!(parse_budget("high"), Budget::High));
+    }
+
+    #[test]
+    fn test_parse_budget_case_insensitive() {
+        assert!(matches!(parse_budget("LOW"), Budget::Low));
+        assert!(matches!(parse_budget("MID"), Budget::Mid));
+        assert!(matches!(parse_budget("HIGH"), Budget::High));
+        assert!(matches!(parse_budget("Low"), Budget::Low));
+        assert!(matches!(parse_budget("High"), Budget::High));
+    }
+
+    #[test]
+    fn test_parse_budget_defaults_to_mid() {
+        assert!(matches!(parse_budget("invalid"), Budget::Mid));
+        assert!(matches!(parse_budget(""), Budget::Mid));
+        assert!(matches!(parse_budget("unknown"), Budget::Mid));
+    }
+}

hindsight-cli/src/config.rs

@@ -8,6 +8,7 @@ const DEFAULT_API_URL: &str = "http://localhost:8888";
 const CONFIG_FILE_NAME: &str = "config";
 const CONFIG_DIR_NAME: &str = ".hindsight";
 
+#[derive(Debug)]
 pub struct Config {
     pub api_url: String,
     pub api_key: Option<String>,
@@ -174,3 +175,156 @@ pub fn generate_doc_id() -> String {
     let now = chrono::Local::now();
     format!("cli_put_{}", now.format("%Y%m%d_%H%M%S"))
 }
+
+/// Parse a simple TOML-like config line and extract value.
+/// Handles both quoted and unquoted values.
+pub fn parse_config_value(line: &str, key: &str) -> Option<String> {
+    let line = line.trim();
+    if !line.starts_with(key) {
+        return None;
+    }
+    line.split('=').nth(1).map(|value| {
+        value.trim().trim_matches('"').trim_matches('\'').to_string()
+    }).filter(|v| !v.is_empty())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_config_source_display() {
+        assert_eq!(format!("{}", ConfigSource::LocalFile), "config file");
+        assert_eq!(format!("{}", ConfigSource::Environment), "environment variable");
+        assert_eq!(format!("{}", ConfigSource::Default), "default");
+    }
+
+    #[test]
+    fn test_validate_and_create_valid_http() {
+        let config = Config::validate_and_create(
+            "http://localhost:8888".to_string(),
+            None,
+            ConfigSource::Default,
+        );
+        assert!(config.is_ok());
+        let config = config.unwrap();
+        assert_eq!(config.api_url, "http://localhost:8888");
+        assert_eq!(config.source, ConfigSource::Default);
+    }
+
+    #[test]
+    fn test_validate_and_create_valid_https() {
+        let config = Config::validate_and_create(
+            "https://api.example.com".to_string(),
+            Some("secret-key".to_string()),
+            ConfigSource::Environment,
+        );
+        assert!(config.is_ok());
+        let config = config.unwrap();
+        assert_eq!(config.api_url, "https://api.example.com");
+        assert_eq!(config.api_key, Some("secret-key".to_string()));
+        assert_eq!(config.source, ConfigSource::Environment);
+    }
+
+    #[test]
+    fn test_validate_and_create_invalid_url() {
+        let config = Config::validate_and_create(
+            "localhost:8888".to_string(),
+            None,
+            ConfigSource::Default,
+        );
+        assert!(config.is_err());
+        let err = config.unwrap_err().to_string();
+        assert!(err.contains("Invalid API URL"));
+        assert!(err.contains("Must start with http:// or https://"));
+    }
+
+    #[test]
+    fn test_validate_and_create_ftp_url() {
+        let config = Config::validate_and_create(
+            "ftp://example.com".to_string(),
+            None,
+            ConfigSource::Default,
+        );
+        assert!(config.is_err());
+    }
+
+    #[test]
+    fn test_generate_doc_id_format() {
+        let doc_id = generate_doc_id();
+        assert!(doc_id.starts_with("cli_put_"));
+        // Should be cli_put_YYYYMMDD_HHMMSS format
+        assert!(doc_id.len() > 20); // cli_put_ (8) + date (8) + _ (1) + time (6) = 23
+    }
+
+    #[test]
+    fn test_generate_doc_id_uniqueness() {
+        let id1 = generate_doc_id();
+        std::thread::sleep(std::time::Duration::from_secs(1));
+        let id2 = generate_doc_id();
+        // IDs generated at different times should be different
+        assert_ne!(id1, id2);
+    }
+
+    #[test]
+    fn test_parse_config_value_quoted() {
+        assert_eq!(
+            parse_config_value(r#"api_url = "http://localhost:8888""#, "api_url"),
+            Some("http://localhost:8888".to_string())
+        );
+    }
+
+    #[test]
+    fn test_parse_config_value_single_quoted() {
+        assert_eq!(
+            parse_config_value("api_url = 'http://localhost:8888'", "api_url"),
+            Some("http://localhost:8888".to_string())
+        );
+    }
+
+    #[test]
+    fn test_parse_config_value_unquoted() {
+        assert_eq!(
+            parse_config_value("api_url = http://localhost:8888", "api_url"),
+            Some("http://localhost:8888".to_string())
+        );
+    }
+
+    #[test]
+    fn test_parse_config_value_with_spaces() {
+        assert_eq!(
+            parse_config_value("  api_url  =  \"http://localhost:8888\"  ", "api_url"),
+            Some("http://localhost:8888".to_string())
+        );
+    }
+
+    #[test]
+    fn test_parse_config_value_wrong_key() {
+        assert_eq!(
+            parse_config_value("api_key = secret", "api_url"),
+            None
+        );
+    }
+
+    #[test]
+    fn test_parse_config_value_empty() {
+        assert_eq!(
+            parse_config_value("api_url = ", "api_url"),
+            None
+        );
+        assert_eq!(
+            parse_config_value("api_url = \"\"", "api_url"),
+            None
+        );
+    }
+
+    #[test]
+    fn test_config_api_url_accessor() {
+        let config = Config {
+            api_url: "http://test:8080".to_string(),
+            api_key: None,
+            source: ConfigSource::Default,
+        };
+        assert_eq!(config.api_url(), "http://test:8080");
+    }
+}

hindsight-cli/src/output.rs

@@ -8,13 +8,35 @@ pub enum OutputFormat {
     Yaml,
 }
 
+impl OutputFormat {
+    /// Parse output format from string
+    pub fn from_str(s: &str) -> Option<Self> {
+        match s.to_lowercase().as_str() {
+            "json" => Some(OutputFormat::Json),
+            "yaml" | "yml" => Some(OutputFormat::Yaml),
+            "pretty" | "text" => Some(OutputFormat::Pretty),
+            _ => None,
+        }
+    }
+}
+
+/// Format data as JSON string
+pub fn to_json<T: Serialize>(data: &T) -> Result<String> {
+    Ok(serde_json::to_string_pretty(data)?)
+}
+
+/// Format data as YAML string
+pub fn to_yaml<T: Serialize>(data: &T) -> Result<String> {
+    Ok(serde_yaml::to_string(data)?)
+}
+
 pub fn print_output<T: Serialize>(data: &T, format: OutputFormat) -> Result<()> {
     match format {
         OutputFormat::Json => {
-            println!("{}", serde_json::to_string_pretty(data)?);
+            println!("{}", to_json(data)?);
         }
         OutputFormat::Yaml => {
-            println!("{}", serde_yaml::to_string(data)?);
+            println!("{}", to_yaml(data)?);
         }
         OutputFormat::Pretty => {
             // This should not be called - pretty printing is handled in ui.rs
@@ -23,3 +45,121 @@ pub fn print_output<T: Serialize>(data: &T, format: OutputFormat) -> Result<()>
     }
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde::{Deserialize, Serialize};
+
+    #[derive(Debug, Serialize, Deserialize, PartialEq)]
+    struct TestData {
+        name: String,
+        count: i32,
+        active: bool,
+    }
+
+    #[test]
+    fn test_output_format_from_str_json() {
+        assert_eq!(OutputFormat::from_str("json"), Some(OutputFormat::Json));
+        assert_eq!(OutputFormat::from_str("JSON"), Some(OutputFormat::Json));
+        assert_eq!(OutputFormat::from_str("Json"), Some(OutputFormat::Json));
+    }
+
+    #[test]
+    fn test_output_format_from_str_yaml() {
+        assert_eq!(OutputFormat::from_str("yaml"), Some(OutputFormat::Yaml));
+        assert_eq!(OutputFormat::from_str("YAML"), Some(OutputFormat::Yaml));
+        assert_eq!(OutputFormat::from_str("yml"), Some(OutputFormat::Yaml));
+        assert_eq!(OutputFormat::from_str("YML"), Some(OutputFormat::Yaml));
+    }
+
+    #[test]
+    fn test_output_format_from_str_pretty() {
+        assert_eq!(OutputFormat::from_str("pretty"), Some(OutputFormat::Pretty));
+        assert_eq!(OutputFormat::from_str("PRETTY"), Some(OutputFormat::Pretty));
+        assert_eq!(OutputFormat::from_str("text"), Some(OutputFormat::Pretty));
+    }
+
+    #[test]
+    fn test_output_format_from_str_invalid() {
+        assert_eq!(OutputFormat::from_str("xml"), None);
+        assert_eq!(OutputFormat::from_str("csv"), None);
+        assert_eq!(OutputFormat::from_str(""), None);
+    }
+
+    #[test]
+    fn test_to_json() {
+        let data = TestData {
+            name: "test".to_string(),
+            count: 42,
+            active: true,
+        };
+        let json = to_json(&data).unwrap();
+        assert!(json.contains("\"name\": \"test\""));
+        assert!(json.contains("\"count\": 42"));
+        assert!(json.contains("\"active\": true"));
+    }
+
+    #[test]
+    fn test_to_yaml() {
+        let data = TestData {
+            name: "test".to_string(),
+            count: 42,
+            active: true,
+        };
+        let yaml = to_yaml(&data).unwrap();
+        assert!(yaml.contains("name: test"));
+        assert!(yaml.contains("count: 42"));
+        assert!(yaml.contains("active: true"));
+    }
+
+    #[test]
+    fn test_to_json_array() {
+        let data = vec![
+            TestData { name: "a".to_string(), count: 1, active: true },
+            TestData { name: "b".to_string(), count: 2, active: false },
+        ];
+        let json = to_json(&data).unwrap();
+        assert!(json.contains("\"name\": \"a\""));
+        assert!(json.contains("\"name\": \"b\""));
+    }
+
+    #[test]
+    fn test_to_yaml_array() {
+        let data = vec![
+            TestData { name: "a".to_string(), count: 1, active: true },
+            TestData { name: "b".to_string(), count: 2, active: false },
+        ];
+        let yaml = to_yaml(&data).unwrap();
+        assert!(yaml.contains("name: a"));
+        assert!(yaml.contains("name: b"));
+    }
+
+    #[test]
+    fn test_output_format_equality() {
+        assert_eq!(OutputFormat::Json, OutputFormat::Json);
+        assert_ne!(OutputFormat::Json, OutputFormat::Yaml);
+        assert_ne!(OutputFormat::Yaml, OutputFormat::Pretty);
+    }
+
+    #[test]
+    fn test_output_format_clone() {
+        let format = OutputFormat::Json;
+        let cloned = format.clone();
+        assert_eq!(format, cloned);
+    }
+
+    #[test]
+    fn test_to_json_special_chars() {
+        let data = TestData {
+            name: "test\"with\\special\nchars".to_string(),
+            count: 0,
+            active: false,
+        };
+        let json = to_json(&data).unwrap();
+        // JSON should properly escape special characters
+        assert!(json.contains("\\\""));
+        assert!(json.contains("\\\\"));
+        assert!(json.contains("\\n"));
+    }
+}

hindsight-cli/src/ui.rs

@@ -172,8 +172,11 @@ pub fn print_think_response(response: &ReflectResponse) {
     println!("{}", response.text);
     println!();
 
-    if !response.based_on.is_empty() {
-        println!("{}", dim(&format!("Based on {} memory units", response.based_on.len())));
+    if let Some(based_on) = &response.based_on {
+        let count = based_on.memories.len() + based_on.mental_models.len();
+        if count > 0 {
+            println!("{}", dim(&format!("Based on {} memory units", count)));
+        }
     }
 
     // Display structured output if present
@@ -322,10 +325,10 @@ pub fn print_disposition(profile: &BankProfileResponse) {
     println!("{} {}", dim("Name:"), gradient_start(&profile.name));
     println!();
 
-    // Print background if available
-    if !profile.background.is_empty() {
-        println!("{}", gradient_mid("Background:"));
-        for line in profile.background.lines() {
+    // Print mission if available
+    if !profile.mission.is_empty() {
+        println!("{}", gradient_mid("Mission:"));
+        for line in profile.mission.lines() {
             println!("{}", line);
         }
         println!();

hindsight-clients/python/.openapi-generator/FILES

@@ -4,6 +4,7 @@ hindsight_client_api/api/banks_api.py
 hindsight_client_api/api/documents_api.py
 hindsight_client_api/api/entities_api.py
 hindsight_client_api/api/memory_api.py
+hindsight_client_api/api/mental_models_api.py
 hindsight_client_api/api/monitoring_api.py
 hindsight_client_api/api/operations_api.py
 hindsight_client_api/api_client.py
@@ -12,6 +13,7 @@ hindsight_client_api/configuration.py
 hindsight_client_api/exceptions.py
 hindsight_client_api/models/__init__.py
 hindsight_client_api/models/add_background_request.py
+hindsight_client_api/models/async_operation_submit_response.py
 hindsight_client_api/models/background_response.py
 hindsight_client_api/models/bank_list_item.py
 hindsight_client_api/models/bank_list_response.py
@@ -23,6 +25,8 @@ hindsight_client_api/models/chunk_data.py
 hindsight_client_api/models/chunk_include_options.py
 hindsight_client_api/models/chunk_response.py
 hindsight_client_api/models/create_bank_request.py
+hindsight_client_api/models/create_mental_model_request.py
+hindsight_client_api/models/created_mental_model.py
 hindsight_client_api/models/delete_document_response.py
 hindsight_client_api/models/delete_response.py
 hindsight_client_api/models/disposition_traits.py
@@ -41,20 +45,35 @@ hindsight_client_api/models/list_documents_response.py
 hindsight_client_api/models/list_memory_units_response.py
 hindsight_client_api/models/list_tags_response.py
 hindsight_client_api/models/memory_item.py
+hindsight_client_api/models/mental_model_freshness_response.py
+hindsight_client_api/models/mental_model_list_response.py
+hindsight_client_api/models/mental_model_observation_response.py
+hindsight_client_api/models/mental_model_response.py
+hindsight_client_api/models/observation_evidence_response.py
+hindsight_client_api/models/observation_input.py
 hindsight_client_api/models/operation_response.py
+hindsight_client_api/models/operation_status_response.py
 hindsight_client_api/models/operations_list_response.py
 hindsight_client_api/models/recall_request.py
 hindsight_client_api/models/recall_response.py
 hindsight_client_api/models/recall_result.py
+hindsight_client_api/models/reflect_based_on.py
 hindsight_client_api/models/reflect_fact.py
 hindsight_client_api/models/reflect_include_options.py
+hindsight_client_api/models/reflect_llm_call.py
+hindsight_client_api/models/reflect_mental_model.py
 hindsight_client_api/models/reflect_request.py
 hindsight_client_api/models/reflect_response.py
+hindsight_client_api/models/reflect_tool_call.py
+hindsight_client_api/models/reflect_trace.py
+hindsight_client_api/models/refresh_mental_models_request.py
 hindsight_client_api/models/retain_request.py
 hindsight_client_api/models/retain_response.py
 hindsight_client_api/models/tag_item.py
 hindsight_client_api/models/token_usage.py
+hindsight_client_api/models/tool_calls_include_options.py
 hindsight_client_api/models/update_disposition_request.py
+hindsight_client_api/models/update_mental_model_request.py
 hindsight_client_api/models/validation_error.py
 hindsight_client_api/models/validation_error_loc_inner.py
 hindsight_client_api/rest.py

hindsight-clients/python/hindsight_client/hindsight_client.py

@@ -115,6 +115,7 @@ class Hindsight:
         document_id: Optional[str] = None,
         metadata: Optional[Dict[str, str]] = None,
         entities: Optional[List[Dict[str, str]]] = None,
+        tags: Optional[List[str]] = None,
     ) -> RetainResponse:
         """
         Store a single memory (simplified interface).
@@ -127,13 +128,14 @@ class Hindsight:
             document_id: Optional document ID for grouping
             metadata: Optional user-defined metadata
             entities: Optional list of entities [{"text": "...", "type": "..."}]
+            tags: Optional list of tags for this memory
 
         Returns:
             RetainResponse with success status
         """
         return self.retain_batch(
             bank_id=bank_id,
-            items=[{"content": content, "timestamp": timestamp, "context": context, "metadata": metadata, "entities": entities}],
+            items=[{"content": content, "timestamp": timestamp, "context": context, "metadata": metadata, "entities": entities, "tags": tags}],
             document_id=document_id,
         )
 
@@ -143,15 +145,17 @@ class Hindsight:
         items: List[Dict[str, Any]],
         document_id: Optional[str] = None,
         retain_async: bool = False,
+        document_tags: Optional[List[str]] = None,
     ) -> RetainResponse:
         """
         Store multiple memories in batch.
 
         Args:
             bank_id: The memory bank ID
-            items: List of memory items with 'content' and optional 'timestamp', 'context', 'metadata', 'document_id', 'entities'
+            items: List of memory items with 'content' and optional 'timestamp', 'context', 'metadata', 'document_id', 'entities', 'tags'
             document_id: Optional document ID for grouping memories (applied to items that don't have their own)
             retain_async: If True, process asynchronously in background (default: False)
+            document_tags: Optional list of tags to apply to all memories in this batch
 
         Returns:
             RetainResponse with success status and item count
@@ -175,12 +179,14 @@ class Hindsight:
                     # Use item's document_id if provided, otherwise fall back to batch-level document_id
                     document_id=item.get("document_id") or document_id,
                     entities=entities,
+                    tags=item.get("tags"),
                 )
             )
 
         request_obj = retain_request.RetainRequest(
             items=memory_items,
             async_=retain_async,
+            document_tags=document_tags,
         )
 
         return _run_async(self._memory_api.retain_memories(bank_id, request_obj))
@@ -198,6 +204,8 @@ class Hindsight:
         max_entity_tokens: int = 500,
         include_chunks: bool = False,
         max_chunk_tokens: int = 8192,
+        tags: Optional[List[str]] = None,
+        tags_match: str = "any",
     ) -> RecallResponse:
         """
         Recall memories using semantic similarity.
@@ -214,6 +222,9 @@ class Hindsight:
             max_entity_tokens: Maximum tokens for entity observations (default: 500)
             include_chunks: Include raw text chunks in results (default: False)
             max_chunk_tokens: Maximum tokens for chunks (default: 8192)
+            tags: Optional list of tags to filter memories by
+            tags_match: How to match tags: 'any' (OR, includes untagged), 'all' (AND, includes untagged),
+                'any_strict' (OR, excludes untagged), 'all_strict' (AND, excludes untagged). Default: 'any'
 
         Returns:
             RecallResponse with results, optional entities, optional chunks, and optional trace
@@ -233,6 +244,8 @@ class Hindsight:
             trace=trace,
             query_timestamp=query_timestamp,
             include=include_opts,
+            tags=tags,
+            tags_match=tags_match,
         )
 
         return _run_async(self._memory_api.recall_memories(bank_id, request_obj))
@@ -245,6 +258,8 @@ class Hindsight:
         context: Optional[str] = None,
         max_tokens: Optional[int] = None,
         response_schema: Optional[Dict[str, Any]] = None,
+        tags: Optional[List[str]] = None,
+        tags_match: str = "any",
     ) -> ReflectResponse:
         """
         Generate a contextual answer based on bank identity and memories.
@@ -258,6 +273,9 @@ class Hindsight:
             response_schema: Optional JSON Schema for structured output. When provided,
                 the response will include a 'structured_output' field with the LLM
                 response parsed according to this schema.
+            tags: Optional list of tags to filter memories by
+            tags_match: How to match tags: 'any' (OR, includes untagged), 'all' (AND, includes untagged),
+                'any_strict' (OR, excludes untagged), 'all_strict' (AND, excludes untagged). Default: 'any'
 
         Returns:
             ReflectResponse with answer text, optionally facts used, and optionally
@@ -269,6 +287,8 @@ class Hindsight:
             context=context,
             max_tokens=max_tokens,
             response_schema=response_schema,
+            tags=tags,
+            tags_match=tags_match,
         )
 
         return _run_async(self._memory_api.reflect(bank_id, request_obj))

hindsight-clients/python/hindsight_client_api/__init__.py

@@ -21,6 +21,7 @@ from hindsight_client_api.api.banks_api import BanksApi
 from hindsight_client_api.api.documents_api import DocumentsApi
 from hindsight_client_api.api.entities_api import EntitiesApi
 from hindsight_client_api.api.memory_api import MemoryApi
+from hindsight_client_api.api.mental_models_api import MentalModelsApi
 from hindsight_client_api.api.monitoring_api import MonitoringApi
 from hindsight_client_api.api.operations_api import OperationsApi
 
@@ -37,6 +38,7 @@ from hindsight_client_api.exceptions import ApiException
 
 # import models into sdk package
 from hindsight_client_api.models.add_background_request import AddBackgroundRequest
+from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
 from hindsight_client_api.models.background_response import BackgroundResponse
 from hindsight_client_api.models.bank_list_item import BankListItem
 from hindsight_client_api.models.bank_list_response import BankListResponse
@@ -48,6 +50,8 @@ from hindsight_client_api.models.chunk_data import ChunkData
 from hindsight_client_api.models.chunk_include_options import ChunkIncludeOptions
 from hindsight_client_api.models.chunk_response import ChunkResponse
 from hindsight_client_api.models.create_bank_request import CreateBankRequest
+from hindsight_client_api.models.create_mental_model_request import CreateMentalModelRequest
+from hindsight_client_api.models.created_mental_model import CreatedMentalModel
 from hindsight_client_api.models.delete_document_response import DeleteDocumentResponse
 from hindsight_client_api.models.delete_response import DeleteResponse
 from hindsight_client_api.models.disposition_traits import DispositionTraits
@@ -66,19 +70,34 @@ from hindsight_client_api.models.list_documents_response import ListDocumentsRes
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
 from hindsight_client_api.models.list_tags_response import ListTagsResponse
 from hindsight_client_api.models.memory_item import MemoryItem
+from hindsight_client_api.models.mental_model_freshness_response import MentalModelFreshnessResponse
+from hindsight_client_api.models.mental_model_list_response import MentalModelListResponse
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
+from hindsight_client_api.models.observation_evidence_response import ObservationEvidenceResponse
+from hindsight_client_api.models.observation_input import ObservationInput
 from hindsight_client_api.models.operation_response import OperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 from hindsight_client_api.models.recall_request import RecallRequest
 from hindsight_client_api.models.recall_response import RecallResponse
 from hindsight_client_api.models.recall_result import RecallResult
+from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
 from hindsight_client_api.models.reflect_fact import ReflectFact
 from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
+from hindsight_client_api.models.reflect_llm_call import ReflectLLMCall
+from hindsight_client_api.models.reflect_mental_model import ReflectMentalModel
 from hindsight_client_api.models.reflect_request import ReflectRequest
 from hindsight_client_api.models.reflect_response import ReflectResponse
+from hindsight_client_api.models.reflect_tool_call import ReflectToolCall
+from hindsight_client_api.models.reflect_trace import ReflectTrace
+from hindsight_client_api.models.refresh_mental_models_request import RefreshMentalModelsRequest
 from hindsight_client_api.models.retain_request import RetainRequest
 from hindsight_client_api.models.retain_response import RetainResponse
 from hindsight_client_api.models.tag_item import TagItem
 from hindsight_client_api.models.token_usage import TokenUsage
+from hindsight_client_api.models.tool_calls_include_options import ToolCallsIncludeOptions
 from hindsight_client_api.models.update_disposition_request import UpdateDispositionRequest
+from hindsight_client_api.models.update_mental_model_request import UpdateMentalModelRequest
 from hindsight_client_api.models.validation_error import ValidationError
 from hindsight_client_api.models.validation_error_loc_inner import ValidationErrorLocInner

hindsight-clients/python/hindsight_client_api/api/__init__.py

@@ -5,6 +5,7 @@ from hindsight_client_api.api.banks_api import BanksApi
 from hindsight_client_api.api.documents_api import DocumentsApi
 from hindsight_client_api.api.entities_api import EntitiesApi
 from hindsight_client_api.api.memory_api import MemoryApi
+from hindsight_client_api.api.mental_models_api import MentalModelsApi
 from hindsight_client_api.api.monitoring_api import MonitoringApi
 from hindsight_client_api.api.operations_api import OperationsApi
 

hindsight-clients/python/hindsight_client_api/api/banks_api.py

@@ -64,9 +64,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> BackgroundResponse:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -95,6 +95,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -140,9 +141,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> ApiResponse[BackgroundResponse]:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -171,6 +172,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -216,9 +218,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> RESTResponseType:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -247,6 +249,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -372,7 +375,7 @@ class BanksApi:
     ) -> BankProfileResponse:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -448,7 +451,7 @@ class BanksApi:
     ) -> ApiResponse[BankProfileResponse]:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -524,7 +527,7 @@ class BanksApi:
     ) -> RESTResponseType:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1233,7 +1236,7 @@ class BanksApi:
     ) -> BankProfileResponse:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1305,7 +1308,7 @@ class BanksApi:
     ) -> ApiResponse[BankProfileResponse]:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1377,7 +1380,7 @@ class BanksApi:
     ) -> RESTResponseType:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1754,6 +1757,312 @@ class BanksApi:
 
 
 
+    @validate_call
+    async def update_bank(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> BankProfileResponse:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def update_bank_with_http_info(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[BankProfileResponse]:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def update_bank_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _update_bank_serialize(
+        self,
+        bank_id,
+        create_bank_request,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        # process the query parameters
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+        if create_bank_request is not None:
+            _body_params = create_bank_request
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+        # set the HTTP header `Content-Type`
+        if _content_type:
+            _header_params['Content-Type'] = _content_type
+        else:
+            _default_content_type = (
+                self.api_client.select_header_content_type(
+                    [
+                        'application/json'
+                    ]
+                )
+            )
+            if _default_content_type is not None:
+                _header_params['Content-Type'] = _default_content_type
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='PATCH',
+            resource_path='/v1/default/banks/{bank_id}',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def update_bank_disposition(
         self,

hindsight-clients/python/hindsight_client_api/api/entities_api.py

@@ -664,9 +664,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> EntityDetailResponse:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -695,6 +695,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,
@@ -740,9 +741,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> ApiResponse[EntityDetailResponse]:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -771,6 +772,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,
@@ -816,9 +818,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> RESTResponseType:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -847,6 +849,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,

hindsight-clients/python/hindsight_client_api/api/operations_api.py

@@ -19,6 +19,7 @@ from typing_extensions import Annotated
 from pydantic import StrictStr
 from typing import Optional
 from hindsight_client_api.models.cancel_operation_response import CancelOperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 
 from hindsight_client_api.api_client import ApiClient, RequestSerialized
@@ -332,6 +333,299 @@ class OperationsApi:
 
 
 
+    @validate_call
+    async def get_operation_status(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> OperationStatusResponse:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def get_operation_status_with_http_info(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[OperationStatusResponse]:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def get_operation_status_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _get_operation_status_serialize(
+        self,
+        bank_id,
+        operation_id,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        if operation_id is not None:
+            _path_params['operation_id'] = operation_id
+        # process the query parameters
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='GET',
+            resource_path='/v1/default/banks/{bank_id}/operations/{operation_id}',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def list_operations(
         self,

hindsight-clients/python/hindsight_client_api/models/__init__.py

@@ -15,6 +15,7 @@
 
 # import models into model package
 from hindsight_client_api.models.add_background_request import AddBackgroundRequest
+from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
 from hindsight_client_api.models.background_response import BackgroundResponse
 from hindsight_client_api.models.bank_list_item import BankListItem
 from hindsight_client_api.models.bank_list_response import BankListResponse
@@ -26,6 +27,8 @@ from hindsight_client_api.models.chunk_data import ChunkData
 from hindsight_client_api.models.chunk_include_options import ChunkIncludeOptions
 from hindsight_client_api.models.chunk_response import ChunkResponse
 from hindsight_client_api.models.create_bank_request import CreateBankRequest
+from hindsight_client_api.models.create_mental_model_request import CreateMentalModelRequest
+from hindsight_client_api.models.created_mental_model import CreatedMentalModel
 from hindsight_client_api.models.delete_document_response import DeleteDocumentResponse
 from hindsight_client_api.models.delete_response import DeleteResponse
 from hindsight_client_api.models.disposition_traits import DispositionTraits
@@ -44,19 +47,34 @@ from hindsight_client_api.models.list_documents_response import ListDocumentsRes
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
 from hindsight_client_api.models.list_tags_response import ListTagsResponse
 from hindsight_client_api.models.memory_item import MemoryItem
+from hindsight_client_api.models.mental_model_freshness_response import MentalModelFreshnessResponse
+from hindsight_client_api.models.mental_model_list_response import MentalModelListResponse
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
+from hindsight_client_api.models.observation_evidence_response import ObservationEvidenceResponse
+from hindsight_client_api.models.observation_input import ObservationInput
 from hindsight_client_api.models.operation_response import OperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 from hindsight_client_api.models.recall_request import RecallRequest
 from hindsight_client_api.models.recall_response import RecallResponse
 from hindsight_client_api.models.recall_result import RecallResult
+from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
 from hindsight_client_api.models.reflect_fact import ReflectFact
 from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
+from hindsight_client_api.models.reflect_llm_call import ReflectLLMCall
+from hindsight_client_api.models.reflect_mental_model import ReflectMentalModel
 from hindsight_client_api.models.reflect_request import ReflectRequest
 from hindsight_client_api.models.reflect_response import ReflectResponse
+from hindsight_client_api.models.reflect_tool_call import ReflectToolCall
+from hindsight_client_api.models.reflect_trace import ReflectTrace
+from hindsight_client_api.models.refresh_mental_models_request import RefreshMentalModelsRequest
 from hindsight_client_api.models.retain_request import RetainRequest
 from hindsight_client_api.models.retain_response import RetainResponse
 from hindsight_client_api.models.tag_item import TagItem
 from hindsight_client_api.models.token_usage import TokenUsage
+from hindsight_client_api.models.tool_calls_include_options import ToolCallsIncludeOptions
 from hindsight_client_api.models.update_disposition_request import UpdateDispositionRequest
+from hindsight_client_api.models.update_mental_model_request import UpdateMentalModelRequest
 from hindsight_client_api.models.validation_error import ValidationError
 from hindsight_client_api.models.validation_error_loc_inner import ValidationErrorLocInner

hindsight-clients/python/hindsight_client_api/models/add_background_request.py

@@ -24,10 +24,10 @@ from typing_extensions import Self
 
 class AddBackgroundRequest(BaseModel):
     """
-    Request model for adding/merging background information.
+    Request model for adding/merging background information. Deprecated: use SetMissionRequest instead.
     """ # noqa: E501
     content: StrictStr = Field(description="New background information to add or merge")
-    update_disposition: Optional[StrictBool] = Field(default=True, description="If true, infer disposition traits from the merged background (default: true)")
+    update_disposition: Optional[StrictBool] = Field(default=True, description="Deprecated - disposition is no longer auto-inferred from mission")
     __properties: ClassVar[List[str]] = ["content", "update_disposition"]
 
     model_config = ConfigDict(

hindsight-clients/python/hindsight_client_api/models/async_operation_submit_response.py

@@ -0,0 +1,89 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class AsyncOperationSubmitResponse(BaseModel):
+    """
+    Response model for submitting an async operation.
+    """ # noqa: E501
+    operation_id: StrictStr
+    status: StrictStr
+    __properties: ClassVar[List[str]] = ["operation_id", "status"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of AsyncOperationSubmitResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of AsyncOperationSubmitResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "operation_id": obj.get("operation_id"),
+            "status": obj.get("status")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/background_response.py

@@ -25,11 +25,12 @@ from typing_extensions import Self
 
 class BackgroundResponse(BaseModel):
     """
-    Response model for background update.
+    Response model for background update. Deprecated: use MissionResponse instead.
     """ # noqa: E501
-    background: StrictStr
+    mission: StrictStr
+    background: Optional[StrictStr] = None
     disposition: Optional[DispositionTraits] = None
-    __properties: ClassVar[List[str]] = ["background", "disposition"]
+    __properties: ClassVar[List[str]] = ["mission", "background", "disposition"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -73,6 +74,11 @@ class BackgroundResponse(BaseModel):
         # override the default output from pydantic by calling `to_dict()` of disposition
         if self.disposition:
             _dict['disposition'] = self.disposition.to_dict()
+        # set to None if background (nullable) is None
+        # and model_fields_set contains the field
+        if self.background is None and "background" in self.model_fields_set:
+            _dict['background'] = None
+
         # set to None if disposition (nullable) is None
         # and model_fields_set contains the field
         if self.disposition is None and "disposition" in self.model_fields_set:
@@ -90,6 +96,7 @@ class BackgroundResponse(BaseModel):
             return cls.model_validate(obj)
 
         _obj = cls.model_validate({
+            "mission": obj.get("mission"),
             "background": obj.get("background"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None
         })

hindsight-clients/python/hindsight_client_api/models/bank_list_item.py

@@ -30,10 +30,10 @@ class BankListItem(BaseModel):
     bank_id: StrictStr
     name: Optional[StrictStr] = None
     disposition: DispositionTraits
-    background: Optional[StrictStr] = None
+    mission: Optional[StrictStr] = None
     created_at: Optional[StrictStr] = None
     updated_at: Optional[StrictStr] = None
-    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "background", "created_at", "updated_at"]
+    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "mission", "created_at", "updated_at"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -82,10 +82,10 @@ class BankListItem(BaseModel):
         if self.name is None and "name" in self.model_fields_set:
             _dict['name'] = None
 
-        # set to None if background (nullable) is None
+        # set to None if mission (nullable) is None
         # and model_fields_set contains the field
-        if self.background is None and "background" in self.model_fields_set:
-            _dict['background'] = None
+        if self.mission is None and "mission" in self.model_fields_set:
+            _dict['mission'] = None
 
         # set to None if created_at (nullable) is None
         # and model_fields_set contains the field
@@ -112,7 +112,7 @@ class BankListItem(BaseModel):
             "bank_id": obj.get("bank_id"),
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
-            "background": obj.get("background"),
+            "mission": obj.get("mission"),
             "created_at": obj.get("created_at"),
             "updated_at": obj.get("updated_at")
         })

hindsight-clients/python/hindsight_client_api/models/bank_profile_response.py

@@ -17,8 +17,8 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictStr
-from typing import Any, ClassVar, Dict, List
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
 from hindsight_client_api.models.disposition_traits import DispositionTraits
 from typing import Optional, Set
 from typing_extensions import Self
@@ -30,8 +30,9 @@ class BankProfileResponse(BaseModel):
     bank_id: StrictStr
     name: StrictStr
     disposition: DispositionTraits
-    background: StrictStr
-    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "background"]
+    mission: StrictStr = Field(description="The agent's mission - who they are and what they're trying to accomplish")
+    background: Optional[StrictStr] = None
+    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "mission", "background"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -75,6 +76,11 @@ class BankProfileResponse(BaseModel):
         # override the default output from pydantic by calling `to_dict()` of disposition
         if self.disposition:
             _dict['disposition'] = self.disposition.to_dict()
+        # set to None if background (nullable) is None
+        # and model_fields_set contains the field
+        if self.background is None and "background" in self.model_fields_set:
+            _dict['background'] = None
+
         return _dict
 
     @classmethod
@@ -90,6 +96,7 @@ class BankProfileResponse(BaseModel):
             "bank_id": obj.get("bank_id"),
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
+            "mission": obj.get("mission"),
             "background": obj.get("background")
         })
         return _obj

hindsight-clients/python/hindsight_client_api/models/create_bank_request.py

@@ -29,8 +29,9 @@ class CreateBankRequest(BaseModel):
     """ # noqa: E501
     name: Optional[StrictStr] = None
     disposition: Optional[DispositionTraits] = None
+    mission: Optional[StrictStr] = None
     background: Optional[StrictStr] = None
-    __properties: ClassVar[List[str]] = ["name", "disposition", "background"]
+    __properties: ClassVar[List[str]] = ["name", "disposition", "mission", "background"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -84,6 +85,11 @@ class CreateBankRequest(BaseModel):
         if self.disposition is None and "disposition" in self.model_fields_set:
             _dict['disposition'] = None
 
+        # set to None if mission (nullable) is None
+        # and model_fields_set contains the field
+        if self.mission is None and "mission" in self.model_fields_set:
+            _dict['mission'] = None
+
         # set to None if background (nullable) is None
         # and model_fields_set contains the field
         if self.background is None and "background" in self.model_fields_set:
@@ -103,6 +109,7 @@ class CreateBankRequest(BaseModel):
         _obj = cls.model_validate({
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
+            "mission": obj.get("mission"),
             "background": obj.get("background")
         })
         return _obj

hindsight-clients/python/hindsight_client_api/models/create_mental_model_request.py

@@ -0,0 +1,108 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.observation_input import ObservationInput
+from typing import Optional, Set
+from typing_extensions import Self
+
+class CreateMentalModelRequest(BaseModel):
+    """
+    Request model for creating a mental model.
+    """ # noqa: E501
+    name: StrictStr = Field(description="Human-readable name for the mental model")
+    description: StrictStr = Field(description="One-liner description for quick scanning")
+    subtype: Optional[StrictStr] = Field(default='pinned', description="Type of mental model: 'pinned' (observations LLM-generated) or 'directive' (observations user-provided)")
+    observations: Optional[List[ObservationInput]] = None
+    tags: Optional[List[StrictStr]] = Field(default=None, description="Tags for scoped visibility")
+    __properties: ClassVar[List[str]] = ["name", "description", "subtype", "observations", "tags"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of CreateMentalModelRequest from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in observations (list)
+        _items = []
+        if self.observations:
+            for _item_observations in self.observations:
+                if _item_observations:
+                    _items.append(_item_observations.to_dict())
+            _dict['observations'] = _items
+        # set to None if observations (nullable) is None
+        # and model_fields_set contains the field
+        if self.observations is None and "observations" in self.model_fields_set:
+            _dict['observations'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of CreateMentalModelRequest from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "name": obj.get("name"),
+            "description": obj.get("description"),
+            "subtype": obj.get("subtype") if obj.get("subtype") is not None else 'pinned',
+            "observations": [ObservationInput.from_dict(_item) for _item in obj["observations"]] if obj.get("observations") is not None else None,
+            "tags": obj.get("tags")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/created_mental_model.py

@@ -0,0 +1,91 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class CreatedMentalModel(BaseModel):
+    """
+    A mental model created during reflection.
+    """ # noqa: E501
+    id: StrictStr = Field(description="Mental model ID")
+    name: StrictStr = Field(description="Human-readable name")
+    description: StrictStr = Field(description="What this model tracks")
+    __properties: ClassVar[List[str]] = ["id", "name", "description"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of CreatedMentalModel from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of CreatedMentalModel from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "id": obj.get("id"),
+            "name": obj.get("name"),
+            "description": obj.get("description")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_freshness_response.py

@@ -0,0 +1,98 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictInt, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelFreshnessResponse(BaseModel):
+    """
+    Freshness information for a mental model.
+    """ # noqa: E501
+    is_up_to_date: StrictBool = Field(description="Whether the model has been refreshed since the last memory was added")
+    last_refresh_at: Optional[StrictStr]
+    memories_since_refresh: StrictInt = Field(description="Number of memories added since last refresh")
+    reasons: Optional[List[StrictStr]] = Field(default=None, description="Reasons why the model needs refresh (empty if up to date). Possible values: never_refreshed, new_memories, mission_changed, disposition_changed, directives_changed")
+    __properties: ClassVar[List[str]] = ["is_up_to_date", "last_refresh_at", "memories_since_refresh", "reasons"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelFreshnessResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # set to None if last_refresh_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.last_refresh_at is None and "last_refresh_at" in self.model_fields_set:
+            _dict['last_refresh_at'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelFreshnessResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "is_up_to_date": obj.get("is_up_to_date"),
+            "last_refresh_at": obj.get("last_refresh_at"),
+            "memories_since_refresh": obj.get("memories_since_refresh"),
+            "reasons": obj.get("reasons")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_list_response.py

@@ -0,0 +1,95 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict
+from typing import Any, ClassVar, Dict, List
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelListResponse(BaseModel):
+    """
+    Response model for listing mental models.
+    """ # noqa: E501
+    items: List[MentalModelResponse]
+    __properties: ClassVar[List[str]] = ["items"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelListResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in items (list)
+        _items = []
+        if self.items:
+            for _item_items in self.items:
+                if _item_items:
+                    _items.append(_item_items.to_dict())
+            _dict['items'] = _items
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelListResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "items": [MentalModelResponse.from_dict(_item) for _item in obj["items"]] if obj.get("items") is not None else None
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_observation_response.py

@@ -0,0 +1,107 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.observation_evidence_response import ObservationEvidenceResponse
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelObservationResponse(BaseModel):
+    """
+    An observation within a mental model with its supporting evidence.
+    """ # noqa: E501
+    title: StrictStr = Field(description="Short summary title for the observation")
+    content: StrictStr = Field(description="The observation content - detailed explanation")
+    evidence: Optional[List[ObservationEvidenceResponse]] = Field(default=None, description="Supporting evidence with quotes")
+    created_at: StrictStr = Field(description="When this observation was first created (ISO format)")
+    trend: StrictStr = Field(description="Computed trend: stable, strengthening, weakening, new, stale")
+    evidence_count: StrictInt = Field(description="Number of evidence items supporting this observation")
+    evidence_span: Dict[str, Any] = Field(description="Time span of evidence: {from: iso_date, to: iso_date}")
+    __properties: ClassVar[List[str]] = ["title", "content", "evidence", "created_at", "trend", "evidence_count", "evidence_span"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelObservationResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in evidence (list)
+        _items = []
+        if self.evidence:
+            for _item_evidence in self.evidence:
+                if _item_evidence:
+                    _items.append(_item_evidence.to_dict())
+            _dict['evidence'] = _items
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelObservationResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "title": obj.get("title"),
+            "content": obj.get("content"),
+            "evidence": [ObservationEvidenceResponse.from_dict(_item) for _item in obj["evidence"]] if obj.get("evidence") is not None else None,
+            "created_at": obj.get("created_at"),
+            "trend": obj.get("trend"),
+            "evidence_count": obj.get("evidence_count"),
+            "evidence_span": obj.get("evidence_span")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_response.py

@@ -0,0 +1,145 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.mental_model_freshness_response import MentalModelFreshnessResponse
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelResponse(BaseModel):
+    """
+    Response model for a mental model.
+    """ # noqa: E501
+    id: StrictStr
+    bank_id: StrictStr
+    subtype: StrictStr
+    name: StrictStr
+    description: StrictStr
+    observations: Optional[List[MentalModelObservationResponse]] = Field(default=None, description="Structured observations with per-observation fact attribution")
+    version: Optional[StrictInt] = Field(default=0, description="Version number of the mental model observations")
+    entity_id: Optional[StrictStr] = None
+    links: Optional[List[StrictStr]] = None
+    tags: Optional[List[StrictStr]] = None
+    last_updated: Optional[StrictStr] = None
+    last_refresh_at: Optional[StrictStr] = None
+    freshness: Optional[MentalModelFreshnessResponse] = None
+    created_at: StrictStr
+    __properties: ClassVar[List[str]] = ["id", "bank_id", "subtype", "name", "description", "observations", "version", "entity_id", "links", "tags", "last_updated", "last_refresh_at", "freshness", "created_at"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in observations (list)
+        _items = []
+        if self.observations:
+            for _item_observations in self.observations:
+                if _item_observations:
+                    _items.append(_item_observations.to_dict())
+            _dict['observations'] = _items
+        # override the default output from pydantic by calling `to_dict()` of freshness
+        if self.freshness:
+            _dict['freshness'] = self.freshness.to_dict()
+        # set to None if entity_id (nullable) is None
+        # and model_fields_set contains the field
+        if self.entity_id is None and "entity_id" in self.model_fields_set:
+            _dict['entity_id'] = None
+
+        # set to None if last_updated (nullable) is None
+        # and model_fields_set contains the field
+        if self.last_updated is None and "last_updated" in self.model_fields_set:
+            _dict['last_updated'] = None
+
+        # set to None if last_refresh_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.last_refresh_at is None and "last_refresh_at" in self.model_fields_set:
+            _dict['last_refresh_at'] = None
+
+        # set to None if freshness (nullable) is None
+        # and model_fields_set contains the field
+        if self.freshness is None and "freshness" in self.model_fields_set:
+            _dict['freshness'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "id": obj.get("id"),
+            "bank_id": obj.get("bank_id"),
+            "subtype": obj.get("subtype"),
+            "name": obj.get("name"),
+            "description": obj.get("description"),
+            "observations": [MentalModelObservationResponse.from_dict(_item) for _item in obj["observations"]] if obj.get("observations") is not None else None,
+            "version": obj.get("version") if obj.get("version") is not None else 0,
+            "entity_id": obj.get("entity_id"),
+            "links": obj.get("links"),
+            "tags": obj.get("tags"),
+            "last_updated": obj.get("last_updated"),
+            "last_refresh_at": obj.get("last_refresh_at"),
+            "freshness": MentalModelFreshnessResponse.from_dict(obj["freshness"]) if obj.get("freshness") is not None else None,
+            "created_at": obj.get("created_at")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/observation_evidence_response.py

@@ -0,0 +1,93 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ObservationEvidenceResponse(BaseModel):
+    """
+    A single piece of evidence supporting an observation.
+    """ # noqa: E501
+    memory_id: StrictStr = Field(description="ID of the memory unit this evidence comes from")
+    quote: StrictStr = Field(description="Exact quote from the memory supporting the observation")
+    relevance: StrictStr = Field(description="Brief explanation of how this quote supports the observation")
+    timestamp: StrictStr = Field(description="When the source memory was created (ISO format)")
+    __properties: ClassVar[List[str]] = ["memory_id", "quote", "relevance", "timestamp"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ObservationEvidenceResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ObservationEvidenceResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "memory_id": obj.get("memory_id"),
+            "quote": obj.get("quote"),
+            "relevance": obj.get("relevance"),
+            "timestamp": obj.get("timestamp")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/observation_input.py

@@ -0,0 +1,89 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ObservationInput(BaseModel):
+    """
+    Input model for a single observation.
+    """ # noqa: E501
+    title: StrictStr = Field(description="Short title/header for the observation")
+    content: StrictStr = Field(description="Content of the observation")
+    __properties: ClassVar[List[str]] = ["title", "content"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ObservationInput from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ObservationInput from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "title": obj.get("title"),
+            "content": obj.get("content")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/operation_response.py

@@ -29,7 +29,7 @@ class OperationResponse(BaseModel):
     id: StrictStr
     task_type: StrictStr
     items_count: StrictInt
-    document_id: Optional[StrictStr]
+    document_id: Optional[StrictStr] = None
     created_at: StrictStr
     status: StrictStr
     error_message: Optional[StrictStr]

hindsight-clients/python/hindsight_client_api/models/operation_status_response.py

@@ -0,0 +1,131 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, StrictStr, field_validator
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class OperationStatusResponse(BaseModel):
+    """
+    Response model for getting a single operation status.
+    """ # noqa: E501
+    operation_id: StrictStr
+    status: StrictStr
+    operation_type: Optional[StrictStr] = None
+    created_at: Optional[StrictStr] = None
+    updated_at: Optional[StrictStr] = None
+    completed_at: Optional[StrictStr] = None
+    error_message: Optional[StrictStr] = None
+    __properties: ClassVar[List[str]] = ["operation_id", "status", "operation_type", "created_at", "updated_at", "completed_at", "error_message"]
+
+    @field_validator('status')
+    def status_validate_enum(cls, value):
+        """Validates the enum"""
+        if value not in set(['pending', 'completed', 'failed', 'not_found']):
+            raise ValueError("must be one of enum values ('pending', 'completed', 'failed', 'not_found')")
+        return value
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of OperationStatusResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # set to None if operation_type (nullable) is None
+        # and model_fields_set contains the field
+        if self.operation_type is None and "operation_type" in self.model_fields_set:
+            _dict['operation_type'] = None
+
+        # set to None if created_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.created_at is None and "created_at" in self.model_fields_set:
+            _dict['created_at'] = None
+
+        # set to None if updated_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.updated_at is None and "updated_at" in self.model_fields_set:
+            _dict['updated_at'] = None
+
+        # set to None if completed_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.completed_at is None and "completed_at" in self.model_fields_set:
+            _dict['completed_at'] = None
+
+        # set to None if error_message (nullable) is None
+        # and model_fields_set contains the field
+        if self.error_message is None and "error_message" in self.model_fields_set:
+            _dict['error_message'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of OperationStatusResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "operation_id": obj.get("operation_id"),
+            "status": obj.get("status"),
+            "operation_type": obj.get("operation_type"),
+            "created_at": obj.get("created_at"),
+            "updated_at": obj.get("updated_at"),
+            "completed_at": obj.get("completed_at"),
+            "error_message": obj.get("error_message")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/operations_list_response.py

@@ -17,7 +17,7 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictStr
+from pydantic import BaseModel, ConfigDict, StrictInt, StrictStr
 from typing import Any, ClassVar, Dict, List
 from hindsight_client_api.models.operation_response import OperationResponse
 from typing import Optional, Set
@@ -28,8 +28,9 @@ class OperationsListResponse(BaseModel):
     Response model for list operations endpoint.
     """ # noqa: E501
     bank_id: StrictStr
+    total: StrictInt
     operations: List[OperationResponse]
-    __properties: ClassVar[List[str]] = ["bank_id", "operations"]
+    __properties: ClassVar[List[str]] = ["bank_id", "total", "operations"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -90,6 +91,7 @@ class OperationsListResponse(BaseModel):
 
         _obj = cls.model_validate({
             "bank_id": obj.get("bank_id"),
+            "total": obj.get("total"),
             "operations": [OperationResponse.from_dict(_item) for _item in obj["operations"]] if obj.get("operations") is not None else None
         })
         return _obj