fix

* doc: update expired Slack invite link

* feat: add cloud mode to skill installer for team memory sharing

Adds support for Hindsight Cloud in the skill installer, enabling teams
to share memories about a codebase. Changes include:

- Add `--mode cloud` option to get-skill installer
- Install hindsight CLI binary for cloud mode (via get-cli)
- Configure ~/.hindsight/config with API URL and key
- Generate cloud-specific SKILL.md with team-aware guidance
- Distinguish between project conventions and individual preferences
- Update skills.md documentation with cloud setup instructions

Cloud mode workflow:
1. Team admin creates a bank in Hindsight Cloud
2. Each developer runs: curl ... | bash -s -- --mode cloud
3. All team members share the same memory bank
4. Knowledge retained by one member benefits everyone

.pgbouncer/pgbouncer.ini

@@ -0,0 +1,39 @@
+[databases]
+; Connect to pg0 on port 5433
+; The actual pg0 database is called "hindsight"
+hindsight = host=127.0.0.1 port=5433 dbname=hindsight user=hindsight password=hindsight
+
+[pgbouncer]
+listen_addr = 127.0.0.1
+listen_port = 6432
+
+; Use md5 authentication (matches pg0's auth)
+auth_type = md5
+auth_file = /Users/nicoloboschi/dev/memory-poc/.pgbouncer/userlist.txt
+
+; Transaction pooling mode (recommended for hindsight)
+pool_mode = transaction
+
+; Reset connection state after each transaction
+server_reset_query = DISCARD ALL
+
+; Pool sizing
+default_pool_size = 20
+max_client_conn = 200
+min_pool_size = 5
+
+; Timeouts
+server_idle_timeout = 600
+server_lifetime = 3600
+query_timeout = 120
+
+; Logging
+log_connections = 1
+log_disconnections = 1
+log_pooler_errors = 1
+
+; Stats
+stats_period = 60
+
+; Admin console
+admin_users = admin

.pgbouncer/userlist.txt

@@ -0,0 +1,2 @@
+"hindsight" "md5d842ccb6249bcd3c53b2f648378092a6"
+"admin" ""

CLAUDE.md

@@ -174,6 +174,25 @@ This runs the same checks as the pre-commit hook (Ruff for Python, ESLint/Pretti
 - Multi-bank queries are client responsibility to orchestrate
 - Disposition traits only affect reflect, not recall
 
+### Control Plane API Routes
+
+When adding or modifying parameters in the dataplane API (hindsight-api), you must also update the control plane routes that proxy to it:
+
+1. **API Routes** (`hindsight-control-plane/src/app/api/`):
+   - `recall/route.ts` - proxies to `/v1/default/banks/{bank_id}/memories/recall`
+   - `reflect/route.ts` - proxies to `/v1/default/banks/{bank_id}/reflect`
+   - `memories/retain/route.ts` - proxies to `/v1/default/banks/{bank_id}/memories/retain`
+   - Other routes follow the same pattern
+
+2. **Client types** (`hindsight-control-plane/src/lib/api.ts`):
+   - Update the TypeScript type definitions for `recall()`, `reflect()`, `retain()` etc.
+
+3. **Checklist when adding new API parameters**:
+   - Add parameter extraction in the route handler (destructure from `body`)
+   - Pass the parameter to the SDK call
+   - Update the client type definition in `lib/api.ts`
+   - Update any UI components that need to use the new parameter
+
 ### Python Style
 - Python 3.11+, type hints required
 - Async throughout (asyncpg, async FastAPI)

README.md

@@ -5,7 +5,7 @@
 [Documentation](https://hindsight.vectorize.io) • [Paper](https://arxiv.org/abs/2512.12818) • [Cookbook](https://hindsight.vectorize.io/cookbook) • [Hindsight Cloud](https://vectorize.io/hindsight/cloud)
 
 [![CI](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml/badge.svg)](https://github.com/vectorize-io/hindsight/actions/workflows/release.yml)
-[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3klo21kua-VUCC_zHP5rIcXFB1_5yw6A)
+[![Slack Community](https://img.shields.io/badge/Slack-Join%20Community-4A154B?logo=slack)](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 ![PyPI - Downloads](https://img.shields.io/pypi/dm/hindsight-api?label=PyPI)
 ![NPM Downloads](https://img.shields.io/npm/dm/%40vectorize-io%2Fhindsight-client?logoColor=orange&label=NPM&color=blue&link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40vectorize-io%2Fhindsight-client)
@@ -242,7 +242,7 @@ client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 - [CLI](https://hindsight.vectorize.io/sdks/cli)
 
 **Community:**
-- [Slack](https://join.slack.com/t/hindsight-space/shared_invite/zt-3klo21kua-VUCC_zHP5rIcXFB1_5yw6A)
+- [Slack](https://join.slack.com/t/hindsight-space/shared_invite/zt-3nhbm4w29-LeSJ5Ixi6j8PdiYOCPlOgg)
 - [GitHub Issues](https://github.com/vectorize-io/hindsight/issues)
 
 ---

helm/hindsight/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: hindsight
 description: Hindsight helm chart
 type: application
-version: 0.2.1
-appVersion: "0.2.1"
+version: 0.3.0
+appVersion: "0.3.0"
 keywords:
   - ai
   - memory

hindsight-api/hindsight_api/alembic/versions/g2a3b4c5d6e7_add_tags_column.py

@@ -0,0 +1,48 @@
+"""add_tags_column
+
+Revision ID: g2a3b4c5d6e7
+Revises: f1a2b3c4d5e6
+Create Date: 2025-01-13
+
+Add tags column to memory_units and documents tables for visibility scoping.
+Tags enable filtering memories by scope (e.g., user IDs, session IDs) during recall/reflect.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "g2a3b4c5d6e7"
+down_revision: str | Sequence[str] | None = "f1a2b3c4d5e6"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (e.g., 'tenant_x.' or '' for public)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Add tags column to memory_units and documents tables."""
+    schema = _get_schema_prefix()
+
+    # Add tags column to memory_units table
+    op.execute(f"ALTER TABLE {schema}memory_units ADD COLUMN IF NOT EXISTS tags VARCHAR[] NOT NULL DEFAULT '{{}}'")
+
+    # Create GIN index for efficient array containment queries (tags && ARRAY['x'])
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_memory_units_tags ON {schema}memory_units USING GIN (tags)")
+
+    # Add tags column to documents table for document-level tags
+    op.execute(f"ALTER TABLE {schema}documents ADD COLUMN IF NOT EXISTS tags VARCHAR[] NOT NULL DEFAULT '{{}}'")
+
+
+def downgrade() -> None:
+    """Remove tags columns and index."""
+    schema = _get_schema_prefix()
+
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_tags")
+    op.execute(f"ALTER TABLE {schema}memory_units DROP COLUMN IF EXISTS tags")
+    op.execute(f"ALTER TABLE {schema}documents DROP COLUMN IF EXISTS tags")

hindsight-api/hindsight_api/alembic/versions/h3c4d5e6f7g8_mental_models_v4.py

@@ -0,0 +1,112 @@
+"""mental_models_v4
+
+Revision ID: h3c4d5e6f7g8
+Revises: g2a3b4c5d6e7
+Create Date: 2026-01-08 00:00:00.000000
+
+This migration implements the v4 mental models system:
+1. Deletes existing observation memory_units (observations now in mental models)
+2. Adds mission column to banks (replacing background)
+3. Creates mental_models table with final schema
+
+Mental models can reference entities when an entity is "promoted" to a mental model.
+Summary content is stored as JSONB observations with per-observation fact attribution.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "h3c4d5e6f7g8"
+down_revision: str | Sequence[str] | None = "g2a3b4c5d6e7"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Apply mental models v4 changes."""
+    schema = _get_schema_prefix()
+
+    # Step 1: Delete observation memory_units (cascades to unit_entities links)
+    # Observations are now handled through mental models, not memory_units
+    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'observation'")
+
+    # Step 2: Drop observation-specific index (if it exists)
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_memory_units_observation_date")
+
+    # Step 3: Add mission column to banks (replacing background)
+    op.execute(f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS mission TEXT")
+
+    # Migrate: copy background to mission if background column exists
+    # Use DO block to check column existence first (idempotent for re-runs)
+    schema_name = context.config.get_main_option("target_schema") or "public"
+    op.execute(f"""
+        DO $$
+        BEGIN
+            IF EXISTS (
+                SELECT 1 FROM information_schema.columns
+                WHERE table_schema = '{schema_name}' AND table_name = 'banks' AND column_name = 'background'
+            ) THEN
+                UPDATE {schema}banks
+                SET mission = background
+                WHERE mission IS NULL;
+            END IF;
+        END $$;
+    """)
+
+    # Remove background column (replaced by mission)
+    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS background")
+
+    # Step 4: Create mental_models table with final v4 schema (if not exists)
+    op.execute(f"""
+        CREATE TABLE IF NOT EXISTS {schema}mental_models (
+            id VARCHAR(64) NOT NULL,
+            bank_id VARCHAR(64) NOT NULL,
+            subtype VARCHAR(32) NOT NULL,
+            name VARCHAR(256) NOT NULL,
+            description TEXT NOT NULL,
+            entity_id UUID,
+            observations JSONB DEFAULT '{{"observations": []}}'::jsonb,
+            links VARCHAR[],
+            tags VARCHAR[] DEFAULT '{{}}',
+            last_updated TIMESTAMP WITH TIME ZONE,
+            created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(),
+            PRIMARY KEY (id, bank_id),
+            FOREIGN KEY (bank_id) REFERENCES {schema}banks(bank_id) ON DELETE CASCADE,
+            FOREIGN KEY (entity_id) REFERENCES {schema}entities(id) ON DELETE SET NULL,
+            CONSTRAINT ck_mental_models_subtype CHECK (subtype IN ('structural', 'emergent', 'pinned', 'learned'))
+        )
+    """)
+
+    # Step 5: Create indexes for efficient queries (if not exist)
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_bank_id ON {schema}mental_models(bank_id)")
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_subtype ON {schema}mental_models(bank_id, subtype)")
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_entity_id ON {schema}mental_models(entity_id)")
+    # GIN index for efficient tags array filtering
+    op.execute(f"CREATE INDEX IF NOT EXISTS idx_mental_models_tags ON {schema}mental_models USING GIN(tags)")
+
+
+def downgrade() -> None:
+    """Revert mental models v4 changes."""
+    schema = _get_schema_prefix()
+
+    # Drop mental_models table (cascades to indexes)
+    op.execute(f"DROP TABLE IF EXISTS {schema}mental_models CASCADE")
+
+    # Add back background column to banks
+    op.execute(f"ALTER TABLE {schema}banks ADD COLUMN IF NOT EXISTS background TEXT")
+
+    # Migrate mission back to background
+    op.execute(f"UPDATE {schema}banks SET background = mission WHERE background IS NULL")
+
+    # Remove mission column
+    op.execute(f"ALTER TABLE {schema}banks DROP COLUMN IF EXISTS mission")
+
+    # Note: Cannot restore deleted observations - they are lost on downgrade

hindsight-api/hindsight_api/alembic/versions/i4d5e6f7g8h9_delete_opinions.py

@@ -0,0 +1,41 @@
+"""delete_opinions
+
+Revision ID: i4d5e6f7g8h9
+Revises: h3c4d5e6f7g8
+Create Date: 2026-01-15 00:00:00.000000
+
+This migration removes opinion facts from memory_units.
+Opinions are no longer a separate fact type - they are now represented
+through mental model observations with confidence scores.
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+# revision identifiers, used by Alembic.
+revision: str = "i4d5e6f7g8h9"
+down_revision: str | Sequence[str] | None = "h3c4d5e6f7g8"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Delete opinion memory_units."""
+    schema = _get_schema_prefix()
+
+    # Delete opinion memory_units (cascades to unit_entities links)
+    # Opinions are now handled through mental model observations
+    op.execute(f"DELETE FROM {schema}memory_units WHERE fact_type = 'opinion'")
+
+
+def downgrade() -> None:
+    """Cannot restore deleted opinions."""
+    # Note: Cannot restore deleted opinions - they are lost on downgrade
+    pass

hindsight-api/hindsight_api/api/mcp.py

@@ -196,7 +196,7 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
         Each bank is an isolated memory store (like a separate "brain").
 
         Returns:
-            JSON list of banks with their IDs, names, dispositions, and backgrounds.
+            JSON list of banks with their IDs, names, dispositions, and missions.
         """
         try:
             banks = await memory.list_banks(request_context=RequestContext())
@@ -206,7 +206,7 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
             return f'{{"error": "{e}", "banks": []}}'
 
     @mcp.tool()
-    async def create_bank(bank_id: str, name: str | None = None, background: str | None = None) -> str:
+    async def create_bank(bank_id: str, name: str | None = None, mission: str | None = None) -> str:
         """
         Create a new memory bank or get an existing one.
 
@@ -216,18 +216,18 @@ def create_mcp_server(memory: MemoryEngine) -> FastMCP:
         Args:
             bank_id: Unique identifier for the bank (e.g., 'user-123', 'agent-alpha')
             name: Optional human-friendly name for the bank
-            background: Optional background context about the bank's owner/purpose
+            mission: Optional mission describing who the agent is and what they're trying to accomplish
         """
         try:
             # get_bank_profile auto-creates bank if it doesn't exist
             profile = await memory.get_bank_profile(bank_id, request_context=RequestContext())
 
-            # Update name/background if provided
-            if name is not None or background is not None:
+            # Update name/mission if provided
+            if name is not None or mission is not None:
                 await memory.update_bank(
                     bank_id,
                     name=name,
-                    background=background,
+                    mission=mission,
                     request_context=RequestContext(),
                 )
                 # Fetch updated profile

hindsight-api/hindsight_api/config.py

@@ -41,10 +41,19 @@ ENV_EMBEDDINGS_LOCAL_MODEL = "HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL"
 ENV_EMBEDDINGS_TEI_URL = "HINDSIGHT_API_EMBEDDINGS_TEI_URL"
 ENV_EMBEDDINGS_OPENAI_API_KEY = "HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY"
 ENV_EMBEDDINGS_OPENAI_MODEL = "HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL"
+ENV_EMBEDDINGS_OPENAI_BASE_URL = "HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL"
 
 ENV_COHERE_API_KEY = "HINDSIGHT_API_COHERE_API_KEY"
 ENV_EMBEDDINGS_COHERE_MODEL = "HINDSIGHT_API_EMBEDDINGS_COHERE_MODEL"
+ENV_EMBEDDINGS_COHERE_BASE_URL = "HINDSIGHT_API_EMBEDDINGS_COHERE_BASE_URL"
 ENV_RERANKER_COHERE_MODEL = "HINDSIGHT_API_RERANKER_COHERE_MODEL"
+ENV_RERANKER_COHERE_BASE_URL = "HINDSIGHT_API_RERANKER_COHERE_BASE_URL"
+
+# LiteLLM gateway configuration (for embeddings and reranker via LiteLLM proxy)
+ENV_LITELLM_API_BASE = "HINDSIGHT_API_LITELLM_API_BASE"
+ENV_LITELLM_API_KEY = "HINDSIGHT_API_LITELLM_API_KEY"
+ENV_EMBEDDINGS_LITELLM_MODEL = "HINDSIGHT_API_EMBEDDINGS_LITELLM_MODEL"
+ENV_RERANKER_LITELLM_MODEL = "HINDSIGHT_API_RERANKER_LITELLM_MODEL"
 
 ENV_RERANKER_PROVIDER = "HINDSIGHT_API_RERANKER_PROVIDER"
 ENV_RERANKER_LOCAL_MODEL = "HINDSIGHT_API_RERANKER_LOCAL_MODEL"
@@ -67,6 +76,7 @@ ENV_RECALL_MAX_CONCURRENT = "HINDSIGHT_API_RECALL_MAX_CONCURRENT"
 ENV_RECALL_CONNECTION_BUDGET = "HINDSIGHT_API_RECALL_CONNECTION_BUDGET"
 ENV_MCP_LOCAL_BANK_ID = "HINDSIGHT_API_MCP_LOCAL_BANK_ID"
 ENV_MCP_INSTRUCTIONS = "HINDSIGHT_API_MCP_INSTRUCTIONS"
+ENV_MENTAL_MODEL_REFRESH_CONCURRENCY = "HINDSIGHT_API_MENTAL_MODEL_REFRESH_CONCURRENCY"
 
 # Observation thresholds
 ENV_OBSERVATION_MIN_FACTS = "HINDSIGHT_API_OBSERVATION_MIN_FACTS"
@@ -97,6 +107,9 @@ ENV_TASK_BACKEND = "HINDSIGHT_API_TASK_BACKEND"
 ENV_TASK_BACKEND_MEMORY_BATCH_SIZE = "HINDSIGHT_API_TASK_BACKEND_MEMORY_BATCH_SIZE"
 ENV_TASK_BACKEND_MEMORY_BATCH_INTERVAL = "HINDSIGHT_API_TASK_BACKEND_MEMORY_BATCH_INTERVAL"
 
+# Reflect agent settings
+ENV_REFLECT_MAX_ITERATIONS = "HINDSIGHT_API_REFLECT_MAX_ITERATIONS"
+
 # Default values
 DEFAULT_DATABASE_URL = "pg0"
 DEFAULT_LLM_PROVIDER = "openai"
@@ -121,6 +134,11 @@ DEFAULT_RERANKER_FLASHRANK_CACHE_DIR = None  # Use default cache directory
 DEFAULT_EMBEDDINGS_COHERE_MODEL = "embed-english-v3.0"
 DEFAULT_RERANKER_COHERE_MODEL = "rerank-english-v3.0"
 
+# LiteLLM defaults
+DEFAULT_LITELLM_API_BASE = "http://localhost:4000"
+DEFAULT_EMBEDDINGS_LITELLM_MODEL = "text-embedding-3-small"
+DEFAULT_RERANKER_LITELLM_MODEL = "cohere/rerank-english-v3.0"
+
 DEFAULT_HOST = "0.0.0.0"
 DEFAULT_PORT = 8888
 DEFAULT_LOG_LEVEL = "info"
@@ -131,6 +149,7 @@ DEFAULT_MPFP_TOP_K_NEIGHBORS = 20  # Fan-out limit per node in MPFP graph traver
 DEFAULT_RECALL_MAX_CONCURRENT = 32  # Max concurrent recall operations per worker
 DEFAULT_RECALL_CONNECTION_BUDGET = 4  # Max concurrent DB connections per recall operation
 DEFAULT_MCP_LOCAL_BANK_ID = "mcp"
+DEFAULT_MENTAL_MODEL_REFRESH_CONCURRENCY = 8  # Max concurrent mental model refreshes
 
 # Observation thresholds
 DEFAULT_OBSERVATION_MIN_FACTS = 5  # Min facts required to generate entity observations
@@ -158,6 +177,9 @@ DEFAULT_TASK_BACKEND = "memory"  # Options: "memory", "noop"
 DEFAULT_TASK_BACKEND_MEMORY_BATCH_SIZE = 10
 DEFAULT_TASK_BACKEND_MEMORY_BATCH_INTERVAL = 1.0  # seconds
 
+# Reflect agent settings
+DEFAULT_REFLECT_MAX_ITERATIONS = 10  # Max tool call iterations before forcing response
+
 # Default MCP tool descriptions (can be customized via env vars)
 DEFAULT_MCP_RETAIN_DESCRIPTION = """Store important information to long-term memory.
 
@@ -224,6 +246,8 @@ class HindsightConfig:
     embeddings_provider: str
     embeddings_local_model: str
     embeddings_tei_url: str | None
+    embeddings_openai_base_url: str | None
+    embeddings_cohere_base_url: str | None
 
     # Reranker
     reranker_provider: str
@@ -232,6 +256,7 @@ class HindsightConfig:
     reranker_tei_batch_size: int
     reranker_tei_max_concurrent: int
     reranker_max_candidates: int
+    reranker_cohere_base_url: str | None
 
     # Server
     host: str
@@ -244,6 +269,7 @@ class HindsightConfig:
     mpfp_top_k_neighbors: int
     recall_max_concurrent: int
     recall_connection_budget: int
+    mental_model_refresh_concurrency: int
 
     # Observation thresholds
     observation_min_facts: int
@@ -274,6 +300,9 @@ class HindsightConfig:
     task_backend_memory_batch_size: int
     task_backend_memory_batch_interval: float
 
+    # Reflect agent settings
+    reflect_max_iterations: int
+
     @classmethod
     def from_env(cls) -> "HindsightConfig":
         """Create configuration from environment variables."""
@@ -300,6 +329,8 @@ class HindsightConfig:
             embeddings_provider=os.getenv(ENV_EMBEDDINGS_PROVIDER, DEFAULT_EMBEDDINGS_PROVIDER),
             embeddings_local_model=os.getenv(ENV_EMBEDDINGS_LOCAL_MODEL, DEFAULT_EMBEDDINGS_LOCAL_MODEL),
             embeddings_tei_url=os.getenv(ENV_EMBEDDINGS_TEI_URL),
+            embeddings_openai_base_url=os.getenv(ENV_EMBEDDINGS_OPENAI_BASE_URL) or None,
+            embeddings_cohere_base_url=os.getenv(ENV_EMBEDDINGS_COHERE_BASE_URL) or None,
             # Reranker
             reranker_provider=os.getenv(ENV_RERANKER_PROVIDER, DEFAULT_RERANKER_PROVIDER),
             reranker_local_model=os.getenv(ENV_RERANKER_LOCAL_MODEL, DEFAULT_RERANKER_LOCAL_MODEL),
@@ -309,6 +340,7 @@ class HindsightConfig:
                 os.getenv(ENV_RERANKER_TEI_MAX_CONCURRENT, str(DEFAULT_RERANKER_TEI_MAX_CONCURRENT))
             ),
             reranker_max_candidates=int(os.getenv(ENV_RERANKER_MAX_CANDIDATES, str(DEFAULT_RERANKER_MAX_CANDIDATES))),
+            reranker_cohere_base_url=os.getenv(ENV_RERANKER_COHERE_BASE_URL) or None,
             # Server
             host=os.getenv(ENV_HOST, DEFAULT_HOST),
             port=int(os.getenv(ENV_PORT, DEFAULT_PORT)),
@@ -321,6 +353,9 @@ class HindsightConfig:
             recall_connection_budget=int(
                 os.getenv(ENV_RECALL_CONNECTION_BUDGET, str(DEFAULT_RECALL_CONNECTION_BUDGET))
             ),
+            mental_model_refresh_concurrency=int(
+                os.getenv(ENV_MENTAL_MODEL_REFRESH_CONCURRENCY, str(DEFAULT_MENTAL_MODEL_REFRESH_CONCURRENCY))
+            ),
             # Optimization flags
             skip_llm_verification=os.getenv(ENV_SKIP_LLM_VERIFICATION, "false").lower() == "true",
             lazy_reranker=os.getenv(ENV_LAZY_RERANKER, "false").lower() == "true",
@@ -360,6 +395,8 @@ class HindsightConfig:
             task_backend_memory_batch_interval=float(
                 os.getenv(ENV_TASK_BACKEND_MEMORY_BATCH_INTERVAL, str(DEFAULT_TASK_BACKEND_MEMORY_BATCH_INTERVAL))
             ),
+            # Reflect agent settings
+            reflect_max_iterations=int(os.getenv(ENV_REFLECT_MAX_ITERATIONS, str(DEFAULT_REFLECT_MAX_ITERATIONS))),
         )
 
     def get_llm_base_url(self) -> str:

hindsight-api/hindsight_api/engine/cross_encoder.py

@@ -15,18 +15,24 @@ from concurrent.futures import ThreadPoolExecutor
 import httpx
 
 from ..config import (
+    DEFAULT_LITELLM_API_BASE,
     DEFAULT_RERANKER_COHERE_MODEL,
     DEFAULT_RERANKER_FLASHRANK_CACHE_DIR,
     DEFAULT_RERANKER_FLASHRANK_MODEL,
+    DEFAULT_RERANKER_LITELLM_MODEL,
     DEFAULT_RERANKER_LOCAL_MAX_CONCURRENT,
     DEFAULT_RERANKER_LOCAL_MODEL,
     DEFAULT_RERANKER_PROVIDER,
     DEFAULT_RERANKER_TEI_BATCH_SIZE,
     DEFAULT_RERANKER_TEI_MAX_CONCURRENT,
     ENV_COHERE_API_KEY,
+    ENV_LITELLM_API_BASE,
+    ENV_LITELLM_API_KEY,
+    ENV_RERANKER_COHERE_BASE_URL,
     ENV_RERANKER_COHERE_MODEL,
     ENV_RERANKER_FLASHRANK_CACHE_DIR,
     ENV_RERANKER_FLASHRANK_MODEL,
+    ENV_RERANKER_LITELLM_MODEL,
     ENV_RERANKER_LOCAL_MAX_CONCURRENT,
     ENV_RERANKER_LOCAL_MODEL,
     ENV_RERANKER_PROVIDER,
@@ -392,6 +398,7 @@ class CohereCrossEncoder(CrossEncoderModel):
         self,
         api_key: str,
         model: str = DEFAULT_RERANKER_COHERE_MODEL,
+        base_url: str | None = None,
         timeout: float = 60.0,
     ):
         """
@@ -400,10 +407,12 @@ class CohereCrossEncoder(CrossEncoderModel):
         Args:
             api_key: Cohere API key
             model: Cohere rerank model name (default: rerank-english-v3.0)
+            base_url: Custom base URL for Cohere-compatible API (e.g., Azure-hosted endpoint)
             timeout: Request timeout in seconds (default: 60.0)
         """
         self.api_key = api_key
         self.model = model
+        self.base_url = base_url
         self.timeout = timeout
         self._client = None
 
@@ -421,8 +430,14 @@ class CohereCrossEncoder(CrossEncoderModel):
         except ImportError:
             raise ImportError("cohere is required for CohereCrossEncoder. Install it with: pip install cohere")
 
-        logger.info(f"Reranker: initializing Cohere provider with model {self.model}")
-        self._client = cohere.Client(api_key=self.api_key, timeout=self.timeout)
+        base_url_msg = f" at {self.base_url}" if self.base_url else ""
+        logger.info(f"Reranker: initializing Cohere provider with model {self.model}{base_url_msg}")
+
+        # Build client kwargs, only including base_url if set (for Azure or custom endpoints)
+        client_kwargs = {"api_key": self.api_key, "timeout": self.timeout}
+        if self.base_url:
+            client_kwargs["base_url"] = self.base_url
+        self._client = cohere.Client(**client_kwargs)
         logger.info("Reranker: Cohere provider initialized")
 
     async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
@@ -641,6 +656,116 @@ class FlashRankCrossEncoder(CrossEncoderModel):
         return await loop.run_in_executor(FlashRankCrossEncoder._executor, self._predict_sync, pairs)
 
 
+class LiteLLMCrossEncoder(CrossEncoderModel):
+    """
+    LiteLLM cross-encoder implementation using LiteLLM proxy's /rerank endpoint.
+
+    LiteLLM provides a unified interface for multiple reranking providers via
+    the Cohere-compatible /rerank endpoint.
+    See: https://docs.litellm.ai/docs/rerank
+
+    Supported providers via LiteLLM:
+    - Cohere (rerank-english-v3.0, etc.) - prefix with cohere/
+    - Together AI - prefix with together_ai/
+    - Azure AI - prefix with azure_ai/
+    - Jina AI - prefix with jina_ai/
+    - AWS Bedrock - prefix with bedrock/
+    - Voyage AI - prefix with voyage/
+    """
+
+    def __init__(
+        self,
+        api_base: str = DEFAULT_LITELLM_API_BASE,
+        api_key: str | None = None,
+        model: str = DEFAULT_RERANKER_LITELLM_MODEL,
+        timeout: float = 60.0,
+    ):
+        """
+        Initialize LiteLLM cross-encoder client.
+
+        Args:
+            api_base: Base URL of the LiteLLM proxy (default: http://localhost:4000)
+            api_key: API key for the LiteLLM proxy (optional, depends on proxy config)
+            model: Reranking model name (default: cohere/rerank-english-v3.0)
+                   Use provider prefix (e.g., cohere/, together_ai/, voyage/)
+            timeout: Request timeout in seconds (default: 60.0)
+        """
+        self.api_base = api_base.rstrip("/")
+        self.api_key = api_key
+        self.model = model
+        self.timeout = timeout
+        self._async_client: httpx.AsyncClient | None = None
+
+    @property
+    def provider_name(self) -> str:
+        return "litellm"
+
+    async def initialize(self) -> None:
+        """Initialize the async HTTP client."""
+        if self._async_client is not None:
+            return
+
+        logger.info(f"Reranker: initializing LiteLLM provider at {self.api_base} with model {self.model}")
+
+        headers = {"Content-Type": "application/json"}
+        if self.api_key:
+            headers["Authorization"] = f"Bearer {self.api_key}"
+
+        self._async_client = httpx.AsyncClient(timeout=self.timeout, headers=headers)
+        logger.info("Reranker: LiteLLM provider initialized")
+
+    async def predict(self, pairs: list[tuple[str, str]]) -> list[float]:
+        """
+        Score query-document pairs using the LiteLLM proxy's /rerank endpoint.
+
+        Args:
+            pairs: List of (query, document) tuples to score
+
+        Returns:
+            List of relevance scores
+        """
+        if self._async_client is None:
+            raise RuntimeError("Reranker not initialized. Call initialize() first.")
+
+        if not pairs:
+            return []
+
+        # Group pairs by query (LiteLLM rerank expects one query with multiple documents)
+        query_groups: dict[str, list[tuple[int, str]]] = {}
+        for idx, (query, text) in enumerate(pairs):
+            if query not in query_groups:
+                query_groups[query] = []
+            query_groups[query].append((idx, text))
+
+        all_scores = [0.0] * len(pairs)
+
+        for query, indexed_texts in query_groups.items():
+            texts = [text for _, text in indexed_texts]
+            indices = [idx for idx, _ in indexed_texts]
+
+            # LiteLLM /rerank follows Cohere API format
+            response = await self._async_client.post(
+                f"{self.api_base}/rerank",
+                json={
+                    "model": self.model,
+                    "query": query,
+                    "documents": texts,
+                    "top_n": len(texts),  # Return all scores
+                },
+            )
+            response.raise_for_status()
+            result = response.json()
+
+            # Map scores back to original positions
+            # Response format: {"results": [{"index": 0, "relevance_score": 0.9}, ...]}
+            for item in result.get("results", []):
+                original_idx = item["index"]
+                score = item.get("relevance_score", item.get("score", 0.0))
+                all_scores[indices[original_idx]] = score
+
+        return all_scores
+
+
 def create_cross_encoder_from_env() -> CrossEncoderModel:
     """
     Create a CrossEncoderModel instance based on environment variables.
@@ -671,14 +796,20 @@ def create_cross_encoder_from_env() -> CrossEncoderModel:
         if not api_key:
             raise ValueError(f"{ENV_COHERE_API_KEY} is required when {ENV_RERANKER_PROVIDER} is 'cohere'")
         model = os.environ.get(ENV_RERANKER_COHERE_MODEL, DEFAULT_RERANKER_COHERE_MODEL)
-        return CohereCrossEncoder(api_key=api_key, model=model)
+        base_url = os.environ.get(ENV_RERANKER_COHERE_BASE_URL) or None
+        return CohereCrossEncoder(api_key=api_key, model=model, base_url=base_url)
     elif provider == "flashrank":
         model = os.environ.get(ENV_RERANKER_FLASHRANK_MODEL, DEFAULT_RERANKER_FLASHRANK_MODEL)
         cache_dir = os.environ.get(ENV_RERANKER_FLASHRANK_CACHE_DIR, DEFAULT_RERANKER_FLASHRANK_CACHE_DIR)
         return FlashRankCrossEncoder(model_name=model, cache_dir=cache_dir)
+    elif provider == "litellm":
+        api_base = os.environ.get(ENV_LITELLM_API_BASE, DEFAULT_LITELLM_API_BASE)
+        api_key = os.environ.get(ENV_LITELLM_API_KEY)
+        model = os.environ.get(ENV_RERANKER_LITELLM_MODEL, DEFAULT_RERANKER_LITELLM_MODEL)
+        return LiteLLMCrossEncoder(api_base=api_base, api_key=api_key, model=model)
     elif provider == "rrf":
         return RRFPassthroughCrossEncoder()
     else:
         raise ValueError(
-            f"Unknown reranker provider: {provider}. Supported: 'local', 'tei', 'cohere', 'flashrank', 'rrf'"
+            f"Unknown reranker provider: {provider}. Supported: 'local', 'tei', 'cohere', 'flashrank', 'litellm', 'rrf'"
         )

hindsight-api/hindsight_api/engine/embeddings.py

@@ -17,16 +17,23 @@ import httpx
 
 from ..config import (
     DEFAULT_EMBEDDINGS_COHERE_MODEL,
+    DEFAULT_EMBEDDINGS_LITELLM_MODEL,
     DEFAULT_EMBEDDINGS_LOCAL_MODEL,
     DEFAULT_EMBEDDINGS_OPENAI_MODEL,
     DEFAULT_EMBEDDINGS_PROVIDER,
+    DEFAULT_LITELLM_API_BASE,
     ENV_COHERE_API_KEY,
+    ENV_EMBEDDINGS_COHERE_BASE_URL,
     ENV_EMBEDDINGS_COHERE_MODEL,
+    ENV_EMBEDDINGS_LITELLM_MODEL,
     ENV_EMBEDDINGS_LOCAL_MODEL,
     ENV_EMBEDDINGS_OPENAI_API_KEY,
+    ENV_EMBEDDINGS_OPENAI_BASE_URL,
     ENV_EMBEDDINGS_OPENAI_MODEL,
     ENV_EMBEDDINGS_PROVIDER,
     ENV_EMBEDDINGS_TEI_URL,
+    ENV_LITELLM_API_BASE,
+    ENV_LITELLM_API_KEY,
     ENV_LLM_API_KEY,
 )
 
@@ -322,6 +329,7 @@ class OpenAIEmbeddings(Embeddings):
         self,
         api_key: str,
         model: str = DEFAULT_EMBEDDINGS_OPENAI_MODEL,
+        base_url: str | None = None,
         batch_size: int = 100,
         max_retries: int = 3,
     ):
@@ -331,11 +339,13 @@ class OpenAIEmbeddings(Embeddings):
         Args:
             api_key: OpenAI API key
             model: OpenAI embedding model name (default: text-embedding-3-small)
+            base_url: Custom base URL for OpenAI-compatible API (e.g., Azure OpenAI endpoint)
             batch_size: Maximum batch size for embedding requests (default: 100)
             max_retries: Maximum number of retries for failed requests (default: 3)
         """
         self.api_key = api_key
         self.model = model
+        self.base_url = base_url
         self.batch_size = batch_size
         self.max_retries = max_retries
         self._client = None
@@ -361,8 +371,14 @@ class OpenAIEmbeddings(Embeddings):
         except ImportError:
             raise ImportError("openai is required for OpenAIEmbeddings. Install it with: pip install openai")
 
-        logger.info(f"Embeddings: initializing OpenAI provider with model {self.model}")
-        self._client = OpenAI(api_key=self.api_key, max_retries=self.max_retries)
+        base_url_msg = f" at {self.base_url}" if self.base_url else ""
+        logger.info(f"Embeddings: initializing OpenAI provider with model {self.model}{base_url_msg}")
+
+        # Build client kwargs, only including base_url if set (for Azure or custom endpoints)
+        client_kwargs = {"api_key": self.api_key, "max_retries": self.max_retries}
+        if self.base_url:
+            client_kwargs["base_url"] = self.base_url
+        self._client = OpenAI(**client_kwargs)
 
         # Try to get dimension from known models, otherwise do a test embedding
         if self.model in self.MODEL_DIMENSIONS:
@@ -435,6 +451,7 @@ class CohereEmbeddings(Embeddings):
         self,
         api_key: str,
         model: str = DEFAULT_EMBEDDINGS_COHERE_MODEL,
+        base_url: str | None = None,
         batch_size: int = 96,
         timeout: float = 60.0,
         input_type: str = "search_document",
@@ -445,6 +462,7 @@ class CohereEmbeddings(Embeddings):
         Args:
             api_key: Cohere API key
             model: Cohere embedding model name (default: embed-english-v3.0)
+            base_url: Custom base URL for Cohere-compatible API (e.g., Azure-hosted endpoint)
             batch_size: Maximum batch size for embedding requests (default: 96, Cohere's limit)
             timeout: Request timeout in seconds (default: 60.0)
             input_type: Input type for embeddings (default: search_document).
@@ -452,6 +470,7 @@ class CohereEmbeddings(Embeddings):
         """
         self.api_key = api_key
         self.model = model
+        self.base_url = base_url
         self.batch_size = batch_size
         self.timeout = timeout
         self.input_type = input_type
@@ -478,8 +497,14 @@ class CohereEmbeddings(Embeddings):
         except ImportError:
             raise ImportError("cohere is required for CohereEmbeddings. Install it with: pip install cohere")
 
-        logger.info(f"Embeddings: initializing Cohere provider with model {self.model}")
-        self._client = cohere.Client(api_key=self.api_key, timeout=self.timeout)
+        base_url_msg = f" at {self.base_url}" if self.base_url else ""
+        logger.info(f"Embeddings: initializing Cohere provider with model {self.model}{base_url_msg}")
+
+        # Build client kwargs, only including base_url if set (for Azure or custom endpoints)
+        client_kwargs = {"api_key": self.api_key, "timeout": self.timeout}
+        if self.base_url:
+            client_kwargs["base_url"] = self.base_url
+        self._client = cohere.Client(**client_kwargs)
 
         # Try to get dimension from known models, otherwise do a test embedding
         if self.model in self.MODEL_DIMENSIONS:
@@ -529,6 +554,123 @@ class CohereEmbeddings(Embeddings):
         return all_embeddings
 
 
+class LiteLLMEmbeddings(Embeddings):
+    """
+    LiteLLM embeddings implementation using LiteLLM proxy's /embeddings endpoint.
+
+    LiteLLM provides a unified interface for multiple embedding providers.
+    The proxy exposes an OpenAI-compatible /embeddings endpoint.
+    See: https://docs.litellm.ai/docs/embedding/supported_embedding
+
+    Supported providers via LiteLLM:
+    - OpenAI (text-embedding-3-small, text-embedding-ada-002, etc.)
+    - Cohere (embed-english-v3.0, etc.) - prefix with cohere/
+    - Vertex AI (textembedding-gecko, etc.) - prefix with vertex_ai/
+    - HuggingFace, Mistral, Voyage AI, etc.
+
+    The embedding dimension is auto-detected from the model at initialization.
+    """
+
+    def __init__(
+        self,
+        api_base: str = DEFAULT_LITELLM_API_BASE,
+        api_key: str | None = None,
+        model: str = DEFAULT_EMBEDDINGS_LITELLM_MODEL,
+        batch_size: int = 100,
+        timeout: float = 60.0,
+    ):
+        """
+        Initialize LiteLLM embeddings client.
+
+        Args:
+            api_base: Base URL of the LiteLLM proxy (default: http://localhost:4000)
+            api_key: API key for the LiteLLM proxy (optional, depends on proxy config)
+            model: Embedding model name (default: text-embedding-3-small)
+                   Use provider prefix for non-OpenAI models (e.g., cohere/embed-english-v3.0)
+            batch_size: Maximum batch size for embedding requests (default: 100)
+            timeout: Request timeout in seconds (default: 60.0)
+        """
+        self.api_base = api_base.rstrip("/")
+        self.api_key = api_key
+        self.model = model
+        self.batch_size = batch_size
+        self.timeout = timeout
+        self._client: httpx.Client | None = None
+        self._dimension: int | None = None
+
+    @property
+    def provider_name(self) -> str:
+        return "litellm"
+
+    @property
+    def dimension(self) -> int:
+        if self._dimension is None:
+            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
+        return self._dimension
+
+    async def initialize(self) -> None:
+        """Initialize the HTTP client and detect embedding dimension."""
+        if self._client is not None:
+            return
+
+        logger.info(f"Embeddings: initializing LiteLLM provider at {self.api_base} with model {self.model}")
+
+        headers = {"Content-Type": "application/json"}
+        if self.api_key:
+            headers["Authorization"] = f"Bearer {self.api_key}"
+
+        self._client = httpx.Client(timeout=self.timeout, headers=headers)
+
+        # Do a test embedding to detect dimension
+        try:
+            response = self._client.post(
+                f"{self.api_base}/embeddings",
+                json={"model": self.model, "input": ["test"]},
+            )
+            response.raise_for_status()
+            result = response.json()
+            if result.get("data") and len(result["data"]) > 0:
+                self._dimension = len(result["data"][0]["embedding"])
+            logger.info(f"Embeddings: LiteLLM provider initialized (model: {self.model}, dim: {self._dimension})")
+        except httpx.HTTPError as e:
+            raise RuntimeError(f"Failed to connect to LiteLLM proxy at {self.api_base}: {e}")
+
+    def encode(self, texts: list[str]) -> list[list[float]]:
+        """
+        Generate embeddings using the LiteLLM proxy.
+
+        Args:
+            texts: List of text strings to encode
+
+        Returns:
+            List of embedding vectors
+        """
+        if self._client is None:
+            raise RuntimeError("Embeddings not initialized. Call initialize() first.")
+
+        if not texts:
+            return []
+
+        all_embeddings = []
+
+        # Process in batches
+        for i in range(0, len(texts), self.batch_size):
+            batch = texts[i : i + self.batch_size]
+
+            response = self._client.post(
+                f"{self.api_base}/embeddings",
+                json={"model": self.model, "input": batch},
+            )
+            response.raise_for_status()
+            result = response.json()
+
+            # Sort by index to ensure correct order
+            batch_embeddings = sorted(result["data"], key=lambda x: x["index"])
+            all_embeddings.extend([e["embedding"] for e in batch_embeddings])
+
+        return all_embeddings
+
+
 def create_embeddings_from_env() -> Embeddings:
     """
     Create an Embeddings instance based on environment variables.
@@ -558,12 +700,21 @@ def create_embeddings_from_env() -> Embeddings:
                 f"when {ENV_EMBEDDINGS_PROVIDER} is 'openai'"
             )
         model = os.environ.get(ENV_EMBEDDINGS_OPENAI_MODEL, DEFAULT_EMBEDDINGS_OPENAI_MODEL)
-        return OpenAIEmbeddings(api_key=api_key, model=model)
+        base_url = os.environ.get(ENV_EMBEDDINGS_OPENAI_BASE_URL) or None
+        return OpenAIEmbeddings(api_key=api_key, model=model, base_url=base_url)
     elif provider == "cohere":
         api_key = os.environ.get(ENV_COHERE_API_KEY)
         if not api_key:
             raise ValueError(f"{ENV_COHERE_API_KEY} is required when {ENV_EMBEDDINGS_PROVIDER} is 'cohere'")
         model = os.environ.get(ENV_EMBEDDINGS_COHERE_MODEL, DEFAULT_EMBEDDINGS_COHERE_MODEL)
-        return CohereEmbeddings(api_key=api_key, model=model)
+        base_url = os.environ.get(ENV_EMBEDDINGS_COHERE_BASE_URL) or None
+        return CohereEmbeddings(api_key=api_key, model=model, base_url=base_url)
+    elif provider == "litellm":
+        api_base = os.environ.get(ENV_LITELLM_API_BASE, DEFAULT_LITELLM_API_BASE)
+        api_key = os.environ.get(ENV_LITELLM_API_KEY)
+        model = os.environ.get(ENV_EMBEDDINGS_LITELLM_MODEL, DEFAULT_EMBEDDINGS_LITELLM_MODEL)
+        return LiteLLMEmbeddings(api_base=api_base, api_key=api_key, model=model)
     else:
-        raise ValueError(f"Unknown embeddings provider: {provider}. Supported: 'local', 'tei', 'openai', 'cohere'")
+        raise ValueError(
+            f"Unknown embeddings provider: {provider}. Supported: 'local', 'tei', 'openai', 'cohere', 'litellm'"
+        )

hindsight-api/hindsight_api/engine/interface.py

@@ -160,14 +160,14 @@ class MemoryEngineInterface(ABC):
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Get bank profile including disposition and background.
+        Get bank profile including disposition and mission.
 
         Args:
             bank_id: The memory bank ID.
             request_context: Request context for authentication.
 
         Returns:
-            Bank profile dict.
+            Bank profile dict with bank_id, name, disposition, and mission.
         """
         ...
 
@@ -190,25 +190,44 @@ class MemoryEngineInterface(ABC):
         ...
 
     @abstractmethod
-    async def merge_bank_background(
+    async def merge_bank_mission(
         self,
         bank_id: str,
         new_info: str,
         *,
-        update_disposition: bool = True,
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Merge new background information into bank profile.
+        Merge new mission information into bank profile.
 
         Args:
             bank_id: The memory bank ID.
-            new_info: New background information to merge.
-            update_disposition: Whether to infer disposition from background.
+            new_info: New mission information to merge.
             request_context: Request context for authentication.
 
         Returns:
-            Updated background info.
+            Updated mission info.
+        """
+        ...
+
+    @abstractmethod
+    async def set_bank_mission(
+        self,
+        bank_id: str,
+        mission: str,
+        *,
+        request_context: "RequestContext",
+    ) -> dict[str, Any]:
+        """
+        Set the bank's mission (replaces existing).
+
+        Args:
+            bank_id: The memory bank ID.
+            mission: The mission text.
+            request_context: Request context for authentication.
+
+        Returns:
+            Dict with bank_id and mission.
         """
         ...
 
@@ -518,7 +537,7 @@ class MemoryEngineInterface(ABC):
         bank_id: str,
         *,
         request_context: "RequestContext",
-    ) -> list[dict[str, Any]]:
+    ) -> dict[str, Any]:
         """
         List async operations for a bank.
 
@@ -527,7 +546,7 @@ class MemoryEngineInterface(ABC):
             request_context: Request context for authentication.
 
         Returns:
-            List of operation dicts with id, task_type, status, etc.
+            Dict with 'total' (int) and 'operations' (list of operation dicts).
         """
         ...
 
@@ -561,16 +580,16 @@ class MemoryEngineInterface(ABC):
         bank_id: str,
         *,
         name: str | None = None,
-        background: str | None = None,
+        mission: str | None = None,
         request_context: "RequestContext",
     ) -> dict[str, Any]:
         """
-        Update bank name and/or background.
+        Update bank name and/or mission.
 
         Args:
             bank_id: The memory bank ID.
             name: New bank name (optional).
-            background: New background text (optional, replaces existing).
+            mission: New mission text (optional, replaces existing).
             request_context: Request context for authentication.
 
         Returns:

hindsight-api/hindsight_api/engine/llm_wrapper.py

@@ -209,10 +209,10 @@ class LLMProvider:
             OutputTooLongError: If output exceeds token limits.
             Exception: Re-raises API errors after retries exhausted.
         """
-        queue_start_time = time.time()
+        semaphore_start = time.time()
         async with _global_llm_semaphore:
+            semaphore_wait_time = time.time() - semaphore_start
             start_time = time.time()
-            semaphore_wait_time = start_time - queue_start_time
 
             # Handle Mock provider (for testing)
             if self.provider == "mock":
@@ -318,43 +318,44 @@ class LLMProvider:
 
             last_exception = None
 
+            # Prepare response format ONCE before the retry loop
+            # (to avoid appending schema to messages on every retry)
+            if response_format is not None:
+                schema = None
+                if hasattr(response_format, "model_json_schema"):
+                    schema = response_format.model_json_schema()
+
+                if strict_schema and schema is not None:
+                    # Use OpenAI's strict JSON schema enforcement
+                    # This guarantees all required fields are returned
+                    call_params["response_format"] = {
+                        "type": "json_schema",
+                        "json_schema": {
+                            "name": "response",
+                            "strict": True,
+                            "schema": schema,
+                        },
+                    }
+                else:
+                    # Soft enforcement: add schema to prompt and use json_object mode
+                    if schema is not None:
+                        schema_msg = f"\n\nYou must respond with valid JSON matching this schema:\n{json.dumps(schema, indent=2)}"
+
+                        if call_params["messages"] and call_params["messages"][0].get("role") == "system":
+                            call_params["messages"][0]["content"] += schema_msg
+                        elif call_params["messages"]:
+                            call_params["messages"][0]["content"] = (
+                                schema_msg + "\n\n" + call_params["messages"][0]["content"]
+                            )
+                    if self.provider not in ("lmstudio", "ollama"):
+                        # LM Studio and Ollama don't support json_object response format reliably
+                        # We rely on the schema in the system message instead
+                        call_params["response_format"] = {"type": "json_object"}
+
             for attempt in range(max_retries + 1):
                 try:
                     if response_format is not None:
-                        schema = None
-                        if hasattr(response_format, "model_json_schema"):
-                            schema = response_format.model_json_schema()
-
-                        if strict_schema and schema is not None:
-                            # Use OpenAI's strict JSON schema enforcement
-                            # This guarantees all required fields are returned
-                            call_params["response_format"] = {
-                                "type": "json_schema",
-                                "json_schema": {
-                                    "name": "response",
-                                    "strict": True,
-                                    "schema": schema,
-                                },
-                            }
-                        else:
-                            # Soft enforcement: add schema to prompt and use json_object mode
-                            if schema is not None:
-                                schema_msg = f"\n\nYou must respond with valid JSON matching this schema:\n{json.dumps(schema, indent=2)}"
-
-                                if call_params["messages"] and call_params["messages"][0].get("role") == "system":
-                                    call_params["messages"][0]["content"] += schema_msg
-                                elif call_params["messages"]:
-                                    call_params["messages"][0]["content"] = (
-                                        schema_msg + "\n\n" + call_params["messages"][0]["content"]
-                                    )
-                            if self.provider not in ("lmstudio", "ollama"):
-                                # LM Studio and Ollama don't support json_object response format reliably
-                                # We rely on the schema in the system message instead
-                                call_params["response_format"] = {"type": "json_object"}
-
-                        logger.debug(f"Sending request to {self.provider}/{self.model} (timeout={self.timeout})")
                         response = await self._client.chat.completions.create(**call_params)
-                        logger.debug(f"Received response from {self.provider}/{self.model}")
 
                         content = response.choices[0].message.content
 
@@ -467,13 +468,11 @@ class LLMProvider:
 
                 except APIConnectionError as e:
                     last_exception = e
+                    status_code = getattr(e, "status_code", None) or getattr(
+                        getattr(e, "response", None), "status_code", None
+                    )
+                    logger.warning(f"APIConnectionError (HTTP {status_code}), attempt {attempt + 1}: {str(e)[:200]}")
                     if attempt < max_retries:
-                        status_code = getattr(e, "status_code", None) or getattr(
-                            getattr(e, "response", None), "status_code", None
-                        )
-                        logger.warning(
-                            f"Connection error, retrying... (attempt {attempt + 1}/{max_retries + 1}) - status_code={status_code}, message={e}"
-                        )
                         backoff = min(initial_backoff * (2**attempt), max_backoff)
                         await asyncio.sleep(backoff)
                         continue
@@ -487,6 +486,45 @@ class LLMProvider:
                         logger.error(f"Auth error (HTTP {e.status_code}), not retrying: {str(e)}")
                         raise
 
+                    # Handle tool_use_failed error - model outputted in tool call format
+                    # Convert to expected JSON format and continue
+                    if e.status_code == 400 and response_format is not None:
+                        try:
+                            error_body = e.body if hasattr(e, "body") else {}
+                            if isinstance(error_body, dict):
+                                error_info: dict[str, Any] = error_body.get("error") or {}
+                                if error_info.get("code") == "tool_use_failed":
+                                    failed_gen = error_info.get("failed_generation", "")
+                                    if failed_gen:
+                                        # Parse the tool call format and convert to actions format
+                                        tool_call = json.loads(failed_gen)
+                                        tool_name = tool_call.get("name", "")
+                                        tool_args = tool_call.get("arguments", {})
+                                        # Convert to actions format: {"actions": [{"tool": "name", ...args}]}
+                                        converted = {"actions": [{"tool": tool_name, **tool_args}]}
+                                        if skip_validation:
+                                            result = converted
+                                        else:
+                                            result = response_format.model_validate(converted)
+
+                                        # Record metrics for this successful recovery
+                                        duration = time.time() - start_time
+                                        metrics = get_metrics_collector()
+                                        metrics.record_llm_call(
+                                            provider=self.provider,
+                                            model=self.model,
+                                            scope=scope,
+                                            duration=duration,
+                                            input_tokens=0,
+                                            output_tokens=0,
+                                            success=True,
+                                        )
+                                        if return_usage:
+                                            return result, TokenUsage(input_tokens=0, output_tokens=0, total_tokens=0)
+                                        return result
+                        except (json.JSONDecodeError, KeyError, TypeError):
+                            pass  # Failed to parse tool_use_failed, continue with normal retry
+
                     last_exception = e
                     if attempt < max_retries:
                         backoff = min(initial_backoff * (2**attempt), max_backoff)
@@ -497,14 +535,416 @@ class LLMProvider:
                         logger.error(f"API error after {max_retries + 1} attempts: {str(e)}")
                         raise
 
-                except Exception as e:
-                    logger.error(f"Unexpected error during LLM call: {type(e).__name__}: {str(e)}")
+                except Exception:
                     raise
 
             if last_exception:
                 raise last_exception
             raise RuntimeError("LLM call failed after all retries with no exception captured")
 
+    async def call_with_tools(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_completion_tokens: int | None = None,
+        temperature: float | None = None,
+        scope: str = "tools",
+        max_retries: int = 5,
+        initial_backoff: float = 1.0,
+        max_backoff: float = 30.0,
+        tool_choice: str | dict[str, Any] = "auto",
+    ) -> "LLMToolCallResult":
+        """
+        Make an LLM API call with tool/function calling support.
+
+        Args:
+            messages: List of message dicts. Can include tool results with role='tool'.
+            tools: List of tool definitions in OpenAI format.
+            max_completion_tokens: Maximum tokens in response.
+            temperature: Sampling temperature (0.0-2.0).
+            scope: Scope identifier for tracking.
+            max_retries: Maximum retry attempts.
+            initial_backoff: Initial backoff time in seconds.
+            max_backoff: Maximum backoff time in seconds.
+            tool_choice: How to choose tools - "auto", "none", "required", or {"type": "function", "function": {"name": "..."}}
+
+        Returns:
+            LLMToolCallResult with content and/or tool_calls.
+        """
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        async with _global_llm_semaphore:
+            start_time = time.time()
+
+            # Handle Mock provider
+            if self.provider == "mock":
+                return await self._call_with_tools_mock(messages, tools, scope)
+
+            # Handle Anthropic separately (uses different tool format)
+            if self.provider == "anthropic":
+                return await self._call_with_tools_anthropic(
+                    messages, tools, max_completion_tokens, max_retries, initial_backoff, max_backoff, start_time, scope
+                )
+
+            # Handle Gemini (convert to Gemini tool format)
+            if self.provider == "gemini":
+                return await self._call_with_tools_gemini(
+                    messages, tools, max_retries, initial_backoff, max_backoff, start_time, scope
+                )
+
+            # OpenAI-compatible providers (OpenAI, Groq, Ollama, LMStudio)
+            call_params: dict[str, Any] = {
+                "model": self.model,
+                "messages": messages,
+                "tools": tools,
+                "tool_choice": tool_choice,
+            }
+
+            if max_completion_tokens is not None:
+                call_params["max_completion_tokens"] = max_completion_tokens
+            if temperature is not None:
+                call_params["temperature"] = temperature
+
+            # Provider-specific parameters
+            if self.provider == "groq":
+                call_params["seed"] = DEFAULT_LLM_SEED
+
+            last_exception = None
+
+            for attempt in range(max_retries + 1):
+                try:
+                    response = await self._client.chat.completions.create(**call_params)
+
+                    message = response.choices[0].message
+                    finish_reason = response.choices[0].finish_reason
+
+                    # Extract tool calls if present
+                    tool_calls: list[LLMToolCall] = []
+                    if message.tool_calls:
+                        for tc in message.tool_calls:
+                            try:
+                                args = json.loads(tc.function.arguments) if tc.function.arguments else {}
+                            except json.JSONDecodeError:
+                                args = {"_raw": tc.function.arguments}
+                            tool_calls.append(LLMToolCall(id=tc.id, name=tc.function.name, arguments=args))
+
+                    content = message.content
+
+                    # Record metrics
+                    duration = time.time() - start_time
+                    usage = response.usage
+                    input_tokens = usage.prompt_tokens or 0 if usage else 0
+                    output_tokens = usage.completion_tokens or 0 if usage else 0
+
+                    metrics = get_metrics_collector()
+                    metrics.record_llm_call(
+                        provider=self.provider,
+                        model=self.model,
+                        scope=scope,
+                        duration=duration,
+                        input_tokens=input_tokens,
+                        output_tokens=output_tokens,
+                        success=True,
+                    )
+
+                    return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+                except APIConnectionError as e:
+                    last_exception = e
+                    if attempt < max_retries:
+                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                        continue
+                    raise
+
+                except APIStatusError as e:
+                    if e.status_code in (401, 403):
+                        raise
+                    last_exception = e
+                    if attempt < max_retries:
+                        await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                        continue
+                    raise
+
+                except Exception:
+                    raise
+
+            if last_exception:
+                raise last_exception
+            raise RuntimeError("Tool call failed after all retries")
+
+    async def _call_with_tools_mock(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle mock tool calls for testing."""
+        from .response_models import LLMToolCallResult
+
+        call_record = {
+            "provider": self.provider,
+            "model": self.model,
+            "messages": messages,
+            "tools": [t.get("function", {}).get("name") for t in tools],
+            "scope": scope,
+        }
+        self._mock_calls.append(call_record)
+
+        if self._mock_response is not None:
+            if isinstance(self._mock_response, LLMToolCallResult):
+                return self._mock_response
+            # Allow setting just tool calls as a list
+            if isinstance(self._mock_response, list):
+                from .response_models import LLMToolCall
+
+                return LLMToolCallResult(
+                    tool_calls=[
+                        LLMToolCall(id=f"mock_{i}", name=tc["name"], arguments=tc.get("arguments", {}))
+                        for i, tc in enumerate(self._mock_response)
+                    ],
+                    finish_reason="tool_calls",
+                )
+
+        return LLMToolCallResult(content="mock response", finish_reason="stop")
+
+    async def _call_with_tools_anthropic(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_completion_tokens: int | None,
+        max_retries: int,
+        initial_backoff: float,
+        max_backoff: float,
+        start_time: float,
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle Anthropic tool calling."""
+        from anthropic import APIConnectionError, APIStatusError
+
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        # Convert OpenAI tool format to Anthropic format
+        anthropic_tools = []
+        for tool in tools:
+            func = tool.get("function", {})
+            anthropic_tools.append(
+                {
+                    "name": func.get("name", ""),
+                    "description": func.get("description", ""),
+                    "input_schema": func.get("parameters", {"type": "object", "properties": {}}),
+                }
+            )
+
+        # Convert messages - handle tool results
+        system_prompt = None
+        anthropic_messages = []
+        for msg in messages:
+            role = msg.get("role", "user")
+            content = msg.get("content", "")
+
+            if role == "system":
+                system_prompt = (system_prompt + "\n\n" + content) if system_prompt else content
+            elif role == "tool":
+                # Anthropic uses tool_result blocks
+                anthropic_messages.append(
+                    {
+                        "role": "user",
+                        "content": [
+                            {"type": "tool_result", "tool_use_id": msg.get("tool_call_id", ""), "content": content}
+                        ],
+                    }
+                )
+            elif role == "assistant" and msg.get("tool_calls"):
+                # Convert assistant tool calls
+                tool_use_blocks = []
+                for tc in msg["tool_calls"]:
+                    tool_use_blocks.append(
+                        {
+                            "type": "tool_use",
+                            "id": tc.get("id", ""),
+                            "name": tc.get("function", {}).get("name", ""),
+                            "input": json.loads(tc.get("function", {}).get("arguments", "{}")),
+                        }
+                    )
+                anthropic_messages.append({"role": "assistant", "content": tool_use_blocks})
+            else:
+                anthropic_messages.append({"role": role, "content": content})
+
+        call_params: dict[str, Any] = {
+            "model": self.model,
+            "messages": anthropic_messages,
+            "tools": anthropic_tools,
+            "max_tokens": max_completion_tokens or 4096,
+        }
+        if system_prompt:
+            call_params["system"] = system_prompt
+
+        last_exception = None
+        for attempt in range(max_retries + 1):
+            try:
+                response = await self._anthropic_client.messages.create(**call_params)
+
+                # Extract content and tool calls
+                content_parts = []
+                tool_calls: list[LLMToolCall] = []
+
+                for block in response.content:
+                    if block.type == "text":
+                        content_parts.append(block.text)
+                    elif block.type == "tool_use":
+                        tool_calls.append(LLMToolCall(id=block.id, name=block.name, arguments=block.input or {}))
+
+                content = "".join(content_parts) if content_parts else None
+                finish_reason = "tool_calls" if tool_calls else "stop"
+
+                # Record metrics
+                metrics = get_metrics_collector()
+                metrics.record_llm_call(
+                    provider=self.provider,
+                    model=self.model,
+                    scope=scope,
+                    duration=time.time() - start_time,
+                    input_tokens=response.usage.input_tokens or 0,
+                    output_tokens=response.usage.output_tokens or 0,
+                    success=True,
+                )
+
+                return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+            except (APIConnectionError, APIStatusError) as e:
+                if isinstance(e, APIStatusError) and e.status_code in (401, 403):
+                    raise
+                last_exception = e
+                if attempt < max_retries:
+                    await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                    continue
+                raise
+
+        if last_exception:
+            raise last_exception
+        raise RuntimeError("Anthropic tool call failed")
+
+    async def _call_with_tools_gemini(
+        self,
+        messages: list[dict[str, Any]],
+        tools: list[dict[str, Any]],
+        max_retries: int,
+        initial_backoff: float,
+        max_backoff: float,
+        start_time: float,
+        scope: str,
+    ) -> "LLMToolCallResult":
+        """Handle Gemini tool calling."""
+        from .response_models import LLMToolCall, LLMToolCallResult
+
+        # Convert tools to Gemini format
+        gemini_tools = []
+        for tool in tools:
+            func = tool.get("function", {})
+            gemini_tools.append(
+                genai_types.Tool(
+                    function_declarations=[
+                        genai_types.FunctionDeclaration(
+                            name=func.get("name", ""),
+                            description=func.get("description", ""),
+                            parameters=func.get("parameters"),
+                        )
+                    ]
+                )
+            )
+
+        # Convert messages
+        system_instruction = None
+        gemini_contents = []
+        for msg in messages:
+            role = msg.get("role", "user")
+            content = msg.get("content", "")
+
+            if role == "system":
+                system_instruction = (system_instruction + "\n\n" + content) if system_instruction else content
+            elif role == "tool":
+                # Gemini uses function_response
+                gemini_contents.append(
+                    genai_types.Content(
+                        role="user",
+                        parts=[
+                            genai_types.Part(
+                                function_response=genai_types.FunctionResponse(
+                                    name=msg.get("name", ""),
+                                    response={"result": content},
+                                )
+                            )
+                        ],
+                    )
+                )
+            elif role == "assistant":
+                gemini_contents.append(genai_types.Content(role="model", parts=[genai_types.Part(text=content)]))
+            else:
+                gemini_contents.append(genai_types.Content(role="user", parts=[genai_types.Part(text=content)]))
+
+        config = genai_types.GenerateContentConfig(
+            system_instruction=system_instruction,
+            tools=gemini_tools,
+        )
+
+        last_exception = None
+        for attempt in range(max_retries + 1):
+            try:
+                response = await self._gemini_client.aio.models.generate_content(
+                    model=self.model,
+                    contents=gemini_contents,
+                    config=config,
+                )
+
+                # Extract content and tool calls
+                content = None
+                tool_calls: list[LLMToolCall] = []
+
+                if response.candidates and response.candidates[0].content:
+                    for part in response.candidates[0].content.parts:
+                        if hasattr(part, "text") and part.text:
+                            content = part.text
+                        if hasattr(part, "function_call") and part.function_call:
+                            fc = part.function_call
+                            tool_calls.append(
+                                LLMToolCall(
+                                    id=f"gemini_{len(tool_calls)}",
+                                    name=fc.name,
+                                    arguments=dict(fc.args) if fc.args else {},
+                                )
+                            )
+
+                finish_reason = "tool_calls" if tool_calls else "stop"
+
+                # Record metrics
+                metrics = get_metrics_collector()
+                input_tokens = response.usage_metadata.prompt_token_count if response.usage_metadata else 0
+                output_tokens = response.usage_metadata.candidates_token_count if response.usage_metadata else 0
+                metrics.record_llm_call(
+                    provider=self.provider,
+                    model=self.model,
+                    scope=scope,
+                    duration=time.time() - start_time,
+                    input_tokens=input_tokens,
+                    output_tokens=output_tokens,
+                    success=True,
+                )
+
+                return LLMToolCallResult(content=content, tool_calls=tool_calls, finish_reason=finish_reason)
+
+            except genai_errors.APIError as e:
+                if e.code in (401, 403):
+                    raise
+                last_exception = e
+                if attempt < max_retries:
+                    await asyncio.sleep(min(initial_backoff * (2**attempt), max_backoff))
+                    continue
+                raise
+
+        if last_exception:
+            raise last_exception
+        raise RuntimeError("Gemini tool call failed")
+
     async def _call_anthropic(
         self,
         messages: list[dict[str, str]],

hindsight-api/hindsight_api/engine/mental_models/__init__.py

@@ -0,0 +1,18 @@
+"""
+Mental models module for Hindsight.
+
+Mental models are synthesized summaries that represent understanding. They come
+in different subtypes based on how they were created:
+
+- Structural: Derived from the bank's mission (e.g., "Be a PM for engineering team")
+  These are created upfront based on what any agent with this role would need.
+
+- Emergent: Discovered from data patterns (named entities, temporal clusters, etc.)
+  These surface organically as facts are retained.
+
+- Pinned: User-defined models that persist across refreshes.
+"""
+
+from .models import MentalModel, MentalModelSubtype
+
+__all__ = ["MentalModel", "MentalModelSubtype"]

hindsight-api/hindsight_api/engine/mental_models/emergent.py

@@ -0,0 +1,311 @@
+"""
+Emergent mental model detection and promotion.
+
+Emergent models are discovered from data patterns:
+- Named entity extraction (people, projects, systems)
+- Temporal clustering (events with multiple references)
+- Causal patterns ("Because X, we do Y")
+- Behavioral anchors ("After X, we started Y")
+- Reference frequency (anything mentioned repeatedly)
+
+When a pattern is detected, it goes through a mission filter to check relevance,
+and if relevant, is promoted to a mental model.
+"""
+
+import logging
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel, Field
+
+from .models import EmergentCandidate
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMConfig
+
+logger = logging.getLogger(__name__)
+
+
+class MissionFilterCandidate(BaseModel):
+    """Result of mission filtering for a single candidate."""
+
+    name: str
+    promote: bool = Field(description="True if this is a specific named entity worth tracking")
+    reason: str = Field(description="Brief explanation for the decision")
+
+
+class MissionFilterResponse(BaseModel):
+    """Response from LLM for mission filtering."""
+
+    candidates: list[MissionFilterCandidate] = Field(description="Filtering decision for each candidate")
+
+
+def build_mission_filter_prompt(mission: str, candidates: list[EmergentCandidate]) -> str:
+    """Build the prompt for filtering candidates by mission relevance."""
+    candidate_list = "\n".join(
+        [f"- {c.name} (mentions: {c.mention_count}, method: {c.detection_method})" for c in candidates]
+    )
+
+    return f"""Filter these detected entities. For each one, decide: promote=true or promote=false.
+
+MISSION: {mission}
+
+DETECTED ENTITIES:
+{candidate_list}
+
+=== DECISION RULES ===
+
+Set promote=true ONLY for specific, named entities:
+- Person names: "John", "Maria", "Alice Chen", "Dr. Smith"
+- Named organizations: "Google", "Acme Corp", "Frontend Team"
+- Named places: "Central Park Zoo", "NYC Office", "Building A"
+- Named projects: "Project Phoenix", "Auth Service v2"
+
+Set promote=false for EVERYTHING ELSE, including:
+- Common English words: user, support, help, family, kids, parents, friends, people, team, photo, nature, park, office, home, work, school, joy, love, hope, fear, anger, gratitude, kindness, passion, motivation, inspiration, encouragement, positivity, energy, community, connection, commitment, collaboration, growth, impact, difference, success, progress, change, education, volunteering, veterans, homeless, shelter, meeting, project, system, process, event
+- Generic categories (even capitalized): Users, Customers, Team, Family, Kids, Veterans, Community
+- Abstract concepts: motivation, inspiration, gratitude, commitment, resilience
+
+THE TEST: Is this a specific name you'd find in a contact list or org chart?
+- "John" → YES (promote=true)
+- "kids" → NO (promote=false)
+- "community" → NO (promote=false)
+- "Maria" → YES (promote=true)
+- "park" → NO (promote=false)
+
+When in doubt, set promote=false."""
+
+
+def get_mission_filter_system_message() -> str:
+    """System message for mission filtering."""
+    return """You filter entities for promotion. Output JSON with 'candidates' array.
+
+Rules:
+- promote=true ONLY for specific names (people, organizations, named places/projects)
+- promote=false for common words, generic categories, abstract concepts
+
+Examples:
+- "John" → promote=true (person name)
+- "kids" → promote=false (generic category)
+- "community" → promote=false (abstract concept)
+- "Google" → promote=true (organization name)
+- "motivation" → promote=false (abstract concept)
+
+When in doubt, promote=false. Most entities should be rejected."""
+
+
+async def filter_candidates_by_mission(
+    llm_config: "LLMConfig",
+    mission: str,
+    candidates: list[EmergentCandidate],
+) -> list[EmergentCandidate]:
+    """
+    Filter emergent candidates to keep only specific, named entities.
+
+    Args:
+        llm_config: LLM configuration
+        mission: The bank's mission (used for context)
+        candidates: List of detected candidates
+
+    Returns:
+        Filtered list of candidates that are specific named entities
+    """
+    if not candidates:
+        return []
+
+    if not mission:
+        # No mission = no filtering, keep all candidates
+        logger.debug("[EMERGENT] No mission set, skipping filter")
+        return candidates
+
+    prompt = build_mission_filter_prompt(mission, candidates)
+
+    try:
+        result = await llm_config.call(
+            messages=[
+                {"role": "system", "content": get_mission_filter_system_message()},
+                {"role": "user", "content": prompt},
+            ],
+            response_format=MissionFilterResponse,
+            scope="mental_model_mission_filter",
+        )
+
+        # Build name -> promote map
+        promote_map = {c.name: c.promote for c in result.candidates}
+
+        # Filter candidates
+        filtered = []
+        for candidate in candidates:
+            if candidate.name in promote_map:
+                if promote_map[candidate.name]:
+                    filtered.append(candidate)
+                    logger.debug(f"[EMERGENT] Promoting '{candidate.name}'")
+                else:
+                    logger.debug(f"[EMERGENT] Rejecting '{candidate.name}'")
+            else:
+                # Candidate not in response - reject by default
+                logger.debug(f"[EMERGENT] '{candidate.name}' not in response, rejecting")
+
+        logger.info(f"[EMERGENT] Mission filter: {len(filtered)}/{len(candidates)} candidates promoted")
+        return filtered
+
+    except Exception as e:
+        logger.warning(f"[EMERGENT] Mission filter failed, rejecting all candidates: {e}")
+        return []
+
+
+async def evaluate_emergent_models(
+    llm_config: "LLMConfig",
+    models: list[dict],
+) -> list[str]:
+    """
+    Evaluate existing emergent models to check if they should be kept.
+
+    This re-evaluates emergent models using the same filtering criteria
+    as new candidates. Models that are generic/abstract will be removed.
+
+    Args:
+        llm_config: LLM configuration
+        models: List of existing emergent model dicts with 'name', 'id'
+
+    Returns:
+        List of model IDs that should be REMOVED (no longer valid)
+    """
+    if not models:
+        return []
+
+    # Convert existing models to candidates for evaluation
+    candidates = [
+        EmergentCandidate(
+            name=m["name"],
+            detection_method="existing_emergent_model",
+            mention_count=0,
+        )
+        for m in models
+    ]
+
+    # Build a simple prompt for re-evaluation
+    names_list = "\n".join([f"- {m['name']}" for m in models])
+    prompt = f"""Re-evaluate these existing mental models. For each one, decide: promote=true (keep) or promote=false (remove).
+
+EXISTING MODELS:
+{names_list}
+
+=== DECISION RULES ===
+
+Set promote=true ONLY for specific, named entities:
+- Person names: "John", "Maria", "Alice Chen", "Dr. Smith"
+- Named organizations: "Google", "Acme Corp", "Frontend Team"
+- Named places: "Central Park Zoo", "NYC Office", "Building A"
+- Named projects: "Project Phoenix", "Auth Service v2"
+
+Set promote=false for EVERYTHING ELSE, including:
+- Common English words: user, support, help, family, kids, parents, friends, people, team, photo, nature, park, office, home, work, school, joy, love, hope, fear, anger, gratitude, kindness, passion, motivation, inspiration, encouragement, positivity, energy, community, connection, commitment, collaboration, growth, impact, difference, success, progress, change, education, volunteering, veterans, homeless, shelter, meeting, project, system, process, event
+- Generic categories (even capitalized): Users, Customers, Team, Family, Kids, Veterans, Community
+- Abstract concepts: motivation, inspiration, gratitude, commitment, resilience
+
+THE TEST: Is this a specific name you'd find in a contact list or org chart?
+- "John" → YES (promote=true)
+- "kids" → NO (promote=false)
+- "community" → NO (promote=false)
+
+When in doubt, set promote=false."""
+
+    try:
+        result = await llm_config.call(
+            messages=[
+                {"role": "system", "content": get_mission_filter_system_message()},
+                {"role": "user", "content": prompt},
+            ],
+            response_format=MissionFilterResponse,
+            scope="mental_model_emergent_evaluation",
+        )
+
+        # Build name -> promote map
+        promote_map = {c.name: c.promote for c in result.candidates}
+
+        # Find models to remove
+        models_to_remove = []
+        for model in models:
+            name = model["name"]
+            if name in promote_map:
+                if not promote_map[name]:
+                    models_to_remove.append(model["id"])
+                else:
+                    logger.debug(f"[EMERGENT] Keeping '{name}'")
+            else:
+                # Model not in response - remove to be safe
+                logger.info(f"[EMERGENT] '{name}' not in evaluation response, marking for removal")
+                models_to_remove.append(model["id"])
+
+        logger.info(f"[EMERGENT] Evaluation: {len(models_to_remove)}/{len(models)} emergent models marked for removal")
+        return models_to_remove
+
+    except Exception as e:
+        logger.warning(f"[EMERGENT] Evaluation failed, keeping all models: {e}")
+        return []
+
+
+async def detect_entity_candidates(
+    pool,
+    bank_id: str,
+    min_mentions: int = 5,
+    top_percent: int = 20,
+) -> list[EmergentCandidate]:
+    """
+    Detect entities that are candidates for promotion to mental models.
+
+    Args:
+        pool: Database connection pool
+        bank_id: Bank identifier
+        min_mentions: Minimum mention count to consider
+        top_percent: Only consider top X% by mention count
+
+    Returns:
+        List of entity candidates
+    """
+    from ..db_utils import acquire_with_retry
+    from ..memory_engine import fq_table
+
+    candidates = []
+
+    async with acquire_with_retry(pool) as conn:
+        # Get entities that meet criteria and don't already have mental models
+        rows = await conn.fetch(
+            f"""
+            WITH ranked AS (
+                SELECT
+                    e.id,
+                    e.canonical_name,
+                    e.mention_count,
+                    PERCENT_RANK() OVER (ORDER BY e.mention_count DESC) as rank_pct
+                FROM {fq_table("entities")} e
+                LEFT JOIN {fq_table("mental_models")} mm
+                    ON mm.entity_id = e.id AND mm.bank_id = e.bank_id
+                WHERE e.bank_id = $1
+                  AND e.mention_count >= $2
+                  AND mm.id IS NULL  -- Not already a mental model
+            )
+            SELECT id, canonical_name, mention_count
+            FROM ranked
+            WHERE rank_pct <= $3
+            ORDER BY mention_count DESC
+            LIMIT 50
+            """,
+            bank_id,
+            min_mentions,
+            top_percent / 100.0,
+        )
+
+        for row in rows:
+            candidates.append(
+                EmergentCandidate(
+                    name=row["canonical_name"],
+                    detection_method="named_entity_extraction",
+                    mention_count=row["mention_count"],
+                    entity_id=str(row["id"]),
+                    relevance_score=0.0,
+                )
+            )
+
+    logger.debug(f"[EMERGENT] Detected {len(candidates)} entity candidates")
+    return candidates

hindsight-api/hindsight_api/engine/mental_models/models.py

@@ -0,0 +1,97 @@
+"""
+Pydantic models for mental models.
+"""
+
+from datetime import datetime, timezone
+from enum import Enum
+
+from pydantic import BaseModel, Field
+
+
+class MentalModelSubtype(str, Enum):
+    """Subtype of mental model - how it was created."""
+
+    STRUCTURAL = "structural"  # Derived from mission, created upfront
+    EMERGENT = "emergent"  # Discovered from data patterns
+    LEARNED = "learned"  # Formed through reflection
+    PINNED = "pinned"  # User-defined, persists across refreshes
+
+
+class MentalModel(BaseModel):
+    """
+    A mental model representing synthesized understanding.
+
+    Mental models are the agent's consolidated knowledge. Unlike raw facts,
+    mental models provide:
+    - A one-liner description for quick scanning/retrieval
+    - A full summary for deep understanding
+    - Links to related mental models
+    """
+
+    id: str = Field(description="Unique identifier within the bank")
+    bank_id: str = Field(description="Bank this mental model belongs to")
+    subtype: MentalModelSubtype = Field(description="How this model was created")
+    name: str = Field(description="Human-readable name")
+    description: str = Field(description="One-liner for quick scanning and retrieval matching")
+    summary: str | None = Field(default=None, description="Full synthesized understanding")
+
+    # References
+    entity_id: str | None = Field(default=None, description="Reference to entities table when type=entity")
+    source_facts: list[str] = Field(default_factory=list, description="Fact IDs used to generate summary")
+    links: list[str] = Field(default_factory=list, description="Related mental model IDs")
+
+    # Tags for scoped visibility (similar to document tags)
+    tags: list[str] = Field(default_factory=list, description="Tags for scoped visibility filtering")
+
+    # Timestamps
+    last_updated: datetime | None = Field(default=None, description="When summary was last regenerated")
+    created_at: datetime = Field(
+        default_factory=lambda: datetime.now(timezone.utc), description="When this model was created"
+    )
+
+
+class StructuralModelTemplate(BaseModel):
+    """
+    A template for a structural mental model.
+
+    Generated by LLM based on the bank's mission. Represents what any agent
+    with this role would need to track.
+    """
+
+    id: str = Field(default="", description="Existing model ID to keep, or empty for new models")
+    name: str = Field(description="Human-readable name")
+    description: str = Field(description="What this model should track")
+    initial_probes: list[str] = Field(default_factory=list, description="Initial search queries to populate this model")
+
+
+class StructuralModelDerivationResponse(BaseModel):
+    """Response from LLM for structural model derivation."""
+
+    templates: list[StructuralModelTemplate] = Field(description="Structural model templates derived from the mission")
+
+
+class EmergentCandidate(BaseModel):
+    """
+    A candidate for promotion to emergent mental model.
+
+    Detected through pattern analysis of facts.
+    """
+
+    name: str = Field(description="Name of the detected pattern/entity")
+    detection_method: str = Field(description="How this candidate was detected")
+    mention_count: int = Field(default=0, description="How many times referenced")
+    entity_id: str | None = Field(default=None, description="Entity ID if detected as entity")
+    relevance_score: float = Field(default=0.0, description="Score from mission filter (0-1)")
+
+
+class ResearchResult(BaseModel):
+    """
+    Result from the research endpoint.
+
+    Contains the answer along with the mental models and facts used.
+    """
+
+    answer: str = Field(description="The synthesized answer")
+    mental_models_used: list[str] = Field(default_factory=list, description="IDs of mental models that contributed")
+    facts_used: list[str] = Field(default_factory=list, description="Fact IDs that contributed")
+    question_type: str | None = Field(default=None, description="Detected question type (WHO, WHAT, HOW, etc.)")

hindsight-api/hindsight_api/engine/mental_models/structural.py

@@ -0,0 +1,228 @@
+"""
+Structural mental model derivation from bank mission.
+
+Structural models are derived from the bank's mission - they represent what
+any agent with this role would need to track. For example:
+
+Mission: "Be a PM for engineering team"
+Structural models:
+  - Team Structure (who's on the team, roles)
+  - Project Overview (current projects, status)
+  - Processes (how releases work, how decisions are made)
+  - Key Systems (what we own, dependencies)
+"""
+
+import logging
+from typing import TYPE_CHECKING
+
+from pydantic import BaseModel, Field
+
+from .models import StructuralModelTemplate
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMConfig
+
+logger = logging.getLogger(__name__)
+
+
+class StructuralDerivationResponse(BaseModel):
+    """Response from LLM for structural model derivation."""
+
+    templates: list[StructuralModelTemplate] = Field(description="Structural model templates derived from the mission")
+
+
+class StructuralRelevanceResult(BaseModel):
+    """Result of evaluating a structural model's relevance to the mission."""
+
+    name: str
+    relevant: bool
+    reason: str
+
+
+class StructuralRelevanceResponse(BaseModel):
+    """Response from LLM for structural model relevance evaluation."""
+
+    models: list[StructuralRelevanceResult] = Field(description="Relevance evaluation for each model")
+
+
+def build_structural_derivation_prompt(mission: str, existing_models: list[dict] | None = None) -> str:
+    """Build the prompt for deriving structural models from a mission."""
+    existing_section = ""
+    if existing_models:
+        model_list = "\n".join([f"- id='{m['id']}' name='{m['name']}': {m['description']}" for m in existing_models])
+        existing_section = f"""
+EXISTING STRUCTURAL MODELS:
+{model_list}
+
+IMPORTANT: If keeping an existing model, you MUST return its EXACT 'id' value.
+Models not included in your output will be REMOVED.
+"""
+
+    return f"""Given this agent mission, identify the KEY THINGS to track to achieve it.
+
+MISSION: {mission}
+{existing_section}
+IMPORTANT CONSTRAINTS:
+- Return 0-3 structural models MAXIMUM (less is better!)
+- Only include models for SPECIFIC, CONCRETE things the agent needs to track
+- Each model must be DIRECTLY tied to achieving the mission
+- If the mission is simple, return 0 models (empty array is fine)
+- If existing models are provided and you want to keep one, use its EXACT id
+- Do NOT create near-duplicates (e.g., don't create "topic-map" if "topic-connections" exists)
+
+GOOD examples (specific, actionable):
+- Mission: "Be a PM for engineering team" → "Team Members" (track who's on the team)
+- Mission: "Track customer feedback" → "Customer Issues" (track specific complaints/requests)
+- Mission: "Manage project X" → "Project X Milestones" (track progress)
+
+BAD examples (too generic, don't create these):
+- "Processes", "Workflows", "Key Systems", "Important Events"
+- "Communication", "Collaboration", "Progress", "Status"
+- Generic role-based models not tied to the specific mission
+
+For each model:
+1. id: Use EXACT existing id if keeping a model, or leave empty for new models
+2. name: Short, specific name (e.g., "Team Members", "Sprint Goals")
+3. description: One line describing what to track
+4. initial_probes: 2-3 search queries to find relevant information
+
+Return ONLY the models that should exist. Existing models not in your output will be deleted."""
+
+
+def get_structural_derivation_system_message() -> str:
+    """System message for structural model derivation."""
+    return """You identify the key things to track for a mission. Be VERY selective.
+
+Rules:
+- Maximum 3 models (prefer fewer)
+- Only SPECIFIC, CONCRETE things - not generic categories
+- Each must DIRECTLY help achieve the mission
+- Empty array is valid if no models are truly needed
+- If existing models are shown and you want to keep one, return its EXACT id
+- Never create duplicates - if a similar model exists, keep the existing one
+
+Output JSON with 'templates' array (can be empty)."""
+
+
+def _normalize_id(text: str) -> str:
+    """Normalize a string to a canonical form for comparison.
+
+    Removes common suffixes, pluralization, and normalizes separators.
+    """
+    # Lowercase and normalize separators
+    normalized = text.lower().replace(" ", "-").replace("_", "-")
+
+    # Remove common suffixes that indicate the same concept
+    suffixes_to_remove = ["-map", "-list", "-overview", "-tracker", "-s"]
+    for suffix in suffixes_to_remove:
+        if normalized.endswith(suffix) and len(normalized) > len(suffix):
+            normalized = normalized[: -len(suffix)]
+
+    return normalized
+
+
+def _find_similar_existing_id(new_id: str, existing_models: list[dict]) -> str | None:
+    """Find an existing model ID that is similar to the new ID.
+
+    Returns the existing ID if a similar one is found, None otherwise.
+    """
+    if not existing_models:
+        return None
+
+    new_normalized = _normalize_id(new_id)
+
+    for model in existing_models:
+        existing_id = model.get("id", "")
+        existing_normalized = _normalize_id(existing_id)
+
+        # Check if one is a prefix of the other (normalized)
+        if new_normalized.startswith(existing_normalized) or existing_normalized.startswith(new_normalized):
+            return existing_id
+
+        # Check if they're the same when normalized
+        if new_normalized == existing_normalized:
+            return existing_id
+
+    return None
+
+
+async def derive_structural_models(
+    llm_config: "LLMConfig",
+    mission: str,
+    existing_models: list[dict] | None = None,
+) -> tuple[list[StructuralModelTemplate], list[str]]:
+    """
+    Derive structural model templates from a bank's mission.
+
+    This combines derivation and evaluation in one call. The LLM sees existing
+    models and decides which to keep. Any existing model not in the output
+    will be marked for removal.
+
+    Args:
+        llm_config: LLM configuration for calling the model
+        mission: The bank's mission (e.g., "Be a PM for engineering team")
+        existing_models: Optional list of existing model dicts with 'name', 'description', 'id'
+
+    Returns:
+        Tuple of (templates to create/keep, IDs of existing models to remove)
+
+    Raises:
+        Exception: If LLM call fails
+    """
+    prompt = build_structural_derivation_prompt(mission, existing_models)
+
+    result = await llm_config.call(
+        messages=[
+            {"role": "system", "content": get_structural_derivation_system_message()},
+            {"role": "user", "content": prompt},
+        ],
+        response_format=StructuralDerivationResponse,
+        scope="mental_model_structural_derivation",
+    )
+
+    templates = result.templates
+    logger.info(f"[STRUCTURAL] LLM returned {len(templates)} structural models")
+
+    # Build set of existing IDs for quick lookup
+    existing_ids = {m["id"] for m in existing_models} if existing_models else set()
+
+    # Process templates: validate IDs, deduplicate, assign stable IDs
+    processed_templates: list[StructuralModelTemplate] = []
+    kept_existing_ids: set[str] = set()
+
+    for template in templates:
+        # If LLM returned an ID, check if it's a valid existing ID
+        if template.id and template.id in existing_ids:
+            # LLM is keeping an existing model
+            kept_existing_ids.add(template.id)
+            processed_templates.append(template)
+            logger.info(f"[STRUCTURAL] Keeping existing model: {template.id}")
+        else:
+            # New model or LLM didn't return a valid ID
+            # Generate ID from name
+            generated_id = template.name.lower().replace(" ", "-").replace("_", "-")
+
+            # Check for similar existing models to prevent near-duplicates
+            similar_id = _find_similar_existing_id(generated_id, existing_models)
+            if similar_id and similar_id not in kept_existing_ids:
+                # Use the existing similar model instead of creating a new one
+                logger.info(f"[STRUCTURAL] Detected near-duplicate: '{generated_id}' matches existing '{similar_id}'")
+                template.id = similar_id
+                kept_existing_ids.add(similar_id)
+            else:
+                template.id = generated_id
+
+            processed_templates.append(template)
+
+    # Find existing models to remove (not kept in LLM output)
+    models_to_remove = []
+    if existing_models:
+        for model in existing_models:
+            if model["id"] not in kept_existing_ids:
+                logger.info(f"[STRUCTURAL] Marking '{model['name']}' (id={model['id']}) for removal")
+                models_to_remove.append(model["id"])
+
+    if models_to_remove:
+        logger.info(f"[STRUCTURAL] {len(models_to_remove)} existing models will be removed")
+
+    return processed_templates, models_to_remove

hindsight-api/hindsight_api/engine/reflect/__init__.py

@@ -0,0 +1,20 @@
+"""
+Reflect agent module for agentic reflection with tools.
+
+The reflect agent uses an iterative loop with tools to:
+1. Lookup mental models (existing knowledge)
+2. Recall facts (semantic + temporal search)
+3. Learn new insights (create/update mental models)
+4. Expand memories (get chunk/document context)
+"""
+
+from .agent import ReflectAgentResult, run_reflect_agent
+from .models import MentalModelInput, ReflectAction, ReflectActionBatch
+
+__all__ = [
+    "run_reflect_agent",
+    "ReflectAgentResult",
+    "ReflectAction",
+    "ReflectActionBatch",
+    "MentalModelInput",
+]

hindsight-api/hindsight_api/engine/reflect/agent.py

@@ -0,0 +1,731 @@
+"""
+Reflect agent - agentic loop for reflection with native tool calling.
+"""
+
+import asyncio
+import json
+import logging
+import time
+from typing import TYPE_CHECKING, Any, Awaitable, Callable, Literal
+
+from .models import LLMCall, MentalModelInput, Observation, ReflectAgentResult, ToolCall
+from .prompts import FINAL_SYSTEM_PROMPT, build_final_prompt, build_system_prompt_for_tools
+from .tools_schema import get_reflect_tools
+
+if TYPE_CHECKING:
+    from ..llm_wrapper import LLMProvider
+    from ..response_models import LLMToolCall
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_MAX_ITERATIONS = 10
+
+
+async def _generate_structured_output(
+    answer: str,
+    response_schema: dict,
+    llm_config: "LLMProvider",
+    reflect_id: str,
+) -> dict[str, Any] | None:
+    """Generate structured output from an answer using the provided JSON schema.
+
+    Args:
+        answer: The text answer to extract structured data from
+        response_schema: JSON Schema for the expected output structure
+        llm_config: LLM provider for making the extraction call
+        reflect_id: Reflect ID for logging
+
+    Returns:
+        Structured output dict if successful, None otherwise
+    """
+    try:
+        from typing import Any as TypingAny
+
+        from pydantic import create_model
+
+        def _json_schema_type_to_python(field_schema: dict) -> type:
+            """Map JSON schema type to Python type for better LLM guidance."""
+            json_type = field_schema.get("type", "string")
+            if json_type == "array":
+                return list
+            elif json_type == "object":
+                return dict
+            elif json_type == "integer":
+                return int
+            elif json_type == "number":
+                return float
+            elif json_type == "boolean":
+                return bool
+            else:
+                return str
+
+        # Build fields from JSON schema properties
+        schema_props = response_schema.get("properties", {})
+        required_fields = set(response_schema.get("required", []))
+        fields: dict[str, TypingAny] = {}
+        for field_name, field_schema in schema_props.items():
+            field_type = _json_schema_type_to_python(field_schema)
+            default = ... if field_name in required_fields else None
+            fields[field_name] = (field_type, default)
+
+        if not fields:
+            return None
+
+        DynamicModel = create_model("StructuredResponse", **fields)
+
+        # Include the full schema in the prompt for better LLM guidance
+        schema_str = json.dumps(response_schema, indent=2)
+
+        # Call LLM with the answer to extract structured data
+        structured_prompt = f"""Based on this answer, extract the information into the requested structured format.
+
+Answer: {answer}
+
+JSON Schema to follow:
+```json
+{schema_str}
+```
+
+Return ONLY a valid JSON object that matches this exact schema. Pay special attention to field types:
+- "type": "array" means the value must be a JSON array/list, NOT a string
+- "type": "string" means the value must be a string
+- "type": "object" means the value must be a JSON object
+
+Do not include any explanation, only the JSON object."""
+
+        structured_result = await llm_config.call(
+            messages=[
+                {
+                    "role": "system",
+                    "content": "Extract structured data from the given answer. Return only valid JSON matching the provided schema exactly.",
+                },
+                {"role": "user", "content": structured_prompt},
+            ],
+            response_format=DynamicModel,
+            scope="reflect_structured",
+            skip_validation=True,  # We'll handle the dict ourselves
+        )
+
+        # Convert to dict
+        if hasattr(structured_result, "model_dump"):
+            structured_output = structured_result.model_dump()
+        elif isinstance(structured_result, dict):
+            structured_output = structured_result
+        else:
+            # Try to parse as JSON
+            structured_output = json.loads(str(structured_result))
+
+        logger.info(f"[REFLECT {reflect_id}] Generated structured output with {len(structured_output)} fields")
+        return structured_output
+
+    except Exception as e:
+        logger.warning(f"[REFLECT {reflect_id}] Failed to generate structured output: {e}")
+        return None
+
+
+async def run_reflect_agent(
+    llm_config: "LLMProvider",
+    bank_id: str,
+    query: str,
+    bank_profile: dict[str, Any],
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+    context: str | None = None,
+    max_iterations: int = DEFAULT_MAX_ITERATIONS,
+    max_tokens: int | None = None,
+    response_schema: dict | None = None,
+    output_mode: Literal["answer", "observations"] = "answer",
+) -> ReflectAgentResult:
+    """
+    Execute the reflect agent loop using native tool calling.
+
+    The agent iteratively calls tools to gather information and learn,
+    then provides a final answer via the done() tool.
+
+    Args:
+        llm_config: LLM provider for agent calls
+        bank_id: Bank identifier
+        query: Question to answer
+        bank_profile: Bank profile with name and mission
+        lookup_fn: Tool callback for lookup (model_id) -> result
+        recall_fn: Tool callback for recall (query, max_tokens) -> result
+        expand_fn: Tool callback for expand (memory_id, depth) -> result
+        learn_fn: Optional tool callback for learn (MentalModelInput) -> result.
+                  If None, learn tool is disabled.
+        context: Optional additional context
+        max_iterations: Maximum number of iterations before forcing response
+        max_tokens: Maximum tokens for the final response
+        response_schema: Optional JSON Schema for structured output in final response
+        output_mode: "answer" returns final text, "observations" returns structured observations
+
+    Returns:
+        ReflectAgentResult with final answer and metadata
+    """
+    enable_learn = learn_fn is not None
+    reflect_id = f"{bank_id[:8]}-{int(time.time() * 1000) % 100000}"
+    start_time = time.time()
+
+    # Get tools for this agent
+    tools = get_reflect_tools(enable_learn=enable_learn, output_mode=output_mode)
+
+    # Build initial messages
+    system_prompt = build_system_prompt_for_tools(bank_profile, context, output_mode=output_mode)
+    messages: list[dict[str, Any]] = [
+        {"role": "system", "content": system_prompt},
+        {"role": "user", "content": query},
+    ]
+
+    # Tracking
+    mental_models_created: list[str] = []
+    total_tools_called = 0
+    tool_trace: list[ToolCall] = []
+    tool_trace_summary: list[dict[str, Any]] = []
+    llm_trace: list[dict[str, Any]] = []
+    context_history: list[dict[str, Any]] = []  # For final prompt fallback
+
+    # Track available IDs for validation (prevents hallucinated citations)
+    available_memory_ids: set[str] = set()
+    available_model_ids: set[str] = set()
+
+    # In answer mode, pre-fetch mental models so the agent always starts with this knowledge
+    if output_mode == "answer":
+        prefetch_start = time.time()
+        models_result = await lookup_fn(None)  # List all mental models
+        prefetch_duration = int((time.time() - prefetch_start) * 1000)
+
+        # Track available model IDs
+        if isinstance(models_result, dict) and "models" in models_result:
+            for model in models_result["models"]:
+                if "id" in model:
+                    available_model_ids.add(model["id"])
+
+        # Add to context history for the agent
+        context_history.append({"tool": "list_mental_models", "output": models_result})
+
+        # Add to tool trace
+        tool_trace.append(
+            ToolCall(
+                tool="list_mental_models",
+                input={"tool": "list_mental_models"},
+                output=models_result,
+                duration_ms=prefetch_duration,
+                iteration=0,
+            )
+        )
+        tool_trace_summary.append(
+            {
+                "tool": "list_mental_models",
+                "input_summary": "(prefetch)",
+                "duration_ms": prefetch_duration,
+                "output_chars": len(json.dumps(models_result, default=str)),
+            }
+        )
+        total_tools_called += 1
+
+        # Include in the user message so the agent sees it
+        models_info = json.dumps(models_result, indent=2, default=str)
+        messages[1]["content"] = f"{query}\n\n## Available Mental Models (pre-fetched)\n```json\n{models_info}\n```"
+
+    def _get_llm_trace() -> list[LLMCall]:
+        return [LLMCall(scope=c["scope"], duration_ms=c["duration_ms"]) for c in llm_trace]
+
+    def _log_completion(answer: str, iterations: int, forced: bool = False):
+        elapsed_ms = int((time.time() - start_time) * 1000)
+        tools_summary = (
+            ", ".join(
+                f"{t['tool']}({t['input_summary']})={t['duration_ms']}ms/{t.get('output_chars', 0)}c"
+                for t in tool_trace_summary
+            )
+            or "none"
+        )
+        llm_summary = ", ".join(f"{c['scope']}={c['duration_ms']}ms" for c in llm_trace) or "none"
+        total_llm_ms = sum(c["duration_ms"] for c in llm_trace)
+        total_tools_ms = sum(t["duration_ms"] for t in tool_trace_summary)
+
+        answer_preview = answer[:100] + "..." if len(answer) > 100 else answer
+        mode = "forced" if forced else "done"
+        logger.info(
+            f"[REFLECT {reflect_id}] {mode} | "
+            f"query='{query[:50]}...' | "
+            f"iterations={iterations} | "
+            f"llm=[{llm_summary}] ({total_llm_ms}ms) | "
+            f"tools=[{tools_summary}] ({total_tools_ms}ms) | "
+            f"answer='{answer_preview}' | "
+            f"total={elapsed_ms}ms"
+        )
+
+    for iteration in range(max_iterations):
+        is_last = iteration == max_iterations - 1
+
+        if is_last:
+            # Force text response on last iteration - no tools
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+            )
+
+        # Call LLM with tools
+        llm_start = time.time()
+
+        try:
+            result = await llm_config.call_with_tools(
+                messages=messages,
+                tools=tools,
+                scope="reflect_agent",
+                tool_choice="required" if iteration == 0 else "auto",  # Force tool use on first iteration
+            )
+            llm_duration = int((time.time() - llm_start) * 1000)
+            llm_trace.append({"scope": f"agent_{iteration + 1}", "duration_ms": llm_duration})
+
+        except Exception:
+            llm_trace.append(
+                {"scope": f"agent_{iteration + 1}_err", "duration_ms": int((time.time() - llm_start) * 1000)}
+            )
+            # Guardrail: If no evidence gathered yet, retry
+            has_gathered_evidence = bool(available_memory_ids) or bool(available_model_ids)
+            if not has_gathered_evidence and iteration < max_iterations - 1:
+                continue
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+            )
+
+        # No tool calls - LLM wants to respond with text
+        if not result.tool_calls:
+            if result.content:
+                answer = result.content.strip()
+
+                # Generate structured output if schema provided
+                structured_output = None
+                if response_schema and answer:
+                    structured_output = await _generate_structured_output(
+                        answer, response_schema, llm_config, reflect_id
+                    )
+
+                _log_completion(answer, iteration + 1)
+                return ReflectAgentResult(
+                    text=answer,
+                    structured_output=structured_output,
+                    iterations=iteration + 1,
+                    tools_called=total_tools_called,
+                    mental_models_created=mental_models_created,
+                    tool_trace=tool_trace,
+                    llm_trace=_get_llm_trace(),
+                )
+            # Empty response, force final
+            prompt = build_final_prompt(query, context_history, bank_profile, context)
+            llm_start = time.time()
+            response = await llm_config.call(
+                messages=[
+                    {"role": "system", "content": FINAL_SYSTEM_PROMPT},
+                    {"role": "user", "content": prompt},
+                ],
+                scope="reflect_agent_final",
+                max_completion_tokens=max_tokens,
+            )
+            llm_trace.append({"scope": "final", "duration_ms": int((time.time() - llm_start) * 1000)})
+            answer = response.strip()
+
+            # Generate structured output if schema provided
+            structured_output = None
+            if response_schema and answer:
+                structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+            _log_completion(answer, iteration + 1, forced=True)
+            return ReflectAgentResult(
+                text=answer,
+                structured_output=structured_output,
+                iterations=iteration + 1,
+                tools_called=total_tools_called,
+                mental_models_created=mental_models_created,
+                tool_trace=tool_trace,
+                llm_trace=_get_llm_trace(),
+            )
+
+        # Check for done tool call
+        done_call = next((tc for tc in result.tool_calls if tc.name == "done"), None)
+        if done_call:
+            # Guardrail: Require evidence before done
+            has_gathered_evidence = bool(available_memory_ids) or bool(available_model_ids)
+            if not has_gathered_evidence and iteration < max_iterations - 1:
+                # Add assistant message and fake tool result asking for evidence
+                messages.append(
+                    {
+                        "role": "assistant",
+                        "tool_calls": [_tool_call_to_dict(done_call)],
+                    }
+                )
+                messages.append(
+                    {
+                        "role": "tool",
+                        "tool_call_id": done_call.id,
+                        "content": json.dumps(
+                            {
+                                "error": "You must call recall() or list_mental_models() to gather evidence before providing your final answer."
+                            }
+                        ),
+                    }
+                )
+                continue
+
+            # Process done tool
+            return await _process_done_tool(
+                done_call,
+                output_mode,
+                available_memory_ids,
+                available_model_ids,
+                iteration + 1,
+                total_tools_called,
+                mental_models_created,
+                tool_trace,
+                _get_llm_trace(),
+                _log_completion,
+                reflect_id,
+                llm_config=llm_config,
+                response_schema=response_schema,
+            )
+
+        # Execute other tools in parallel
+        other_tools = [tc for tc in result.tool_calls if tc.name != "done"]
+        if other_tools:
+            # Add assistant message with tool calls
+            messages.append(
+                {
+                    "role": "assistant",
+                    "tool_calls": [_tool_call_to_dict(tc) for tc in other_tools],
+                }
+            )
+
+            # Execute tools in parallel
+            tool_tasks = [
+                _execute_tool_with_timing(tc, lookup_fn, recall_fn, expand_fn, learn_fn) for tc in other_tools
+            ]
+            tool_results = await asyncio.gather(*tool_tasks, return_exceptions=True)
+            total_tools_called += len(other_tools)
+
+            # Process results and add to messages
+            for tc, result_data in zip(other_tools, tool_results):
+                if isinstance(result_data, Exception):
+                    # Tool execution failed - log and raise to fail the request
+                    logger.error(f"[REFLECT {reflect_id}] Tool {tc.name} failed with exception: {result_data}")
+                    raise RuntimeError(f"Reflect tool '{tc.name}' failed: {result_data}")
+
+                output, duration_ms = result_data
+
+                # Check if tool returned an error response
+                if isinstance(output, dict) and "error" in output:
+                    logger.error(f"[REFLECT {reflect_id}] Tool {tc.name} returned error: {output['error']}")
+                    raise RuntimeError(f"Reflect tool '{tc.name}' error: {output['error']}")
+
+                # Track created mental models
+                if tc.name == "learn" and isinstance(output, dict) and "model_id" in output:
+                    mental_models_created.append(output["model_id"])
+
+                # Track available memory IDs from recall
+                if tc.name == "recall" and isinstance(output, dict) and "memories" in output:
+                    for memory in output["memories"]:
+                        if "id" in memory:
+                            available_memory_ids.add(memory["id"])
+
+                # Track available model IDs
+                if tc.name in ("list_mental_models", "get_mental_model") and isinstance(output, dict):
+                    if output.get("found") and "model" in output:
+                        model_id = output["model"].get("id")
+                        if model_id:
+                            available_model_ids.add(model_id)
+                    elif "models" in output:
+                        for model in output["models"]:
+                            if "id" in model:
+                                available_model_ids.add(model["id"])
+
+                # Add tool result message
+                messages.append(
+                    {
+                        "role": "tool",
+                        "tool_call_id": tc.id,
+                        "content": json.dumps(output, default=str),
+                    }
+                )
+
+                # Track for logging and context history
+                input_dict = {"tool": tc.name, **tc.arguments}
+                input_summary = _summarize_input(tc.name, tc.arguments)
+
+                tool_trace.append(
+                    ToolCall(
+                        tool=tc.name, input=input_dict, output=output, duration_ms=duration_ms, iteration=iteration + 1
+                    )
+                )
+
+                try:
+                    output_chars = len(json.dumps(output))
+                except (TypeError, ValueError):
+                    output_chars = len(str(output))
+
+                tool_trace_summary.append(
+                    {
+                        "tool": tc.name,
+                        "input_summary": input_summary,
+                        "duration_ms": duration_ms,
+                        "output_chars": output_chars,
+                    }
+                )
+
+                # Keep context history for fallback final prompt
+                context_history.append({"tool": tc.name, "input": input_dict, "output": output})
+
+    # Should not reach here
+    answer = "I was unable to formulate a complete answer within the iteration limit."
+    _log_completion(answer, max_iterations, forced=True)
+    return ReflectAgentResult(
+        text=answer,
+        iterations=max_iterations,
+        tools_called=total_tools_called,
+        mental_models_created=mental_models_created,
+        tool_trace=tool_trace,
+        llm_trace=_get_llm_trace(),
+    )
+
+
+def _tool_call_to_dict(tc: "LLMToolCall") -> dict[str, Any]:
+    """Convert LLMToolCall to OpenAI message format."""
+    return {
+        "id": tc.id,
+        "type": "function",
+        "function": {
+            "name": tc.name,
+            "arguments": json.dumps(tc.arguments),
+        },
+    }
+
+
+async def _process_done_tool(
+    done_call: "LLMToolCall",
+    output_mode: str,
+    available_memory_ids: set[str],
+    available_model_ids: set[str],
+    iterations: int,
+    total_tools_called: int,
+    mental_models_created: list[str],
+    tool_trace: list[ToolCall],
+    llm_trace: list[LLMCall],
+    log_completion: Callable,
+    reflect_id: str,
+    llm_config: "LLMProvider | None" = None,
+    response_schema: dict | None = None,
+) -> ReflectAgentResult:
+    """Process the done tool call and return the result."""
+    args = done_call.arguments
+
+    if output_mode == "observations" and "observations" in args:
+        # Process observations - handle both list and nested {"observations": [...]} format
+        observations: list[Observation] = []
+        used_memory_ids: list[str] = []
+
+        obs_list = args["observations"]
+        # Handle nested format where LLM outputs {"observations": [...]} instead of just [...]
+        if isinstance(obs_list, dict) and "observations" in obs_list:
+            obs_list = obs_list["observations"]
+
+        for obs_data in obs_list:
+            validated_mids = []
+            for mid in obs_data.get("memory_ids", []):
+                if mid in available_memory_ids:
+                    validated_mids.append(mid)
+                    if mid not in used_memory_ids:
+                        used_memory_ids.append(mid)
+
+            observations.append(
+                Observation(
+                    title=obs_data.get("title", ""),
+                    text=obs_data.get("text", ""),
+                    memory_ids=validated_mids,
+                )
+            )
+
+        # Build text from observations
+        text_parts = []
+        for obs in observations:
+            if obs.title:
+                text_parts.append(f"## {obs.title}\n{obs.text}")
+            else:
+                text_parts.append(obs.text)
+        answer = "\n\n".join(text_parts)
+
+        log_completion(answer, iterations)
+        return ReflectAgentResult(
+            text=answer,
+            observations=observations,
+            iterations=iterations,
+            tools_called=total_tools_called,
+            mental_models_created=mental_models_created,
+            tool_trace=tool_trace,
+            llm_trace=llm_trace,
+            used_memory_ids=used_memory_ids,
+        )
+
+    # Default: answer mode
+    answer = args.get("answer", "").strip()
+    if not answer:
+        answer = "No answer provided."
+
+    # Validate IDs
+    used_memory_ids = [mid for mid in args.get("memory_ids", []) if mid in available_memory_ids]
+    used_model_ids = [mid for mid in args.get("model_ids", []) if mid in available_model_ids]
+
+    # Generate structured output if schema provided
+    structured_output = None
+    if response_schema and llm_config and answer:
+        structured_output = await _generate_structured_output(answer, response_schema, llm_config, reflect_id)
+
+    log_completion(answer, iterations)
+    return ReflectAgentResult(
+        text=answer,
+        structured_output=structured_output,
+        iterations=iterations,
+        tools_called=total_tools_called,
+        mental_models_created=mental_models_created,
+        tool_trace=tool_trace,
+        llm_trace=llm_trace,
+        used_memory_ids=used_memory_ids,
+        used_model_ids=used_model_ids,
+    )
+
+
+async def _execute_tool_with_timing(
+    tc: "LLMToolCall",
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+) -> tuple[dict[str, Any], int]:
+    """Execute a tool call and return result with timing."""
+    start = time.time()
+    result = await _execute_tool(tc.name, tc.arguments, lookup_fn, recall_fn, expand_fn, learn_fn)
+    duration_ms = int((time.time() - start) * 1000)
+    return result, duration_ms
+
+
+async def _execute_tool(
+    tool_name: str,
+    args: dict[str, Any],
+    lookup_fn: Callable[[str | None], Awaitable[dict[str, Any]]],
+    recall_fn: Callable[[str, int], Awaitable[dict[str, Any]]],
+    expand_fn: Callable[[list[str], str], Awaitable[dict[str, Any]]],
+    learn_fn: Callable[[MentalModelInput], Awaitable[dict[str, Any]]] | None = None,
+) -> dict[str, Any]:
+    """Execute a single tool by name."""
+    if tool_name == "list_mental_models":
+        return await lookup_fn(None)
+
+    elif tool_name == "get_mental_model":
+        model_id = args.get("model_id")
+        if not model_id:
+            return {"error": "get_mental_model requires model_id"}
+        return await lookup_fn(model_id)
+
+    elif tool_name == "recall":
+        query = args.get("query")
+        if not query:
+            return {"error": "recall requires a query parameter"}
+        max_tokens = max(args.get("max_tokens") or 2048, 1000)  # Default 2048, min 1000
+        return await recall_fn(query, max_tokens)
+
+    elif tool_name == "learn":
+        if learn_fn is None:
+            return {"error": "learn tool is not available"}
+        name = args.get("name")
+        description = args.get("description")
+        if not name or not description:
+            return {"error": "learn requires name and description"}
+        return await learn_fn(MentalModelInput(name=name, description=description))
+
+    elif tool_name == "expand":
+        memory_ids = args.get("memory_ids", [])
+        if not memory_ids:
+            return {"error": "expand requires memory_ids"}
+        depth = args.get("depth", "chunk")
+        return await expand_fn(memory_ids, depth)
+
+    else:
+        return {"error": f"Unknown tool: {tool_name}"}
+
+
+def _summarize_input(tool_name: str, args: dict[str, Any]) -> str:
+    """Create a summary of tool input for logging, showing all params."""
+    if tool_name == "list_mental_models":
+        return "()"
+    elif tool_name == "get_mental_model":
+        return f"(model_id={args.get('model_id', '?')})"
+    elif tool_name == "recall":
+        query = args.get("query", "")
+        query_preview = f"'{query[:30]}...'" if len(query) > 30 else f"'{query}'"
+        # Show actual value used (default 2048, min 1000)
+        max_tokens = max(args.get("max_tokens") or 2048, 1000)
+        return f"(query={query_preview}, max_tokens={max_tokens})"
+    elif tool_name == "learn":
+        name = args.get("name", "?")
+        desc = args.get("description", "")
+        desc_preview = f"'{desc[:20]}...'" if len(desc) > 20 else f"'{desc}'"
+        return f"(name='{name}', description={desc_preview})"
+    elif tool_name == "expand":
+        memory_ids = args.get("memory_ids", [])
+        depth = args.get("depth", "chunk")
+        return f"(memory_ids=[{len(memory_ids)} ids], depth={depth})"
+    elif tool_name == "done":
+        answer = args.get("answer", "")
+        answer_preview = f"'{answer[:30]}...'" if len(answer) > 30 else f"'{answer}'"
+        memory_ids = args.get("memory_ids", [])
+        model_ids = args.get("model_ids", [])
+        return f"(answer={answer_preview}, memory_ids={len(memory_ids)}, model_ids={len(model_ids)})"
+    return str(args)

hindsight-api/hindsight_api/engine/reflect/models.py

@@ -0,0 +1,114 @@
+"""
+Pydantic models for the reflect agent.
+"""
+
+from typing import Any, Literal
+
+from pydantic import BaseModel, Field
+
+
+class MentalModelObservation(BaseModel):
+    """An observation within a mental model with its supporting memories."""
+
+    title: str = Field(description="Observation header (can be empty for intro)")
+    text: str = Field(description="Observation content - no headers, use lists/tables/bold")
+    memory_ids: list[str] = Field(default_factory=list, description="Memory IDs supporting this observation")
+
+
+class MentalModelInput(BaseModel):
+    """Input for the learn tool to create a mental model placeholder.
+
+    The agent only specifies name and description - the actual content/observations
+    are generated during refresh, similar to pinned models.
+    """
+
+    name: str = Field(description="Human-readable name for the mental model")
+    description: str = Field(description="What to track - used as prompt for content generation during refresh")
+    entity_id: str | None = Field(default=None, description="Optional link to existing entity ID")
+
+
+class AnswerSection(BaseModel):
+    """A section of the answer with its supporting evidence (DEPRECATED)."""
+
+    title: str = Field(description="Section header/title")
+    text: str = Field(description="Section content")
+    memory_ids: list[str] = Field(default_factory=list, description="Memory IDs supporting this section")
+    model_ids: list[str] = Field(default_factory=list, description="Mental model IDs supporting this section")
+
+
+class ReflectAction(BaseModel):
+    """Single action the reflect agent can take."""
+
+    tool: Literal["list_mental_models", "get_mental_model", "recall", "learn", "expand", "done"] = Field(
+        description="Tool to invoke: list_mental_models, get_mental_model, recall, learn, expand, or done"
+    )
+    # Tool-specific parameters
+    model_id: str | None = Field(default=None, description="Mental model ID for get_mental_model")
+    query: str | None = Field(default=None, description="Search query for recall")
+    max_tokens: int | None = Field(default=None, description="Max tokens for recall results (default 2048)")
+    mental_model: MentalModelInput | None = Field(default=None, description="Mental model to create/update for learn")
+    memory_ids: list[str] | None = Field(default=None, description="Memory unit IDs for expand (batched)")
+    depth: Literal["chunk", "document"] | None = Field(default=None, description="Expansion depth for expand")
+    sections: list[AnswerSection] | None = Field(default=None, description="DEPRECATED: Use answer field instead")
+    observations: list[MentalModelObservation] | None = Field(
+        default=None, description="Observations for done action (when output_mode=observations)"
+    )
+    # Plain text answer fields (for output_mode=answer)
+    answer: str | None = Field(default=None, description="Plain text answer for done action (no markdown)")
+    answer_memory_ids: list[str] | None = Field(
+        default=None, description="Memory IDs supporting the answer", alias="memory_ids"
+    )
+    answer_model_ids: list[str] | None = Field(
+        default=None, description="Mental model IDs supporting the answer", alias="model_ids"
+    )
+    reasoning: str | None = Field(default=None, description="Brief reasoning for this action")
+
+
+class ReflectActionBatch(BaseModel):
+    """Batch of actions for parallel execution."""
+
+    actions: list[ReflectAction] = Field(description="List of actions to execute in parallel")
+
+
+class ToolCall(BaseModel):
+    """A single tool call made during reflect."""
+
+    tool: str = Field(description="Tool name: lookup, recall, learn, expand")
+    input: dict = Field(description="Tool input parameters")
+    output: dict = Field(description="Tool output/result")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+    iteration: int = Field(default=0, description="Iteration number (1-based) when this tool was called")
+
+
+class LLMCall(BaseModel):
+    """A single LLM call made during reflect."""
+
+    scope: str = Field(description="Call scope: agent_1, agent_2, final, etc.")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+
+
+class Observation(BaseModel):
+    """A single observation with supporting memories."""
+
+    title: str = Field(description="Observation title/header")
+    text: str = Field(description="Observation content")
+    memory_ids: list[str] = Field(default_factory=list, description="Memory IDs supporting this observation")
+
+
+class ReflectAgentResult(BaseModel):
+    """Result from the reflect agent."""
+
+    text: str = Field(description="Final answer text")
+    observations: list[Observation] = Field(
+        default_factory=list, description="Structured observations (when output_mode=observations)"
+    )
+    structured_output: dict[str, Any] | None = Field(
+        default=None, description="Structured output parsed according to provided response_schema"
+    )
+    iterations: int = Field(default=0, description="Number of iterations taken")
+    tools_called: int = Field(default=0, description="Total number of tool calls made")
+    mental_models_created: list[str] = Field(default_factory=list, description="IDs of mental models created/updated")
+    tool_trace: list[ToolCall] = Field(default_factory=list, description="Trace of all tool calls made")
+    llm_trace: list[LLMCall] = Field(default_factory=list, description="Trace of all LLM calls made")
+    used_memory_ids: list[str] = Field(default_factory=list, description="Validated memory IDs actually used in answer")
+    used_model_ids: list[str] = Field(default_factory=list, description="Validated model IDs actually used in answer")

hindsight-api/hindsight_api/engine/reflect/prompts.py

@@ -0,0 +1,312 @@
+"""
+System prompts for the reflect agent.
+"""
+
+import json
+from typing import Any
+
+
+def build_system_prompt_for_tools(
+    bank_profile: dict[str, Any],
+    context: str | None = None,
+    output_mode: str = "answer",
+) -> str:
+    """
+    Build the system prompt for tool-calling reflect agent.
+
+    This is a simplified prompt since tools are defined separately via the tools parameter.
+
+    Args:
+        bank_profile: Bank profile with name and mission
+        context: Optional additional context
+        output_mode: "answer" for plain text response, "observations" for structured observations
+    """
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    # Build critical rules based on mode
+    if output_mode == "observations":
+        no_info_rule = "- Only say 'I don't have information' AFTER trying recall with no relevant results"
+    else:
+        no_info_rule = (
+            "- Only say 'I don't have information' AFTER trying list_mental_models AND recall with no relevant results"
+        )
+
+    parts = [
+        "You are a reflection agent that answers questions by reasoning over retrieved memories.",
+        "",
+        "## CRITICAL RULES",
+        "- You must NEVER fabricate information that has no basis in retrieved data",
+        "- You SHOULD synthesize, infer, and reason from the retrieved memories",
+        "- You MUST call recall() before saying you don't have information",
+        no_info_rule,
+        "",
+        "## How to Reason",
+        "- If memories mention someone did an activity, you can infer they likely enjoyed it",
+        "- Synthesize a coherent narrative from related memories",
+        "- Be a thoughtful interpreter, not just a literal repeater",
+        "- When the exact answer isn't stated, use what IS stated to give the best answer",
+        "",
+        "## Query Strategy (IMPORTANT)",
+        "recall() uses semantic search. NEVER just echo the user's question - decompose it into targeted searches:",
+        "",
+        "BAD: User asks 'recurring lesson themes between students' → recall('recurring lesson themes between students')",
+        "GOOD: Break it down into component searches:",
+        "  1. recall('lessons') - find all lesson-related memories",
+        "  2. recall('teaching sessions') - alternative phrasing",
+        "  3. recall('student progress') - find student-related memories",
+        "  4. recall('topics taught') - find subject matter",
+        "",
+        "Think: What ENTITIES and CONCEPTS does this question involve? Search for each separately.",
+        "- Questions about patterns → search for the individual instances first",
+        "- Questions comparing things → search for each thing separately",
+        "- Questions about relationships → search for each party involved",
+        "",
+        "## Workflow",
+    ]
+
+    # Mode-specific workflow and output format
+    if output_mode == "observations":
+        # Observations mode: for mental model generation - no mental model lookup tools
+        parts.extend(
+            [
+                "1. DECOMPOSE the topic into component searches (see Query Strategy above)",
+                "   - Don't search for the topic name itself - search for related concepts",
+                "   - Example for 'Coffee preferences': search 'coffee', 'drinks', 'morning routine', 'caffeine'",
+                "2. Run multiple recall() calls with varied, targeted queries",
+                "3. IMPORTANT: Use expand(memory_ids, 'chunk') to verify memories before using them",
+                "   - Always verify the source chunk to confirm the memory is actually relevant",
+                "   - Don't assume a memory is relevant based on the summary alone",
+                "   - Only include memories you've verified via expand()",
+                "4. When ready, call done() with MULTIPLE structured observations",
+                "",
+                "## Output Format: MULTIPLE Structured Observations",
+                "",
+                "CRITICAL: You MUST create MULTIPLE separate observations in the array - one for each theme.",
+                "Do NOT put all content in a single observation!",
+                "",
+                "- Create 3-8 separate observations, each as its OWN item in the observations array",
+                "- Each observation covers ONE specific theme (preferences, history, relationships, etc.)",
+                "- Each observation has: title (short header), text (content), memory_ids (full UUIDs)",
+                "",
+                "Text format for each observation:",
+                "- Main insight or finding (no markdown headers)",
+                "- End with 'Key evidence:' section containing DIRECT QUOTES from memories in *italics*",
+                "- Quote the actual memory text, don't summarize - use *italics* for citations",
+                "",
+                "Example done() call with MULTIPLE observations:",
+                "```json",
+                "{",
+                '  "observations": [',
+                "    {",
+                '      "title": "Work Preferences",',
+                '      "text": "Prefers async communication and flexible schedules.\\n\\nKey evidence:\\n- *I prefer Slack over calls for most communication*\\n- *Flexible hours help me do my best work*",',
+                '      "memory_ids": ["abc123-full-uuid", "def456-full-uuid"]',
+                "    },",
+                "    {",
+                '      "title": "Technical Background",',
+                '      "text": "Has extensive ML experience spanning a decade.\\n\\nKey evidence:\\n- *I have 10 years of experience in machine learning*\\n- *Led the ML team at my previous company*",',
+                '      "memory_ids": ["ghi789-full-uuid"]',
+                "    }",
+                "  ]",
+                "}",
+                "```",
+            ]
+        )
+    else:
+        # Answer mode: include mental model lookup in workflow
+        parts.extend(
+            [
+                "1. Review the pre-fetched mental models for relevant synthesized knowledge",
+                "2. If relevant, call get_mental_model(model_id) for full observations",
+                "3. DECOMPOSE the question into component searches (see Query Strategy above)",
+                "   - Identify entities and concepts in the question",
+                "   - Search for each separately with targeted queries",
+                "4. Run multiple recall() calls - don't just echo the user's question",
+                "5. Use expand() if you need more context on specific memories",
+                "6. If you discover an important recurring topic worth tracking, use learn() to create a mental model",
+                "7. When ready, call done() with your answer and supporting memory_ids",
+                "",
+                "## When to Use learn()",
+                "Use learn() to create a new mental model when you discover:",
+                "- A person, project, or concept that appears frequently in memories",
+                "- An important topic the user seems to care about but has no mental model for",
+                "- A pattern or relationship worth synthesizing for future reference",
+                "Example: learn(name='Project Alpha', description='Track goals, status, and key decisions for Project Alpha')",
+                "",
+                "## Output Format: Plain Text Answer",
+                "Call done() with a plain text 'answer' field.",
+                "- Do NOT use markdown formatting",
+                "- NEVER include memory IDs, UUIDs, or 'Memory references' in the answer text",
+                "- Put memory IDs ONLY in the memory_ids array parameter, not in the answer",
+            ]
+        )
+
+    parts.append("")
+    parts.append(f"## Memory Bank: {name}")
+
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    if context:
+        parts.append(f"\n## Additional Context\n{context}")
+
+    return "\n".join(parts)
+
+
+def build_agent_prompt(
+    query: str,
+    context_history: list[dict],
+    bank_profile: dict,
+    additional_context: str | None = None,
+) -> str:
+    """Build the user prompt for the reflect agent."""
+    parts = []
+
+    # Bank identity
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    parts.append(f"## Memory Bank Context\nName: {name}")
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits if present
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    # Additional context from caller
+    if additional_context:
+        parts.append(f"\n## Additional Context\n{additional_context}")
+
+    # Tool call history
+    if context_history:
+        parts.append("\n## Tool Results (synthesize and reason from this data)")
+        for i, entry in enumerate(context_history, 1):
+            tool = entry["tool"]
+            output = entry["output"]
+            # Format as proper JSON for LLM readability
+            try:
+                output_str = json.dumps(output, indent=2, default=str)
+            except (TypeError, ValueError):
+                output_str = str(output)
+            parts.append(f"\n### Call {i}: {tool}\n```json\n{output_str}\n```")
+
+    # The question
+    parts.append(f"\n## Question\n{query}")
+
+    # Instructions
+    if context_history:
+        parts.append(
+            "\n## Instructions\n"
+            "Based on the tool results above, either call more tools or provide your final answer. "
+            "Synthesize and reason from the data - make reasonable inferences when helpful. "
+            "If you have related information, use it to give the best possible answer."
+        )
+    else:
+        parts.append(
+            "\n## Instructions\n"
+            "Start by calling list_mental_models() to see available mental models - they contain pre-synthesized knowledge. "
+            "If a relevant model exists, use get_mental_model(model_id) to get its observations. "
+            "Then use recall(query) for specific details not covered by mental models."
+        )
+
+    return "\n".join(parts)
+
+
+def build_final_prompt(
+    query: str,
+    context_history: list[dict],
+    bank_profile: dict,
+    additional_context: str | None = None,
+) -> str:
+    """Build the final prompt when forcing a text response (no tools)."""
+    parts = []
+
+    # Bank identity
+    name = bank_profile.get("name", "Assistant")
+    mission = bank_profile.get("mission", "")
+
+    parts.append(f"## Memory Bank Context\nName: {name}")
+    if mission:
+        parts.append(f"Mission: {mission}")
+
+    # Disposition traits if present
+    disposition = bank_profile.get("disposition", {})
+    if disposition:
+        traits = []
+        if "skepticism" in disposition:
+            traits.append(f"skepticism={disposition['skepticism']}")
+        if "literalism" in disposition:
+            traits.append(f"literalism={disposition['literalism']}")
+        if "empathy" in disposition:
+            traits.append(f"empathy={disposition['empathy']}")
+        if traits:
+            parts.append(f"Disposition: {', '.join(traits)}")
+
+    # Additional context from caller
+    if additional_context:
+        parts.append(f"\n## Additional Context\n{additional_context}")
+
+    # Tool call history
+    if context_history:
+        parts.append("\n## Retrieved Data (synthesize and reason from this data)")
+        for entry in context_history:
+            tool = entry["tool"]
+            output = entry["output"]
+            # Format as proper JSON for LLM readability
+            try:
+                output_str = json.dumps(output, indent=2, default=str)
+            except (TypeError, ValueError):
+                output_str = str(output)
+            parts.append(f"\n### From {tool}:\n```json\n{output_str}\n```")
+    else:
+        parts.append("\n## Retrieved Data\nNo data was retrieved.")
+
+    # The question
+    parts.append(f"\n## Question\n{query}")
+
+    # Final instructions
+    parts.append(
+        "\n## Instructions\n"
+        "Provide a thoughtful answer by synthesizing and reasoning from the retrieved data above. "
+        "You can make reasonable inferences from the memories, but don't completely fabricate information."
+        "If the exact answer isn't stated, use what IS stated to give the best possible answer. "
+        "Only say 'I don't have information' if the retrieved data is truly unrelated to the question."
+    )
+
+    return "\n".join(parts)
+
+
+FINAL_SYSTEM_PROMPT = """You are a thoughtful assistant that synthesizes answers from retrieved memories.
+
+Your approach:
+- Reason over the retrieved memories to answer the question
+- Make reasonable inferences when the exact answer isn't explicitly stated
+- Connect related memories to form a complete picture
+- Be helpful - if you have related information, use it to give the best possible answer
+
+Only say "I don't have information" if the retrieved data is truly unrelated to the question.
+Do NOT fabricate information that has no basis in the retrieved data."""

hindsight-api/hindsight_api/engine/reflect/tools.py

@@ -0,0 +1,425 @@
+"""
+Tool implementations for the reflect agent.
+"""
+
+import logging
+import re
+import uuid
+from typing import TYPE_CHECKING, Any
+
+from .models import MentalModelInput
+
+if TYPE_CHECKING:
+    from asyncpg import Connection
+
+    from ...api.http import RequestContext
+    from ..memory_engine import MemoryEngine
+
+logger = logging.getLogger(__name__)
+
+
+def generate_model_id(name: str) -> str:
+    """Generate a stable ID from mental model name."""
+    # Normalize: lowercase, replace spaces/special chars with hyphens
+    normalized = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
+    # Truncate to reasonable length
+    return normalized[:50]
+
+
+async def tool_lookup(
+    conn: "Connection",
+    bank_id: str,
+    model_id: str | None = None,
+    tags: list[str] | None = None,
+    tags_match: str = "any",
+) -> dict[str, Any]:
+    """
+    List or get mental models.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        model_id: Optional specific model ID to get (if None, lists all)
+        tags: Optional tags to filter models (when listing)
+        tags_match: How to match tags - "any" (OR), "all" (AND)
+
+    Returns:
+        Dict with either a list of models or a single model's details
+    """
+    if model_id:
+        # Get specific mental model with full details including observations
+        row = await conn.fetchrow(
+            """
+            SELECT id, subtype, name, description, observations, entity_id, last_updated
+            FROM mental_models
+            WHERE id = $1 AND bank_id = $2
+            """,
+            model_id,
+            bank_id,
+        )
+        if row:
+            # Parse observations JSON
+            obs_data = row["observations"] or {"observations": []}
+            if isinstance(obs_data, str):
+                import json
+
+                obs_data = json.loads(obs_data)
+            observations_raw = obs_data.get("observations", []) if isinstance(obs_data, dict) else obs_data
+
+            # Normalize observation format: map memory_ids/fact_ids to based_on
+            observations = []
+            for obs in observations_raw:
+                if isinstance(obs, dict):
+                    based_on = obs.get("memory_ids") or obs.get("fact_ids") or []
+                    observations.append(
+                        {
+                            "title": obs.get("title", ""),
+                            "text": obs.get("text", ""),
+                            "based_on": based_on,
+                        }
+                    )
+
+            return {
+                "found": True,
+                "model": {
+                    "id": row["id"],
+                    "subtype": row["subtype"],
+                    "name": row["name"],
+                    "description": row["description"],
+                    "observations": observations,  # [{title, text, based_on}, ...]
+                    "entity_id": str(row["entity_id"]) if row["entity_id"] else None,
+                    "last_updated": row["last_updated"].isoformat() if row["last_updated"] else None,
+                },
+            }
+        return {"found": False, "model_id": model_id}
+    else:
+        # List mental models (compact: id, name, description only)
+        # Full observations are retrieved via get_mental_model(model_id)
+        # Filter by tags if provided
+        if tags:
+            if tags_match == "all":
+                # All tags must match
+                rows = await conn.fetch(
+                    """
+                    SELECT id, subtype, name, description
+                    FROM mental_models
+                    WHERE bank_id = $1 AND tags @> $2::varchar[]
+                    ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                    """,
+                    bank_id,
+                    tags,
+                )
+            else:
+                # Any tag matches (OR) - default
+                rows = await conn.fetch(
+                    """
+                    SELECT id, subtype, name, description
+                    FROM mental_models
+                    WHERE bank_id = $1 AND tags && $2::varchar[]
+                    ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                    """,
+                    bank_id,
+                    tags,
+                )
+        else:
+            rows = await conn.fetch(
+                """
+                SELECT id, subtype, name, description
+                FROM mental_models
+                WHERE bank_id = $1
+                ORDER BY last_updated DESC NULLS LAST, created_at DESC
+                """,
+                bank_id,
+            )
+
+        return {
+            "count": len(rows),
+            "models": [
+                {
+                    "id": row["id"],
+                    "subtype": row["subtype"],
+                    "name": row["name"],
+                    "description": row["description"],
+                }
+                for row in rows
+            ],
+        }
+
+
+async def tool_recall(
+    memory_engine: "MemoryEngine",
+    bank_id: str,
+    query: str,
+    request_context: "RequestContext",
+    max_tokens: int = 2048,
+    max_results: int = 50,
+    tags: list[str] | None = None,
+    tags_match: str = "any",
+    connection_budget: int = 1,
+) -> dict[str, Any]:
+    """
+    Search memories using TEMPR retrieval.
+
+    Args:
+        memory_engine: Memory engine instance
+        bank_id: Bank identifier
+        query: Search query
+        request_context: Request context for authentication
+        max_tokens: Maximum tokens for results (default 2048)
+        max_results: Maximum number of results
+        tags: Filter by tags (includes untagged memories)
+        tags_match: How to match tags - "any" (OR), "all" (AND), or "exact"
+        connection_budget: Max DB connections for this recall (default 1 for internal ops)
+
+    Returns:
+        Dict with list of matching memories
+    """
+    result = await memory_engine.recall_async(
+        bank_id=bank_id,
+        query=query,
+        fact_type=["experience", "world"],  # Exclude opinions
+        max_tokens=max_tokens,
+        enable_trace=False,
+        request_context=request_context,
+        tags=tags,
+        tags_match=tags_match,
+        _connection_budget=connection_budget,
+    )
+
+    memories = []
+    for m in result.results[:max_results]:
+        memories.append(
+            {
+                "id": str(m.id),
+                "text": m.text,
+                "type": m.fact_type,
+                "entities": m.entities or [],
+                "occurred": m.occurred_start,  # Already ISO format string
+            }
+        )
+
+    return {
+        "query": query,
+        "count": len(memories),
+        "memories": memories,
+    }
+
+
+async def tool_learn(
+    conn: "Connection",
+    bank_id: str,
+    input: MentalModelInput,
+    tags: list[str] | None = None,
+) -> dict[str, Any]:
+    """
+    Create a mental model placeholder with subtype='learned'.
+
+    The agent only specifies name and description - actual observations are generated
+    in the background via refresh, similar to pinned models.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        input: Mental model input data (name, description, optional entity_id)
+        tags: Tags to apply to new mental models (from reflect context)
+
+    Returns:
+        Dict with created model info including model_id for background generation
+    """
+    model_id = generate_model_id(input.name)
+
+    # Parse entity_id if provided
+    entity_uuid = None
+    if input.entity_id:
+        try:
+            entity_uuid = uuid.UUID(input.entity_id)
+        except ValueError:
+            logger.warning(f"Invalid entity_id format: {input.entity_id}")
+
+    # Check if model exists
+    existing = await conn.fetchrow(
+        "SELECT id FROM mental_models WHERE id = $1 AND bank_id = $2",
+        model_id,
+        bank_id,
+    )
+
+    if existing:
+        # Update description only - observations will be regenerated
+        await conn.execute(
+            """
+            UPDATE mental_models SET
+                description = $3,
+                entity_id = $4
+            WHERE id = $1 AND bank_id = $2
+            """,
+            model_id,
+            bank_id,
+            input.description,
+            entity_uuid,
+        )
+        status = "updated"
+    else:
+        # Insert new model placeholder - observations will be generated in background
+        await conn.execute(
+            """
+            INSERT INTO mental_models (id, bank_id, subtype, name, description, observations, entity_id, tags, created_at)
+            VALUES ($1, $2, 'learned', $3, $4, '{}'::jsonb, $5, $6, NOW())
+            """,
+            model_id,
+            bank_id,
+            input.name,
+            input.description,
+            entity_uuid,
+            tags or [],
+        )
+        status = "created"
+
+    logger.info(f"[REFLECT] Mental model '{model_id}' {status} in bank {bank_id} - pending background generation")
+
+    return {
+        "status": status,
+        "model_id": model_id,
+        "name": input.name,
+        "pending_generation": True,
+    }
+
+
+async def tool_expand(
+    conn: "Connection",
+    bank_id: str,
+    memory_ids: list[str],
+    depth: str,
+) -> dict[str, Any]:
+    """
+    Expand multiple memories to get chunk or document context.
+
+    Args:
+        conn: Database connection
+        bank_id: Bank identifier
+        memory_ids: List of memory unit IDs
+        depth: "chunk" or "document"
+
+    Returns:
+        Dict with results array, each containing memory, chunk, and optionally document data
+    """
+    if not memory_ids:
+        return {"error": "memory_ids is required and must not be empty"}
+
+    # Validate and convert UUIDs
+    valid_uuids: list[uuid.UUID] = []
+    errors: dict[str, str] = {}
+    for mid in memory_ids:
+        try:
+            valid_uuids.append(uuid.UUID(mid))
+        except ValueError:
+            errors[mid] = f"Invalid memory_id format: {mid}"
+
+    if not valid_uuids:
+        return {"error": "No valid memory IDs provided", "details": errors}
+
+    # Batch fetch all memory units
+    memories = await conn.fetch(
+        """
+        SELECT id, text, chunk_id, document_id, fact_type, context
+        FROM memory_units
+        WHERE id = ANY($1) AND bank_id = $2
+        """,
+        valid_uuids,
+        bank_id,
+    )
+    memory_map = {row["id"]: row for row in memories}
+
+    # Collect chunk_ids and document_ids for batch fetching
+    chunk_ids = [m["chunk_id"] for m in memories if m["chunk_id"]]
+    doc_ids_from_chunks: set[str] = set()
+    doc_ids_direct: set[str] = set()
+
+    # Batch fetch all chunks
+    chunk_map: dict[str, Any] = {}
+    if chunk_ids:
+        chunks = await conn.fetch(
+            """
+            SELECT chunk_id, chunk_text, chunk_index, document_id
+            FROM chunks
+            WHERE chunk_id = ANY($1)
+            """,
+            chunk_ids,
+        )
+        chunk_map = {row["chunk_id"]: row for row in chunks}
+        if depth == "document":
+            doc_ids_from_chunks = {c["document_id"] for c in chunks if c["document_id"]}
+
+    # Collect direct document IDs (memories without chunks)
+    if depth == "document":
+        for m in memories:
+            if not m["chunk_id"] and m["document_id"]:
+                doc_ids_direct.add(m["document_id"])
+
+    # Batch fetch all documents
+    doc_map: dict[str, Any] = {}
+    all_doc_ids = list(doc_ids_from_chunks | doc_ids_direct)
+    if all_doc_ids:
+        docs = await conn.fetch(
+            """
+            SELECT id, original_text, metadata, retain_params
+            FROM documents
+            WHERE id = ANY($1) AND bank_id = $2
+            """,
+            all_doc_ids,
+            bank_id,
+        )
+        doc_map = {row["id"]: row for row in docs}
+
+    # Build results
+    results: list[dict[str, Any]] = []
+    for mid, mem_uuid in zip(memory_ids, valid_uuids):
+        if mid in errors:
+            results.append({"memory_id": mid, "error": errors[mid]})
+            continue
+
+        memory = memory_map.get(mem_uuid)
+        if not memory:
+            results.append({"memory_id": mid, "error": f"Memory not found: {mid}"})
+            continue
+
+        item: dict[str, Any] = {
+            "memory_id": mid,
+            "memory": {
+                "id": str(memory["id"]),
+                "text": memory["text"],
+                "type": memory["fact_type"],
+                "context": memory["context"],
+            },
+        }
+
+        # Add chunk if available
+        if memory["chunk_id"] and memory["chunk_id"] in chunk_map:
+            chunk = chunk_map[memory["chunk_id"]]
+            item["chunk"] = {
+                "id": chunk["chunk_id"],
+                "text": chunk["chunk_text"],
+                "index": chunk["chunk_index"],
+                "document_id": chunk["document_id"],
+            }
+            # Add document if depth=document
+            if depth == "document" and chunk["document_id"] in doc_map:
+                doc = doc_map[chunk["document_id"]]
+                item["document"] = {
+                    "id": doc["id"],
+                    "full_text": doc["original_text"],
+                    "metadata": doc["metadata"],
+                    "retain_params": doc["retain_params"],
+                }
+        elif memory["document_id"] and depth == "document" and memory["document_id"] in doc_map:
+            # No chunk, but has document_id
+            doc = doc_map[memory["document_id"]]
+            item["document"] = {
+                "id": doc["id"],
+                "full_text": doc["original_text"],
+                "metadata": doc["metadata"],
+                "retain_params": doc["retain_params"],
+            }
+
+        results.append(item)
+
+    return {"results": results, "count": len(results)}

hindsight-api/hindsight_api/engine/reflect/tools_schema.py

@@ -0,0 +1,212 @@
+"""
+Tool schema definitions for the reflect agent.
+
+These are OpenAI-format tool definitions used with native tool calling.
+"""
+
+from typing import Literal
+
+# Tool definitions in OpenAI format
+TOOL_LIST_MENTAL_MODELS = {
+    "type": "function",
+    "function": {
+        "name": "list_mental_models",
+        "description": "List all available mental models - your synthesized knowledge about entities, concepts, and events. Returns an array of models with id, name, and description.",
+        "parameters": {
+            "type": "object",
+            "properties": {},
+            "required": [],
+        },
+    },
+}
+
+TOOL_GET_MENTAL_MODEL = {
+    "type": "function",
+    "function": {
+        "name": "get_mental_model",
+        "description": "Get full details of a specific mental model including all observations and memory references.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "model_id": {
+                    "type": "string",
+                    "description": "ID of the mental model (from list_mental_models results)",
+                },
+            },
+            "required": ["model_id"],
+        },
+    },
+}
+
+TOOL_RECALL = {
+    "type": "function",
+    "function": {
+        "name": "recall",
+        "description": "Search memories using semantic + temporal retrieval. Returns relevant memories from experience and world knowledge, each with an 'id' you can reference.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "Search query string",
+                },
+                "max_tokens": {
+                    "type": "integer",
+                    "description": "Optional limit on result size (default 2048). Use higher values for broader searches.",
+                },
+            },
+            "required": ["query"],
+        },
+    },
+}
+
+TOOL_LEARN = {
+    "type": "function",
+    "function": {
+        "name": "learn",
+        "description": "Create a new mental model to track an important recurring topic. Use when you discover a person, project, concept, or pattern that appears frequently and would benefit from synthesized knowledge. The model content will be generated automatically.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "name": {
+                    "type": "string",
+                    "description": "Human-readable name (e.g., 'Project Alpha', 'John Smith', 'Product Strategy')",
+                },
+                "description": {
+                    "type": "string",
+                    "description": "What to track and synthesize (e.g., 'Track goals, milestones, blockers, and key decisions for Project Alpha')",
+                },
+            },
+            "required": ["name", "description"],
+        },
+    },
+}
+
+TOOL_EXPAND = {
+    "type": "function",
+    "function": {
+        "name": "expand",
+        "description": "Get more context for one or more memories. Memory hierarchy: memory -> chunk -> document.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "memory_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of memory IDs from recall results (batch multiple for efficiency)",
+                },
+                "depth": {
+                    "type": "string",
+                    "enum": ["chunk", "document"],
+                    "description": "chunk: surrounding text chunk, document: full source document",
+                },
+            },
+            "required": ["memory_ids", "depth"],
+        },
+    },
+}
+
+TOOL_DONE_ANSWER = {
+    "type": "function",
+    "function": {
+        "name": "done",
+        "description": "Signal completion with your final answer. Use this when you have gathered enough information to answer the question.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "answer": {
+                    "type": "string",
+                    "description": "Your response as plain text. Do NOT use markdown formatting. NEVER include memory IDs, UUIDs, or 'Memory references' in this text - put IDs only in memory_ids array.",
+                },
+                "memory_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of memory IDs that support your answer (put IDs here, NOT in answer text)",
+                },
+                "model_ids": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Array of mental model IDs that support your answer",
+                },
+            },
+            "required": ["answer"],
+        },
+    },
+}
+
+TOOL_DONE_OBSERVATIONS = {
+    "type": "function",
+    "function": {
+        "name": "done",
+        "description": "Signal completion with MULTIPLE structured observations. Each observation must be a SEPARATE item in the array covering ONE theme. Do NOT combine all content into a single observation.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "observations": {
+                    "type": "array",
+                    "minItems": 3,
+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "title": {
+                                "type": "string",
+                                "description": "Short header for this observation's theme (e.g., 'Work Style', 'Technical Skills')",
+                            },
+                            "text": {
+                                "type": "string",
+                                "description": "Observation content about ONE theme. End with 'Key evidence:' containing text citations (summaries of what memories say), NOT memory IDs.",
+                            },
+                            "memory_ids": {
+                                "type": "array",
+                                "items": {"type": "string"},
+                                "description": "Full UUIDs of memories supporting this observation (put IDs here, not in text)",
+                            },
+                        },
+                        "required": ["title", "text", "memory_ids"],
+                    },
+                    "description": "Array of 3-8 observations, each covering a DIFFERENT aspect/theme. Do NOT put everything in one observation.",
+                },
+            },
+            "required": ["observations"],
+        },
+    },
+}
+
+
+def get_reflect_tools(
+    enable_learn: bool = True, output_mode: Literal["answer", "observations"] = "answer"
+) -> list[dict]:
+    """
+    Get the list of tools for the reflect agent.
+
+    Args:
+        enable_learn: Whether to include the learn tool
+        output_mode: "answer" or "observations" - determines done tool format
+                     In observations mode, mental model tools are excluded to avoid
+                     using potentially outdated models during regeneration.
+
+    Returns:
+        List of tool definitions in OpenAI format
+    """
+    tools = []
+
+    # In answer mode, include mental model tools for lookup
+    # In observations mode (mental model generation), exclude them to avoid circular references
+    if output_mode == "answer":
+        tools.append(TOOL_LIST_MENTAL_MODELS)
+        tools.append(TOOL_GET_MENTAL_MODEL)
+
+    tools.append(TOOL_RECALL)
+
+    if enable_learn:
+        tools.append(TOOL_LEARN)
+
+    tools.append(TOOL_EXPAND)
+
+    # Add appropriate done tool based on output mode
+    if output_mode == "observations":
+        tools.append(TOOL_DONE_OBSERVATIONS)
+    else:
+        tools.append(TOOL_DONE_ANSWER)
+
+    return tools

hindsight-api/hindsight_api/engine/response_models.py

@@ -10,8 +10,52 @@ from typing import Any
 
 from pydantic import BaseModel, ConfigDict, Field
 
-# Valid fact types for recall operations (excludes 'observation' which is internal)
-VALID_RECALL_FACT_TYPES = frozenset(["world", "experience", "opinion"])
+# Valid fact types for recall operations (excludes 'observation' which is internal, and 'opinion' which is deprecated)
+VALID_RECALL_FACT_TYPES = frozenset(["world", "experience"])
+
+
+class LLMToolCall(BaseModel):
+    """A tool call requested by the LLM."""
+
+    id: str = Field(description="Unique identifier for this tool call")
+    name: str = Field(description="Name of the tool to call")
+    arguments: dict[str, Any] = Field(description="Arguments to pass to the tool")
+
+
+class LLMToolCallResult(BaseModel):
+    """Result from an LLM call that may include tool calls."""
+
+    content: str | None = Field(default=None, description="Text content if any")
+    tool_calls: list[LLMToolCall] = Field(default_factory=list, description="Tool calls requested by the LLM")
+    finish_reason: str | None = Field(default=None, description="Reason the LLM stopped: 'stop', 'tool_calls', etc.")
+
+
+class ToolCallTrace(BaseModel):
+    """A single tool call made during reflect."""
+
+    tool: str = Field(description="Tool name: lookup, recall, learn, expand")
+    input: dict = Field(description="Tool input parameters")
+    output: dict = Field(description="Tool output/result")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+    iteration: int = Field(default=0, description="Iteration number (1-based) when this tool was called")
+
+
+class LLMCallTrace(BaseModel):
+    """A single LLM call made during reflect."""
+
+    scope: str = Field(description="Call scope: agent_1, agent_2, final, etc.")
+    duration_ms: int = Field(description="Execution time in milliseconds")
+
+
+class MentalModelRef(BaseModel):
+    """Reference to a mental model accessed during reflect."""
+
+    id: str = Field(description="Mental model ID")
+    name: str = Field(description="Mental model name")
+    type: str = Field(description="Mental model type: entity, concept, event")
+    subtype: str = Field(description="Mental model subtype: structural, emergent, learned")
+    description: str = Field(description="Brief description")
+    summary: str | None = Field(default=None, description="Full summary (when looked up in detail)")
 
 
 class TokenUsage(BaseModel):
@@ -85,6 +129,7 @@ class MemoryFact(BaseModel):
                 "metadata": {"source": "slack"},
                 "chunk_id": "bank123_session_abc123_0",
                 "activation": 0.95,
+                "tags": ["user_a", "session_123"],
             }
         }
     )
@@ -102,6 +147,7 @@ class MemoryFact(BaseModel):
     chunk_id: str | None = Field(
         None, description="ID of the chunk this fact was extracted from (format: bank_id_document_id_chunk_index)"
     )
+    tags: list[str] | None = Field(None, description="Visibility scope tags associated with this fact")
 
 
 class ChunkInfo(BaseModel):
@@ -196,6 +242,18 @@ class ReflectResult(BaseModel):
         default=None,
         description="Token usage metrics for the LLM calls made during this reflect operation.",
     )
+    tool_trace: list[ToolCallTrace] = Field(
+        default_factory=list,
+        description="Trace of tool calls made during reflection. Only present when include.tool_calls is enabled.",
+    )
+    llm_trace: list[LLMCallTrace] = Field(
+        default_factory=list,
+        description="Trace of LLM calls made during reflection. Only present when include.tool_calls is enabled.",
+    )
+    mental_models: list[MentalModelRef] = Field(
+        default_factory=list,
+        description="Mental models accessed during reflection. Only present when include.facts is enabled.",
+    )
 
 
 class Opinion(BaseModel):
@@ -259,3 +317,32 @@ class EntityState(BaseModel):
     observations: list[EntityObservation] = Field(
         default_factory=list, description="List of observations about this entity"
     )
+
+
+class MentalModel(BaseModel):
+    """
+    A manually configured mental model for tracking specific topics/areas.
+
+    Mental models are user-defined focus areas that the agent should track
+    and maintain summaries for, unlike auto-extracted entities.
+    """
+
+    model_config = ConfigDict(
+        json_schema_extra={
+            "example": {
+                "id": "team-dynamics",
+                "name": "Team Dynamics",
+                "description": "Track how the team collaborates, communication patterns, conflicts, and resolutions",
+                "summary": "The team has strong collaboration...",
+                "summary_updated_at": "2024-01-15T10:30:00Z",
+                "created_at": "2024-01-10T08:00:00Z",
+            }
+        }
+    )
+
+    id: str = Field(description="Unique identifier (alphanumeric lowercase)")
+    name: str = Field(description="Display name for the mental model")
+    description: str = Field(description="Prompt/directions for what to track and summarize")
+    summary: str | None = Field(None, description="Generated summary based on relevant facts")
+    summary_updated_at: str | None = Field(None, description="ISO format date when summary was last updated")
+    created_at: str = Field(description="ISO format date when the mental model was created")

hindsight-api/hindsight_api/engine/retain/bank_utils.py

@@ -1,5 +1,5 @@
 """
-bank profile utilities for disposition and background management.
+bank profile utilities for disposition and mission management.
 """
 
 import json
@@ -27,19 +27,18 @@ class BankProfile(TypedDict):
 
     name: str
     disposition: DispositionTraits
-    background: str
+    mission: str
 
 
-class BackgroundMergeResponse(BaseModel):
-    """LLM response for background merge with disposition inference."""
+class MissionMergeResponse(BaseModel):
+    """LLM response for mission merge."""
 
-    background: str = Field(description="Merged background in first person perspective")
-    disposition: DispositionTraits = Field(description="Inferred disposition traits (skepticism, literalism, empathy)")
+    mission: str = Field(description="Merged mission in first person perspective")
 
 
 async def get_bank_profile(pool, bank_id: str) -> BankProfile:
     """
-    Get bank profile (name, disposition + background).
+    Get bank profile (name, disposition + mission).
     Auto-creates bank with default values if not exists.
 
     Args:
@@ -47,13 +46,13 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
         bank_id: bank IDentifier
 
     Returns:
-        BankProfile with name, typed DispositionTraits, and background
+        BankProfile with name, typed DispositionTraits, and mission
     """
     async with acquire_with_retry(pool) as conn:
         # Try to get existing bank
         row = await conn.fetchrow(
             f"""
-            SELECT name, disposition, background
+            SELECT name, disposition, mission
             FROM {fq_table("banks")} WHERE bank_id = $1
             """,
             bank_id,
@@ -66,13 +65,15 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
                 disposition_data = json.loads(disposition_data)
 
             return BankProfile(
-                name=row["name"], disposition=DispositionTraits(**disposition_data), background=row["background"]
+                name=row["name"],
+                disposition=DispositionTraits(**disposition_data),
+                mission=row["mission"] or "",
             )
 
         # Bank doesn't exist, create with defaults
         await conn.execute(
             f"""
-            INSERT INTO {fq_table("banks")} (bank_id, name, disposition, background)
+            INSERT INTO {fq_table("banks")} (bank_id, name, disposition, mission)
             VALUES ($1, $2, $3::jsonb, $4)
             ON CONFLICT (bank_id) DO NOTHING
             """,
@@ -82,7 +83,7 @@ async def get_bank_profile(pool, bank_id: str) -> BankProfile:
             "",
         )
 
-        return BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), background="")
+        return BankProfile(name=bank_id, disposition=DispositionTraits(**DEFAULT_DISPOSITION), mission="")
 
 
 async def update_bank_disposition(pool, bank_id: str, disposition: dict[str, int]) -> None:
@@ -110,244 +111,121 @@ async def update_bank_disposition(pool, bank_id: str, disposition: dict[str, int
         )
 
 
-async def merge_bank_background(pool, llm_config, bank_id: str, new_info: str, update_disposition: bool = True) -> dict:
+async def set_bank_mission(pool, bank_id: str, mission: str) -> None:
     """
-    Merge new background information with existing background using LLM.
-    Normalizes to first person ("I") and resolves conflicts.
-    Optionally infers disposition traits from the merged background.
+    Set bank mission (replacing any existing mission).
 
     Args:
         pool: Database connection pool
-        llm_config: LLM configuration for background merging
         bank_id: bank IDentifier
-        new_info: New background information to add/merge
-        update_disposition: If True, infer Big Five traits from background (default: True)
+        mission: The mission text
+    """
+    # Ensure bank exists first
+    await get_bank_profile(pool, bank_id)
+
+    async with acquire_with_retry(pool) as conn:
+        await conn.execute(
+            f"""
+            UPDATE {fq_table("banks")}
+            SET mission = $2,
+                updated_at = NOW()
+            WHERE bank_id = $1
+            """,
+            bank_id,
+            mission,
+        )
+
+
+async def merge_bank_mission(pool, llm_config, bank_id: str, new_info: str) -> dict:
+    """
+    Merge new mission information with existing mission using LLM.
+    Normalizes to first person ("I") and resolves conflicts.
+
+    Args:
+        pool: Database connection pool
+        llm_config: LLM configuration for mission merging
+        bank_id: bank IDentifier
+        new_info: New mission information to add/merge
 
     Returns:
-        Dict with 'background' (str) and optionally 'disposition' (dict) keys
+        Dict with 'mission' (str) key
     """
     # Get current profile
     profile = await get_bank_profile(pool, bank_id)
-    current_background = profile["background"]
+    current_mission = profile["mission"]
 
-    # Use LLM to merge backgrounds and optionally infer disposition
-    result = await _llm_merge_background(llm_config, current_background, new_info, infer_disposition=update_disposition)
+    # Use LLM to merge missions
+    result = await _llm_merge_mission(llm_config, current_mission, new_info)
 
-    merged_background = result["background"]
-    inferred_disposition = result.get("disposition")
+    merged_mission = result["mission"]
 
     # Update in database
     async with acquire_with_retry(pool) as conn:
-        if inferred_disposition:
-            # Update both background and disposition
-            await conn.execute(
-                f"""
-                UPDATE {fq_table("banks")}
-                SET background = $2,
-                    disposition = $3::jsonb,
-                    updated_at = NOW()
-                WHERE bank_id = $1
-                """,
-                bank_id,
-                merged_background,
-                json.dumps(inferred_disposition),
-            )
-        else:
-            # Update only background
-            await conn.execute(
-                f"""
-                UPDATE {fq_table("banks")}
-                SET background = $2,
-                    updated_at = NOW()
-                WHERE bank_id = $1
-                """,
-                bank_id,
-                merged_background,
-            )
+        await conn.execute(
+            f"""
+            UPDATE {fq_table("banks")}
+            SET mission = $2,
+                updated_at = NOW()
+            WHERE bank_id = $1
+            """,
+            bank_id,
+            merged_mission,
+        )
 
-    response = {"background": merged_background}
-    if inferred_disposition:
-        response["disposition"] = inferred_disposition
-
-    return response
+    return {"mission": merged_mission}
 
 
-async def _llm_merge_background(llm_config, current: str, new_info: str, infer_disposition: bool = False) -> dict:
+async def _llm_merge_mission(llm_config, current: str, new_info: str) -> dict:
     """
-    Use LLM to intelligently merge background information.
-    Optionally infer Big Five disposition traits from the merged background.
+    Use LLM to intelligently merge mission information.
 
     Args:
         llm_config: LLM configuration to use
-        current: Current background text
+        current: Current mission text
         new_info: New information to merge
-        infer_disposition: If True, also infer disposition traits
 
     Returns:
-        Dict with 'background' (str) and optionally 'disposition' (dict) keys
+        Dict with 'mission' (str) key
     """
-    if infer_disposition:
-        prompt = f"""You are helping maintain a memory bank's background/profile and infer their disposition. You MUST respond with ONLY valid JSON.
+    prompt = f"""You are helping maintain an agent's mission statement.
 
-Current background: {current if current else "(empty)"}
+Current mission: {current if current else "(empty)"}
 
 New information to add: {new_info}
 
 Instructions:
-1. Merge the new information with the current background
-2. If there are conflicts (e.g., different birthplaces), the NEW information overwrites the old
-3. Keep additions that don't conflict
-4. Output in FIRST PERSON ("I") perspective
-5. Be concise - keep merged background under 500 characters
-6. Infer disposition traits from the merged background (each 1-5 integer):
-   - Skepticism: 1-5 (1=trusting, takes things at face value; 5=skeptical, questions everything)
-   - Literalism: 1-5 (1=flexible interpretation, reads between lines; 5=literal, exact interpretation)
-   - Empathy: 1-5 (1=detached, focuses on facts; 5=empathetic, considers emotional context)
-
-CRITICAL: You MUST respond with ONLY a valid JSON object. No markdown, no code blocks, no explanations. Just the JSON.
-
-Format:
-{{
-  "background": "the merged background text in first person",
-  "disposition": {{
-    "skepticism": 3,
-    "literalism": 3,
-    "empathy": 3
-  }}
-}}
-
-Trait inference examples:
-- "I'm a lawyer" → skepticism: 4, literalism: 5, empathy: 2
-- "I'm a therapist" → skepticism: 2, literalism: 2, empathy: 5
-- "I'm an engineer" → skepticism: 3, literalism: 4, empathy: 3
-- "I've been burned before by trusting people" → skepticism: 5, literalism: 3, empathy: 3
-- "I try to understand what people really mean" → skepticism: 3, literalism: 2, empathy: 4
-- "I take contracts very seriously" → skepticism: 4, literalism: 5, empathy: 2"""
-    else:
-        prompt = f"""You are helping maintain a memory bank's background/profile.
-
-Current background: {current if current else "(empty)"}
-
-New information to add: {new_info}
-
-Instructions:
-1. Merge the new information with the current background
-2. If there are conflicts (e.g., different birthplaces), the NEW information overwrites the old
+1. Merge the new information with the current mission
+2. If there are conflicts, the NEW information overwrites the old
 3. Keep additions that don't conflict
 4. Output in FIRST PERSON ("I") perspective
 5. Be concise - keep it under 500 characters
-6. Return ONLY the merged background text, no explanations
+6. Return ONLY the merged mission text, no explanations
 
-Merged background:"""
+Merged mission:"""
 
     try:
-        # Prepare messages
         messages = [{"role": "user", "content": prompt}]
 
-        if infer_disposition:
-            # Use structured output with Pydantic model for disposition inference
-            try:
-                parsed = await llm_config.call(
-                    messages=messages,
-                    response_format=BackgroundMergeResponse,
-                    scope="bank_background",
-                    temperature=0.3,
-                    max_completion_tokens=8192,
-                )
-                logger.info(f"Successfully got structured response: background={parsed.background[:100]}")
-
-                # Convert Pydantic model to dict format
-                return {"background": parsed.background, "disposition": parsed.disposition.model_dump()}
-            except Exception as e:
-                logger.warning(f"Structured output failed, falling back to manual parsing: {e}")
-                # Fall through to manual parsing below
-
-        # Manual parsing fallback or non-disposition merge
         content = await llm_config.call(
-            messages=messages, scope="bank_background", temperature=0.3, max_completion_tokens=8192
+            messages=messages, scope="bank_mission", temperature=0.3, max_completion_tokens=8192
         )
 
-        logger.info(f"LLM response for background merge (first 500 chars): {content[:500]}")
+        logger.info(f"LLM response for mission merge (first 500 chars): {content[:500]}")
 
-        if infer_disposition:
-            # Parse JSON response - try multiple extraction methods
-            result = None
-
-            # Method 1: Direct parse
-            try:
-                result = json.loads(content)
-                logger.info("Successfully parsed JSON directly")
-            except json.JSONDecodeError:
-                pass
-
-            # Method 2: Extract from markdown code blocks
-            if result is None:
-                # Remove markdown code blocks
-                code_block_match = re.search(r"```(?:json)?\s*(\{.*?\})\s*```", content, re.DOTALL)
-                if code_block_match:
-                    try:
-                        result = json.loads(code_block_match.group(1))
-                        logger.info("Successfully extracted JSON from markdown code block")
-                    except json.JSONDecodeError:
-                        pass
-
-            # Method 3: Find nested JSON structure
-            if result is None:
-                # Look for JSON object with nested structure
-                json_match = re.search(
-                    r'\{[^{}]*"background"[^{}]*"disposition"[^{}]*\{[^{}]*\}[^{}]*\}', content, re.DOTALL
-                )
-                if json_match:
-                    try:
-                        result = json.loads(json_match.group())
-                        logger.info("Successfully extracted JSON using nested pattern")
-                    except json.JSONDecodeError:
-                        pass
-
-            # All parsing methods failed - use fallback
-            if result is None:
-                logger.warning(f"Failed to extract JSON from LLM response. Raw content: {content[:200]}")
-                # Fallback: use new_info as background with default disposition
-                return {
-                    "background": new_info if new_info else current if current else "",
-                    "disposition": DEFAULT_DISPOSITION.copy(),
-                }
-
-            # Validate disposition values
-            disposition = result.get("disposition", {})
-            for key in ["skepticism", "literalism", "empathy"]:
-                if key not in disposition:
-                    disposition[key] = 3  # Default to neutral
-                else:
-                    # Clamp to [1, 5] and convert to int
-                    disposition[key] = max(1, min(5, int(disposition[key])))
-
-            result["disposition"] = disposition
-
-            # Ensure background exists
-            if "background" not in result or not result["background"]:
-                result["background"] = new_info if new_info else ""
-
-            return result
-        else:
-            # Just background merge
-            merged = content
-            if not merged or merged.lower() in ["(empty)", "none", "n/a"]:
-                merged = new_info if new_info else ""
-            return {"background": merged}
+        merged = content.strip()
+        if not merged or merged.lower() in ["(empty)", "none", "n/a"]:
+            merged = new_info if new_info else ""
+        return {"mission": merged}
 
     except Exception as e:
-        logger.error(f"Error merging background with LLM: {e}")
+        logger.error(f"Error merging mission with LLM: {e}")
         # Fallback: just append new info
         if current:
             merged = f"{current} {new_info}".strip()
         else:
             merged = new_info
 
-        result = {"background": merged}
-        if infer_disposition:
-            result["disposition"] = DEFAULT_DISPOSITION.copy()
-        return result
+        return {"mission": merged}
 
 
 async def list_banks(pool) -> list:
@@ -358,12 +236,12 @@ async def list_banks(pool) -> list:
         pool: Database connection pool
 
     Returns:
-        List of dicts with bank_id, name, disposition, background, created_at, updated_at
+        List of dicts with bank_id, name, disposition, mission, created_at, updated_at
     """
     async with acquire_with_retry(pool) as conn:
         rows = await conn.fetch(
             f"""
-            SELECT bank_id, name, disposition, background, created_at, updated_at
+            SELECT bank_id, name, disposition, mission, created_at, updated_at
             FROM {fq_table("banks")}
             ORDER BY updated_at DESC
             """
@@ -381,7 +259,7 @@ async def list_banks(pool) -> list:
                     "bank_id": row["bank_id"],
                     "name": row["name"],
                     "disposition": disposition_data,
-                    "background": row["background"],
+                    "mission": row["mission"] or "",
                     "created_at": row["created_at"].isoformat() if row["created_at"] else None,
                     "updated_at": row["updated_at"].isoformat() if row["updated_at"] else None,
                 }

hindsight-api/hindsight_api/engine/retain/fact_extraction.py

@@ -1268,6 +1268,7 @@ async def extract_facts_from_contents(
                         # mentioned_at: always the event_date (when the conversation/document occurred)
                         mentioned_at=content.event_date,
                         metadata=content.metadata,
+                        tags=content.tags,
                     )
 
                     extracted_facts.append(extracted_fact)

hindsight-api/hindsight_api/engine/retain/fact_storage.py

@@ -45,6 +45,7 @@ async def insert_facts_batch(
     metadata_jsons = []
     chunk_ids = []
     document_ids = []
+    tags_list = []
 
     for fact in facts:
         fact_texts.append(fact.fact_text)
@@ -65,16 +66,31 @@ async def insert_facts_batch(
         chunk_ids.append(fact.chunk_id)
         # Use per-fact document_id if available, otherwise fallback to batch-level document_id
         document_ids.append(fact.document_id if fact.document_id else document_id)
+        # Convert tags to JSON string for proper batch insertion (PostgreSQL unnest doesn't handle 2D arrays well)
+        tags_list.append(json.dumps(fact.tags if fact.tags else []))
 
     # Batch insert all facts
+    # Note: tags are passed as JSON strings and converted back to varchar[] via jsonb_array_elements_text + array_agg
     results = await conn.fetch(
         f"""
-        INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
-                                 context, fact_type, confidence_score, access_count, metadata, chunk_id, document_id)
-        SELECT $1, * FROM unnest(
-            $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
-            $8::text[], $9::text[], $10::float[], $11::int[], $12::jsonb[], $13::text[], $14::text[]
+        WITH input_data AS (
+            SELECT * FROM unnest(
+                $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
+                $8::text[], $9::text[], $10::float[], $11::int[], $12::jsonb[], $13::text[], $14::text[], $15::jsonb[]
+            ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                   context, fact_type, confidence_score, access_count, metadata, chunk_id, document_id, tags_json)
         )
+        INSERT INTO {fq_table("memory_units")} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                                 context, fact_type, confidence_score, access_count, metadata, chunk_id, document_id, tags)
+        SELECT
+            $1,
+            text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+            context, fact_type, confidence_score, access_count, metadata, chunk_id, document_id,
+            COALESCE(
+                (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
+                '{{}}'::varchar[]
+            )
+        FROM input_data
         RETURNING id
         """,
         bank_id,
@@ -91,6 +107,7 @@ async def insert_facts_batch(
         metadata_jsons,
         chunk_ids,
         document_ids,
+        tags_list,
     )
 
     unit_ids = [str(row["id"]) for row in results]
@@ -109,7 +126,7 @@ async def ensure_bank_exists(conn, bank_id: str) -> None:
     """
     await conn.execute(
         f"""
-        INSERT INTO {fq_table("banks")} (bank_id, disposition, background)
+        INSERT INTO {fq_table("banks")} (bank_id, disposition, mission)
         VALUES ($1, $2::jsonb, $3)
         ON CONFLICT (bank_id) DO UPDATE
         SET updated_at = NOW()
@@ -121,7 +138,13 @@ async def ensure_bank_exists(conn, bank_id: str) -> None:
 
 
 async def handle_document_tracking(
-    conn, bank_id: str, document_id: str, combined_content: str, is_first_batch: bool, retain_params: dict | None = None
+    conn,
+    bank_id: str,
+    document_id: str,
+    combined_content: str,
+    is_first_batch: bool,
+    retain_params: dict | None = None,
+    document_tags: list[str] | None = None,
 ) -> None:
     """
     Handle document tracking in the database.
@@ -133,6 +156,7 @@ async def handle_document_tracking(
         combined_content: Combined content text from all content items
         is_first_batch: Whether this is the first batch (for chunked operations)
         retain_params: Optional parameters passed during retain (context, event_date, etc.)
+        document_tags: Optional list of tags to associate with the document
     """
     import hashlib
 
@@ -149,13 +173,14 @@ async def handle_document_tracking(
     # Insert document (or update if exists from concurrent operations)
     await conn.execute(
         f"""
-        INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, metadata, retain_params)
-        VALUES ($1, $2, $3, $4, $5, $6)
+        INSERT INTO {fq_table("documents")} (id, bank_id, original_text, content_hash, metadata, retain_params, tags)
+        VALUES ($1, $2, $3, $4, $5, $6, $7)
         ON CONFLICT (id, bank_id) DO UPDATE
         SET original_text = EXCLUDED.original_text,
             content_hash = EXCLUDED.content_hash,
             metadata = EXCLUDED.metadata,
             retain_params = EXCLUDED.retain_params,
+            tags = EXCLUDED.tags,
             updated_at = NOW()
         """,
         document_id,
@@ -164,4 +189,5 @@ async def handle_document_tracking(
         content_hash,
         json.dumps({}),  # Empty metadata dict
         json.dumps(retain_params) if retain_params else None,
+        document_tags or [],
     )

hindsight-api/hindsight_api/engine/retain/observation_regeneration.py

@@ -1,254 +0,0 @@
-"""
-Observation regeneration for retain pipeline.
-
-Regenerates entity observations as part of the retain transaction.
-"""
-
-import logging
-import time
-import uuid
-from datetime import UTC, datetime
-
-from ...config import get_config
-from ..memory_engine import fq_table
-from ..search import observation_utils
-from . import embedding_utils
-from .types import EntityLink
-
-logger = logging.getLogger(__name__)
-
-
-def utcnow():
-    """Get current UTC time."""
-    return datetime.now(UTC)
-
-
-# Simple dataclass-like container for facts (avoid importing from memory_engine)
-class MemoryFactForObservation:
-    def __init__(self, id: str, text: str, fact_type: str, context: str, occurred_start: str | None):
-        self.id = id
-        self.text = text
-        self.fact_type = fact_type
-        self.context = context
-        self.occurred_start = occurred_start
-
-
-async def regenerate_observations_batch(
-    conn, embeddings_model, llm_config, bank_id: str, entity_links: list[EntityLink], log_buffer: list[str] = None
-) -> None:
-    """
-    Regenerate observations for top entities in this batch.
-
-    Called INSIDE the retain transaction for atomicity - if observations
-    fail, the entire retain batch is rolled back.
-
-    Args:
-        conn: Database connection (from the retain transaction)
-        embeddings_model: Embeddings model for generating observation embeddings
-        llm_config: LLM configuration for observation extraction
-        bank_id: Bank identifier
-        entity_links: Entity links from this batch
-        log_buffer: Optional log buffer for timing
-    """
-    config = get_config()
-    TOP_N_ENTITIES = config.observation_top_entities
-    MIN_FACTS_THRESHOLD = config.observation_min_facts
-
-    if not entity_links:
-        return
-
-    # Count mentions per entity in this batch
-    entity_mention_counts: dict[str, int] = {}
-    for link in entity_links:
-        if link.entity_id:
-            entity_id = str(link.entity_id)
-            entity_mention_counts[entity_id] = entity_mention_counts.get(entity_id, 0) + 1
-
-    if not entity_mention_counts:
-        return
-
-    # Sort by mention count descending and take top N
-    sorted_entities = sorted(entity_mention_counts.items(), key=lambda x: x[1], reverse=True)
-    entities_to_process = [e[0] for e in sorted_entities[:TOP_N_ENTITIES]]
-
-    obs_start = time.time()
-
-    # Convert to UUIDs
-    entity_uuids = [uuid.UUID(eid) if isinstance(eid, str) else eid for eid in entities_to_process]
-
-    # Batch query for entity names
-    entity_rows = await conn.fetch(
-        f"""
-        SELECT id, canonical_name FROM {fq_table("entities")}
-        WHERE id = ANY($1) AND bank_id = $2
-        """,
-        entity_uuids,
-        bank_id,
-    )
-    entity_names = {row["id"]: row["canonical_name"] for row in entity_rows}
-
-    # Batch query for fact counts
-    fact_counts = await conn.fetch(
-        f"""
-        SELECT ue.entity_id, COUNT(*) as cnt
-        FROM {fq_table("unit_entities")} ue
-        JOIN {fq_table("memory_units")} mu ON ue.unit_id = mu.id
-        WHERE ue.entity_id = ANY($1) AND mu.bank_id = $2
-        GROUP BY ue.entity_id
-        """,
-        entity_uuids,
-        bank_id,
-    )
-    entity_fact_counts = {row["entity_id"]: row["cnt"] for row in fact_counts}
-
-    # Filter entities that meet the threshold
-    entities_with_names = []
-    for entity_id in entities_to_process:
-        entity_uuid = uuid.UUID(entity_id) if isinstance(entity_id, str) else entity_id
-        if entity_uuid not in entity_names:
-            continue
-        fact_count = entity_fact_counts.get(entity_uuid, 0)
-        if fact_count >= MIN_FACTS_THRESHOLD:
-            entities_with_names.append((entity_id, entity_names[entity_uuid]))
-
-    if not entities_with_names:
-        return
-
-    # Process entities SEQUENTIALLY (asyncpg doesn't allow concurrent queries on same connection)
-    # We must use the same connection to stay in the retain transaction
-    total_observations = 0
-
-    for entity_id, entity_name in entities_with_names:
-        try:
-            obs_ids = await _regenerate_entity_observations(
-                conn, embeddings_model, llm_config, bank_id, entity_id, entity_name
-            )
-            total_observations += len(obs_ids)
-        except Exception as e:
-            logger.error(f"[OBSERVATIONS] Error processing entity {entity_id}: {e}")
-
-    obs_time = time.time() - obs_start
-    if log_buffer is not None:
-        log_buffer.append(
-            f"[11] Observations: {total_observations} observations for {len(entities_with_names)} entities in {obs_time:.3f}s"
-        )
-
-
-async def _regenerate_entity_observations(
-    conn, embeddings_model, llm_config, bank_id: str, entity_id: str, entity_name: str
-) -> list[str]:
-    """
-    Regenerate observations for a single entity.
-
-    Uses the provided connection (part of retain transaction).
-
-    Args:
-        conn: Database connection (from the retain transaction)
-        embeddings_model: Embeddings model
-        llm_config: LLM configuration
-        bank_id: Bank identifier
-        entity_id: Entity UUID
-        entity_name: Canonical name of the entity
-
-    Returns:
-        List of created observation IDs
-    """
-    entity_uuid = uuid.UUID(entity_id) if isinstance(entity_id, str) else entity_id
-
-    # Get all facts mentioning this entity (exclude observations themselves)
-    rows = await conn.fetch(
-        f"""
-        SELECT mu.id, mu.text, mu.context, mu.occurred_start, mu.fact_type
-        FROM {fq_table("memory_units")} mu
-        JOIN {fq_table("unit_entities")} ue ON mu.id = ue.unit_id
-        WHERE mu.bank_id = $1
-          AND ue.entity_id = $2
-          AND mu.fact_type IN ('world', 'experience')
-        ORDER BY mu.occurred_start DESC
-        LIMIT 50
-        """,
-        bank_id,
-        entity_uuid,
-    )
-
-    if not rows:
-        return []
-
-    # Convert to fact objects for observation extraction
-    facts = []
-    for row in rows:
-        occurred_start = row["occurred_start"].isoformat() if row["occurred_start"] else None
-        facts.append(
-            MemoryFactForObservation(
-                id=str(row["id"]),
-                text=row["text"],
-                fact_type=row["fact_type"],
-                context=row["context"],
-                occurred_start=occurred_start,
-            )
-        )
-
-    # Extract observations using LLM
-    observations = await observation_utils.extract_observations_from_facts(llm_config, entity_name, facts)
-
-    if not observations:
-        return []
-
-    # Delete old observations for this entity
-    await conn.execute(
-        f"""
-        DELETE FROM {fq_table("memory_units")}
-        WHERE id IN (
-            SELECT mu.id
-            FROM {fq_table("memory_units")} mu
-            JOIN {fq_table("unit_entities")} ue ON mu.id = ue.unit_id
-            WHERE mu.bank_id = $1
-              AND mu.fact_type = 'observation'
-              AND ue.entity_id = $2
-        )
-        """,
-        bank_id,
-        entity_uuid,
-    )
-
-    # Generate embeddings for new observations
-    embeddings = await embedding_utils.generate_embeddings_batch(embeddings_model, observations)
-
-    # Insert new observations
-    current_time = utcnow()
-    created_ids = []
-
-    for obs_text, embedding in zip(observations, embeddings):
-        result = await conn.fetchrow(
-            f"""
-            INSERT INTO {fq_table("memory_units")} (
-                bank_id, text, embedding, context, event_date,
-                occurred_start, occurred_end, mentioned_at,
-                fact_type, access_count
-            )
-            VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'observation', 0)
-            RETURNING id
-            """,
-            bank_id,
-            obs_text,
-            str(embedding),
-            f"observation about {entity_name}",
-            current_time,
-            current_time,
-            current_time,
-            current_time,
-        )
-        obs_id = str(result["id"])
-        created_ids.append(obs_id)
-
-        # Link observation to entity
-        await conn.execute(
-            f"""
-            INSERT INTO {fq_table("unit_entities")} (unit_id, entity_id)
-            VALUES ($1, $2)
-            """,
-            uuid.UUID(obs_id),
-            entity_uuid,
-        )
-
-    return created_ids

hindsight-api/hindsight_api/engine/retain/orchestrator.py

@@ -9,7 +9,6 @@ import time
 import uuid
 from datetime import UTC, datetime
 
-from ...config import get_config
 from ..db_utils import acquire_with_retry
 from . import bank_utils
 
@@ -28,9 +27,8 @@ from . import (
     fact_extraction,
     fact_storage,
     link_creation,
-    observation_regeneration,
 )
-from .types import ExtractedFact, ProcessedFact, RetainContent, RetainContentDict
+from .types import EntityLink, ExtractedFact, ProcessedFact, RetainContent, RetainContentDict
 
 logger = logging.getLogger(__name__)
 
@@ -40,7 +38,6 @@ async def retain_batch(
     embeddings_model,
     llm_config,
     entity_resolver,
-    task_backend,
     format_date_fn,
     duplicate_checker_fn,
     bank_id: str,
@@ -49,6 +46,7 @@ async def retain_batch(
     is_first_batch: bool = True,
     fact_type_override: str | None = None,
     confidence_score: float | None = None,
+    document_tags: list[str] | None = None,
 ) -> tuple[list[list[str]], TokenUsage]:
     """
     Process a batch of content through the retain pipeline.
@@ -58,7 +56,6 @@ async def retain_batch(
         embeddings_model: Embeddings model for generating embeddings
         llm_config: LLM configuration for fact extraction
         entity_resolver: Entity resolver for entity processing
-        task_backend: Task backend for background jobs
         format_date_fn: Function to format datetime to readable string
         duplicate_checker_fn: Function to check for duplicate facts
         bank_id: Bank identifier
@@ -67,6 +64,7 @@ async def retain_batch(
         is_first_batch: Whether this is the first batch
         fact_type_override: Override fact type for all facts
         confidence_score: Confidence score for opinions
+        document_tags: Tags applied to all items in this batch
 
     Returns:
         Tuple of (unit ID lists, token usage for fact extraction)
@@ -88,12 +86,16 @@ async def retain_batch(
     # Convert dicts to RetainContent objects
     contents = []
     for item in contents_dicts:
+        # Merge item-level tags with document-level tags
+        item_tags = item.get("tags", []) or []
+        merged_tags = list(set(item_tags + (document_tags or [])))
         content = RetainContent(
             content=item["content"],
             context=item.get("context", ""),
             event_date=item.get("event_date") or utcnow(),
             metadata=item.get("metadata", {}),
             entities=item.get("entities", []),
+            tags=merged_tags,
         )
         contents.append(content)
 
@@ -131,7 +133,7 @@ async def retain_batch(
                         if first_item.get("metadata"):
                             retain_params["metadata"] = first_item["metadata"]
                     await fact_storage.handle_document_tracking(
-                        conn, bank_id, document_id, combined_content, is_first_batch, retain_params
+                        conn, bank_id, document_id, combined_content, is_first_batch, retain_params, document_tags
                     )
                 else:
                     # Check for per-item document_ids
@@ -159,7 +161,7 @@ async def retain_batch(
                             if first_item.get("metadata"):
                                 retain_params["metadata"] = first_item["metadata"]
                         await fact_storage.handle_document_tracking(
-                            conn, bank_id, doc_id, combined_content, is_first_batch, retain_params
+                            conn, bank_id, doc_id, combined_content, is_first_batch, retain_params, document_tags
                         )
 
         total_time = time.time() - start_time
@@ -225,7 +227,7 @@ async def retain_batch(
                         retain_params["metadata"] = first_item["metadata"]
 
                 await fact_storage.handle_document_tracking(
-                    conn, bank_id, document_id, combined_content, is_first_batch, retain_params
+                    conn, bank_id, document_id, combined_content, is_first_batch, retain_params, document_tags
                 )
                 document_ids_added.append(document_id)
                 doc_id_mapping[None] = document_id  # For backwards compatibility
@@ -269,7 +271,13 @@ async def retain_batch(
                                     retain_params["metadata"] = first_item["metadata"]
 
                             await fact_storage.handle_document_tracking(
-                                conn, bank_id, actual_doc_id, combined_content, is_first_batch, retain_params
+                                conn,
+                                bank_id,
+                                actual_doc_id,
+                                combined_content,
+                                is_first_batch,
+                                retain_params,
+                                document_tags,
                             )
                             document_ids_added.append(actual_doc_id)
 
@@ -396,27 +404,9 @@ async def retain_batch(
             causal_link_count = await link_creation.create_causal_links_batch(conn, unit_ids, non_duplicate_facts)
             log_buffer.append(f"[10] Causal links: {causal_link_count} links in {time.time() - step_start:.3f}s")
 
-            # Regenerate observations - sync (in transaction) or async (background task)
-            config = get_config()
-            if config.retain_observations_async:
-                # Queue for async processing after transaction commits
-                entity_ids_for_async = list(set(link.entity_id for link in entity_links)) if entity_links else []
-                log_buffer.append(
-                    f"[11] Observations: queued {len(entity_ids_for_async)} entities for async processing"
-                )
-            else:
-                # Run synchronously inside transaction for atomicity
-                await observation_regeneration.regenerate_observations_batch(
-                    conn, embeddings_model, llm_config, bank_id, entity_links, log_buffer
-                )
-                entity_ids_for_async = []
-
             # Map results back to original content items
             result_unit_ids = _map_results_to_contents(contents, extracted_facts, is_duplicate_flags, unit_ids)
 
-        # Trigger background tasks AFTER transaction commits
-        await _trigger_background_tasks(task_backend, bank_id, unit_ids, non_duplicate_facts, entity_ids_for_async)
-
         # Log final summary
         total_time = time.time() - start_time
         log_buffer.append(f"{'=' * 60}")
@@ -458,35 +448,3 @@ def _map_results_to_contents(
         result_unit_ids.append(content_unit_ids)
 
     return result_unit_ids
-
-
-async def _trigger_background_tasks(
-    task_backend,
-    bank_id: str,
-    unit_ids: list[str],
-    facts: list[ProcessedFact],
-    entity_ids_for_observations: list[str] | None = None,
-) -> None:
-    """Trigger background tasks after transaction commits."""
-    # Trigger opinion reinforcement if there are entities
-    fact_entities = [[e.name for e in fact.entities] for fact in facts]
-    if any(fact_entities):
-        await task_backend.submit_task(
-            {
-                "type": "reinforce_opinion",
-                "bank_id": bank_id,
-                "created_unit_ids": unit_ids,
-                "unit_texts": [fact.fact_text for fact in facts],
-                "unit_entities": fact_entities,
-            }
-        )
-
-    # Trigger observation regeneration if async mode is enabled
-    if entity_ids_for_observations:
-        await task_backend.submit_task(
-            {
-                "type": "regenerate_observations",
-                "bank_id": bank_id,
-                "entity_ids": entity_ids_for_observations,
-            }
-        )

hindsight-api/hindsight_api/engine/retain/types.py

@@ -21,6 +21,7 @@ class RetainContentDict(TypedDict, total=False):
         metadata: Custom key-value metadata (optional)
         document_id: Document ID for this content item (optional)
         entities: User-provided entities to merge with extracted entities (optional)
+        tags: Visibility scope tags for this content item (optional)
     """
 
     content: str  # Required
@@ -29,6 +30,7 @@ class RetainContentDict(TypedDict, total=False):
     metadata: dict[str, str]
     document_id: str
     entities: list[dict[str, str]]  # [{"text": "...", "type": "..."}]
+    tags: list[str]  # Visibility scope tags
 
 
 def _now_utc() -> datetime:
@@ -49,6 +51,7 @@ class RetainContent:
     event_date: datetime = field(default_factory=_now_utc)
     metadata: dict[str, str] = field(default_factory=dict)
     entities: list[dict[str, str]] = field(default_factory=list)  # User-provided entities
+    tags: list[str] = field(default_factory=list)  # Visibility scope tags
 
 
 @dataclass
@@ -113,6 +116,7 @@ class ExtractedFact:
     context: str = ""
     mentioned_at: datetime | None = None
     metadata: dict[str, str] = field(default_factory=dict)
+    tags: list[str] = field(default_factory=list)  # Visibility scope tags
 
 
 @dataclass
@@ -158,6 +162,9 @@ class ProcessedFact:
     # Track which content this fact came from (for user entity merging)
     content_index: int = 0
 
+    # Visibility scope tags
+    tags: list[str] = field(default_factory=list)
+
     @property
     def is_duplicate(self) -> bool:
         """Check if this fact was marked as a duplicate."""
@@ -201,6 +208,7 @@ class ProcessedFact:
             causal_relations=extracted_fact.causal_relations,
             chunk_id=chunk_id,
             content_index=extracted_fact.content_index,
+            tags=extracted_fact.tags,
         )
 
 
@@ -232,6 +240,7 @@ class RetainBatch:
     document_id: str | None = None
     fact_type_override: str | None = None
     confidence_score: float | None = None
+    document_tags: list[str] = field(default_factory=list)  # Tags applied to all items
 
     # Extracted data (populated during processing)
     extracted_facts: list[ExtractedFact] = field(default_factory=list)

hindsight-api/hindsight_api/engine/search/graph_retrieval.py

@@ -11,6 +11,7 @@ from abc import ABC, abstractmethod
 
 from ..db_utils import acquire_with_retry
 from ..memory_engine import fq_table
+from .tags import TagsMatch, filter_results_by_tags
 from .types import MPFPTimings, RetrievalResult
 
 logger = logging.getLogger(__name__)
@@ -43,6 +44,8 @@ class GraphRetriever(ABC):
         semantic_seeds: list[RetrievalResult] | None = None,
         temporal_seeds: list[RetrievalResult] | None = None,
         adjacency=None,  # TypedAdjacency, optional pre-loaded graph
+        tags: list[str] | None = None,  # Visibility scope tags for filtering
+        tags_match: TagsMatch = "any",  # How to match tags: 'any' (OR) or 'all' (AND)
     ) -> tuple[list[RetrievalResult], MPFPTimings | None]:
         """
         Retrieve relevant facts via graph traversal.
@@ -57,6 +60,7 @@ class GraphRetriever(ABC):
             semantic_seeds: Pre-computed semantic entry points (from semantic retrieval)
             temporal_seeds: Pre-computed temporal entry points (from temporal retrieval)
             adjacency: Pre-loaded typed adjacency graph (optional, for MPFP)
+            tags: Optional list of tags for visibility filtering (OR matching)
 
         Returns:
             Tuple of (List of RetrievalResult with activation scores, optional timing info)
@@ -114,6 +118,8 @@ class BFSGraphRetriever(GraphRetriever):
         semantic_seeds: list[RetrievalResult] | None = None,
         temporal_seeds: list[RetrievalResult] | None = None,
         adjacency=None,  # Not used by BFS
+        tags: list[str] | None = None,
+        tags_match: TagsMatch = "any",
     ) -> tuple[list[RetrievalResult], MPFPTimings | None]:
         """
         Retrieve facts using BFS spreading activation.
@@ -129,7 +135,9 @@ class BFSGraphRetriever(GraphRetriever):
         for interface compatibility but not used.
         """
         async with acquire_with_retry(pool) as conn:
-            results = await self._retrieve_with_conn(conn, query_embedding_str, bank_id, fact_type, budget)
+            results = await self._retrieve_with_conn(
+                conn, query_embedding_str, bank_id, fact_type, budget, tags=tags, tags_match=tags_match
+            )
             return results, None
 
     async def _retrieve_with_conn(
@@ -139,33 +147,46 @@ class BFSGraphRetriever(GraphRetriever):
         bank_id: str,
         fact_type: str,
         budget: int,
+        tags: list[str] | None = None,
+        tags_match: TagsMatch = "any",
     ) -> list[RetrievalResult]:
         """Internal implementation with connection."""
+        from .tags import build_tags_where_clause_simple
+
+        tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
+        params = [query_embedding_str, bank_id, fact_type, self.entry_point_threshold, self.entry_point_limit]
+        if tags:
+            params.append(tags)
 
         # Step 1: Find entry points
         entry_points = await conn.fetch(
             f"""
             SELECT id, text, context, event_date, occurred_start, occurred_end,
-                   mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+                   mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    1 - (embedding <=> $1::vector) AS similarity
             FROM {fq_table("memory_units")}
             WHERE bank_id = $2
               AND embedding IS NOT NULL
               AND fact_type = $3
               AND (1 - (embedding <=> $1::vector)) >= $4
+              {tags_clause}
             ORDER BY embedding <=> $1::vector
             LIMIT $5
             """,
-            query_embedding_str,
-            bank_id,
-            fact_type,
-            self.entry_point_threshold,
-            self.entry_point_limit,
+            *params,
         )
 
         if not entry_points:
+            logger.debug(
+                f"[BFS] No entry points found for fact_type={fact_type} (tags={tags}, tags_match={tags_match})"
+            )
             return []
 
+        logger.debug(
+            f"[BFS] Found {len(entry_points)} entry points for fact_type={fact_type} "
+            f"(tags={tags}, tags_match={tags_match})"
+        )
+
         # Step 2: BFS spreading activation
         visited = set()
         results = []
@@ -196,7 +217,7 @@ class BFSGraphRetriever(GraphRetriever):
                     f"""
                     SELECT mu.id, mu.text, mu.context, mu.occurred_start, mu.occurred_end,
                            mu.mentioned_at, mu.access_count, mu.embedding, mu.fact_type,
-                           mu.document_id, mu.chunk_id,
+                           mu.document_id, mu.chunk_id, mu.tags,
                            ml.weight, ml.link_type, ml.from_unit_id
                     FROM {fq_table("memory_links")} ml
                     JOIN {fq_table("memory_units")} mu ON ml.to_unit_id = mu.id
@@ -236,4 +257,8 @@ class BFSGraphRetriever(GraphRetriever):
                             neighbor_result = RetrievalResult.from_db_row(dict(n))
                             queue.append((neighbor_result, new_activation))
 
+        # Apply tags filtering (BFS may traverse into memories that don't match tags criteria)
+        if tags:
+            results = filter_results_by_tags(results, tags, match=tags_match)
+
         return results

hindsight-api/hindsight_api/engine/search/link_expansion_retrieval.py

@@ -18,6 +18,7 @@ import time
 from ..db_utils import acquire_with_retry
 from ..memory_engine import fq_table
 from .graph_retrieval import GraphRetriever
+from .tags import TagsMatch, filter_results_by_tags
 from .types import MPFPTimings, RetrievalResult
 
 logger = logging.getLogger(__name__)
@@ -30,26 +31,32 @@ async def _find_semantic_seeds(
     fact_type: str,
     limit: int = 20,
     threshold: float = 0.3,
+    tags: list[str] | None = None,
+    tags_match: TagsMatch = "any",
 ) -> list[RetrievalResult]:
     """Find semantic seeds via embedding search."""
+    from .tags import build_tags_where_clause_simple
+
+    tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
+    params = [query_embedding_str, bank_id, fact_type, threshold, limit]
+    if tags:
+        params.append(tags)
+
     rows = await conn.fetch(
         f"""
         SELECT id, text, context, event_date, occurred_start, occurred_end,
-               mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+               mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                1 - (embedding <=> $1::vector) AS similarity
         FROM {fq_table("memory_units")}
         WHERE bank_id = $2
           AND embedding IS NOT NULL
           AND fact_type = $3
           AND (1 - (embedding <=> $1::vector)) >= $4
+          {tags_clause}
         ORDER BY embedding <=> $1::vector
         LIMIT $5
         """,
-        query_embedding_str,
-        bank_id,
-        fact_type,
-        threshold,
-        limit,
+        *params,
     )
     return [RetrievalResult.from_db_row(dict(r)) for r in rows]
 
@@ -95,6 +102,8 @@ class LinkExpansionRetriever(GraphRetriever):
         semantic_seeds: list[RetrievalResult] | None = None,
         temporal_seeds: list[RetrievalResult] | None = None,
         adjacency=None,
+        tags: list[str] | None = None,
+        tags_match: TagsMatch = "any",
     ) -> tuple[list[RetrievalResult], MPFPTimings | None]:
         """
         Retrieve facts by expanding links from seeds.
@@ -109,6 +118,7 @@ class LinkExpansionRetriever(GraphRetriever):
             semantic_seeds: Pre-computed semantic entry points
             temporal_seeds: Pre-computed temporal entry points
             adjacency: Unused, kept for interface compatibility
+            tags: Optional list of tags for visibility filtering (OR matching)
 
         Returns:
             Tuple of (results, timings)
@@ -125,15 +135,27 @@ class LinkExpansionRetriever(GraphRetriever):
             else:
                 seeds_start = time.time()
                 all_seeds = await _find_semantic_seeds(
-                    conn, query_embedding_str, bank_id, fact_type, limit=20, threshold=0.3
+                    conn,
+                    query_embedding_str,
+                    bank_id,
+                    fact_type,
+                    limit=20,
+                    threshold=0.3,
+                    tags=tags,
+                    tags_match=tags_match,
                 )
                 timings.seeds_time = time.time() - seeds_start
+                logger.debug(
+                    f"[LinkExpansion] Found {len(all_seeds)} semantic seeds for fact_type={fact_type} "
+                    f"(tags={tags}, tags_match={tags_match})"
+                )
 
             # Add temporal seeds if provided
             if temporal_seeds:
                 all_seeds.extend(temporal_seeds)
 
             if not all_seeds:
+                logger.debug("[LinkExpansion] No seeds found, returning empty results")
                 return [], timings
 
             seed_ids = list({s.id for s in all_seeds})
@@ -147,7 +169,7 @@ class LinkExpansionRetriever(GraphRetriever):
                 SELECT
                     mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start,
                     mu.occurred_end, mu.mentioned_at, mu.access_count, mu.embedding,
-                    mu.fact_type, mu.document_id, mu.chunk_id,
+                    mu.fact_type, mu.document_id, mu.chunk_id, mu.tags,
                     COUNT(*)::float AS score
                 FROM {fq_table("unit_entities")} seed_ue
                 JOIN {fq_table("entities")} e ON seed_ue.entity_id = e.id
@@ -172,7 +194,7 @@ class LinkExpansionRetriever(GraphRetriever):
                 SELECT DISTINCT ON (mu.id)
                     mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start,
                     mu.occurred_end, mu.mentioned_at, mu.access_count, mu.embedding,
-                    mu.fact_type, mu.document_id, mu.chunk_id,
+                    mu.fact_type, mu.document_id, mu.chunk_id, mu.tags,
                     ml.weight + 1.0 AS score
                 FROM {fq_table("memory_links")} ml
                 JOIN {fq_table("memory_units")} mu ON ml.to_unit_id = mu.id
@@ -219,6 +241,10 @@ class LinkExpansionRetriever(GraphRetriever):
             result.activation = row["score"]
             results.append(result)
 
+        # Apply tags filtering (graph expansion may reach untagged memories)
+        if tags:
+            results = filter_results_by_tags(results, tags, match=tags_match)
+
         timings.result_count = len(results)
         timings.traverse = time.time() - start_time
 

hindsight-api/hindsight_api/engine/search/mpfp_retrieval.py

@@ -23,6 +23,7 @@ from dataclasses import dataclass, field
 from ..db_utils import acquire_with_retry
 from ..memory_engine import fq_table
 from .graph_retrieval import GraphRetriever
+from .tags import TagsMatch
 from .types import MPFPTimings, RetrievalResult
 
 logger = logging.getLogger(__name__)
@@ -448,7 +449,7 @@ async def fetch_memory_units_by_ids(
         rows = await conn.fetch(
             f"""
             SELECT id, text, context, event_date, occurred_start, occurred_end,
-                   mentioned_at, access_count, embedding, fact_type, document_id, chunk_id
+                   mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags
             FROM {fq_table("memory_units")}
             WHERE id = ANY($1::uuid[])
               AND fact_type = $2
@@ -503,6 +504,8 @@ class MPFPGraphRetriever(GraphRetriever):
         semantic_seeds: list[RetrievalResult] | None = None,
         temporal_seeds: list[RetrievalResult] | None = None,
         adjacency=None,  # Ignored - kept for interface compatibility
+        tags: list[str] | None = None,
+        tags_match: TagsMatch = "any",
     ) -> tuple[list[RetrievalResult], MPFPTimings | None]:
         """
         Retrieve facts using MPFP algorithm with lazy edge loading.
@@ -517,6 +520,7 @@ class MPFPGraphRetriever(GraphRetriever):
             semantic_seeds: Pre-computed semantic entry points
             temporal_seeds: Pre-computed temporal entry points
             adjacency: Ignored (kept for interface compatibility)
+            tags: Optional list of tags for visibility filtering (OR matching)
 
         Returns:
             Tuple of (List of RetrievalResult with activation scores, MPFPTimings)
@@ -532,8 +536,13 @@ class MPFPGraphRetriever(GraphRetriever):
         # If no semantic seeds provided, fall back to finding our own
         if not semantic_seed_nodes:
             seeds_start = time.time()
-            semantic_seed_nodes = await self._find_semantic_seeds(pool, query_embedding_str, bank_id, fact_type)
+            semantic_seed_nodes = await self._find_semantic_seeds(
+                pool, query_embedding_str, bank_id, fact_type, tags=tags, tags_match=tags_match
+            )
             timings.seeds_time = time.time() - seeds_start
+            logger.debug(
+                f"[MPFP] Found {len(semantic_seed_nodes)} semantic seeds for fact_type={fact_type} (tags={tags}, tags_match={tags_match})"
+            )
 
         # Collect all pattern jobs
         pattern_jobs = []
@@ -549,6 +558,9 @@ class MPFPGraphRetriever(GraphRetriever):
                 pattern_jobs.append((temporal_seed_nodes, pattern))
 
         if not pattern_jobs:
+            logger.debug(
+                f"[MPFP] No pattern jobs (semantic_seeds={len(semantic_seed_nodes)}, temporal_seeds={len(temporal_seed_nodes)})"
+            )
             return [], timings
 
         timings.pattern_count = len(pattern_jobs)
@@ -587,6 +599,7 @@ class MPFPGraphRetriever(GraphRetriever):
         timings.fusion = time.time() - step_start
 
         if not fused:
+            logger.debug(f"[MPFP] No fused results after RRF fusion (pattern_count={len(pattern_results)})")
             return [], timings
 
         # Get top result IDs
@@ -596,6 +609,13 @@ class MPFPGraphRetriever(GraphRetriever):
         step_start = time.time()
         results = await fetch_memory_units_by_ids(pool, result_ids, fact_type)
         timings.fetch = time.time() - step_start
+
+        # Filter results by tags (graph traversal may have picked up unfiltered memories)
+        if tags:
+            from .tags import filter_results_by_tags
+
+            results = filter_results_by_tags(results, tags, match=tags_match)
+
         timings.result_count = len(results)
 
         # Add activation scores from fusion
@@ -634,8 +654,17 @@ class MPFPGraphRetriever(GraphRetriever):
         fact_type: str,
         limit: int = 20,
         threshold: float = 0.3,
+        tags: list[str] | None = None,
+        tags_match: TagsMatch = "any",
     ) -> list[SeedNode]:
         """Fallback: find semantic seeds via embedding search."""
+        from .tags import build_tags_where_clause_simple
+
+        tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
+        params = [query_embedding_str, bank_id, fact_type, threshold, limit]
+        if tags:
+            params.append(tags)
+
         async with acquire_with_retry(pool) as conn:
             rows = await conn.fetch(
                 f"""
@@ -645,14 +674,11 @@ class MPFPGraphRetriever(GraphRetriever):
                   AND embedding IS NOT NULL
                   AND fact_type = $3
                   AND (1 - (embedding <=> $1::vector)) >= $4
+                  {tags_clause}
                 ORDER BY embedding <=> $1::vector
                 LIMIT $5
                 """,
-                query_embedding_str,
-                bank_id,
-                fact_type,
-                threshold,
-                limit,
+                *params,
             )
 
         return [SeedNode(node_id=str(r["id"]), score=r["similarity"]) for r in rows]

hindsight-api/hindsight_api/engine/search/observation_utils.py

@@ -1,125 +0,0 @@
-"""
-Observation utilities for generating entity observations from facts.
-
-Observations are objective facts synthesized from multiple memory facts
-about an entity, without personality influence.
-"""
-
-import logging
-
-from pydantic import BaseModel, Field
-
-from ..response_models import MemoryFact
-
-logger = logging.getLogger(__name__)
-
-
-class Observation(BaseModel):
-    """An observation about an entity."""
-
-    observation: str = Field(description="The observation text - a factual statement about the entity")
-
-
-class ObservationExtractionResponse(BaseModel):
-    """Response containing extracted observations."""
-
-    observations: list[Observation] = Field(default_factory=list, description="List of observations about the entity")
-
-
-def format_facts_for_observation_prompt(facts: list[MemoryFact]) -> str:
-    """Format facts as text for observation extraction prompt."""
-    import json
-
-    if not facts:
-        return "[]"
-    formatted = []
-    for fact in facts:
-        fact_obj = {"text": fact.text}
-
-        # Add context if available
-        if fact.context:
-            fact_obj["context"] = fact.context
-
-        # Add occurred_start if available
-        if fact.occurred_start:
-            fact_obj["occurred_at"] = fact.occurred_start
-
-        formatted.append(fact_obj)
-
-    return json.dumps(formatted, indent=2)
-
-
-def build_observation_prompt(
-    entity_name: str,
-    facts_text: str,
-) -> str:
-    """Build the observation extraction prompt for the LLM."""
-    return f"""Based on the following facts about "{entity_name}", generate a list of key observations.
-
-FACTS ABOUT {entity_name.upper()}:
-{facts_text}
-
-Your task: Synthesize the facts into clear, objective observations about {entity_name}.
-
-GUIDELINES:
-1. Each observation should be a factual statement about {entity_name}
-2. Combine related facts into single observations where appropriate
-3. Be objective - do not add opinions, judgments, or interpretations
-4. Focus on what we KNOW about {entity_name}, not what we assume
-5. Include observations about: identity, characteristics, roles, relationships, activities
-6. Write in third person (e.g., "John is..." not "I think John is...")
-7. If there are conflicting facts, note the most recent or most supported one
-
-EXAMPLES of good observations:
-- "John works at Google as a software engineer"
-- "John is detail-oriented and methodical in his approach"
-- "John collaborates frequently with Sarah on the AI project"
-- "John joined the company in 2023"
-
-EXAMPLES of bad observations (avoid these):
-- "John seems like a good person" (opinion/judgment)
-- "John probably likes his job" (assumption)
-- "I believe John is reliable" (first-person opinion)
-
-Generate 3-7 observations based on the available facts. If there are very few facts, generate fewer observations."""
-
-
-def get_observation_system_message() -> str:
-    """Get the system message for observation extraction."""
-    return "You are an objective observer synthesizing facts about an entity. Generate clear, factual observations without opinions or personality influence. Be concise and accurate."
-
-
-async def extract_observations_from_facts(llm_config, entity_name: str, facts: list[MemoryFact]) -> list[str]:
-    """
-    Extract observations from facts about an entity using LLM.
-
-    Args:
-        llm_config: LLM configuration to use
-        entity_name: Name of the entity to generate observations about
-        facts: List of facts mentioning the entity
-
-    Returns:
-        List of observation strings
-    """
-    if not facts:
-        return []
-
-    facts_text = format_facts_for_observation_prompt(facts)
-    prompt = build_observation_prompt(entity_name, facts_text)
-
-    try:
-        result = await llm_config.call(
-            messages=[
-                {"role": "system", "content": get_observation_system_message()},
-                {"role": "user", "content": prompt},
-            ],
-            response_format=ObservationExtractionResponse,
-            scope="memory_extract_observation",
-        )
-
-        observations = [op.observation for op in result.observations]
-        return observations
-
-    except Exception as e:
-        logger.warning(f"Failed to extract observations for {entity_name}: {str(e)}")
-        return []

hindsight-api/hindsight_api/engine/search/retrieval.py

@@ -20,6 +20,7 @@ from ..memory_engine import fq_table
 from .graph_retrieval import BFSGraphRetriever, GraphRetriever
 from .link_expansion_retrieval import LinkExpansionRetriever
 from .mpfp_retrieval import MPFPGraphRetriever
+from .tags import TagsMatch, build_tags_where_clause_simple
 from .types import MPFPTimings, RetrievalResult
 
 logger = logging.getLogger(__name__)
@@ -85,7 +86,12 @@ def set_default_graph_retriever(retriever: GraphRetriever) -> None:
 
 
 async def retrieve_semantic(
-    conn, query_emb_str: str, bank_id: str, fact_type: str, limit: int
+    conn,
+    query_emb_str: str,
+    bank_id: str,
+    fact_type: str,
+    limit: int,
+    tags: list[str] | None = None,
 ) -> list[RetrievalResult]:
     """
     Semantic retrieval via vector similarity.
@@ -96,31 +102,44 @@ async def retrieve_semantic(
         agent_id: bank ID
         fact_type: Fact type to filter
         limit: Maximum results to return
+        tags: Optional list of tags for visibility filtering (OR matching)
 
     Returns:
         List of RetrievalResult objects
     """
+    from .tags import TagsMatch, build_tags_where_clause_simple
+
+    tags_clause = build_tags_where_clause_simple(tags, 5)
+    params = [query_emb_str, bank_id, fact_type, limit]
+    if tags:
+        params.append(tags)
+
     results = await conn.fetch(
         f"""
-        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                1 - (embedding <=> $1::vector) AS similarity
         FROM {fq_table("memory_units")}
         WHERE bank_id = $2
           AND embedding IS NOT NULL
           AND fact_type = $3
           AND (1 - (embedding <=> $1::vector)) >= 0.3
+          {tags_clause}
         ORDER BY embedding <=> $1::vector
         LIMIT $4
         """,
-        query_emb_str,
-        bank_id,
-        fact_type,
-        limit,
+        *params,
     )
     return [RetrievalResult.from_db_row(dict(r)) for r in results]
 
 
-async def retrieve_bm25(conn, query_text: str, bank_id: str, fact_type: str, limit: int) -> list[RetrievalResult]:
+async def retrieve_bm25(
+    conn,
+    query_text: str,
+    bank_id: str,
+    fact_type: str,
+    limit: int,
+    tags: list[str] | None = None,
+) -> list[RetrievalResult]:
     """
     BM25 keyword retrieval via full-text search.
 
@@ -130,12 +149,15 @@ async def retrieve_bm25(conn, query_text: str, bank_id: str, fact_type: str, lim
         agent_id: bank ID
         fact_type: Fact type to filter
         limit: Maximum results to return
+        tags: Optional list of tags for visibility filtering (OR matching)
 
     Returns:
         List of RetrievalResult objects
     """
     import re
 
+    from .tags import TagsMatch, build_tags_where_clause_simple
+
     # Sanitize query text: remove special characters that have meaning in tsquery
     # Keep only alphanumeric characters and spaces
     sanitized_text = re.sub(r"[^\w\s]", " ", query_text.lower())
@@ -151,21 +173,24 @@ async def retrieve_bm25(conn, query_text: str, bank_id: str, fact_type: str, lim
     # This prevents empty results when some terms are missing
     query_tsquery = " | ".join(tokens)
 
+    tags_clause = build_tags_where_clause_simple(tags, 5)
+    params = [query_tsquery, bank_id, fact_type, limit]
+    if tags:
+        params.append(tags)
+
     results = await conn.fetch(
         f"""
-        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                ts_rank_cd(search_vector, to_tsquery('english', $1)) AS bm25_score
         FROM {fq_table("memory_units")}
         WHERE bank_id = $2
           AND fact_type = $3
           AND search_vector @@ to_tsquery('english', $1)
+          {tags_clause}
         ORDER BY bm25_score DESC
         LIMIT $4
         """,
-        query_tsquery,
-        bank_id,
-        fact_type,
-        limit,
+        *params,
     )
     return [RetrievalResult.from_db_row(dict(r)) for r in results]
 
@@ -177,6 +202,8 @@ async def retrieve_semantic_bm25_combined(
     bank_id: str,
     fact_types: list[str],
     limit: int,
+    tags: list[str] | None = None,
+    tags_match: TagsMatch = "any",
 ) -> dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]]:
     """
     Combined semantic + BM25 retrieval for multiple fact types in a single query.
@@ -203,10 +230,14 @@ async def retrieve_semantic_bm25_combined(
 
     # If no valid tokens for BM25, just run semantic
     if not tokens:
+        tags_clause = build_tags_where_clause_simple(tags, 5, match=tags_match)
+        params = [query_emb_str, bank_id, fact_types, limit]
+        if tags:
+            params.append(tags)
         results = await conn.fetch(
             f"""
             WITH semantic_ranked AS (
-                SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+                SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                        1 - (embedding <=> $1::vector) AS similarity,
                        NULL::float AS bm25_score,
                        'semantic' AS source,
@@ -216,16 +247,14 @@ async def retrieve_semantic_bm25_combined(
                   AND embedding IS NOT NULL
                   AND fact_type = ANY($3)
                   AND (1 - (embedding <=> $1::vector)) >= 0.3
+                  {tags_clause}
             )
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    similarity, bm25_score, source
             FROM semantic_ranked
             WHERE rn <= $4
             """,
-            query_emb_str,
-            bank_id,
-            fact_types,
-            limit,
+            *params,
         )
         # Group by fact_type
         result_dict: dict[str, tuple[list[RetrievalResult], list[RetrievalResult]]] = {
@@ -241,12 +270,18 @@ async def retrieve_semantic_bm25_combined(
 
     query_tsquery = " | ".join(tokens)
 
+    # Build tags clause - param 6 if tags provided
+    tags_clause = build_tags_where_clause_simple(tags, 6, match=tags_match)
+    params = [query_emb_str, bank_id, fact_types, limit, query_tsquery]
+    if tags:
+        params.append(tags)
+
     # Combined CTE query for both semantic and BM25 across all fact types
     # Uses window functions to limit per fact_type per method
     results = await conn.fetch(
         f"""
         WITH semantic_ranked AS (
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    1 - (embedding <=> $1::vector) AS similarity,
                    NULL::float AS bm25_score,
                    'semantic' AS source,
@@ -256,9 +291,10 @@ async def retrieve_semantic_bm25_combined(
               AND embedding IS NOT NULL
               AND fact_type = ANY($3)
               AND (1 - (embedding <=> $1::vector)) >= 0.3
+              {tags_clause}
         ),
         bm25_ranked AS (
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    NULL::float AS similarity,
                    ts_rank_cd(search_vector, to_tsquery('english', $5)) AS bm25_score,
                    'bm25' AS source,
@@ -267,14 +303,15 @@ async def retrieve_semantic_bm25_combined(
             WHERE bank_id = $2
               AND fact_type = ANY($3)
               AND search_vector @@ to_tsquery('english', $5)
+              {tags_clause}
         ),
         semantic AS (
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    similarity, bm25_score, source
             FROM semantic_ranked WHERE rn <= $4
         ),
         bm25 AS (
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    similarity, bm25_score, source
             FROM bm25_ranked WHERE rn <= $4
         )
@@ -282,11 +319,7 @@ async def retrieve_semantic_bm25_combined(
         UNION ALL
         SELECT * FROM bm25
         """,
-        query_emb_str,
-        bank_id,
-        fact_types,
-        limit,
-        query_tsquery,
+        *params,
     )
 
     # Group results by fact_type and source
@@ -313,6 +346,8 @@ async def retrieve_temporal_combined(
     end_date: datetime,
     budget: int,
     semantic_threshold: float = 0.1,
+    tags: list[str] | None = None,
+    tags_match: TagsMatch = "any",
 ) -> dict[str, list[RetrievalResult]]:
     """
     Temporal retrieval for multiple fact types in a single query.
@@ -341,11 +376,17 @@ async def retrieve_temporal_combined(
     if end_date.tzinfo is None:
         end_date = end_date.replace(tzinfo=UTC)
 
+    # Build tags clause
+    tags_clause = build_tags_where_clause_simple(tags, 7, match=tags_match)
+    params = [query_emb_str, bank_id, fact_types, start_date, end_date, semantic_threshold]
+    if tags:
+        params.append(tags)
+
     # Batch query: Get entry points for ALL fact types at once with window function
     entry_points = await conn.fetch(
         f"""
         WITH ranked_entries AS (
-            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+            SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                    1 - (embedding <=> $1::vector) AS similarity,
                    ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY COALESCE(occurred_start, mentioned_at, occurred_end) DESC, embedding <=> $1::vector) AS rn
             FROM {fq_table("memory_units")}
@@ -363,17 +404,13 @@ async def retrieve_temporal_combined(
                   (occurred_end IS NOT NULL AND occurred_end BETWEEN $4 AND $5)
               )
               AND (1 - (embedding <=> $1::vector)) >= $6
+              {tags_clause}
         )
-        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, similarity
+        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags, similarity
         FROM ranked_entries
         WHERE rn <= 10
         """,
-        query_emb_str,
-        bank_id,
-        fact_types,
-        start_date,
-        end_date,
-        semantic_threshold,
+        *params,
     )
 
     if not entry_points:
@@ -436,13 +473,20 @@ async def retrieve_temporal_combined(
         budget_remaining = budget - len(ft_entry_points)
         batch_size = 20
 
+        # Build tags clause for spreading (use param 6 since 1-5 are used)
+        spreading_tags_clause = build_tags_where_clause_simple(tags, 6, table_alias="mu.", match=tags_match)
+
         while frontier and budget_remaining > 0:
             batch_ids = frontier[:batch_size]
             frontier = frontier[batch_size:]
 
+            spreading_params = [query_emb_str, batch_ids, ft, semantic_threshold, batch_size * 10]
+            if tags:
+                spreading_params.append(tags)
+
             neighbors = await conn.fetch(
                 f"""
-                SELECT mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start, mu.occurred_end, mu.mentioned_at, mu.access_count, mu.embedding, mu.fact_type, mu.document_id, mu.chunk_id,
+                SELECT mu.id, mu.text, mu.context, mu.event_date, mu.occurred_start, mu.occurred_end, mu.mentioned_at, mu.access_count, mu.embedding, mu.fact_type, mu.document_id, mu.chunk_id, mu.tags,
                        ml.weight, ml.link_type, ml.from_unit_id,
                        1 - (mu.embedding <=> $1::vector) AS similarity
                 FROM {fq_table("memory_links")} ml
@@ -453,14 +497,11 @@ async def retrieve_temporal_combined(
                   AND mu.fact_type = $3
                   AND mu.embedding IS NOT NULL
                   AND (1 - (mu.embedding <=> $1::vector)) >= $4
+                  {spreading_tags_clause}
                 ORDER BY ml.weight DESC
                 LIMIT $5
                 """,
-                query_emb_str,
-                batch_ids,
-                ft,
-                semantic_threshold,
-                batch_size * 10,
+                *spreading_params,
             )
 
             for n in neighbors:
@@ -529,6 +570,7 @@ async def retrieve_temporal(
     end_date: datetime,
     budget: int,
     semantic_threshold: float = 0.1,
+    tags: list[str] | None = None,
 ) -> list[RetrievalResult]:
     """
     Temporal retrieval with spreading activation.
@@ -547,6 +589,7 @@ async def retrieve_temporal(
         end_date: End of time range
         budget: Node budget for spreading
         semantic_threshold: Minimum semantic similarity to include
+        tags: Optional list of tags for visibility filtering (OR matching)
 
     Returns:
         List of RetrievalResult objects with temporal scores
@@ -558,9 +601,16 @@ async def retrieve_temporal(
     if end_date.tzinfo is None:
         end_date = end_date.replace(tzinfo=UTC)
 
+    from .tags import TagsMatch, build_tags_where_clause_simple
+
+    tags_clause = build_tags_where_clause_simple(tags, 7)
+    params = [query_emb_str, bank_id, fact_type, start_date, end_date, semantic_threshold]
+    if tags:
+        params.append(tags)
+
     entry_points = await conn.fetch(
         f"""
-        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id,
+        SELECT id, text, context, event_date, occurred_start, occurred_end, mentioned_at, access_count, embedding, fact_type, document_id, chunk_id, tags,
                1 - (embedding <=> $1::vector) AS similarity
         FROM {fq_table("memory_units")}
         WHERE bank_id = $2
@@ -580,15 +630,11 @@ async def retrieve_temporal(
               (occurred_end IS NOT NULL AND occurred_end BETWEEN $4 AND $5)
           )
           AND (1 - (embedding <=> $1::vector)) >= $6
+          {tags_clause}
         ORDER BY COALESCE(occurred_start, mentioned_at, occurred_end) DESC, (embedding <=> $1::vector) ASC
         LIMIT 10
         """,
-        query_emb_str,
-        bank_id,
-        fact_type,
-        start_date,
-        end_date,
-        semantic_threshold,
+        *params,
     )
 
     if not entry_points:
@@ -740,6 +786,7 @@ async def retrieve_parallel(
     query_analyzer: Optional["QueryAnalyzer"] = None,
     graph_retriever: GraphRetriever | None = None,
     temporal_constraint: tuple | None = None,  # Pre-extracted temporal constraint
+    tags: list[str] | None = None,  # Visibility scope tags for filtering
 ) -> ParallelRetrievalResult:
     """
     Run 3-way or 4-way parallel retrieval (adds temporal if detected).
@@ -755,6 +802,7 @@ async def retrieve_parallel(
         query_analyzer: Query analyzer to use (defaults to TransformerQueryAnalyzer)
         graph_retriever: Graph retrieval strategy (defaults to configured retriever)
         temporal_constraint: Pre-extracted temporal constraint (optional)
+        tags: Optional list of tags for visibility filtering (OR matching)
 
     Returns:
         ParallelRetrievalResult with semantic, bm25, graph, temporal results and timings
@@ -775,6 +823,7 @@ async def retrieve_parallel(
             retriever,
             question_date,
             query_analyzer,
+            tags=tags,
         )
     else:
         # For BFS, extract temporal constraint upfront (legacy path)
@@ -785,7 +834,15 @@ async def retrieve_parallel(
                 query_text, reference_date=question_date, analyzer=query_analyzer
             )
         return await _retrieve_parallel_bfs(
-            pool, query_text, query_embedding_str, bank_id, fact_type, thinking_budget, temporal_constraint, retriever
+            pool,
+            query_text,
+            query_embedding_str,
+            bank_id,
+            fact_type,
+            thinking_budget,
+            temporal_constraint,
+            retriever,
+            tags=tags,
         )
 
 
@@ -809,6 +866,7 @@ async def _retrieve_parallel_mpfp(
     retriever: GraphRetriever,
     question_date: datetime | None = None,
     query_analyzer=None,
+    tags: list[str] | None = None,
 ) -> ParallelRetrievalResult:
     """
     MPFP retrieval with true parallelization.
@@ -830,7 +888,9 @@ async def _retrieve_parallel_mpfp(
         acquire_start = time.time()
         async with acquire_with_retry(pool) as conn:
             conn_wait = time.time() - acquire_start
-            results = await retrieve_semantic(conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget)
+            results = await retrieve_semantic(
+                conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget, tags=tags
+            )
         return _TimedResult(results, time.time() - start, conn_wait)
 
     async def run_bm25() -> _TimedResult:
@@ -839,7 +899,7 @@ async def _retrieve_parallel_mpfp(
         acquire_start = time.time()
         async with acquire_with_retry(pool) as conn:
             conn_wait = time.time() - acquire_start
-            results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget)
+            results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget, tags=tags)
         return _TimedResult(results, time.time() - start, conn_wait)
 
     async def run_graph() -> tuple[list[RetrievalResult], float, MPFPTimings | None]:
@@ -857,6 +917,7 @@ async def _retrieve_parallel_mpfp(
             query_text=query_text,
             semantic_seeds=None,  # Let MPFP find its own seeds
             temporal_seeds=None,  # Don't wait for temporal extraction
+            tags=tags,
         )
         return results, time.time() - start, mpfp_timing
 
@@ -1028,6 +1089,7 @@ async def _retrieve_parallel_bfs(
     thinking_budget: int,
     temporal_constraint: tuple | None,
     retriever: GraphRetriever,
+    tags: list[str] | None = None,
 ) -> ParallelRetrievalResult:
     """BFS retrieval: all methods run in parallel (original behavior)."""
     import time
@@ -1035,13 +1097,15 @@ async def _retrieve_parallel_bfs(
     async def run_semantic() -> _TimedResult:
         start = time.time()
         async with acquire_with_retry(pool) as conn:
-            results = await retrieve_semantic(conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget)
+            results = await retrieve_semantic(
+                conn, query_embedding_str, bank_id, fact_type, limit=thinking_budget, tags=tags
+            )
         return _TimedResult(results, time.time() - start)
 
     async def run_bm25() -> _TimedResult:
         start = time.time()
         async with acquire_with_retry(pool) as conn:
-            results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget)
+            results = await retrieve_bm25(conn, query_text, bank_id, fact_type, limit=thinking_budget, tags=tags)
         return _TimedResult(results, time.time() - start)
 
     async def run_graph() -> _TimedResult:
@@ -1053,6 +1117,7 @@ async def _retrieve_parallel_bfs(
             fact_type=fact_type,
             budget=thinking_budget,
             query_text=query_text,
+            tags=tags,
         )
         return _TimedResult(results, time.time() - start)
 
@@ -1068,6 +1133,7 @@ async def _retrieve_parallel_bfs(
                 tc_end,
                 budget=thinking_budget,
                 semantic_threshold=0.1,
+                tags=tags,
             )
         return _TimedResult(results, time.time() - start)
 
@@ -1122,6 +1188,8 @@ async def retrieve_all_fact_types_parallel(
     question_date: datetime | None = None,
     query_analyzer: Optional["QueryAnalyzer"] = None,
     graph_retriever: GraphRetriever | None = None,
+    tags: list[str] | None = None,
+    tags_match: TagsMatch = "any",
 ) -> MultiFactTypeRetrievalResult:
     """
     Optimized retrieval for multiple fact types using batched queries.
@@ -1171,7 +1239,14 @@ async def retrieve_all_fact_types_parallel(
 
         # Semantic + BM25 combined
         semantic_bm25_results = await retrieve_semantic_bm25_combined(
-            conn, query_embedding_str, query_text, bank_id, fact_types, thinking_budget
+            conn,
+            query_embedding_str,
+            query_text,
+            bank_id,
+            fact_types,
+            thinking_budget,
+            tags=tags,
+            tags_match=tags_match,
         )
         semantic_bm25_time = time.time() - semantic_bm25_start
 
@@ -1188,6 +1263,8 @@ async def retrieve_all_fact_types_parallel(
                 tc_end,
                 budget=thinking_budget,
                 semantic_threshold=0.1,
+                tags=tags,
+                tags_match=tags_match,
             )
             temporal_time = time.time() - temporal_start
 
@@ -1206,6 +1283,8 @@ async def retrieve_all_fact_types_parallel(
             query_text=query_text,
             semantic_seeds=None,
             temporal_seeds=None,
+            tags=tags,
+            tags_match=tags_match,
         )
         return ft, results, time.time() - graph_start, mpfp_timing
 

hindsight-api/hindsight_api/engine/search/tags.py

@@ -0,0 +1,172 @@
+"""
+Tags filtering utilities for retrieval.
+
+Provides SQL building functions for filtering memories by tags.
+Supports four matching modes via TagsMatch enum:
+- "any": OR matching, includes untagged memories (default, backward compatible)
+- "all": AND matching, includes untagged memories
+- "any_strict": OR matching, excludes untagged memories
+- "all_strict": AND matching, excludes untagged memories
+
+OR matching (any/any_strict): Memory matches if ANY of its tags overlap with request tags
+AND matching (all/all_strict): Memory matches if ALL request tags are present in its tags
+"""
+
+from typing import Literal
+
+TagsMatch = Literal["any", "all", "any_strict", "all_strict"]
+
+
+def _parse_tags_match(match: TagsMatch) -> tuple[str, bool]:
+    """
+    Parse TagsMatch into operator and include_untagged flag.
+
+    Returns:
+        Tuple of (operator, include_untagged)
+        - operator: "&&" for any/any_strict, "@>" for all/all_strict
+        - include_untagged: True for any/all, False for any_strict/all_strict
+    """
+    if match == "any":
+        return "&&", True
+    elif match == "all":
+        return "@>", True
+    elif match == "any_strict":
+        return "&&", False
+    elif match == "all_strict":
+        return "@>", False
+    else:
+        # Default to "any" behavior
+        return "&&", True
+
+
+def build_tags_where_clause(
+    tags: list[str] | None,
+    param_offset: int = 1,
+    table_alias: str = "",
+    match: TagsMatch = "any",
+) -> tuple[str, list, int]:
+    """
+    Build a SQL WHERE clause for filtering by tags.
+
+    Supports four matching modes:
+    - "any" (default): OR matching, includes untagged memories
+    - "all": AND matching, includes untagged memories
+    - "any_strict": OR matching, excludes untagged memories
+    - "all_strict": AND matching, excludes untagged memories
+
+    Args:
+        tags: List of tags to filter by. If None or empty, returns empty clause (no filtering).
+        param_offset: Starting parameter number for SQL placeholders (default 1).
+        table_alias: Optional table alias prefix (e.g., "mu." for "memory_units mu").
+        match: Matching mode. Defaults to "any".
+
+    Returns:
+        Tuple of (sql_clause, params, next_param_offset):
+        - sql_clause: SQL WHERE clause string
+        - params: List of parameter values to bind
+        - next_param_offset: Next available parameter number
+
+    Example:
+        >>> clause, params, next_offset = build_tags_where_clause(['user_a'], 3, 'mu.', 'any_strict')
+        >>> print(clause)  # "AND mu.tags IS NOT NULL AND mu.tags != '{}' AND mu.tags && $3"
+    """
+    if not tags:
+        return "", [], param_offset
+
+    column = f"{table_alias}tags" if table_alias else "tags"
+    operator, include_untagged = _parse_tags_match(match)
+
+    if include_untagged:
+        # Include untagged memories (NULL or empty array) OR matching tags
+        clause = f"AND ({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_offset})"
+    else:
+        # Strict: only memories with matching tags (exclude NULL and empty)
+        clause = f"AND {column} IS NOT NULL AND {column} != '{{}}' AND {column} {operator} ${param_offset}"
+
+    return clause, [tags], param_offset + 1
+
+
+def build_tags_where_clause_simple(
+    tags: list[str] | None,
+    param_num: int,
+    table_alias: str = "",
+    match: TagsMatch = "any",
+) -> str:
+    """
+    Build a simple SQL WHERE clause for tags filtering.
+
+    This is a convenience version that returns just the clause string,
+    assuming the caller will add the tags array to their params list.
+
+    Args:
+        tags: List of tags to filter by. If None or empty, returns empty string.
+        param_num: Parameter number to use in the clause.
+        table_alias: Optional table alias prefix.
+        match: Matching mode. Defaults to "any".
+
+    Returns:
+        SQL clause string or empty string.
+    """
+    if not tags:
+        return ""
+
+    column = f"{table_alias}tags" if table_alias else "tags"
+    operator, include_untagged = _parse_tags_match(match)
+
+    if include_untagged:
+        # Include untagged memories (NULL or empty array) OR matching tags
+        return f"AND ({column} IS NULL OR {column} = '{{}}' OR {column} {operator} ${param_num})"
+    else:
+        # Strict: only memories with matching tags (exclude NULL and empty)
+        return f"AND {column} IS NOT NULL AND {column} != '{{}}' AND {column} {operator} ${param_num}"
+
+
+def filter_results_by_tags(
+    results: list,
+    tags: list[str] | None,
+    match: TagsMatch = "any",
+) -> list:
+    """
+    Filter retrieval results by tags in Python (for post-processing).
+
+    Used when SQL filtering isn't possible (e.g., graph traversal results).
+
+    Args:
+        results: List of RetrievalResult objects with a 'tags' attribute.
+        tags: List of tags to filter by. If None or empty, returns all results.
+        match: Matching mode. Defaults to "any".
+
+    Returns:
+        Filtered list of results.
+    """
+    if not tags:
+        return results
+
+    _, include_untagged = _parse_tags_match(match)
+    is_any_match = match in ("any", "any_strict")
+
+    tags_set = set(tags)
+    filtered = []
+
+    for result in results:
+        result_tags = getattr(result, "tags", None)
+
+        # Check if untagged
+        is_untagged = result_tags is None or len(result_tags) == 0
+
+        if is_untagged:
+            if include_untagged:
+                filtered.append(result)
+            # else: skip untagged
+        else:
+            result_tags_set = set(result_tags)
+            if is_any_match:
+                # Any overlap
+                if result_tags_set & tags_set:
+                    filtered.append(result)
+            else:
+                # All tags must be present
+                if tags_set <= result_tags_set:
+                    filtered.append(result)
+
+    return filtered

hindsight-api/hindsight_api/engine/search/think_utils.py

@@ -3,31 +3,13 @@ Think operation utilities for formulating answers based on agent and world facts
 """
 
 import logging
-import re
 from datetime import datetime
 
-from pydantic import BaseModel, Field
-
 from ..response_models import DispositionTraits, MemoryFact
 
 logger = logging.getLogger(__name__)
 
 
-class Opinion(BaseModel):
-    """An opinion formed by the bank."""
-
-    opinion: str = Field(description="The opinion or perspective with reasoning included")
-    confidence: float = Field(description="Confidence score for this opinion (0.0 to 1.0, where 1.0 is very confident)")
-
-
-class OpinionExtractionResponse(BaseModel):
-    """Response containing extracted opinions."""
-
-    opinions: list[Opinion] = Field(
-        default_factory=list, description="List of opinions formed with their supporting reasons and confidence scores"
-    )
-
-
 def describe_trait_level(value: int) -> str:
     """Convert trait value (1-5) to descriptive text."""
     levels = {1: "very low", 2: "low", 3: "moderate", 4: "high", 5: "very high"}
@@ -93,17 +75,46 @@ def format_facts_for_prompt(facts: list[MemoryFact]) -> str:
     return json.dumps(formatted, indent=2)
 
 
+def format_entity_summaries_for_prompt(entities: dict) -> str:
+    """Format entity summaries for inclusion in the reflect prompt.
+
+    Args:
+        entities: Dict mapping entity name to EntityState objects
+
+    Returns:
+        Formatted string with entity summaries, or empty string if no summaries
+    """
+    if not entities:
+        return ""
+
+    summaries = []
+    for name, state in entities.items():
+        # Get summary from observations (summary is stored as single observation)
+        if state.observations:
+            summary_text = state.observations[0].text
+            summaries.append(f"## {name}\n{summary_text}")
+
+    if not summaries:
+        return ""
+
+    return "\n\n".join(summaries)
+
+
 def build_think_prompt(
     agent_facts_text: str,
     world_facts_text: str,
-    opinion_facts_text: str,
     query: str,
     name: str,
     disposition: DispositionTraits,
     background: str,
     context: str | None = None,
+    entity_summaries_text: str | None = None,
 ) -> str:
-    """Build the think prompt for the LLM."""
+    """Build the think prompt for the LLM.
+
+    Note: opinion_facts_text parameter removed - opinions are now stored as mental models
+    and included via entity_summaries_text.
+    """
     disposition_desc = build_disposition_description(disposition)
 
     name_section = f"""
@@ -125,6 +136,14 @@ Your background:
 ADDITIONAL CONTEXT:
 {context}
 
+"""
+
+    entity_section = ""
+    if entity_summaries_text:
+        entity_section = f"""
+KEY PEOPLE, PLACES & THINGS I KNOW ABOUT:
+{entity_summaries_text}
+
 """
 
     return f"""Here's what I know and have experienced:
@@ -135,14 +154,11 @@ MY IDENTITY & EXPERIENCES:
 WHAT I KNOW ABOUT THE WORLD:
 {world_facts_text}
 
-MY EXISTING OPINIONS & BELIEFS:
-{opinion_facts_text}
-
-{context_section}{name_section}{disposition_desc}{background_section}
+{entity_section}{context_section}{name_section}{disposition_desc}{background_section}
 
 QUESTION: {query}
 
-Based on everything I know, believe, and who I am (including my name, disposition and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, opinions, and personal traits to give you my honest perspective."""
+Based on everything I know, believe, and who I am (including my name, disposition and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, and personal traits to give you my honest perspective."""
 
 
 def get_system_message(disposition: DispositionTraits) -> str:
@@ -175,122 +191,11 @@ def get_system_message(disposition: DispositionTraits) -> str:
     return f"You are a person with your own thoughts, experiences, opinions, and disposition. When asked a question, respond naturally like a human would - share what you genuinely think based on what you know and have experienced. {disposition_instruction} Be direct, express your views confidently, and use 'I think', 'I believe', 'in my view', etc. Respond in plain text without markdown formatting. IMPORTANT: Detect the language of the question and respond in the SAME language. Do not translate to English if the question is in another language."
 
 
-async def extract_opinions_from_text(llm_config, text: str, query: str) -> list[Opinion]:
-    """
-    Extract opinions with reasons and confidence from text using LLM.
-
-    Args:
-        llm_config: LLM configuration to use
-        text: Text to extract opinions from
-        query: The original query that prompted this response
-
-    Returns:
-        List of Opinion objects with text and confidence
-    """
-    extraction_prompt = f"""Extract any NEW opinions or perspectives from the answer below and rewrite them in FIRST-PERSON as if YOU are stating the opinion directly.
-
-ORIGINAL QUESTION:
-{query}
-
-ANSWER PROVIDED:
-{text}
-
-Your task: Find opinions in the answer and rewrite them AS IF YOU ARE THE ONE SAYING THEM.
-
-An opinion is a judgment, viewpoint, or conclusion that goes beyond just stating facts.
-
-IMPORTANT: Do NOT extract statements like:
-- "I don't have enough information"
-- "The facts don't contain information about X"
-- "I cannot answer because..."
-
-ONLY extract actual opinions about substantive topics.
-
-CRITICAL FORMAT REQUIREMENTS:
-1. **ALWAYS start with first-person phrases**: "I think...", "I believe...", "In my view...", "I've come to believe...", "Previously I thought... but now..."
-2. **NEVER use third-person**: Do NOT say "The speaker thinks..." or "They believe..." - always use "I"
-3. Include the reasoning naturally within the statement
-4. Provide a confidence score (0.0 to 1.0)
-
-CORRECT Examples (✓ FIRST-PERSON):
-- "I think Alice is more reliable because she consistently delivers on time and writes clean code"
-- "Previously I thought all engineers were equal, but now I feel that experience and track record really matter"
-- "I believe reliability is best measured by consistent output over time"
-- "I've come to believe that track records are more important than potential"
-
-WRONG Examples (✗ THIRD-PERSON - DO NOT USE):
-- "The speaker thinks Alice is more reliable"
-- "They believe reliability matters"
-- "It is believed that Alice is better"
-
-If no genuine opinions are expressed (e.g., the response just says "I don't know"), return an empty list."""
-
-    try:
-        result = await llm_config.call(
-            messages=[
-                {
-                    "role": "system",
-                    "content": "You are converting opinions from text into first-person statements. Always use 'I think', 'I believe', 'I feel', etc. NEVER use third-person like 'The speaker' or 'They'.",
-                },
-                {"role": "user", "content": extraction_prompt},
-            ],
-            response_format=OpinionExtractionResponse,
-            scope="memory_extract_opinion",
-        )
-
-        # Format opinions with confidence score and convert to first-person
-        formatted_opinions = []
-        for op in result.opinions:
-            # Convert third-person to first-person if needed
-            opinion_text = op.opinion
-
-            # Replace common third-person patterns with first-person
-            def singularize_verb(verb):
-                if verb.endswith("es"):
-                    return verb[:-1]  # believes -> believe
-                elif verb.endswith("s"):
-                    return verb[:-1]  # thinks -> think
-                return verb
-
-            # Pattern: "The speaker/user [verb]..." -> "I [verb]..."
-            match = re.match(
-                r"^(The speaker|The user|They|It is believed) (believes?|thinks?|feels?|says|asserts?|considers?)(\s+that)?(.*)$",
-                opinion_text,
-                re.IGNORECASE,
-            )
-            if match:
-                verb = singularize_verb(match.group(2))
-                that_part = match.group(3) or ""  # Keep " that" if present
-                rest = match.group(4)
-                opinion_text = f"I {verb}{that_part}{rest}"
-
-            # If still doesn't start with first-person, prepend "I believe that "
-            first_person_starters = [
-                "I think",
-                "I believe",
-                "I feel",
-                "In my view",
-                "I've come to believe",
-                "Previously I",
-            ]
-            if not any(opinion_text.startswith(starter) for starter in first_person_starters):
-                opinion_text = "I believe that " + opinion_text[0].lower() + opinion_text[1:]
-
-            formatted_opinions.append(Opinion(opinion=opinion_text, confidence=op.confidence))
-
-        return formatted_opinions
-
-    except Exception as e:
-        logger.warning(f"Failed to extract opinions: {str(e)}")
-        return []
-
-
 async def reflect(
     llm_config,
     query: str,
     experience_facts: list[str] = None,
     world_facts: list[str] = None,
-    opinion_facts: list[str] = None,
     name: str = "Assistant",
     disposition: DispositionTraits = None,
     background: str = "",
@@ -307,7 +212,6 @@ async def reflect(
         query: Question to answer
         experience_facts: List of experience/agent fact strings
         world_facts: List of world fact strings
-        opinion_facts: List of opinion fact strings
         name: Name of the agent/persona
         disposition: Disposition traits (defaults to neutral)
         background: Background information
@@ -328,18 +232,15 @@ async def reflect(
 
     agent_results = to_memory_facts(experience_facts or [], "experience")
     world_results = to_memory_facts(world_facts or [], "world")
-    opinion_results = to_memory_facts(opinion_facts or [], "opinion")
 
     # Format facts for prompt
     agent_facts_text = format_facts_for_prompt(agent_results)
     world_facts_text = format_facts_for_prompt(world_results)
-    opinion_facts_text = format_facts_for_prompt(opinion_results)
 
     # Build prompt
     prompt = build_think_prompt(
         agent_facts_text=agent_facts_text,
         world_facts_text=world_facts_text,
-        opinion_facts_text=opinion_facts_text,
         query=query,
         name=name,
         disposition=disposition,

hindsight-api/hindsight_api/engine/search/trace.py

@@ -11,6 +11,13 @@ from typing import Any, Literal
 from pydantic import BaseModel, Field
 
 
+class TemporalConstraint(BaseModel):
+    """Detected temporal constraint from query analysis."""
+
+    start: datetime | None = Field(default=None, description="Start of temporal range")
+    end: datetime | None = Field(default=None, description="End of temporal range")
+
+
 class QueryInfo(BaseModel):
     """Information about the search query."""
 
@@ -19,6 +26,11 @@ class QueryInfo(BaseModel):
     timestamp: datetime = Field(description="When the query was executed")
     budget: int = Field(description="Maximum nodes to explore")
     max_tokens: int = Field(description="Maximum tokens to return in results")
+    tags: list[str] | None = Field(default=None, description="Tags filter applied to recall")
+    tags_match: str | None = Field(default=None, description="Tags matching mode: any, all, any_strict, all_strict")
+    temporal_constraint: TemporalConstraint | None = Field(
+        default=None, description="Detected temporal range from query"
+    )
 
 
 class EntryPoint(BaseModel):

hindsight-api/hindsight_api/engine/search/tracer.py

@@ -22,6 +22,7 @@ from .trace import (
     SearchPhaseMetrics,
     SearchSummary,
     SearchTrace,
+    TemporalConstraint,
     WeightComponents,
 )
 
@@ -45,7 +46,14 @@ class SearchTracer:
         json_output = trace.to_json()
     """
 
-    def __init__(self, query: str, budget: int, max_tokens: int):
+    def __init__(
+        self,
+        query: str,
+        budget: int,
+        max_tokens: int,
+        tags: list[str] | None = None,
+        tags_match: str | None = None,
+    ):
         """
         Initialize tracer.
 
@@ -53,10 +61,14 @@ class SearchTracer:
             query: Search query text
             budget: Maximum nodes to explore
             max_tokens: Maximum tokens to return in results
+            tags: Tags filter applied to recall
+            tags_match: Tags matching mode (any, all, any_strict, all_strict)
         """
         self.query_text = query
         self.budget = budget
         self.max_tokens = max_tokens
+        self.tags = tags
+        self.tags_match = tags_match
 
         # Trace data
         self.query_embedding: list[float] | None = None
@@ -66,6 +78,9 @@ class SearchTracer:
         self.pruned: list[PruningDecision] = []
         self.phase_metrics: list[SearchPhaseMetrics] = []
 
+        # Temporal constraint detected from query
+        self.temporal_constraint: TemporalConstraint | None = None
+
         # New 4-way retrieval tracking
         self.retrieval_results: list[RetrievalMethodResults] = []
         self.rrf_merged: list[RRFMergeResult] = []
@@ -88,6 +103,11 @@ class SearchTracer:
         """Record the query embedding."""
         self.query_embedding = embedding
 
+    def record_temporal_constraint(self, start: datetime | None, end: datetime | None):
+        """Record the detected temporal constraint from query analysis."""
+        if start is not None or end is not None:
+            self.temporal_constraint = TemporalConstraint(start=start, end=end)
+
     def add_entry_point(self, node_id: str, text: str, similarity: float, rank: int):
         """
         Record an entry point.
@@ -428,6 +448,9 @@ class SearchTracer:
             timestamp=datetime.now(UTC),
             budget=self.budget,
             max_tokens=self.max_tokens,
+            tags=self.tags,
+            tags_match=self.tags_match,
+            temporal_constraint=self.temporal_constraint,
         )
 
         # Create summary

hindsight-api/hindsight_api/engine/search/types.py

@@ -48,6 +48,7 @@ class RetrievalResult:
     chunk_id: str | None = None
     access_count: int = 0
     embedding: list[float] | None = None
+    tags: list[str] | None = None  # Visibility scope tags
 
     # Retrieval-specific scores (only one will be set depending on retrieval method)
     similarity: float | None = None  # Semantic retrieval
@@ -72,6 +73,7 @@ class RetrievalResult:
             chunk_id=row.get("chunk_id"),
             access_count=row.get("access_count", 0),
             embedding=row.get("embedding"),
+            tags=row.get("tags"),
             similarity=row.get("similarity"),
             bm25_score=row.get("bm25_score"),
             activation=row.get("activation"),
@@ -156,6 +158,7 @@ class ScoredResult:
             "chunk_id": self.retrieval.chunk_id,
             "access_count": self.retrieval.access_count,
             "embedding": self.retrieval.embedding,
+            "tags": self.retrieval.tags,
             "semantic_similarity": self.retrieval.similarity,
             "bm25_score": self.retrieval.bm25_score,
         }

hindsight-api/hindsight_api/main.py

@@ -187,12 +187,15 @@ def main():
             embeddings_provider=config.embeddings_provider,
             embeddings_local_model=config.embeddings_local_model,
             embeddings_tei_url=config.embeddings_tei_url,
+            embeddings_openai_base_url=config.embeddings_openai_base_url,
+            embeddings_cohere_base_url=config.embeddings_cohere_base_url,
             reranker_provider=config.reranker_provider,
             reranker_local_model=config.reranker_local_model,
             reranker_tei_url=config.reranker_tei_url,
             reranker_tei_batch_size=config.reranker_tei_batch_size,
             reranker_tei_max_concurrent=config.reranker_tei_max_concurrent,
             reranker_max_candidates=config.reranker_max_candidates,
+            reranker_cohere_base_url=config.reranker_cohere_base_url,
             host=args.host,
             port=args.port,
             log_level=args.log_level,
@@ -218,6 +221,8 @@ def main():
             task_backend=config.task_backend,
             task_backend_memory_batch_size=config.task_backend_memory_batch_size,
             task_backend_memory_batch_interval=config.task_backend_memory_batch_interval,
+            reflect_max_iterations=config.reflect_max_iterations,
+            mental_model_refresh_concurrency=config.mental_model_refresh_concurrency,
         )
     config.configure_logging()
     if not args.daemon:

hindsight-api/hindsight_api/metrics.py

@@ -28,6 +28,15 @@ from opentelemetry.sdk.resources import Resource
 if TYPE_CHECKING:
     import asyncpg
 
+
+def _get_tenant() -> str:
+    """Get current tenant (schema) from context for metrics labeling."""
+    # Import here to avoid circular imports
+    from hindsight_api.engine.memory_engine import get_current_schema
+
+    return get_current_schema()
+
+
 # Custom bucket boundaries for operation duration (in seconds)
 # Fine granularity in 0-30s range where most operations complete
 DURATION_BUCKETS = (0.1, 0.25, 0.5, 0.75, 1.0, 2.0, 3.0, 5.0, 7.5, 10.0, 15.0, 20.0, 30.0, 60.0, 120.0)
@@ -323,6 +332,7 @@ class MetricsCollector(MetricsCollectorBase):
             "operation": operation,
             "bank_id": bank_id,
             "source": source,
+            "tenant": _get_tenant(),
         }
         if budget:
             attributes["budget"] = budget
@@ -373,6 +383,7 @@ class MetricsCollector(MetricsCollectorBase):
             "model": model,
             "scope": scope,
             "success": str(success).lower(),
+            "tenant": _get_tenant(),
         }
 
         # Record duration
@@ -425,10 +436,14 @@ class MetricsCollector(MetricsCollectorBase):
             status_code = status_code_getter()
             status_class = f"{status_code // 100}xx"
 
+            # Get tenant from context (may be set during request processing)
+            tenant = _get_tenant()
+
             attributes = {
                 **base_attributes,
                 "status_code": str(status_code),
                 "status_class": status_class,
+                "tenant": tenant,
             }
 
             # Record duration and count

hindsight-api/hindsight_api/server.py

@@ -7,6 +7,7 @@ This module provides the ASGI app for uvicorn import string usage:
 For CLI usage, use the hindsight-api command instead.
 """
 
+import logging
 import os
 import warnings
 
@@ -17,6 +18,12 @@ warnings.filterwarnings("ignore", message="websockets.server.WebSocketServerProt
 from hindsight_api import MemoryEngine
 from hindsight_api.api import create_app
 from hindsight_api.config import get_config
+from hindsight_api.extensions import (
+    DefaultExtensionContext,
+    OperationValidatorExtension,
+    TenantExtension,
+    load_extension,
+)
 
 # Disable tokenizers parallelism to avoid warnings
 os.environ["TOKENIZERS_PARALLELISM"] = "false"
@@ -25,10 +32,33 @@ os.environ["TOKENIZERS_PARALLELISM"] = "false"
 config = get_config()
 config.configure_logging()
 
+# Load operation validator extension if configured
+operation_validator = load_extension("OPERATION_VALIDATOR", OperationValidatorExtension)
+if operation_validator:
+    logging.info(f"Loaded operation validator: {operation_validator.__class__.__name__}")
+
+# Load tenant extension if configured
+tenant_extension = load_extension("TENANT", TenantExtension)
+if tenant_extension:
+    logging.info(f"Loaded tenant extension: {tenant_extension.__class__.__name__}")
+
 # Create app at module level (required for uvicorn import string)
 # MemoryEngine reads configuration from environment variables automatically
 # Note: run_migrations=True by default, but migrations are idempotent so safe with workers
-_memory = MemoryEngine(run_migrations=config.run_migrations_on_startup)
+_memory = MemoryEngine(
+    operation_validator=operation_validator,
+    tenant_extension=tenant_extension,
+    run_migrations=config.run_migrations_on_startup,
+)
+
+# Set extension context on tenant extension (needed for schema provisioning)
+if tenant_extension:
+    extension_context = DefaultExtensionContext(
+        database_url=config.database_url,
+        memory_engine=_memory,
+    )
+    tenant_extension.set_context(extension_context)
+    logging.info("Extension context set on tenant extension")
 
 # Create unified app with both HTTP and optionally MCP
 app = create_app(

hindsight-api/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hindsight-api"
-version = "0.2.1"
+version = "0.3.0"
 description = "Hindsight: Agent Memory That Works Like Human Memory"
 readme = "README.md"
 requires-python = ">=3.11"

hindsight-api/tests/test_agents_api.py

@@ -1,5 +1,5 @@
 """
-Tests for agent management API (profile, disposition, background).
+Tests for agent management API (profile, disposition).
 """
 import pytest
 import uuid
@@ -25,15 +25,12 @@ class TestAgentProfile:
 
         assert profile is not None
         assert "disposition" in profile
-        assert "background" in profile
 
         disposition = profile["disposition"]
         assert disposition.skepticism == 3
         assert disposition.literalism == 3
         assert disposition.empathy == 3
 
-        assert profile["background"] == ""
-
     @pytest.mark.asyncio
     async def test_update_agent_disposition(self, memory: MemoryEngine, request_context):
         """Test updating agent disposition traits."""
@@ -76,63 +73,10 @@ class TestAgentProfile:
         for agent in agents:
             assert "bank_id" in agent
             assert "disposition" in agent
-            assert "background" in agent
             assert "created_at" in agent
             assert "updated_at" in agent
 
 
-class TestAgentBackground:
-    """Tests for agent background management."""
-
-    @pytest.mark.asyncio
-    async def test_merge_agent_background(self, memory: MemoryEngine, request_context):
-        """Test merging agent background information."""
-        bank_id = unique_agent_id("test_profile_merge")
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        assert profile["background"] == ""
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I was born in Texas",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result1["background"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "I have 10 years of startup experience",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result2["background"] or "startup" in result2["background"]
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        assert final_profile["background"] != ""
-
-    @pytest.mark.asyncio
-    async def test_merge_background_handles_conflicts(self, memory: MemoryEngine, request_context):
-        """Test that merging background handles conflicts (new overwrites old)."""
-        bank_id = unique_agent_id("test_profile_conflict")
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I was born in Colorado",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Colorado" in result1["background"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "You were born in Texas",
-            update_disposition=False,
-            request_context=request_context,
-        )
-        assert "Texas" in result2["background"]
-
-
 class TestAgentEndpoint:
     """Tests for agent PUT endpoint logic."""
 
@@ -147,7 +91,6 @@ class TestAgentEndpoint:
                 literalism=5,
                 empathy=2
             ),
-            background="I am a creative software engineer"
         )
 
         profile = await memory.get_bank_profile(bank_id, request_context=request_context)
@@ -159,55 +102,10 @@ class TestAgentEndpoint:
                 request_context=request_context,
             )
 
-        if request.background is not None:
-            pool = await memory._get_pool()
-            async with pool.acquire() as conn:
-                await conn.execute(
-                    """
-                    UPDATE banks
-                    SET background = $2,
-                        updated_at = NOW()
-                    WHERE bank_id = $1
-                    """,
-                    bank_id,
-                    request.background
-                )
-
         final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
 
         assert final_profile["disposition"].skepticism == 4
         assert final_profile["disposition"].literalism == 5
-        assert final_profile["background"] == "I am a creative software engineer"
-
-    @pytest.mark.asyncio
-    async def test_put_agent_partial_update(self, memory: MemoryEngine, request_context):
-        """Test updating only background."""
-        bank_id = unique_agent_id("test_put_partial")
-
-        request = CreateBankRequest(
-            background="I am a data scientist"
-        )
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-
-        if request.background is not None:
-            pool = await memory._get_pool()
-            async with pool.acquire() as conn:
-                await conn.execute(
-                    """
-                    UPDATE banks
-                    SET background = $2,
-                        updated_at = NOW()
-                    WHERE bank_id = $1
-                    """,
-                    bank_id,
-                    request.background
-                )
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-
-        assert final_profile["disposition"].skepticism == 3  # Default
-        assert final_profile["background"] == "I am a data scientist"
 
 
 class TestAgentDispositionIntegration:
@@ -225,13 +123,6 @@ class TestAgentDispositionIntegration:
         }
         await memory.update_bank_disposition(bank_id, disposition, request_context=request_context)
 
-        await memory.merge_bank_background(
-            bank_id,
-            "I am a creative artist who values innovation over tradition",
-            update_disposition=False,
-            request_context=request_context,
-        )
-
         await memory.retain_batch_async(
             bank_id=bank_id,
             contents=[

hindsight-api/tests/test_emergent_filtering.py

@@ -0,0 +1,516 @@
+"""Tests for emergent entity filtering."""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock
+
+from hindsight_api.engine.mental_models.emergent import (
+    build_mission_filter_prompt,
+    evaluate_emergent_models,
+    filter_candidates_by_mission,
+    MissionFilterResponse,
+    MissionFilterCandidate,
+)
+from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+
+class TestBuildMissionFilterPrompt:
+    """Test prompt building for mission filtering."""
+
+    def test_prompt_contains_mission(self):
+        """Test that prompt includes the mission."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+        prompt = build_mission_filter_prompt("Be a PM for engineering team", candidates)
+        assert "Be a PM for engineering team" in prompt
+
+    def test_prompt_contains_candidates(self):
+        """Test that prompt includes all candidates."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice Chen",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Project Phoenix",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+        prompt = build_mission_filter_prompt("Track projects", candidates)
+        assert "Alice Chen" in prompt
+        assert "Project Phoenix" in prompt
+
+    def test_prompt_contains_rejection_guidance(self):
+        """Test that prompt contains guidance to reject generic entities."""
+        candidates = [
+            EmergentCandidate(
+                name="test",
+                detection_method="named_entity_extraction",
+                mention_count=1,
+            )
+        ]
+        prompt = build_mission_filter_prompt("Test mission", candidates)
+
+        # Should contain rejection guidance for generic terms
+        assert "promote=false" in prompt
+        assert "kids" in prompt  # Example of generic term to reject
+        assert "community" in prompt  # Example of abstract concept to reject
+        assert "motivation" in prompt  # Example of abstract concept to reject
+
+
+class TestFilterCandidatesByMission:
+    """Test the filter_candidates_by_mission function."""
+
+    @pytest.fixture
+    def mock_llm_config(self):
+        """Create a mock LLM config."""
+        config = MagicMock()
+        config.call = AsyncMock()
+        return config
+
+    async def test_empty_candidates(self, mock_llm_config):
+        """Test with empty candidate list."""
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=[],
+        )
+        assert result == []
+        mock_llm_config.call.assert_not_called()
+
+    async def test_no_mission_keeps_all(self, mock_llm_config):
+        """Test that no mission keeps all candidates (skips filtering)."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="",  # Empty mission
+            candidates=candidates,
+        )
+        assert len(result) == 1
+        assert result[0].name == "Alice"
+        mock_llm_config.call.assert_not_called()
+
+    async def test_filters_by_promote_flag(self, mock_llm_config):
+        """Test that candidates are filtered by promote flag."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice Chen",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="community",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+
+        # Mock LLM response - Alice is promoted, community is not
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice Chen", promote=True, reason="Specific person"),
+                MissionFilterCandidate(name="community", promote=False, reason="Generic abstract concept"),
+            ]
+        )
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a PM for engineering team",
+            candidates=candidates,
+        )
+
+        assert len(result) == 1
+        assert result[0].name == "Alice Chen"
+
+    async def test_rejects_generic_entities(self, mock_llm_config):
+        """Test that generic entities are rejected."""
+        # These are all generic/abstract terms that should be rejected
+        generic_names = [
+            "user", "support", "community", "family", "motivation",
+            "photo", "gratitude", "difference", "volunteering",
+            "kids", "veterans", "impact", "kindness", "encouragement",
+            "education", "nature", "joy", "positivity", "inspiration",
+            "help", "commitment", "passion", "energy", "connection",
+        ]
+        candidates = [
+            EmergentCandidate(
+                name=name,
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+            for name in generic_names
+        ]
+
+        # Add some valid candidates
+        valid_candidates = [
+            EmergentCandidate(
+                name="John",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Maria",
+                detection_method="named_entity_extraction",
+                mention_count=8,
+            ),
+            EmergentCandidate(
+                name="Max",
+                detection_method="named_entity_extraction",
+                mention_count=6,
+            ),
+        ]
+        candidates.extend(valid_candidates)
+
+        # Mock LLM response - reject all generic, promote only specific names
+        response_candidates = [
+            MissionFilterCandidate(name=name, promote=False, reason="Generic/abstract term")
+            for name in generic_names
+        ]
+        response_candidates.extend([
+            MissionFilterCandidate(name=c.name, promote=True, reason="Specific person name")
+            for c in valid_candidates
+        ])
+
+        mock_llm_config.call.return_value = MissionFilterResponse(candidates=response_candidates)
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a health coach",
+            candidates=candidates,
+        )
+
+        # Should only have John, Maria, and Max
+        result_names = {c.name for c in result}
+        assert result_names == {"John", "Maria", "Max"}
+
+    async def test_accepts_specific_named_entities(self, mock_llm_config):
+        """Test that specific named entities are accepted."""
+        # These should all be accepted
+        valid_names = [
+            "Alice Chen",       # Full name
+            "Dr. Smith",        # Title + name
+            "John",             # First name (when it's clearly a person)
+            "Google",           # Organization
+            "Frontend Team",    # Named team
+            "Project Phoenix",  # Named project
+            "NYC Office",       # Named place
+            "Q4 Planning",      # Named event
+            "Sprint 23 Review", # Named meeting
+        ]
+        candidates = [
+            EmergentCandidate(
+                name=name,
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+            for name in valid_names
+        ]
+
+        # Mock LLM response - promote all
+        response_candidates = [
+            MissionFilterCandidate(name=name, promote=True, reason="Specific named entity")
+            for name in valid_names
+        ]
+        mock_llm_config.call.return_value = MissionFilterResponse(candidates=response_candidates)
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Be a PM for engineering team",
+            candidates=candidates,
+        )
+
+        # Should have all valid names
+        result_names = {c.name for c in result}
+        assert result_names == set(valid_names)
+
+    async def test_llm_error_rejects_all_candidates(self, mock_llm_config):
+        """Test that LLM errors result in rejecting all candidates (fail-safe)."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            )
+        ]
+
+        mock_llm_config.call.side_effect = Exception("LLM error")
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=candidates,
+        )
+
+        # Should reject all candidates on error (fail-safe)
+        assert len(result) == 0
+
+    async def test_missing_candidate_in_response_is_rejected(self, mock_llm_config):
+        """Test that candidates not in LLM response are rejected by default."""
+        candidates = [
+            EmergentCandidate(
+                name="Alice",
+                detection_method="named_entity_extraction",
+                mention_count=10,
+            ),
+            EmergentCandidate(
+                name="Bob",
+                detection_method="named_entity_extraction",
+                mention_count=5,
+            ),
+        ]
+
+        # Mock LLM response - only includes Alice, not Bob
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice", promote=True, reason="Specific person"),
+            ]
+        )
+
+        result = await filter_candidates_by_mission(
+            llm_config=mock_llm_config,
+            mission="Test mission",
+            candidates=candidates,
+        )
+
+        # Only Alice should be in result (Bob was missing from response, so rejected)
+        assert len(result) == 1
+        assert result[0].name == "Alice"
+
+
+class TestEvaluateEmergentModels:
+    """Test the evaluate_emergent_models function for cleanup of existing models."""
+
+    @pytest.fixture
+    def mock_llm_config(self):
+        """Create a mock LLM config."""
+        config = MagicMock()
+        config.call = AsyncMock()
+        return config
+
+    async def test_empty_models(self, mock_llm_config):
+        """Test with empty model list."""
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=[],
+        )
+        assert result == []
+        mock_llm_config.call.assert_not_called()
+
+    async def test_removes_generic_models(self, mock_llm_config):
+        """Test that generic/abstract models are marked for removal."""
+        models = [
+            {"id": "id-kids", "name": "kids"},
+            {"id": "id-community", "name": "community"},
+            {"id": "id-motivation", "name": "motivation"},
+            {"id": "id-john", "name": "John"},
+            {"id": "id-maria", "name": "Maria"},
+        ]
+
+        # Mock LLM response - reject generic, keep specific names
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="kids", promote=False, reason="Generic category"),
+                MissionFilterCandidate(name="community", promote=False, reason="Abstract concept"),
+                MissionFilterCandidate(name="motivation", promote=False, reason="Abstract concept"),
+                MissionFilterCandidate(name="John", promote=True, reason="Person name"),
+                MissionFilterCandidate(name="Maria", promote=True, reason="Person name"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Should return IDs of generic models to remove
+        assert set(result) == {"id-kids", "id-community", "id-motivation"}
+
+    async def test_keeps_specific_named_models(self, mock_llm_config):
+        """Test that specific named models are kept."""
+        models = [
+            {"id": "id-john", "name": "John"},
+            {"id": "id-google", "name": "Google"},
+            {"id": "id-project", "name": "Project Phoenix"},
+        ]
+
+        # Mock LLM response - keep all
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="John", promote=True, reason="Person name"),
+                MissionFilterCandidate(name="Google", promote=True, reason="Organization"),
+                MissionFilterCandidate(name="Project Phoenix", promote=True, reason="Named project"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # No models should be removed
+        assert result == []
+
+    async def test_llm_error_keeps_all_models(self, mock_llm_config):
+        """Test that LLM errors result in keeping all models (safe default)."""
+        models = [
+            {"id": "id-kids", "name": "kids"},
+            {"id": "id-john", "name": "John"},
+        ]
+
+        mock_llm_config.call.side_effect = Exception("LLM error")
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Should keep all models on error (return empty removal list)
+        assert result == []
+
+    async def test_missing_model_in_response_is_removed(self, mock_llm_config):
+        """Test that models not in LLM response are marked for removal."""
+        models = [
+            {"id": "id-alice", "name": "Alice"},
+            {"id": "id-bob", "name": "Bob"},
+        ]
+
+        # Mock LLM response - only includes Alice
+        mock_llm_config.call.return_value = MissionFilterResponse(
+            candidates=[
+                MissionFilterCandidate(name="Alice", promote=True, reason="Person name"),
+            ]
+        )
+
+        result = await evaluate_emergent_models(
+            llm_config=mock_llm_config,
+            models=models,
+        )
+
+        # Bob should be marked for removal (missing from response)
+        assert result == ["id-bob"]
+
+
+class TestRemovedEntitiesNotRepromoted:
+    """Test that entities removed by evaluation are not re-promoted.
+
+    This tests the fix for a bug where:
+    1. evaluate_emergent_models returns model IDs to remove (e.g., 'entity-maya')
+    2. We delete those models
+    3. detect_entity_candidates finds the same entities (now eligible since model was deleted)
+    4. filter_candidates_by_goal approves them (different LLM call)
+    5. BUG: We were re-promoting the same entities we just removed
+
+    The fix tracks removed entity_ids and excludes them from promotion.
+    """
+
+    async def test_removed_entity_ids_excluded_from_promotion(self):
+        """Test that entities whose models were removed are not re-promoted."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # Simulate the scenario from the bug:
+        # - existing_emergent has model 'entity-maya' with entity_id='uuid-maya'
+        # - evaluate_emergent_models says to remove 'entity-maya'
+        # - detect_entity_candidates returns 'Maya' with entity_id='uuid-maya' (now eligible)
+        # - filter_candidates_by_goal says to promote 'Maya'
+        # - But we should NOT promote because we just removed it
+
+        existing_emergent = [
+            {"id": "entity-maya", "name": "Maya", "entity_id": "uuid-maya"},
+            {"id": "entity-alex", "name": "Alex", "entity_id": "uuid-alex"},
+            {"id": "entity-john", "name": "John", "entity_id": "uuid-john"},  # This one will be kept
+        ]
+
+        # Models to remove (evaluate_emergent_models would return these)
+        models_to_remove = ["entity-maya", "entity-alex"]
+
+        # Build model_id -> entity_id mapping (this is what the fix does)
+        model_to_entity = {m["id"]: m.get("entity_id") for m in existing_emergent}
+
+        # Track removed entity_ids
+        removed_entity_ids: set[str] = set()
+        for model_id in models_to_remove:
+            entity_id = model_to_entity.get(model_id)
+            if entity_id:
+                removed_entity_ids.add(str(entity_id))
+
+        # Verify we tracked the right entity_ids
+        assert removed_entity_ids == {"uuid-maya", "uuid-alex"}
+
+        # Now simulate candidates that were detected (includes removed entities)
+        candidates = [
+            EmergentCandidate(
+                name="Maya", entity_id="uuid-maya", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Alex", entity_id="uuid-alex", detection_method="named_entity", mention_count=8
+            ),
+            EmergentCandidate(
+                name="NewPerson", entity_id="uuid-new", detection_method="named_entity", mention_count=5
+            ),
+        ]
+
+        # Filter out candidates whose entity was just removed (the fix)
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        # Only NewPerson should remain - Maya and Alex were removed and should not be re-promoted
+        assert len(filtered_candidates) == 1
+        assert filtered_candidates[0].name == "NewPerson"
+        assert filtered_candidates[0].entity_id == "uuid-new"
+
+    async def test_candidates_without_matching_removal_are_kept(self):
+        """Test that candidates not in the removed set are still promoted."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # No models removed
+        removed_entity_ids: set[str] = set()
+
+        candidates = [
+            EmergentCandidate(
+                name="Alice", entity_id="uuid-alice", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Bob", entity_id="uuid-bob", detection_method="named_entity", mention_count=8
+            ),
+        ]
+
+        # Filter (should keep all since nothing was removed)
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        assert len(filtered_candidates) == 2
+        assert {c.name for c in filtered_candidates} == {"Alice", "Bob"}
+
+    async def test_partial_removal_keeps_other_candidates(self):
+        """Test that only removed entities are excluded, others pass through."""
+        from hindsight_api.engine.mental_models.models import EmergentCandidate
+
+        # Only one entity removed
+        removed_entity_ids = {"uuid-removed"}
+
+        candidates = [
+            EmergentCandidate(
+                name="Removed", entity_id="uuid-removed", detection_method="named_entity", mention_count=10
+            ),
+            EmergentCandidate(
+                name="Kept1", entity_id="uuid-kept1", detection_method="named_entity", mention_count=8
+            ),
+            EmergentCandidate(
+                name="Kept2", entity_id="uuid-kept2", detection_method="named_entity", mention_count=5
+            ),
+        ]
+
+        filtered_candidates = [c for c in candidates if c.entity_id not in removed_entity_ids]
+
+        assert len(filtered_candidates) == 2
+        assert {c.name for c in filtered_candidates} == {"Kept1", "Kept2"}

hindsight-api/tests/test_fact_extraction_quality.py

@@ -947,172 +947,3 @@ so the algorithm learns to box out. See you next week!
                     raise e
 
 
-# =============================================================================
-# DISPOSITION INFERENCE TESTS
-# =============================================================================
-
-class TestDispositionInference:
-    """Tests for LLM-based disposition trait inference from background."""
-
-    @pytest.mark.asyncio
-    async def test_background_merge_with_disposition_inference(self, memory, request_context):
-        """Test that background merge infers disposition traits by default."""
-        import uuid
-        bank_id = f"test_infer_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a creative software engineer who loves innovation and trying new technologies",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        assert "background" in result
-        assert "disposition" in result
-
-        background = result["background"]
-        disposition = result["disposition"]
-
-        assert "creative" in background.lower() or "innovation" in background.lower()
-
-        # Check that new traits are present with valid values (1-5)
-        required_traits = ["skepticism", "literalism", "empathy"]
-        for trait in required_traits:
-            assert trait in disposition
-            assert 1 <= disposition[trait] <= 5
-
-    @pytest.mark.asyncio
-    async def test_background_merge_without_disposition_inference(self, memory, request_context):
-        """Test that background merge skips disposition inference when disabled."""
-        import uuid
-        bank_id = f"test_no_infer_{uuid.uuid4().hex[:8]}"
-
-        initial_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        initial_disposition = initial_profile["disposition"]
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a data scientist",
-            update_disposition=False,
-            request_context=request_context,
-        )
-
-        assert "background" in result
-        assert "disposition" not in result
-
-        final_profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        final_disposition = final_profile["disposition"]
-
-        assert initial_disposition == final_disposition
-
-    @pytest.mark.asyncio
-    async def test_disposition_inference_for_lawyer(self, memory, request_context):
-        """Test disposition inference for lawyer profile (high skepticism, high literalism)."""
-        import uuid
-        bank_id = f"test_lawyer_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a lawyer who focuses on contract details and never takes claims at face value",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        disposition = result["disposition"]
-
-        # Lawyers should have higher skepticism and literalism
-        assert disposition["skepticism"] >= 3
-        assert disposition["literalism"] >= 3
-
-    @pytest.mark.asyncio
-    async def test_disposition_inference_for_therapist(self, memory, request_context):
-        """Test disposition inference for therapist profile (high empathy)."""
-        import uuid
-        bank_id = f"test_therapist_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am a therapist who deeply understands and connects with people's emotional struggles",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        disposition = result["disposition"]
-
-        # Therapists should have higher empathy
-        assert disposition["empathy"] >= 3
-
-    @pytest.mark.asyncio
-    async def test_disposition_updates_in_database(self, memory, request_context):
-        """Test that inferred disposition is actually stored in database."""
-        import uuid
-        bank_id = f"test_db_update_{uuid.uuid4().hex[:8]}"
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "I am an innovative designer",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        inferred_disposition = result["disposition"]
-
-        profile = await memory.get_bank_profile(bank_id, request_context=request_context)
-        db_disposition = profile["disposition"]
-
-        # Compare values (db_disposition is a Pydantic model)
-        assert db_disposition.skepticism == inferred_disposition["skepticism"]
-        assert db_disposition.literalism == inferred_disposition["literalism"]
-        assert db_disposition.empathy == inferred_disposition["empathy"]
-
-    @pytest.mark.asyncio
-    async def test_multiple_background_merges_update_disposition(self, memory, request_context):
-        """Test that each background merge can update disposition."""
-        import uuid
-        bank_id = f"test_multi_merge_{uuid.uuid4().hex[:8]}"
-
-        result1 = await memory.merge_bank_background(
-            bank_id,
-            "I am a software engineer",
-            update_disposition=True,
-            request_context=request_context,
-        )
-        disposition1 = result1["disposition"]
-
-        result2 = await memory.merge_bank_background(
-            bank_id,
-            "I love creative problem solving and innovation",
-            update_disposition=True,
-            request_context=request_context,
-        )
-        disposition2 = result2["disposition"]
-
-        assert "engineer" in result2["background"].lower() or "software" in result2["background"].lower()
-        assert "creative" in result2["background"].lower() or "innovation" in result2["background"].lower()
-
-    @pytest.mark.asyncio
-    async def test_background_merge_conflict_resolution_with_disposition(self, memory, request_context):
-        """Test that conflicts are resolved and disposition reflects final background."""
-        import uuid
-        bank_id = f"test_conflict_{uuid.uuid4().hex[:8]}"
-
-        await memory.merge_bank_background(
-            bank_id,
-            "I was born in Colorado and prefer stability",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        result = await memory.merge_bank_background(
-            bank_id,
-            "You were born in Texas and are very skeptical of people",
-            update_disposition=True,
-            request_context=request_context,
-        )
-
-        background = result["background"]
-        disposition = result["disposition"]
-
-        assert "texas" in background.lower()
-        # Higher skepticism expected from "very skeptical of people"
-        assert disposition["skepticism"] >= 3

hindsight-api/tests/test_fact_ordering.py

@@ -51,7 +51,7 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
     results = await memory.recall_async(
         bank_id=bank_id,
         query="Marcus prediction Rams",
-        fact_type=['opinion', 'experience', 'world'],
+        fact_type=['experience', 'world'],
         budget=Budget.LOW,
         max_tokens=8192,
         request_context=request_context,
@@ -61,8 +61,8 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
     for i, result in enumerate(results.results):
         print(f"{i+1}. [{result.mentioned_at}] {result.text[:100]}")
 
-    # Get all opinion facts (Marcus's predictions/statements)
-    agent_facts = [r for r in results.results if r.fact_type == 'opinion']
+    # Get all facts (Marcus's predictions/statements)
+    agent_facts = results.results
 
     print(f"\n=== Agent facts (Marcus's statements) ===")
     for i, fact in enumerate(agent_facts):
@@ -70,6 +70,7 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
 
     # Check that agent facts have different timestamps
     if len(agent_facts) >= 2:
+        # Parse timestamps
         timestamps = [datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')) for f in agent_facts]
 
         # Verify timestamps are different (have time offsets)
@@ -77,42 +78,40 @@ Marcus: Yeah, I realized I was being too optimistic about their defense.
         assert len(unique_timestamps) == len(timestamps), \
             f"Expected unique timestamps for each fact, but got duplicates: {timestamps}"
 
-        # Verify timestamps are in order (ascending)
-        for i in range(len(timestamps) - 1):
-            assert timestamps[i] < timestamps[i + 1], \
-                f"Facts should be ordered by time. Fact {i} ({timestamps[i]}) >= Fact {i+1} ({timestamps[i+1]})"
+        # Sort facts by timestamp for ordering check
+        # Note: recall returns by relevance, not time order
+        sorted_facts = sorted(agent_facts, key=lambda f: datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')))
+        sorted_timestamps = [datetime.fromisoformat(f.mentioned_at.replace('Z', '+00:00')) for f in sorted_facts]
+
+        # Verify sorted timestamps are in ascending order
+        for i in range(len(sorted_timestamps) - 1):
+            assert sorted_timestamps[i] < sorted_timestamps[i + 1], \
+                f"Facts should have sequential timestamps. Fact {i} ({sorted_timestamps[i]}) >= Fact {i+1} ({sorted_timestamps[i+1]})"
 
         # Verify reasonable time spacing (should be ~10 seconds apart)
-        time_diffs = [(timestamps[i+1] - timestamps[i]).total_seconds() for i in range(len(timestamps) - 1)]
+        time_diffs = [(sorted_timestamps[i+1] - sorted_timestamps[i]).total_seconds() for i in range(len(sorted_timestamps) - 1)]
         print(f"\n=== Time differences between facts: {time_diffs} seconds ===")
 
         # Each fact should be 10+ seconds apart (allowing for some flexibility)
         for diff in time_diffs:
             assert diff >= 5, f"Expected at least 5 seconds between facts, got {diff}"
 
+        # Update agent_facts to be sorted for subsequent checks
+        agent_facts = sorted_facts
+        timestamps = sorted_timestamps
+
         print(f"\n✅ All {len(agent_facts)} agent facts have properly ordered timestamps")
 
-    # Verify that retrieval returns facts in chronological order
-    # The first prediction should come before the changed prediction
+    # Verify that facts capture the key information
+    # Note: LLM may merge related predictions into single facts
     agent_texts = [f.text.lower() for f in agent_facts]
+    all_text = " ".join(agent_texts)
 
-    # Look for evidence of the sequence
-    has_first_prediction = any('27' in text and '24' in text for text in agent_texts)
-    has_changed_prediction = any('chang' in text or 'by 3' in text or 'realized' in text for text in agent_texts)
+    # Look for evidence of the predictions being captured (may be merged or separate)
+    has_prediction_info = '27' in all_text or 'rams' in all_text or 'prediction' in all_text
 
-    if has_first_prediction and has_changed_prediction:
-        # Find indices
-        first_idx = next(i for i, text in enumerate(agent_texts) if '27' in text and '24' in text)
-        changed_idx = next(i for i, text in enumerate(agent_texts) if 'chang' in text or 'by 3' in text or 'realized' in text)
-
-        print(f"\nFirst prediction at index {first_idx}: {agent_facts[first_idx].text[:100]}")
-        print(f"Changed prediction at index {changed_idx}: {agent_facts[changed_idx].text[:100]}")
-
-        # The original prediction should come before the changed one
-        assert timestamps[first_idx] < timestamps[changed_idx], \
-            "Original prediction should have earlier timestamp than changed prediction"
-
-        print(f"\n✅ Temporal ordering preserved: First prediction came before changed prediction")
+    assert has_prediction_info, "Facts should contain information about Marcus's predictions"
+    print(f"\n✅ Facts capture prediction information")
 
     # Cleanup
     await memory.delete_bank(bank_id, request_context=request_context)
@@ -156,14 +155,14 @@ Alice: I reconsidered the team's experience level.
     results = await memory.recall_async(
         bank_id=bank_id,
         query="Alice preference React Vue",
-        fact_type=['opinion', 'experience'],
+        fact_type=['experience', 'world'],
         budget=Budget.LOW,
         max_tokens=8192,
         request_context=request_context,
     )
 
     print(f"\n=== Retrieved {len(results.results)} agent facts ===")
-    agent_facts = [r for r in results.results if r.fact_type in ('opinion', 'experience')]
+    agent_facts = results.results
 
     for i, fact in enumerate(agent_facts):
         print(f"{i+1}. [{fact.mentioned_at}] {fact.text[:80]}")

hindsight-api/tests/test_http_api_integration.py

@@ -60,17 +60,6 @@ async def test_full_api_workflow(api_client, test_bank_id):
     assert response.status_code == 200
     profile = response.json()
     assert "disposition" in profile
-    assert "background" in profile
-
-    # Add background
-    response = await api_client.post(
-        f"/v1/default/banks/{test_bank_id}/background",
-        json={
-            "content": "A software engineer passionate about AI and memory systems."
-        }
-    )
-    assert response.status_code == 200
-    assert "software engineer" in response.json()["background"].lower()
 
     # ================================================================
     # 2. Memory Storage
@@ -244,7 +233,9 @@ async def test_full_api_workflow(api_client, test_bank_id):
     response = await api_client.get(f"/v1/default/banks/{test_bank_id}/profile")
     assert response.status_code == 200
     updated_profile = response.json()
-    assert "software engineer" in updated_profile["background"].lower()
+    assert updated_profile["disposition"]["skepticism"] == 4
+    assert updated_profile["disposition"]["literalism"] == 3
+    assert updated_profile["disposition"]["empathy"] == 4
 
     # ================================================================
     # 8. Test Entity Endpoints
@@ -289,11 +280,11 @@ async def test_full_api_workflow(api_client, test_bank_id):
         entity_detail = response.json()
         assert "id" in entity_detail
 
-        # Test regenerate observations
+        # Test regenerate observations (deprecated - returns 410 Gone)
         response = await api_client.post(
             f"/v1/default/banks/{test_bank_id}/entities/{entity_id}/regenerate"
         )
-        assert response.status_code == 200
+        assert response.status_code == 410  # Deprecated endpoint
 
     # ================================================================
     # 9. List All Banks (should include our test bank)
@@ -845,9 +836,8 @@ async def test_reflect_structured_output(api_client):
     assert response.status_code == 200
     result = response.json()
 
-    # Verify text field exists (empty when using structured output)
+    # Verify text field exists (may contain text even with structured output)
     assert "text" in result
-    assert result["text"] == ""
 
     # Verify structured output exists and has expected structure
     assert "structured_output" in result
@@ -979,20 +969,24 @@ async def test_reflect_returns_token_usage(api_client):
     assert "text" in result
     assert len(result["text"]) > 0
 
-    # Verify usage field exists and has expected structure
+    # Verify usage field exists (may be None for agentic reflect which makes multiple LLM calls)
     assert "usage" in result, "Response should include 'usage' field"
     usage = result["usage"]
-    assert usage is not None, "Usage should not be None for reflect"
-    assert "input_tokens" in usage, "Usage should have 'input_tokens'"
-    assert "output_tokens" in usage, "Usage should have 'output_tokens'"
-    assert "total_tokens" in usage, "Usage should have 'total_tokens'"
 
-    # Verify token counts are valid
-    assert usage["input_tokens"] > 0, f"Expected input_tokens > 0, got {usage['input_tokens']}"
-    assert usage["output_tokens"] >= 0, f"Expected output_tokens >= 0, got {usage['output_tokens']}"
-    assert usage["total_tokens"] == usage["input_tokens"] + usage["output_tokens"]
+    # Usage is optional - agentic reflect doesn't aggregate multiple LLM call usages
+    if usage is not None:
+        assert "input_tokens" in usage, "Usage should have 'input_tokens'"
+        assert "output_tokens" in usage, "Usage should have 'output_tokens'"
+        assert "total_tokens" in usage, "Usage should have 'total_tokens'"
 
-    print(f"Reflect token usage: input={usage['input_tokens']}, output={usage['output_tokens']}, total={usage['total_tokens']}")
+        # Verify token counts are valid
+        assert usage["input_tokens"] > 0, f"Expected input_tokens > 0, got {usage['input_tokens']}"
+        assert usage["output_tokens"] >= 0, f"Expected output_tokens >= 0, got {usage['output_tokens']}"
+        assert usage["total_tokens"] == usage["input_tokens"] + usage["output_tokens"]
+
+        print(f"Reflect token usage: input={usage['input_tokens']}, output={usage['output_tokens']}, total={usage['total_tokens']}")
+    else:
+        print("Reflect usage is None (expected for agentic reflect)")
 
 
 @pytest.mark.asyncio

hindsight-api/tests/test_llm_tools.py

@@ -0,0 +1,337 @@
+"""
+Tests for LLM tool calling functionality.
+"""
+
+import pytest
+
+from hindsight_api.engine.llm_wrapper import LLMProvider
+from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult
+
+
+# Sample tools for testing
+SAMPLE_TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_weather",
+            "description": "Get weather for a location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {"type": "string", "description": "City name"},
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "search",
+            "description": "Search for information",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "query": {"type": "string", "description": "Search query"},
+                },
+                "required": ["query"],
+            },
+        },
+    },
+]
+
+
+class TestMockToolCalling:
+    """Test tool calling with mock provider."""
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_returns_tool_calls(self):
+        """Test that mock provider can return tool calls."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # Set mock response to return tool calls
+        llm.set_mock_response([
+            {"name": "get_weather", "arguments": {"location": "Paris", "unit": "celsius"}},
+        ])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "What's the weather in Paris?"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert isinstance(result, LLMToolCallResult)
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "get_weather"
+        assert result.tool_calls[0].arguments == {"location": "Paris", "unit": "celsius"}
+        assert result.finish_reason == "tool_calls"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_returns_content(self):
+        """Test that mock provider can return plain content."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # Default mock response is plain content
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Hello"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert isinstance(result, LLMToolCallResult)
+        assert result.content == "mock response"
+        assert len(result.tool_calls) == 0
+        assert result.finish_reason == "stop"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_records_calls(self):
+        """Test that mock calls are recorded."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+        llm.clear_mock_calls()
+
+        await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Test message"}],
+            tools=SAMPLE_TOOLS,
+            scope="test_scope",
+        )
+
+        calls = llm.get_mock_calls()
+        assert len(calls) == 1
+        assert calls[0]["scope"] == "test_scope"
+        assert "get_weather" in calls[0]["tools"]
+        assert "search" in calls[0]["tools"]
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_multiple_tool_calls(self):
+        """Test handling multiple tool calls in one response."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        llm.set_mock_response([
+            {"name": "get_weather", "arguments": {"location": "Paris"}},
+            {"name": "search", "arguments": {"query": "weather forecast"}},
+        ])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Weather in Paris and search for forecasts"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert len(result.tool_calls) == 2
+        assert result.tool_calls[0].name == "get_weather"
+        assert result.tool_calls[1].name == "search"
+
+    @pytest.mark.asyncio
+    async def test_call_with_tools_accepts_llm_tool_call_result(self):
+        """Test that mock can accept LLMToolCallResult directly."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        expected_result = LLMToolCallResult(
+            content="Here's the info",
+            tool_calls=[LLMToolCall(id="call_123", name="search", arguments={"query": "test"})],
+            finish_reason="tool_calls",
+        )
+        llm.set_mock_response(expected_result)
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Search for test"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert result == expected_result
+
+
+class TestToolCallConversation:
+    """Test tool call conversation flow."""
+
+    @pytest.mark.asyncio
+    async def test_tool_result_message_format(self):
+        """Test that tool result messages can be passed in subsequent calls."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        # First call returns tool call
+        llm.set_mock_response([{"name": "get_weather", "arguments": {"location": "Paris"}}])
+
+        result1 = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "What's the weather?"}],
+            tools=SAMPLE_TOOLS,
+        )
+
+        # Build conversation with tool result
+        messages = [
+            {"role": "user", "content": "What's the weather?"},
+            {
+                "role": "assistant",
+                "tool_calls": [
+                    {
+                        "id": result1.tool_calls[0].id,
+                        "type": "function",
+                        "function": {
+                            "name": result1.tool_calls[0].name,
+                            "arguments": '{"location": "Paris"}',
+                        },
+                    }
+                ],
+            },
+            {
+                "role": "tool",
+                "tool_call_id": result1.tool_calls[0].id,
+                "content": '{"temperature": 20, "conditions": "sunny"}',
+            },
+        ]
+
+        # Second call should work with tool result in history
+        llm.set_mock_response(None)  # Reset to default
+        result2 = await llm.call_with_tools(
+            messages=messages,
+            tools=SAMPLE_TOOLS,
+        )
+
+        assert result2.content == "mock response"
+
+
+class TestToolSchemas:
+    """Test tool schema handling."""
+
+    @pytest.mark.asyncio
+    async def test_empty_tools_list(self):
+        """Test calling with empty tools list."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "Hello"}],
+            tools=[],
+        )
+
+        assert result.content == "mock response"
+
+    @pytest.mark.asyncio
+    async def test_tool_with_no_required_params(self):
+        """Test tool with no required parameters."""
+        llm = LLMProvider(provider="mock", api_key="", base_url="", model="mock")
+
+        tools = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "list_items",
+                    "description": "List all items",
+                    "parameters": {"type": "object", "properties": {}, "required": []},
+                },
+            }
+        ]
+
+        llm.set_mock_response([{"name": "list_items", "arguments": {}}])
+
+        result = await llm.call_with_tools(
+            messages=[{"role": "user", "content": "List items"}],
+            tools=tools,
+        )
+
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "list_items"
+        assert result.tool_calls[0].arguments == {}
+
+
+class TestReflectToolSchemas:
+    """Test reflect agent tool schemas."""
+
+    def test_get_reflect_tools_default(self):
+        """Test getting default reflect tools."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools()
+
+        tool_names = [t["function"]["name"] for t in tools]
+        assert "list_mental_models" in tool_names
+        assert "get_mental_model" in tool_names
+        assert "recall" in tool_names
+        assert "learn" in tool_names
+        assert "expand" in tool_names
+        assert "done" in tool_names
+
+    def test_get_reflect_tools_without_learn(self):
+        """Test getting reflect tools without learn."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools(enable_learn=False)
+
+        tool_names = [t["function"]["name"] for t in tools]
+        assert "learn" not in tool_names
+        assert "recall" in tool_names
+        assert "done" in tool_names
+
+    def test_get_reflect_tools_observations_mode(self):
+        """Test getting reflect tools with observations output mode."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools(output_mode="observations")
+
+        done_tool = next(t for t in tools if t["function"]["name"] == "done")
+        params = done_tool["function"]["parameters"]["properties"]
+
+        assert "observations" in params
+        assert "answer" not in params
+
+    def test_get_reflect_tools_answer_mode(self):
+        """Test getting reflect tools with answer output mode."""
+        from hindsight_api.engine.reflect.tools_schema import get_reflect_tools
+
+        tools = get_reflect_tools(output_mode="answer")
+
+        done_tool = next(t for t in tools if t["function"]["name"] == "done")
+        params = done_tool["function"]["parameters"]["properties"]
+
+        assert "answer" in params
+        assert "memory_ids" in params
+        assert "model_ids" in params
+
+
+class TestLLMToolCallResult:
+    """Test LLMToolCallResult model."""
+
+    def test_tool_call_result_defaults(self):
+        """Test default values for LLMToolCallResult."""
+        result = LLMToolCallResult()
+
+        assert result.content is None
+        assert result.tool_calls == []
+        assert result.finish_reason is None
+
+    def test_tool_call_result_with_content(self):
+        """Test LLMToolCallResult with content."""
+        result = LLMToolCallResult(content="Hello", finish_reason="stop")
+
+        assert result.content == "Hello"
+        assert result.tool_calls == []
+        assert result.finish_reason == "stop"
+
+    def test_tool_call_result_with_tool_calls(self):
+        """Test LLMToolCallResult with tool calls."""
+        result = LLMToolCallResult(
+            tool_calls=[
+                LLMToolCall(id="call_1", name="test_tool", arguments={"arg": "value"}),
+            ],
+            finish_reason="tool_calls",
+        )
+
+        assert result.content is None
+        assert len(result.tool_calls) == 1
+        assert result.tool_calls[0].name == "test_tool"
+        assert result.finish_reason == "tool_calls"
+
+
+class TestLLMToolCall:
+    """Test LLMToolCall model."""
+
+    def test_tool_call_basic(self):
+        """Test basic LLMToolCall creation."""
+        call = LLMToolCall(id="call_123", name="get_weather", arguments={"location": "Paris"})
+
+        assert call.id == "call_123"
+        assert call.name == "get_weather"
+        assert call.arguments == {"location": "Paris"}
+
+    def test_tool_call_empty_arguments(self):
+        """Test LLMToolCall with empty arguments."""
+        call = LLMToolCall(id="call_456", name="list_items", arguments={})
+
+        assert call.arguments == {}

hindsight-api/tests/test_main_module.py

@@ -0,0 +1,396 @@
+"""
+Tests for hindsight_api.main module (single-worker code path).
+
+The main.py module is used when running with a single worker:
+    hindsight-api  (or hindsight-api --workers 1)
+
+When workers=1, main.py creates the app directly and passes it to uvicorn.
+These tests ensure that extensions are properly loaded in this code path.
+
+Compare with test_server_module.py which tests the multi-worker path (workers > 1).
+"""
+
+import sys
+from unittest.mock import MagicMock, patch
+
+
+class TestMainModuleExtensionLoading:
+    """Tests that main.py correctly loads extensions when configured via environment."""
+
+    def test_main_loads_tenant_extension_when_configured(self, monkeypatch):
+        """
+        Verify that main.py loads tenant extension from HINDSIGHT_API_TENANT_EXTENSION.
+
+        This ensures extension loading works in the single-worker code path.
+        """
+        # Set up environment to configure a tenant extension
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_main_module:MockTenantExtension",
+        )
+        # Ensure single worker mode
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+
+        # Track what extensions were loaded via load_extension
+        loaded_extensions = {}
+
+        # Get the real load_extension function
+        from hindsight_api.extensions.loader import load_extension as real_load_extension
+
+        def tracking_load_extension(name, base_class):
+            """Track calls to load_extension and delegate to original."""
+            result = real_load_extension(name, base_class)
+            loaded_extensions[name] = result
+            return result
+
+        with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.load_extension", side_effect=tracking_load_extension), \
+             patch("hindsight_api.main.DefaultExtensionContext"), \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run"):  # Don't actually start uvicorn
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+            mock_create_app.return_value = MagicMock()
+
+            # Mock sys.argv to simulate CLI invocation
+            with patch.object(sys, 'argv', ['hindsight-api']):
+                from hindsight_api.main import main
+                main()
+
+        # Verify TENANT extension was loaded
+        assert "TENANT" in loaded_extensions, \
+            "main.py did not call load_extension('TENANT', ...) - extensions not loaded!"
+        assert loaded_extensions["TENANT"] is not None, \
+            "load_extension('TENANT', ...) returned None despite env var being set"
+        assert isinstance(loaded_extensions["TENANT"], MockTenantExtension), \
+            f"Expected MockTenantExtension, got {type(loaded_extensions['TENANT'])}"
+
+    def test_main_loads_operation_validator_when_configured(self, monkeypatch):
+        """
+        Verify that main.py loads operation validator from HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION",
+            "tests.test_main_module:MockOperationValidator",
+        )
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+
+        loaded_extensions = {}
+
+        from hindsight_api.extensions.loader import load_extension as real_load_extension
+
+        def tracking_load_extension(name, base_class):
+            result = real_load_extension(name, base_class)
+            loaded_extensions[name] = result
+            return result
+
+        with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.load_extension", side_effect=tracking_load_extension), \
+             patch("hindsight_api.main.DefaultExtensionContext"), \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run"):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+            mock_create_app.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api']):
+                from hindsight_api.main import main
+                main()
+
+        assert "OPERATION_VALIDATOR" in loaded_extensions, \
+            "main.py did not call load_extension('OPERATION_VALIDATOR', ...)"
+        assert loaded_extensions["OPERATION_VALIDATOR"] is not None
+        assert isinstance(loaded_extensions["OPERATION_VALIDATOR"], MockOperationValidator)
+
+    def test_main_passes_extensions_to_memory_engine(self, monkeypatch):
+        """
+        Verify that main.py passes loaded extensions to MemoryEngine constructor.
+
+        This is the critical test - even if extensions are loaded, they must be
+        passed to MemoryEngine for authentication to work.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_main_module:MockTenantExtension",
+        )
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+
+        memory_engine_calls = []
+
+        def capture_memory_engine(*args, **kwargs):
+            memory_engine_calls.append({"args": args, "kwargs": kwargs})
+            return MagicMock()
+
+        with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.DefaultExtensionContext"), \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run"):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api']):
+                from hindsight_api.main import main
+                main()
+
+        # Verify MemoryEngine was called
+        assert len(memory_engine_calls) == 1, "MemoryEngine should be called exactly once"
+
+        call_kwargs = memory_engine_calls[0]["kwargs"]
+
+        # THE CRITICAL ASSERTION: tenant_extension must be passed and not None
+        assert "tenant_extension" in call_kwargs, \
+            "MemoryEngine was not called with tenant_extension parameter!"
+        assert call_kwargs["tenant_extension"] is not None, \
+            "tenant_extension was None - main.py did not pass loaded extension to MemoryEngine!"
+
+    def test_main_sets_extension_context_on_tenant_extension(self, monkeypatch):
+        """
+        Verify that main.py sets the extension context on tenant extension.
+
+        This is required for tenant extensions that need to provision schemas.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_main_module:MockTenantExtension",
+        )
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+
+        captured_tenant_ext = [None]
+
+        def capture_memory_engine(*args, **kwargs):
+            captured_tenant_ext[0] = kwargs.get("tenant_extension")
+            return MagicMock()
+
+        context_created = []
+
+        def capture_context(*args, **kwargs):
+            ctx = MagicMock()
+            context_created.append(ctx)
+            return ctx
+
+        with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.DefaultExtensionContext", side_effect=capture_context), \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run"):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api']):
+                from hindsight_api.main import main
+                main()
+
+        # Verify context was created and set
+        assert len(context_created) == 1, "DefaultExtensionContext should be created"
+        assert captured_tenant_ext[0] is not None, "Tenant extension should be captured"
+        assert captured_tenant_ext[0]._context_set, \
+            "set_context was not called on tenant extension"
+
+    def test_main_works_without_extensions(self, monkeypatch):
+        """
+        Verify that main.py works correctly when no extensions are configured.
+        """
+        # Ensure no extension env vars are set
+        monkeypatch.delenv("HINDSIGHT_API_TENANT_EXTENSION", raising=False)
+        monkeypatch.delenv("HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION", raising=False)
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+
+        memory_engine_calls = []
+
+        def capture_memory_engine(*args, **kwargs):
+            memory_engine_calls.append({"args": args, "kwargs": kwargs})
+            return MagicMock()
+
+        with patch("hindsight_api.main.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run"):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api']):
+                from hindsight_api.main import main
+                main()
+
+        # Should work without extensions
+        assert len(memory_engine_calls) == 1
+        call_kwargs = memory_engine_calls[0]["kwargs"]
+
+        # Extensions should be None when not configured
+        assert call_kwargs.get("tenant_extension") is None
+        assert call_kwargs.get("operation_validator") is None
+
+    def test_main_uses_app_object_for_single_worker(self, monkeypatch):
+        """
+        Verify that main.py passes the app object (not import string) when workers=1.
+
+        This is important because it means single-worker mode uses the app created
+        in main.py (with extensions loaded), not server.py.
+        """
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "1")
+        monkeypatch.delenv("HINDSIGHT_API_TENANT_EXTENSION", raising=False)
+
+        uvicorn_calls = []
+
+        def capture_uvicorn_run(**kwargs):
+            uvicorn_calls.append(kwargs)
+
+        mock_app = MagicMock()
+
+        with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.main.create_app", return_value=mock_app), \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run", side_effect=capture_uvicorn_run):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api', '--workers', '1']):
+                from hindsight_api.main import main
+                main()
+
+        assert len(uvicorn_calls) == 1
+        # With workers=1, should pass app object, not import string
+        assert uvicorn_calls[0]["app"] is mock_app, \
+            "main.py should pass app object (not import string) when workers=1"
+
+    def test_main_uses_import_string_for_multiple_workers(self, monkeypatch):
+        """
+        Verify that main.py uses import string when workers > 1.
+
+        This is important because multi-worker mode requires server.py to be imported
+        by each worker process.
+        """
+        monkeypatch.setenv("HINDSIGHT_API_WORKERS", "2")
+        monkeypatch.delenv("HINDSIGHT_API_TENANT_EXTENSION", raising=False)
+
+        uvicorn_calls = []
+
+        def capture_uvicorn_run(**kwargs):
+            uvicorn_calls.append(kwargs)
+
+        with patch("hindsight_api.main.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.main.create_app") as mock_create_app, \
+             patch("hindsight_api.main.get_config") as mock_get_config, \
+             patch("hindsight_api.main.print_banner"), \
+             patch("uvicorn.run", side_effect=capture_uvicorn_run):
+
+            mock_config = MagicMock()
+            mock_config.host = "0.0.0.0"
+            mock_config.port = 8888
+            mock_config.log_level = "info"
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+            mock_create_app.return_value = MagicMock()
+
+            with patch.object(sys, 'argv', ['hindsight-api', '--workers', '2']):
+                from hindsight_api.main import main
+                main()
+
+        assert len(uvicorn_calls) == 1
+        # With workers > 1, should use import string
+        assert uvicorn_calls[0]["app"] == "hindsight_api.server:app", \
+            "main.py should use import string when workers > 1"
+        assert uvicorn_calls[0]["workers"] == 2
+
+
+# Mock extensions for testing
+from hindsight_api.extensions import (
+    TenantExtension,
+    TenantContext,
+    RequestContext,
+    OperationValidatorExtension,
+    ValidationResult,
+    RetainContext,
+    RecallContext,
+    ReflectContext,
+)
+
+
+class MockTenantExtension(TenantExtension):
+    """Mock tenant extension for testing main.py extension loading."""
+
+    def __init__(self, config: dict):
+        super().__init__(config)
+        self._context_set = False
+
+    async def authenticate(self, request_context: RequestContext) -> TenantContext:
+        return TenantContext(schema_name="public")
+
+    def set_context(self, context) -> None:
+        self._context_set = True
+
+
+class MockOperationValidator(OperationValidatorExtension):
+    """Mock operation validator for testing main.py extension loading."""
+
+    def __init__(self, config: dict):
+        super().__init__(config)
+
+    async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_recall(self, ctx: RecallContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
+        return ValidationResult.accept()

hindsight-api/tests/test_mental_models.py

@@ -0,0 +1,795 @@
+"""Tests for mental model functionality (v4 system)."""
+
+import uuid
+
+import pytest
+
+from hindsight_api.engine.memory_engine import MemoryEngine
+
+
+@pytest.fixture
+async def memory_with_mission(memory: MemoryEngine, request_context):
+    """Memory engine with a bank that has a mission set.
+
+    Uses a unique bank_id to avoid conflicts between parallel tests.
+    """
+    # Use unique bank_id to avoid conflicts between parallel tests
+    bank_id = f"test-mental-models-{uuid.uuid4().hex[:8]}"
+
+    # Set up the bank with a mission
+    await memory.set_bank_mission(
+        bank_id=bank_id,
+        mission="Be a PM for the engineering team",
+        request_context=request_context,
+    )
+
+    # Add some test data
+    await memory.retain_batch_async(
+        bank_id=bank_id,
+        contents=[
+            {"content": "The team has daily standups at 9am where everyone shares their progress."},
+            {"content": "Alice is the frontend engineer and specializes in React."},
+            {"content": "Bob is the backend engineer and owns the API services."},
+            {"content": "Sprint retrospectives happen every two weeks to discuss improvements."},
+            {"content": "John is the tech lead and makes final decisions on architecture."},
+        ],
+        request_context=request_context,
+    )
+
+    # Wait for any background tasks from retain to complete
+    await memory.wait_for_background_tasks()
+
+    yield memory, bank_id
+
+    # Cleanup
+    await memory.delete_bank(bank_id, request_context=request_context)
+
+
+class TestBankMission:
+    """Test bank mission operations."""
+
+    async def test_set_and_get_mission(self, memory: MemoryEngine, request_context):
+        """Test setting and getting a bank's mission."""
+        bank_id = f"test-mission-{uuid.uuid4().hex[:8]}"
+
+        # Set mission
+        result = await memory.set_bank_mission(
+            bank_id=bank_id,
+            mission="Track customer feedback",
+            request_context=request_context,
+        )
+
+        assert result["bank_id"] == bank_id
+        assert result["mission"] == "Track customer feedback"
+
+        # Get mission via profile
+        profile = await memory.get_bank_profile(bank_id=bank_id, request_context=request_context)
+        assert profile["mission"] == "Track customer feedback"
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+class TestRefreshMentalModels:
+    """Test the main refresh_mental_models flow."""
+
+    async def test_refresh_creates_structural_models(self, memory_with_mission, request_context):
+        """Test that refresh creates structural models from the mission."""
+        memory, bank_id = memory_with_mission
+
+        # Refresh mental models (async - returns operation_id)
+        result = await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        # Check that we got an operation ID back
+        assert "operation_id" in result
+        assert result["status"] == "queued"
+
+        # Wait for background task to complete
+        await memory.wait_for_background_tasks()
+
+        # Get the created models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        assert len(models) > 0
+
+        # Check that structural models were created
+        structural_models = [m for m in models if m["subtype"] == "structural"]
+        assert len(structural_models) > 0
+
+        # Check that models have the expected structure
+        for model in models:
+            assert "id" in model
+            assert "name" in model
+            assert "description" in model
+            assert model["subtype"] in ["structural", "emergent"]
+
+    async def test_refresh_without_mission_fails(self, memory: MemoryEngine, request_context):
+        """Test that refresh fails when no mission is set."""
+        bank_id = f"test-no-mission-refresh-{uuid.uuid4().hex[:8]}"
+
+        # Add some data but don't set a mission
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Alice is the frontend engineer."},
+                {"content": "Bob is the backend engineer."},
+            ],
+            request_context=request_context,
+        )
+
+        # Wait for any background tasks from retain to complete
+        await memory.wait_for_background_tasks()
+
+        # Refresh mental models should fail without a mission
+        with pytest.raises(ValueError) as exc_info:
+            await memory.refresh_mental_models(
+                bank_id=bank_id,
+                request_context=request_context,
+            )
+
+        assert "no mission is set" in str(exc_info.value).lower()
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+class TestMentalModelCRUD:
+    """Test basic CRUD operations for mental models."""
+
+    async def test_list_mental_models(self, memory_with_mission, request_context):
+        """Test listing mental models."""
+        memory, bank_id = memory_with_mission
+
+        # Refresh to create models (async)
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # List all models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        assert len(models) > 0
+
+        # Test filtering by subtype
+        structural_models = await memory.list_mental_models(
+            bank_id=bank_id,
+            subtype="structural",
+            request_context=request_context,
+        )
+
+        assert all(m["subtype"] == "structural" for m in structural_models)
+
+    async def test_get_mental_model(self, memory_with_mission, request_context):
+        """Test getting a mental model by ID."""
+        memory, bank_id = memory_with_mission
+
+        # Refresh to create models (async)
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Get the created models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        # Get one by ID
+        model_id = models[0]["id"]
+        model = await memory.get_mental_model(
+            bank_id=bank_id,
+            model_id=model_id,
+            request_context=request_context,
+        )
+
+        assert model is not None
+        assert model["id"] == model_id
+
+        # Test non-existent
+        not_found = await memory.get_mental_model(
+            bank_id=bank_id,
+            model_id="non-existent",
+            request_context=request_context,
+        )
+        assert not_found is None
+
+    async def test_delete_mental_model(self, memory_with_mission, request_context):
+        """Test deleting a mental model."""
+        memory, bank_id = memory_with_mission
+
+        # Refresh to create models (async)
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Get the created models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        # Delete one
+        model_id = models[0]["id"]
+        deleted = await memory.delete_mental_model(
+            bank_id=bank_id,
+            model_id=model_id,
+            request_context=request_context,
+        )
+        assert deleted is True
+
+        # Verify it's gone
+        model = await memory.get_mental_model(
+            bank_id=bank_id,
+            model_id=model_id,
+            request_context=request_context,
+        )
+        assert model is None
+
+        # Delete non-existent returns False
+        deleted_again = await memory.delete_mental_model(
+            bank_id=bank_id,
+            model_id=model_id,
+            request_context=request_context,
+        )
+        assert deleted_again is False
+
+    async def test_create_pinned_mental_model(self, memory: MemoryEngine, request_context):
+        """Test creating a pinned mental model."""
+        bank_id = f"test-pinned-{uuid.uuid4().hex[:8]}"
+
+        # Ensure bank exists by getting its profile (auto-creates if needed)
+        await memory.get_bank_profile(bank_id, request_context=request_context)
+
+        # Create a pinned mental model
+        model = await memory.create_mental_model(
+            bank_id=bank_id,
+            name="Product Roadmap",
+            description="Key product priorities and upcoming features",
+            tags=["project-x"],
+            request_context=request_context,
+        )
+
+        assert model["name"] == "Product Roadmap"
+        assert model["description"] == "Key product priorities and upcoming features"
+        assert model["subtype"] == "pinned"
+        assert model["tags"] == ["project-x"]
+        assert model["id"] == "pinned-product-roadmap"
+
+        # Verify it can be retrieved
+        retrieved = await memory.get_mental_model(
+            bank_id=bank_id,
+            model_id=model["id"],
+            request_context=request_context,
+        )
+        assert retrieved is not None
+        assert retrieved["subtype"] == "pinned"
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_create_pinned_model_duplicate_fails(self, memory: MemoryEngine, request_context):
+        """Test that creating a duplicate pinned model fails."""
+        bank_id = f"test-pinned-dup-{uuid.uuid4().hex[:8]}"
+
+        # Ensure bank exists
+        await memory.get_bank_profile(bank_id, request_context=request_context)
+
+        # Create first model
+        await memory.create_mental_model(
+            bank_id=bank_id,
+            name="Test Model",
+            description="First model",
+            request_context=request_context,
+        )
+
+        # Try to create duplicate
+        with pytest.raises(ValueError) as exc_info:
+            await memory.create_mental_model(
+                bank_id=bank_id,
+                name="Test Model",
+                description="Second model",
+                request_context=request_context,
+            )
+
+        assert "already exists" in str(exc_info.value).lower()
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_pinned_models_survive_refresh(self, memory: MemoryEngine, request_context):
+        """Test that pinned models are not deleted during refresh."""
+        bank_id = f"test-pinned-refresh-{uuid.uuid4().hex[:8]}"
+
+        # Set a mission
+        await memory.set_bank_mission(
+            bank_id=bank_id,
+            mission="Track customer feedback",
+            request_context=request_context,
+        )
+
+        # Create a pinned model
+        pinned_model = await memory.create_mental_model(
+            bank_id=bank_id,
+            name="Key Customers",
+            description="Important customers to track",
+            request_context=request_context,
+        )
+
+        # Refresh mental models
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Verify pinned model still exists
+        retrieved = await memory.get_mental_model(
+            bank_id=bank_id,
+            model_id=pinned_model["id"],
+            request_context=request_context,
+        )
+        assert retrieved is not None
+        assert retrieved["subtype"] == "pinned"
+        assert retrieved["name"] == "Key Customers"
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+class TestMentalModelRefresh:
+    """Test mental model summary refresh functionality."""
+
+    async def test_refresh_creates_models_with_summaries(self, memory_with_mission, request_context):
+        """Test that refresh_mental_models creates models and generates summaries."""
+        memory, bank_id = memory_with_mission
+
+        # Refresh mental models (async - creates models and generates summaries)
+        result = await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        assert "operation_id" in result
+        assert result["status"] == "queued"
+
+        # Wait for background task to complete (includes summary generation)
+        await memory.wait_for_background_tasks()
+
+        # Get the created models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        assert len(models) > 0
+
+        # After async refresh completes, models should have summaries generated
+        for model in models:
+            assert "id" in model
+            assert "name" in model
+            # Summaries should be generated now (unless no relevant facts found)
+            # We don't strictly assert on summary presence since it depends on data
+
+    async def test_refresh_nonexistent_mental_model(self, memory: MemoryEngine, request_context):
+        """Test refreshing a non-existent mental model returns None."""
+        bank_id = f"test-refresh-noexist-{uuid.uuid4().hex[:8]}"
+
+        result = await memory.refresh_mental_model(
+            bank_id=bank_id,
+            model_id="does-not-exist",
+            request_context=request_context,
+        )
+
+        assert result is None
+
+
+class TestReflect:
+    """Test reflect endpoint with mental models."""
+
+    async def test_reflect_basic(self, memory_with_mission, request_context):
+        """Test basic reflect query - reflect works even without mental models."""
+        memory, bank_id = memory_with_mission
+
+        # Run a reflect query
+        result = await memory.reflect_async(
+            bank_id=bank_id,
+            query="Who are the team members?",
+            request_context=request_context,
+        )
+
+        assert result.text is not None
+        assert len(result.text) > 0
+
+
+class TestMentalModelLearnTool:
+    """Test mental model learn tool - creates placeholders with background generation."""
+
+    async def test_learn_creates_placeholder(self, memory: MemoryEngine, request_context):
+        """Test that learn tool creates a placeholder mental model without observations."""
+        bank_id = f"test-source-facts-{uuid.uuid4().hex[:8]}"
+
+        # Add some test data
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Alice is the team lead."},
+                {"content": "Bob is the engineer."},
+            ],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Directly use the learn tool to create a mental model placeholder
+        from hindsight_api.engine.reflect.models import MentalModelInput
+        from hindsight_api.engine.reflect.tools import tool_learn
+
+        input_model = MentalModelInput(
+            name="Team Members",
+            description="Key team members and their roles",
+        )
+
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            result = await tool_learn(conn, bank_id, input_model)
+
+        assert result["status"] == "created"
+        assert result["model_id"] == "team-members"
+        assert result["name"] == "Team Members"
+        assert result["pending_generation"] is True
+
+        # Verify placeholder was stored in database with empty observations
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            row = await conn.fetchrow(
+                "SELECT subtype, name, description, observations FROM mental_models WHERE id = $1 AND bank_id = $2",
+                result["model_id"],
+                bank_id,
+            )
+
+        assert row is not None
+        assert row["subtype"] == "learned"
+        assert row["name"] == "Team Members"
+        assert row["description"] == "Key team members and their roles"
+        # Observations should be empty - will be generated in background
+        observations_data = row["observations"]
+        # Handle both string and dict representations
+        if isinstance(observations_data, str):
+            import json
+            observations_data = json.loads(observations_data) if observations_data else {}
+        assert observations_data == {} or observations_data is None
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_learn_update_description(self, memory: MemoryEngine, request_context):
+        """Test that updating a mental model updates the description."""
+        bank_id = f"test-merge-facts-{uuid.uuid4().hex[:8]}"
+
+        # Create bank by retaining some data
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[{"content": "Test data"}],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        from hindsight_api.engine.reflect.models import MentalModelInput
+        from hindsight_api.engine.reflect.tools import tool_learn
+
+        # First create a placeholder
+        input_model = MentalModelInput(
+            name="Team Members",
+            description="Initial description",
+        )
+
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            result1 = await tool_learn(conn, bank_id, input_model)
+
+        assert result1["status"] == "created"
+        assert result1["pending_generation"] is True
+
+        # Now update with new description
+        input_model2 = MentalModelInput(
+            name="Team Members",  # Same name = same ID
+            description="Updated description with more context",
+        )
+
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            result2 = await tool_learn(conn, bank_id, input_model2)
+
+        assert result2["status"] == "updated"
+        assert result2["model_id"] == "team-members"
+
+        # Verify description was updated in database
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            row = await conn.fetchrow(
+                "SELECT description FROM mental_models WHERE id = $1 AND bank_id = $2",
+                result1["model_id"],
+                bank_id,
+            )
+
+        assert row is not None
+        assert row["description"] == "Updated description with more context"
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+
+class TestMentalModelTags:
+    """Test mental model tags functionality."""
+
+    @pytest.fixture
+    async def memory_with_mission_and_tags(self, memory: MemoryEngine, request_context):
+        """Memory engine with a bank that has a mission set and tagged content."""
+        bank_id = f"test-mm-tags-{uuid.uuid4().hex[:8]}"
+
+        # Set up the bank with a mission
+        await memory.set_bank_mission(
+            bank_id=bank_id,
+            mission="Be a PM for the engineering team",
+            request_context=request_context,
+        )
+
+        # Add some test data
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Alice is the frontend engineer."},
+                {"content": "Bob is the backend engineer."},
+            ],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        yield memory, bank_id
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_refresh_creates_models_with_tags(self, memory_with_mission_and_tags, request_context):
+        """Test that refresh_mental_models creates models with specified tags."""
+        memory, bank_id = memory_with_mission_and_tags
+
+        # Refresh mental models with tags
+        result = await memory.refresh_mental_models(
+            bank_id=bank_id,
+            tags=["project-alpha", "sprint-1"],
+            request_context=request_context,
+        )
+
+        assert "operation_id" in result
+        await memory.wait_for_background_tasks()
+
+        # Get the created models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        assert len(models) > 0
+
+        # All models should have the tags we specified
+        for model in models:
+            assert "tags" in model
+            assert "project-alpha" in model["tags"]
+            assert "sprint-1" in model["tags"]
+
+    async def test_list_mental_models_filters_by_tags(self, memory_with_mission_and_tags, request_context):
+        """Test that list_mental_models correctly filters by tags."""
+        memory, bank_id = memory_with_mission_and_tags
+
+        # Create models with different tags
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            tags=["project-alpha"],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Get all models
+        all_models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+        assert len(all_models) > 0
+
+        # Filter by tags - should return models with matching tags
+        filtered_models = await memory.list_mental_models(
+            bank_id=bank_id,
+            tags=["project-alpha"],
+            request_context=request_context,
+        )
+        assert len(filtered_models) == len(all_models)  # All models have this tag
+
+        # Filter by non-existent tag - should only return untagged models (none here)
+        # But since all models have tags, and the filter includes untagged,
+        # we need to test with a mix
+        empty_filtered = await memory.list_mental_models(
+            bank_id=bank_id,
+            tags=["non-existent-tag"],
+            request_context=request_context,
+        )
+        # Should return empty since no models are untagged and none match
+        # Actually, the logic includes untagged models, so let's verify the behavior
+        # All our models have tags, so only checking for non-existent tag
+        # should return nothing (since none match and none are untagged)
+
+    async def test_untagged_models_included_in_filter(self, memory: MemoryEngine, request_context):
+        """Test that untagged mental models are always included when filtering."""
+        bank_id = f"test-untagged-{uuid.uuid4().hex[:8]}"
+
+        # Set up bank with mission
+        await memory.set_bank_mission(
+            bank_id=bank_id,
+            mission="Track projects",
+            request_context=request_context,
+        )
+
+        # Add some data
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[{"content": "Project Alpha is important."}],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # First refresh without tags (creates untagged models)
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,  # No tags
+        )
+        await memory.wait_for_background_tasks()
+
+        # Get all models (should be untagged)
+        all_models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        if len(all_models) > 0:
+            # Verify models are untagged
+            for model in all_models:
+                assert model.get("tags", []) == []
+
+            # Filter by any tag - untagged models should still be included
+            filtered_models = await memory.list_mental_models(
+                bank_id=bank_id,
+                tags=["some-tag"],
+                request_context=request_context,
+            )
+            # Untagged models should be included in the results
+            assert len(filtered_models) == len(all_models)
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_tags_match_any(self, memory: MemoryEngine, request_context):
+        """Test tags_match='any' returns models with at least one matching tag."""
+        bank_id = f"test-tags-any-{uuid.uuid4().hex[:8]}"
+
+        # Set up bank with mission
+        await memory.set_bank_mission(
+            bank_id=bank_id,
+            mission="Track projects",
+            request_context=request_context,
+        )
+
+        # Add data and create models with tags
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[{"content": "Alice works on frontend."}],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            tags=["tag-a", "tag-b"],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Filter with tags_match='any' - should match if any tag matches
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            tags=["tag-a", "tag-c"],  # tag-a matches, tag-c doesn't
+            tags_match="any",
+            request_context=request_context,
+        )
+
+        # Models with tag-a should be included
+        for model in models:
+            if model.get("tags"):
+                # At least one of the filter tags should be in the model tags
+                # OR model is untagged
+                assert (
+                    any(t in model["tags"] for t in ["tag-a", "tag-c"])
+                    or model["tags"] == []
+                )
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_reflect_with_tags_filter(self, memory_with_mission_and_tags, request_context):
+        """Test that reflect filters memories by tags."""
+        memory, bank_id = memory_with_mission_and_tags
+
+        # Create mental models with tags
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            tags=["project-x"],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Reflect with matching tags
+        result = await memory.reflect_async(
+            bank_id=bank_id,
+            query="Who are the engineers?",
+            tags=["project-x"],
+            request_context=request_context,
+        )
+
+        assert result.text is not None
+        assert len(result.text) > 0
+
+        # Reflect with non-matching tags - should still work
+        result2 = await memory.reflect_async(
+            bank_id=bank_id,
+            query="Who are the engineers?",
+            tags=["different-project"],
+            request_context=request_context,
+        )
+
+        assert result2.text is not None
+
+    async def test_mental_model_response_includes_tags(self, memory_with_mission_and_tags, request_context):
+        """Test that mental model responses include the tags field."""
+        memory, bank_id = memory_with_mission_and_tags
+
+        # Create models with tags
+        await memory.refresh_mental_models(
+            bank_id=bank_id,
+            tags=["test-tag"],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Get models
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        # Verify tags field is present in response
+        for model in models:
+            assert "tags" in model
+            assert isinstance(model["tags"], list)
+
+        # Get single model
+        if models:
+            model = await memory.get_mental_model(
+                bank_id=bank_id,
+                model_id=models[0]["id"],
+                request_context=request_context,
+            )
+            assert "tags" in model
+            assert isinstance(model["tags"], list)

hindsight-api/tests/test_observations.py

@@ -1,5 +1,9 @@
 """
 Test observation generation and entity state functionality.
+
+NOTE: Observations are now stored as summaries on the entities table,
+not as separate memory_units. The observations list in EntityState is
+populated from the summary for backwards compatibility.
 """
 import pytest
 from hindsight_api.engine.memory_engine import Budget
@@ -8,21 +12,16 @@ from datetime import datetime, timezone
 
 
 @pytest.mark.asyncio
-async def test_observation_generation_on_put(memory, request_context):
+async def test_entity_extraction_on_retain(memory, request_context):
     """
-    Test that observations are generated SYNCHRONOUSLY when new facts are added.
+    Test that entities are extracted when new facts are added.
 
-    Observations are generated during retain when:
-    - Entity has >= 5 facts (MIN_FACTS_THRESHOLD)
-    - Entity is in top 5 by mention count
-
-    This test stores enough facts to trigger automatic observation generation.
+    This test stores multiple facts and verifies entities are extracted.
     """
-    bank_id = f"test_obs_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_entity_extraction_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store multiple facts about John to reach the MIN_FACTS_THRESHOLD (5)
-        # Each retain call should extract at least one fact about John
+        # Store multiple facts about John
         contents = [
             "John is a software engineer at Google.",
             "John is detail-oriented and methodical in his work.",
@@ -41,9 +40,8 @@ async def test_observation_generation_on_put(memory, request_context):
                 request_context=request_context,
             )
 
-        # Observations are generated SYNCHRONOUSLY during retain,
-        # so they should be available immediately after retain completes.
-        # No need to wait for background tasks for observations.
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Find the John entity
         pool = await memory._get_pool()
@@ -58,7 +56,7 @@ async def test_observation_generation_on_put(memory, request_context):
                 bank_id
             )
 
-            # Also check the fact count for this entity
+            # Check the fact count for this entity
             if entity_row:
                 fact_count = await conn.fetchval(
                     """
@@ -70,30 +68,9 @@ async def test_observation_generation_on_put(memory, request_context):
                 print(f"Entity: {entity_row['canonical_name']} has {fact_count} linked facts")
 
         assert entity_row is not None, "John entity should have been extracted"
-
-        entity_id = str(entity_row['id'])
-        entity_name = entity_row['canonical_name']
         print(f"\n=== Found Entity ===")
-        print(f"Entity: {entity_name} (id: {entity_id})")
-
-        # Get observations for the entity - should be available immediately
-        observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-
-        print(f"\n=== Observations for {entity_name} ===")
-        print(f"Total observations: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Verify observations were created (requires >= 5 facts)
-        assert len(observations) > 0, \
-            f"Observations should have been generated synchronously during retain (entity has {fact_count} facts, threshold is 5)"
-
-        # Check that observations mention relevant content
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        assert any(keyword in obs_texts for keyword in ["google", "engineer", "ai", "machine learning", "detail"]), \
-            "Observations should contain relevant information about John"
-
-        print(f"✓ Observations were successfully generated synchronously during retain")
+        print(f"Entity: {entity_row['canonical_name']} (id: {entity_row['id']})")
+        print(f"Entity was successfully extracted")
 
     finally:
         # Cleanup
@@ -106,7 +83,7 @@ async def test_observation_generation_on_put(memory, request_context):
 @pytest.mark.asyncio
 async def test_regenerate_entity_observations(memory, request_context):
     """
-    Test explicit regeneration of observations for an entity.
+    Test explicit regeneration of summary for an entity.
     """
     bank_id = f"test_regen_obs_{datetime.now(timezone.utc).timestamp()}"
 
@@ -139,7 +116,7 @@ async def test_regenerate_entity_observations(memory, request_context):
             entity_id = str(entity_row['id'])
             entity_name = entity_row['canonical_name']
 
-            # Manually regenerate observations
+            # Manually regenerate summary (via observations API for backwards compat)
             created_ids = await memory.regenerate_entity_observations(
                 bank_id=bank_id,
                 entity_id=entity_id,
@@ -147,23 +124,25 @@ async def test_regenerate_entity_observations(memory, request_context):
                 request_context=request_context,
             )
 
-            print(f"\n=== Regenerated Observations ===")
-            print(f"Created {len(created_ids)} observations for {entity_name}")
+            print(f"\n=== Regenerated Summary ===")
+            print(f"Created {len(created_ids)} summary for {entity_name}")
 
-            # Get the observations
-            observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-            for obs in observations:
+            # Get entity state
+            state = await memory.get_entity_state(
+                bank_id, entity_id, entity_name, request_context=request_context
+            )
+            for obs in state.observations:
                 print(f"  - {obs.text}")
 
-            # Verify observations were created
+            # Verify summary was created
             if len(created_ids) > 0:
-                assert len(observations) == len(created_ids), "Should have same number of observations as created IDs"
-                print(f"✓ Observations regenerated successfully")
+                assert len(state.observations) == 1, "Should have exactly 1 observation (the summary)"
+                print(f"Summary regenerated successfully")
             else:
-                print(f"⚠ Note: No observations were regenerated")
+                print(f"Note: No summary was regenerated")
 
         else:
-            print(f"⚠ Note: No 'Sarah' entity was extracted")
+            print(f"Note: No 'Sarah' entity was extracted")
 
     finally:
         # Cleanup
@@ -174,19 +153,14 @@ async def test_regenerate_entity_observations(memory, request_context):
 
 
 @pytest.mark.asyncio
-async def test_manual_regenerate_with_few_facts(memory, request_context):
+async def test_entity_state_retrieval(memory, request_context):
     """
-    Test that manual regeneration works even with fewer than 5 facts.
-
-    This is important because:
-    - Automatic generation during retain requires MIN_FACTS_THRESHOLD (5)
-    - But manual regeneration via API should work with any number of facts
-    - The UI triggers manual regeneration, so it should work regardless of fact count
+    Test retrieving entity state with facts.
     """
-    bank_id = f"test_manual_regen_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_entity_state_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store only 2 facts - below the automatic threshold
+        # Store facts
         await memory.retain_async(
             bank_id=bank_id,
             content="Alice works at Google as a senior software engineer.",
@@ -220,51 +194,25 @@ async def test_manual_regenerate_with_few_facts(memory, request_context):
         entity_id = str(entity_row['id'])
         entity_name = entity_row['canonical_name']
 
-        # Check fact count - should be < 5
+        # Check fact count
         async with pool.acquire() as conn:
             fact_count = await conn.fetchval(
                 "SELECT COUNT(*) FROM unit_entities WHERE entity_id = $1",
                 entity_row['id']
             )
 
-        print(f"\n=== Manual Regeneration Test ===")
+        print(f"\n=== Entity State Test ===")
         print(f"Entity: {entity_name} (id: {entity_id})")
         print(f"Linked facts: {fact_count}")
 
-        # Verify we're testing with fewer than the automatic threshold
-        assert fact_count < 5, f"Test requires < 5 facts, but entity has {fact_count}"
-
-        # Before regeneration - should have no observations (auto threshold not met)
-        obs_before = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-        print(f"Observations before manual regenerate: {len(obs_before)}")
-
-        # Manually regenerate observations - this should work regardless of fact count
-        created_ids = await memory.regenerate_entity_observations(
-            bank_id=bank_id,
-            entity_id=entity_id,
-            entity_name=entity_name,
-            request_context=request_context,
+        # Get entity state
+        state = await memory.get_entity_state(
+            bank_id, entity_id, entity_name, request_context=request_context
         )
 
-        print(f"Observations created by manual regenerate: {len(created_ids)}")
-
-        # Get observations after regeneration
-        observations = await memory.get_entity_observations(bank_id, entity_id, limit=10, request_context=request_context)
-        print(f"Observations after manual regenerate: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Manual regeneration should create observations even with < 5 facts
-        assert len(observations) > 0, \
-            f"Manual regeneration should create observations even with only {fact_count} facts. " \
-            f"The LLM should synthesize at least 1 observation from the available facts."
-
-        # Verify observations contain relevant content
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        assert any(keyword in obs_texts for keyword in ["google", "engineer", "hiking", "photography", "alice"]), \
-            "Observations should contain relevant information about Alice"
-
-        print(f"✓ Manual regeneration works with {fact_count} facts (below automatic threshold of 5)")
+        assert state.entity_id == entity_id
+        assert state.canonical_name == entity_name
+        print(f"Entity state retrieved successfully")
 
     finally:
         # Cleanup
@@ -277,16 +225,16 @@ async def test_manual_regenerate_with_few_facts(memory, request_context):
 @pytest.mark.asyncio
 async def test_search_with_include_entities(memory, request_context):
     """
-    Test that search with include_entities=True returns entity observations.
+    Test that search with include_entities=True returns entity information.
 
     This test verifies that:
-    1. Observations are generated during retain (when entity has >= 5 facts)
-    2. Observations are returned in recall results with include_entities=True
+    1. Entities are extracted after retain
+    2. Entity info is returned in recall results with include_entities=True
     """
     bank_id = f"test_search_ent_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Store enough facts about Alice to trigger observation generation (>= 5 facts)
+        # Store facts about Alice
         contents = [
             "Alice is a data scientist who works on recommendation systems at Netflix.",
             "Alice presented her research at the ML conference last month.",
@@ -305,7 +253,8 @@ async def test_search_with_include_entities(memory, request_context):
                 request_context=request_context,
             )
 
-        # Observations are generated synchronously during retain, no need to wait
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Search with include_entities=True
         result = await memory.recall_async(
@@ -315,7 +264,7 @@ async def test_search_with_include_entities(memory, request_context):
             budget=Budget.LOW,
             max_tokens=2000,
             include_entities=True,
-            max_entity_tokens=500,
+            max_entity_tokens=5000,
             request_context=request_context,
         )
 
@@ -326,40 +275,28 @@ async def test_search_with_include_entities(memory, request_context):
             if fact.entities:
                 print(f"    Entities: {', '.join(fact.entities)}")
 
-        print(f"\n=== Entity Observations in Recall ===")
-        if result.entities:
-            for name, state in result.entities.items():
-                print(f"\n{name}:")
-                for obs in state.observations:
-                    print(f"  - {obs.text}")
-        else:
-            print("No entity observations returned")
-
         # Verify results
         assert len(result.results) > 0, "Should find some facts"
 
         # Check if entities are included in facts
         facts_with_entities = [f for f in result.results if f.entities]
         assert len(facts_with_entities) > 0, "Some facts should have entity information"
-        print(f"✓ {len(facts_with_entities)} facts have entity information")
+        print(f"{len(facts_with_entities)} facts have entity information")
 
-        # Check if entity observations are included in recall
-        assert result.entities is not None and len(result.entities) > 0, \
-            "Entity observations should be included in recall results"
-        print(f"✓ Entity observations included for {len(result.entities)} entities")
+        # Check if entity info is returned
+        if result.entities:
+            print(f"Entity info included for {len(result.entities)} entities")
 
-        # Verify Alice entity has observations
-        alice_found = False
-        for name, state in result.entities.items():
-            assert state.canonical_name == name, "Entity canonical_name should match key"
-            assert state.entity_id, "Entity should have an ID"
-            if "alice" in name.lower():
-                alice_found = True
-                assert len(state.observations) > 0, \
-                    "Alice should have observations (generated during retain)"
-                print(f"✓ Alice has {len(state.observations)} observations in recall result")
+            # Verify Alice entity is in results
+            alice_found = False
+            for name, state in result.entities.items():
+                assert state.canonical_name == name, "Entity canonical_name should match key"
+                assert state.entity_id, "Entity should have an ID"
+                if "alice" in name.lower():
+                    alice_found = True
+                    print(f"Alice entity found: {name}")
 
-        assert alice_found, "Alice entity should be in recall results"
+            assert alice_found, "Alice entity should be in recall results"
 
     finally:
         # Cleanup
@@ -435,7 +372,10 @@ async def test_get_entity_state(memory, request_context):
 @pytest.mark.asyncio
 async def test_observation_fact_type_in_database(memory, request_context):
     """
-    Test that observations are stored with correct fact_type in database.
+    Test that observations are NOT stored as memory_units with fact_type='observation'.
+
+    NOTE: Observations are now handled via mental models, not as memory_units
+    or entity summaries.
     """
     bank_id = f"test_obs_db_{datetime.now(timezone.utc).timestamp()}"
 
@@ -451,7 +391,7 @@ async def test_observation_fact_type_in_database(memory, request_context):
 
         await memory.wait_for_background_tasks()
 
-        # Check that observations have correct fact_type
+        # Check that NO observations exist in memory_units
         pool = await memory._get_pool()
         async with pool.acquire() as conn:
             observations = await conn.fetch(
@@ -463,17 +403,11 @@ async def test_observation_fact_type_in_database(memory, request_context):
                 bank_id
             )
 
-        print(f"\n=== Observation Records in Database ===")
-        print(f"Found {len(observations)} observation records")
-        for obs in observations:
-            print(f"  - fact_type: {obs['fact_type']}")
-            print(f"    text: {obs['text']}")
-            print(f"    context: {obs['context']}")
+        print(f"\n=== Observation Records in memory_units ===")
+        print(f"Found {len(observations)} observation records (should be 0)")
 
-        if len(observations) > 0:
-            for obs in observations:
-                assert obs['fact_type'] == 'observation', "All observation records should have fact_type='observation'"
-            print(f"✓ All observations have correct fact_type")
+        # Observations are no longer stored as memory_units
+        assert len(observations) == 0, "Observations should NOT be stored as memory_units"
 
     finally:
         # Cleanup
@@ -484,23 +418,183 @@ async def test_observation_fact_type_in_database(memory, request_context):
 
 
 @pytest.mark.asyncio
-async def test_user_entity_prioritized_for_observations(memory, request_context):
+async def test_entity_mention_counts(memory, request_context):
     """
-    Test that the 'user' entity gets observations even when many other entities exist.
+    Test that entity mention counts are tracked correctly.
 
-    The retain pipeline only regenerates observations for TOP_N_ENTITIES (5) entities,
-    sorted by mention count. This test verifies that the most mentioned entity ('user')
-    gets prioritized and receives observations.
-
-    This is critical because 'user' is often the most important entity in personal memory.
+    This test creates entities with varying mention counts and verifies
+    that the counts are accurate.
     """
-    bank_id = f"test_user_priority_{datetime.now(timezone.utc).timestamp()}"
+    bank_id = f"test_mention_counts_{datetime.now(timezone.utc).timestamp()}"
 
     try:
-        # Create content where 'user' (the user) is mentioned many times
-        # along with several other entities
+        # Create content with varying entity mention counts:
+        # - "HighMention Corp" mentioned 10+ times
+        # - "LowMention Ltd" mentioned 1 time
+        contents = [
+            # High mentions - HighMention Corp
+            "HighMention Corp is a tech company based in San Francisco.",
+            "HighMention Corp was founded in 2010 by experienced entrepreneurs.",
+            "HighMention Corp has over 500 employees worldwide.",
+            "HighMention Corp specializes in cloud computing solutions.",
+            "HighMention Corp recently raised $50 million in Series C funding.",
+            "HighMention Corp has partnerships with major tech companies.",
+            "HighMention Corp is known for its innovative culture.",
+            "HighMention Corp offers competitive salaries and benefits.",
+            "HighMention Corp has offices in 5 countries.",
+            "HighMention Corp won the best workplace award last year.",
+            # Low mentions - LowMention Ltd
+            "LowMention Ltd is a small consulting firm.",
+        ]
+
+        for i, content in enumerate(contents):
+            await memory.retain_async(
+                bank_id=bank_id,
+                content=content,
+                context="company info",
+                event_date=datetime(2024, 1, 15 + i, tzinfo=timezone.utc),
+                request_context=request_context,
+            )
+
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
+
+        # Check entity mention counts
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            entities = await conn.fetch(
+                """
+                SELECT e.id, e.canonical_name, e.mention_count
+                FROM entities e
+                WHERE e.bank_id = $1
+                ORDER BY e.mention_count DESC
+                """,
+                bank_id
+            )
+
+        print(f"\n=== Entity Mention Counts Test ===")
+        print(f"Total entities: {len(entities)}")
+
+        high_mention_entity = None
+        low_mention_entity = None
+
+        for entity in entities:
+            name = entity['canonical_name'].lower()
+            mention_count = entity['mention_count']
+
+            print(f"  {entity['canonical_name']}: mentions={mention_count}")
+
+            if "highmention" in name:
+                high_mention_entity = entity
+            elif "lowmention" in name:
+                low_mention_entity = entity
+
+        # Verify HighMention Corp has higher mention count
+        if high_mention_entity and low_mention_entity:
+            assert high_mention_entity['mention_count'] > low_mention_entity['mention_count'], \
+                "HighMention Corp should have more mentions than LowMention Ltd"
+            print("PASS: Entity mention counts are tracked correctly")
+
+    finally:
+        # Cleanup
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", bank_id)
+            await conn.execute("DELETE FROM entities WHERE bank_id = $1", bank_id)
+
+
+@pytest.mark.asyncio
+async def test_entity_mention_ranking(memory, request_context):
+    """
+    Test that entity mention counts correctly rank entities.
+
+    This test:
+    1. Creates an entity with 6 mentions
+    2. Adds more entities with higher mention counts
+    3. Verifies entities are ranked correctly by mention count
+    """
+    bank_id = f"test_ranking_{datetime.now(timezone.utc).timestamp()}"
+
+    try:
+        # Phase 1: Create "OriginalEntity" with 6 mentions
+        print("\n=== Phase 1: Create OriginalEntity with 6 mentions ===")
+        for i in range(6):
+            await memory.retain_async(
+                bank_id=bank_id,
+                content=f"OriginalEntity is mentioned here in fact {i+1}.",
+                context="test",
+                event_date=datetime(2024, 1, 1 + i, tzinfo=timezone.utc),
+                request_context=request_context,
+            )
+
+        await memory.wait_for_background_tasks()
+
+        # Phase 2: Add more entities with MORE mentions
+        print("\n=== Phase 2: Add entities with 10+ mentions each ===")
+        for entity_num in range(3):  # Reduced from 10 to 3 to speed up test
+            entity_name = f"NewEntity{entity_num}"
+            for mention in range(10):
+                await memory.retain_async(
+                    bank_id=bank_id,
+                    content=f"{entity_name} is a very important entity, mention {mention+1}.",
+                    context="test",
+                    event_date=datetime(2024, 2, 1 + mention, tzinfo=timezone.utc),
+                    request_context=request_context,
+                )
+
+        await memory.wait_for_background_tasks()
+
+        # Phase 3: Verify entities are ranked by mention count
+        print("\n=== Phase 3: Check entity ranking ===")
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            all_entities = await conn.fetch(
+                """
+                SELECT canonical_name, mention_count
+                FROM entities
+                WHERE bank_id = $1
+                ORDER BY mention_count DESC
+                """,
+                bank_id
+            )
+
+        print(f"\nAll entities by mention count:")
+        for e in all_entities:
+            print(f"  {e['canonical_name']}: mentions={e['mention_count']}")
+
+        # Verify new entities have higher counts than OriginalEntity
+        original = next((e for e in all_entities if 'originalentity' in e['canonical_name'].lower()), None)
+        new_entities = [e for e in all_entities if 'newentity' in e['canonical_name'].lower()]
+
+        assert original is not None, "OriginalEntity should exist"
+        assert len(new_entities) > 0, "NewEntity entities should exist"
+
+        # Verify entities are created and have mention counts
+        # Note: LLM may merge mentions, so we just check that new entities exist
+        print(f"OriginalEntity mentions: {original['mention_count']}")
+        for new_entity in new_entities:
+            print(f"{new_entity['canonical_name']} mentions: {new_entity['mention_count']}")
+
+        print("PASS: Entities are created with mention counts tracked")
+
+    finally:
+        # Cleanup
+        pool = await memory._get_pool()
+        async with pool.acquire() as conn:
+            await conn.execute("DELETE FROM memory_units WHERE bank_id = $1", bank_id)
+            await conn.execute("DELETE FROM entities WHERE bank_id = $1", bank_id)
+
+
+@pytest.mark.asyncio
+async def test_user_entity_extraction(memory, request_context):
+    """
+    Test that the 'user' entity is correctly extracted when mentioned frequently.
+    """
+    bank_id = f"test_user_entity_{datetime.now(timezone.utc).timestamp()}"
+
+    try:
+        # Create content where 'user' is mentioned many times
         contents = [
-            # User mentioned frequently
             "The user loves hiking in the mountains during summer.",
             "The user works as a software engineer at Microsoft.",
             "The user has a dog named Max who is a golden retriever.",
@@ -510,11 +604,8 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
             # Other entities mentioned fewer times
             "Sarah is a friend who works at Google.",
             "Bob is a colleague from the data science team.",
-            "Tokyo is a city the user visited last year.",
-            "Python is the user's favorite programming language.",
         ]
 
-        # Retain all content in a single batch for efficiency
         for i, content in enumerate(contents):
             await memory.retain_async(
                 bank_id=bank_id,
@@ -524,12 +615,12 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
                 request_context=request_context,
             )
 
-        # Observations are generated synchronously during retain
+        # Wait for background tasks
+        await memory.wait_for_background_tasks()
 
         # Find the 'user' entity
         pool = await memory._get_pool()
         async with pool.acquire() as conn:
-            # Find user entity (may be named "user", "the user", etc.)
             user_entity = await conn.fetchrow(
                 """
                 SELECT e.id, e.canonical_name,
@@ -544,7 +635,7 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
                 bank_id
             )
 
-            # Get all entities with their fact counts to verify prioritization
+            # Get all entities with their fact counts
             all_entities = await conn.fetch(
                 """
                 SELECT e.id, e.canonical_name,
@@ -564,41 +655,10 @@ async def test_user_entity_prioritized_for_observations(memory, request_context)
 
         # Verify user entity exists
         assert user_entity is not None, "User entity should have been extracted"
-        user_entity_id = str(user_entity['id'])
-        user_entity_name = user_entity['canonical_name']
-        user_fact_count = user_entity['fact_count']
-
         print(f"\n=== User Entity ===")
-        print(f"Entity: {user_entity_name} (id: {user_entity_id})")
-        print(f"Fact count: {user_fact_count}")
-
-        # Verify user has enough facts for observations (>= MIN_FACTS_THRESHOLD of 5)
-        assert user_fact_count >= 5, \
-            f"User entity should have at least 5 facts, but has {user_fact_count}"
-
-        # Get observations for user entity
-        observations = await memory.get_entity_observations(bank_id, user_entity_id, limit=10, request_context=request_context)
-
-        print(f"\n=== User Entity Observations ===")
-        print(f"Total observations: {len(observations)}")
-        for obs in observations:
-            print(f"  - {obs.text}")
-
-        # Verify observations were generated for user (critical assertion)
-        assert len(observations) > 0, \
-            f"User entity should have observations (has {user_fact_count} facts, threshold is 5). " \
-            f"This may indicate that 'user' is not being prioritized in the top 5 entities by mention count."
-
-        # Verify observations mention relevant content about the user
-        obs_texts = " ".join([o.text.lower() for o in observations])
-        user_keywords = ["hiking", "software", "engineer", "dog", "max", "cooking",
-                        "italian", "mit", "dune", "microsoft"]
-        matching_keywords = [k for k in user_keywords if k in obs_texts]
-        assert len(matching_keywords) > 0, \
-            f"Observations should contain relevant information about the user. Keywords found: {matching_keywords}"
-
-        print(f"✓ User entity was prioritized and received {len(observations)} observations")
-        print(f"✓ Observations contain relevant keywords: {matching_keywords}")
+        print(f"Entity: {user_entity['canonical_name']} (id: {user_entity['id']})")
+        print(f"Fact count: {user_entity['fact_count']}")
+        print(f"User entity was successfully extracted")
 
     finally:
         # Cleanup

hindsight-api/tests/test_reflect_agent.py

@@ -0,0 +1,983 @@
+"""Tests for the reflect agent and its tools."""
+
+import uuid
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+
+from hindsight_api.engine.reflect.agent import run_reflect_agent
+from hindsight_api.engine.reflect.models import (
+    AnswerSection,
+    MentalModelInput,
+    MentalModelObservation,
+    ReflectAction,
+    ReflectActionBatch,
+    ReflectAgentResult,
+)
+from hindsight_api.engine.reflect.tools import (
+    generate_model_id,
+    tool_expand,
+    tool_learn,
+    tool_lookup,
+    tool_recall,
+)
+from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult
+
+
+class TestGenerateModelId:
+    """Test model ID generation."""
+
+    def test_basic_name(self):
+        """Test simple name conversion."""
+        assert generate_model_id("My Model") == "my-model"
+
+    def test_special_characters(self):
+        """Test name with special characters."""
+        assert generate_model_id("Alice's Project (2024)") == "alice-s-project-2024"
+
+    def test_truncation(self):
+        """Test long name truncation."""
+        long_name = "A" * 100
+        result = generate_model_id(long_name)
+        assert len(result) <= 50
+
+    def test_leading_trailing_hyphens(self):
+        """Test that leading/trailing hyphens are stripped."""
+        assert generate_model_id("--Test--") == "test"
+
+
+class TestToolLookup:
+    """Test the lookup tool."""
+
+    @pytest.fixture
+    def mock_conn(self):
+        """Create a mock database connection."""
+        conn = AsyncMock()
+        return conn
+
+    async def test_list_all_models(self, mock_conn):
+        """Test listing all mental models (compact: id, name, description only)."""
+        mock_conn.fetch.return_value = [
+            {
+                "id": "model-1",
+                "subtype": "learned",
+                "name": "Model 1",
+                "description": "First model",
+            },
+            {
+                "id": "model-2",
+                "subtype": "structural",
+                "name": "Model 2",
+                "description": "Second model",
+            },
+        ]
+
+        result = await tool_lookup(mock_conn, "test-bank")
+
+        assert result["count"] == 2
+        assert len(result["models"]) == 2
+        assert result["models"][0]["id"] == "model-1"
+        assert result["models"][0]["name"] == "Model 1"
+        assert "observation_titles" not in result["models"][0]  # No observation_titles in list view
+        assert result["models"][1]["id"] == "model-2"
+
+    async def test_get_specific_model(self, mock_conn):
+        """Test getting a specific mental model."""
+        mock_conn.fetchrow.return_value = {
+            "id": "model-1",
+            "subtype": "learned",
+            "name": "Model 1",
+            "description": "First model",
+            "observations": {"observations": [{"title": "Overview", "text": "Full summary of model 1", "memory_ids": ["mem-1", "mem-2"]}]},
+            "entity_id": None,
+            "last_updated": MagicMock(isoformat=lambda: "2024-01-01T00:00:00"),
+        }
+
+        result = await tool_lookup(mock_conn, "test-bank", "model-1")
+
+        assert result["found"] is True
+        assert result["model"]["id"] == "model-1"
+        assert len(result["model"]["observations"]) == 1
+        assert result["model"]["observations"][0]["text"] == "Full summary of model 1"
+        # Verify memory_ids are mapped to based_on
+        assert result["model"]["observations"][0]["based_on"] == ["mem-1", "mem-2"]
+
+    async def test_model_not_found(self, mock_conn):
+        """Test looking up non-existent model."""
+        mock_conn.fetchrow.return_value = None
+
+        result = await tool_lookup(mock_conn, "test-bank", "non-existent")
+
+        assert result["found"] is False
+        assert result["model_id"] == "non-existent"
+
+
+class TestToolLearn:
+    """Test the learn tool.
+
+    The learn tool creates placeholder mental models with name/description only.
+    Actual content is generated in the background via refresh.
+    """
+
+    @pytest.fixture
+    def mock_conn(self):
+        """Create a mock database connection."""
+        conn = AsyncMock()
+        return conn
+
+    async def test_create_new_model(self, mock_conn):
+        """Test creating a new mental model placeholder."""
+        mock_conn.fetchrow.return_value = None  # Model doesn't exist
+
+        input_model = MentalModelInput(
+            name="Test Model",
+            description="A test model to track important patterns",
+        )
+
+        result = await tool_learn(mock_conn, "test-bank", input_model)
+
+        assert result["status"] == "created"
+        assert result["model_id"] == "test-model"
+        assert result["name"] == "Test Model"
+        assert result["pending_generation"] is True
+        mock_conn.execute.assert_called_once()
+
+    async def test_update_existing_model(self, mock_conn):
+        """Test updating an existing mental model."""
+        mock_conn.fetchrow.return_value = {"id": "test-model"}  # Model exists
+
+        input_model = MentalModelInput(
+            name="Test Model",
+            description="Updated description",
+        )
+
+        result = await tool_learn(mock_conn, "test-bank", input_model)
+
+        assert result["status"] == "updated"
+        assert result["model_id"] == "test-model"
+
+    async def test_learn_with_entity_id(self, mock_conn):
+        """Test creating model linked to an entity."""
+        mock_conn.fetchrow.return_value = None
+
+        entity_uuid = str(uuid.uuid4())
+        input_model = MentalModelInput(
+            name="Entity Model",
+            description="Model linked to entity",
+            entity_id=entity_uuid,
+        )
+
+        result = await tool_learn(mock_conn, "test-bank", input_model)
+
+        assert result["status"] == "created"
+        assert result["pending_generation"] is True
+        # Verify entity_uuid was passed to the execute call
+        call_args = mock_conn.execute.call_args
+        assert uuid.UUID(entity_uuid) in call_args[0]
+
+    async def test_learn_creates_empty_observations(self, mock_conn):
+        """Test that learn creates model with empty observations (content generated later)."""
+        mock_conn.fetchrow.return_value = None  # Model doesn't exist
+
+        input_model = MentalModelInput(
+            name="Model With Sources",
+            description="A model to track source facts",
+        )
+
+        result = await tool_learn(mock_conn, "test-bank", input_model)
+
+        assert result["status"] == "created"
+        assert result["pending_generation"] is True
+        # Verify the INSERT query was called with empty observations
+        call_args = mock_conn.execute.call_args
+        # observations should be empty JSON
+        assert "'{}'::jsonb" in call_args[0][0]
+
+
+class TestToolExpand:
+    """Test the expand tool."""
+
+    @pytest.fixture
+    def mock_conn(self):
+        """Create a mock database connection."""
+        conn = AsyncMock()
+        return conn
+
+    async def test_empty_memory_ids(self, mock_conn):
+        """Test expand with empty memory_ids list."""
+        result = await tool_expand(mock_conn, "test-bank", [], "chunk")
+
+        assert "error" in result
+        assert "memory_ids is required" in result["error"]
+
+    async def test_invalid_memory_id(self, mock_conn):
+        """Test expand with invalid UUID format."""
+        result = await tool_expand(mock_conn, "test-bank", ["not-a-uuid"], "chunk")
+
+        assert "error" in result
+        assert "No valid memory IDs provided" in result["error"]
+
+    async def test_memory_not_found(self, mock_conn):
+        """Test expand with non-existent memory."""
+        mock_conn.fetch.return_value = []  # No memories found
+        memory_id = str(uuid.uuid4())
+
+        result = await tool_expand(mock_conn, "test-bank", [memory_id], "chunk")
+
+        assert "results" in result
+        assert len(result["results"]) == 1
+        assert "error" in result["results"][0]
+        assert "Memory not found" in result["results"][0]["error"]
+
+    async def test_expand_to_chunk(self, mock_conn):
+        """Test expanding memory to chunk level."""
+        memory_id = uuid.uuid4()
+        # Mock batch fetch for memories
+        mock_conn.fetch.side_effect = [
+            # First call: get memories
+            [
+                {
+                    "id": memory_id,
+                    "text": "Memory text",
+                    "chunk_id": "chunk-1",
+                    "document_id": "doc-1",
+                    "fact_type": "experience",
+                    "context": "some context",
+                }
+            ],
+            # Second call: get chunks
+            [
+                {
+                    "chunk_id": "chunk-1",
+                    "chunk_text": "Full chunk text with more context",
+                    "chunk_index": 0,
+                    "document_id": "doc-1",
+                }
+            ],
+        ]
+
+        result = await tool_expand(mock_conn, "test-bank", [str(memory_id)], "chunk")
+
+        assert "results" in result
+        assert len(result["results"]) == 1
+        assert result["results"][0]["memory"]["text"] == "Memory text"
+        assert result["results"][0]["chunk"]["text"] == "Full chunk text with more context"
+        assert "document" not in result["results"][0]  # depth=chunk doesn't include document
+
+    async def test_expand_to_document(self, mock_conn):
+        """Test expanding memory to document level."""
+        memory_id = uuid.uuid4()
+        mock_conn.fetch.side_effect = [
+            # First call: get memories
+            [
+                {
+                    "id": memory_id,
+                    "text": "Memory text",
+                    "chunk_id": "chunk-1",
+                    "document_id": "doc-1",
+                    "fact_type": "experience",
+                    "context": None,
+                }
+            ],
+            # Second call: get chunks
+            [
+                {
+                    "chunk_id": "chunk-1",
+                    "chunk_text": "Chunk text",
+                    "chunk_index": 0,
+                    "document_id": "doc-1",
+                }
+            ],
+            # Third call: get documents
+            [
+                {
+                    "id": "doc-1",
+                    "original_text": "Full document text here",
+                    "metadata": {"source": "test"},
+                    "retain_params": {},
+                }
+            ],
+        ]
+
+        result = await tool_expand(mock_conn, "test-bank", [str(memory_id)], "document")
+
+        assert "results" in result
+        assert len(result["results"]) == 1
+        assert "memory" in result["results"][0]
+        assert "chunk" in result["results"][0]
+        assert "document" in result["results"][0]
+        assert result["results"][0]["document"]["full_text"] == "Full document text here"
+
+    async def test_expand_multiple_memories(self, mock_conn):
+        """Test expanding multiple memories in a single batch."""
+        memory_id_1 = uuid.uuid4()
+        memory_id_2 = uuid.uuid4()
+        mock_conn.fetch.side_effect = [
+            # First call: get memories
+            [
+                {
+                    "id": memory_id_1,
+                    "text": "Memory 1",
+                    "chunk_id": "chunk-1",
+                    "document_id": "doc-1",
+                    "fact_type": "experience",
+                    "context": None,
+                },
+                {
+                    "id": memory_id_2,
+                    "text": "Memory 2",
+                    "chunk_id": "chunk-2",
+                    "document_id": "doc-1",
+                    "fact_type": "world",
+                    "context": None,
+                },
+            ],
+            # Second call: get chunks
+            [
+                {
+                    "chunk_id": "chunk-1",
+                    "chunk_text": "Chunk 1 text",
+                    "chunk_index": 0,
+                    "document_id": "doc-1",
+                },
+                {
+                    "chunk_id": "chunk-2",
+                    "chunk_text": "Chunk 2 text",
+                    "chunk_index": 1,
+                    "document_id": "doc-1",
+                },
+            ],
+        ]
+
+        result = await tool_expand(mock_conn, "test-bank", [str(memory_id_1), str(memory_id_2)], "chunk")
+
+        assert "results" in result
+        assert result["count"] == 2
+        assert result["results"][0]["memory"]["text"] == "Memory 1"
+        assert result["results"][1]["memory"]["text"] == "Memory 2"
+
+
+class TestToolRecall:
+    """Test the recall tool."""
+
+    async def test_recall_returns_memories(self):
+        """Test recall searches and returns memories."""
+        mock_engine = AsyncMock()
+        mock_result = MagicMock()
+        mock_result.results = [
+            MagicMock(
+                id=uuid.uuid4(),
+                text="Memory 1",
+                fact_type="experience",
+                entities=["Alice"],
+                occurred_start="2024-01-01",
+            ),
+            MagicMock(
+                id=uuid.uuid4(),
+                text="Memory 2",
+                fact_type="world",
+                entities=None,
+                occurred_start=None,
+            ),
+        ]
+        mock_engine.recall_async.return_value = mock_result
+
+        mock_request_context = MagicMock()
+
+        result = await tool_recall(mock_engine, "test-bank", "test query", mock_request_context)
+
+        assert result["query"] == "test query"
+        assert result["count"] == 2
+        assert len(result["memories"]) == 2
+        assert result["memories"][0]["text"] == "Memory 1"
+        assert result["memories"][0]["entities"] == ["Alice"]
+
+        # Verify recall_async was called with correct params
+        mock_engine.recall_async.assert_called_once()
+        call_kwargs = mock_engine.recall_async.call_args[1]
+        assert call_kwargs["bank_id"] == "test-bank"
+        assert call_kwargs["query"] == "test query"
+        assert call_kwargs["fact_type"] == ["experience", "world"]  # No opinions
+
+
+class TestPromptSize:
+    """Test that prompts stay within reasonable size limits.
+
+    Large prompts cause slow LLM responses (120s+ observed in production).
+    The agent should not pre-load all mental models; use lookup() instead.
+    """
+
+    def test_initial_prompt_is_small(self):
+        """Verify the initial prompt (no tool history) is reasonably small."""
+        from hindsight_api.engine.reflect.prompts import build_agent_prompt, build_system_prompt_for_tools
+
+        # Typical bank profile
+        bank_profile = {
+            "name": "Test Assistant",
+            "mission": "A helpful assistant for tracking engineering team activities. Help the team stay organized and informed.",
+        }
+
+        # First iteration: no context history
+        context_history: list[dict] = []
+        query = "Who should take ownership of storing load test scripts in Git?"
+
+        # No additional context (mental models not pre-loaded)
+        prompt = build_agent_prompt(query, context_history, bank_profile, additional_context=None)
+        system_prompt = build_system_prompt_for_tools(bank_profile)
+
+        total_chars = len(prompt) + len(system_prompt)
+        estimated_tokens = total_chars // 4  # Rough estimate
+
+        # Initial prompt should be under 3000 tokens (~12k chars)
+        # This ensures fast LLM responses on the first iteration
+        assert total_chars < 12000, f"Initial prompt too large: {total_chars} chars (~{estimated_tokens} tokens)"
+        assert estimated_tokens < 3000, f"Initial prompt too large: ~{estimated_tokens} tokens"
+
+    def test_prompt_with_tool_history_grows_reasonably(self):
+        """Verify prompts grow reasonably with tool results."""
+        from hindsight_api.engine.reflect.prompts import build_agent_prompt, build_system_prompt_for_tools
+
+        bank_profile = {
+            "name": "Test Assistant",
+            "mission": "A helpful assistant. Help the team.",
+        }
+
+        # Simulate recall result with 50 memories (realistic scenario)
+        recall_result = {
+            "query": "test query",
+            "count": 50,
+            "memories": [
+                {"id": f"mem-{i}", "text": f"This is memory number {i} with some content.", "type": "experience"}
+                for i in range(50)
+            ],
+        }
+
+        context_history = [{"tool": "recall", "input": {"query": "test"}, "output": recall_result}]
+        query = "What do you know about the team?"
+
+        prompt = build_agent_prompt(query, context_history, bank_profile, additional_context=None)
+        system_prompt = build_system_prompt_for_tools(bank_profile)
+
+        total_chars = len(prompt) + len(system_prompt)
+        estimated_tokens = total_chars // 4
+
+        # With tool results, prompt should still be manageable (<20k tokens)
+        assert total_chars < 80000, f"Prompt with tools too large: {total_chars} chars (~{estimated_tokens} tokens)"
+
+
+class TestReflectAgent:
+    """Test the reflect agent loop with native tool calling."""
+
+    @pytest.fixture
+    def mock_llm(self):
+        """Create a mock LLM provider."""
+        llm = AsyncMock()
+        return llm
+
+    @pytest.fixture
+    def bank_profile(self):
+        """Create a test bank profile."""
+        return {
+            "name": "Test Assistant",
+            "mission": "A helpful test assistant. Help with testing.",
+        }
+
+    @pytest.fixture
+    def mock_tools(self):
+        """Create mock tool callbacks."""
+        # Include memory IDs in recall results so guardrail passes
+        memory_id = str(uuid.uuid4())
+        return {
+            "lookup_fn": AsyncMock(return_value={"count": 0, "models": []}),
+            "recall_fn": AsyncMock(return_value={
+                "query": "test",
+                "count": 1,
+                "memories": [{"id": memory_id, "text": "Memory", "type": "experience"}]
+            }),
+            "learn_fn": AsyncMock(return_value={"status": "created", "model_id": "new-model"}),
+            "expand_fn": AsyncMock(return_value={
+                "results": [{"memory_id": "123", "memory": {"id": "123", "text": "Memory text"}}],
+                "count": 1
+            }),
+        }
+
+    def _make_tool_result(self, tool_calls: list[dict]) -> LLMToolCallResult:
+        """Helper to create LLMToolCallResult from tool call dicts."""
+        return LLMToolCallResult(
+            tool_calls=[
+                LLMToolCall(id=f"call_{i}", name=tc["name"], arguments=tc.get("arguments", {}))
+                for i, tc in enumerate(tool_calls)
+            ],
+            finish_reason="tool_calls",
+        )
+
+    async def test_agent_done_immediately_rejected_by_guardrail(self, mock_llm, bank_profile, mock_tools):
+        """Test that guardrail rejects done without evidence gathering."""
+        # First call: agent tries to return done immediately (rejected by guardrail)
+        # Second call: agent gathers evidence
+        # Third call: agent returns done with evidence
+        mock_llm.call_with_tools.side_effect = [
+            self._make_tool_result([{"name": "done", "arguments": {"answer": "The answer is 42."}}]),
+            # After guardrail rejection, agent should gather evidence
+            self._make_tool_result([{"name": "recall", "arguments": {"query": "test query"}}]),
+            # Now with evidence, done is accepted
+            self._make_tool_result([{"name": "done", "arguments": {"answer": "The answer is 42."}}]),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="What is the answer?",
+            bank_profile=bank_profile,
+            **mock_tools,
+        )
+
+        assert isinstance(result, ReflectAgentResult)
+        assert result.text == "The answer is 42."
+        # 3 iterations: rejected done, recall, accepted done
+        assert result.iterations == 3
+        # Tools called: list_mental_models (auto at start of each iteration) + recall
+        # The exact count may vary based on implementation
+        assert result.tools_called >= 1  # At least recall was called
+
+    async def test_agent_calls_tools_then_done(self, mock_llm, bank_profile, mock_tools):
+        """Test agent that calls tools before completing."""
+        # First call: lookup and recall
+        # Second call: done
+        mock_llm.call_with_tools.side_effect = [
+            self._make_tool_result([
+                {"name": "list_mental_models", "arguments": {}},
+                {"name": "recall", "arguments": {"query": "test query"}},
+            ]),
+            self._make_tool_result([
+                {"name": "done", "arguments": {"answer": "Based on my research, the answer is yes."}},
+            ]),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="Is testing important?",
+            bank_profile=bank_profile,
+            **mock_tools,
+        )
+
+        assert result.text == "Based on my research, the answer is yes."
+        assert result.iterations == 2
+        # Tools called: list_mental_models + recall (+ possibly auto list_mental_models)
+        assert result.tools_called >= 2
+        mock_tools["recall_fn"].assert_called_once_with("test query", 2048)
+
+    async def test_agent_learns_model(self, mock_llm, bank_profile, mock_tools):
+        """Test agent that creates a mental model placeholder."""
+        mock_tools["learn_fn"].return_value = {"status": "created", "model_id": "new-insight", "pending_generation": True}
+
+        mock_llm.call_with_tools.side_effect = [
+            # First: gather evidence via recall (required by guardrail)
+            self._make_tool_result([{"name": "recall", "arguments": {"query": "user preferences"}}]),
+            # Then: learn from the gathered evidence
+            self._make_tool_result([{
+                "name": "learn",
+                "arguments": {
+                    "name": "New Insight",
+                    "description": "Track patterns about user preferences and communication style",
+                }
+            }]),
+            # Finally: done with the learning
+            self._make_tool_result([{"name": "done", "arguments": {"answer": "I've learned something new."}}]),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="What can you learn?",
+            bank_profile=bank_profile,
+            **mock_tools,
+        )
+
+        assert result.mental_models_created == ["new-insight"]
+        mock_tools["learn_fn"].assert_called_once()
+        # Verify the learn_fn was called with name and description only
+        call_args = mock_tools["learn_fn"].call_args
+        mental_model_arg = call_args[0][0]
+        assert mental_model_arg.name == "New Insight"
+        assert "preferences" in mental_model_arg.description
+
+    async def test_agent_max_iterations_forces_response(self, mock_llm, bank_profile, mock_tools):
+        """Test that max iterations forces a text response."""
+        # Return tools indefinitely, then final plain text call
+        mock_llm.call_with_tools.side_effect = [
+            self._make_tool_result([{"name": "recall", "arguments": {"query": "query"}}]),
+            self._make_tool_result([{"name": "recall", "arguments": {"query": "query2"}}]),
+        ]
+        # On last iteration, LLM.call is used (not call_with_tools)
+        mock_llm.call.return_value = "Forced final answer after max iterations."
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="Test question",
+            bank_profile=bank_profile,
+            max_iterations=3,
+            **mock_tools,
+        )
+
+        assert result.text == "Forced final answer after max iterations."
+        assert result.iterations == 3
+
+    async def test_agent_handles_tool_error(self, mock_llm, bank_profile, mock_tools):
+        """Test agent propagates tool execution errors."""
+        # Make recall fail
+        mock_tools["recall_fn"].side_effect = Exception("Database error")
+
+        mock_llm.call_with_tools.side_effect = [
+            self._make_tool_result([{"name": "recall", "arguments": {"query": "query"}}]),
+        ]
+
+        # Tool errors are now propagated as RuntimeError
+        with pytest.raises(RuntimeError) as exc_info:
+            await run_reflect_agent(
+                llm_config=mock_llm,
+                bank_id="test-bank",
+                query="Test question",
+                bank_profile=bank_profile,
+                **mock_tools,
+            )
+
+        assert "Database error" in str(exc_info.value)
+
+    async def test_agent_parallel_tool_calls(self, mock_llm, bank_profile, mock_tools):
+        """Test agent executes multiple tools in parallel."""
+        mock_llm.call_with_tools.side_effect = [
+            self._make_tool_result([
+                {"name": "list_mental_models", "arguments": {}},
+                {"name": "recall", "arguments": {"query": "query1"}},
+                {"name": "recall", "arguments": {"query": "query2"}},
+            ]),
+            self._make_tool_result([{"name": "done", "arguments": {"answer": "Done after parallel calls."}}]),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="Test question",
+            bank_profile=bank_profile,
+            **mock_tools,
+        )
+
+        # Tools called: list_mental_models + 2x recall (+ possibly auto list_mental_models)
+        assert result.tools_called >= 3
+        # recall should be called twice
+        assert mock_tools["recall_fn"].call_count == 2
+
+    async def test_agent_returns_validated_memory_ids(self, mock_llm, bank_profile):
+        """Test agent returns only validated memory IDs that were actually recalled."""
+        memory_id_1 = str(uuid.uuid4())
+        memory_id_2 = str(uuid.uuid4())
+
+        # Mock recall returns these specific memory IDs
+        mock_recall = AsyncMock(
+            return_value={
+                "query": "test",
+                "count": 2,
+                "memories": [
+                    {"id": memory_id_1, "text": "Memory 1", "type": "experience"},
+                    {"id": memory_id_2, "text": "Memory 2", "type": "world"},
+                ],
+            }
+        )
+
+        mock_llm.call_with_tools.side_effect = [
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(id="call_0", name="recall", arguments={"query": "test query"})],
+                finish_reason="tool_calls",
+            ),
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(
+                    id="call_1",
+                    name="done",
+                    arguments={"answer": "Based on the evidence...", "memory_ids": [memory_id_1, memory_id_2]}
+                )],
+                finish_reason="tool_calls",
+            ),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="What do we know?",
+            bank_profile=bank_profile,
+            lookup_fn=AsyncMock(return_value={"count": 0, "models": []}),
+            recall_fn=mock_recall,
+            expand_fn=AsyncMock(return_value={}),
+        )
+
+        # Both memory IDs should be in the result (they were recalled)
+        assert memory_id_1 in result.used_memory_ids
+        assert memory_id_2 in result.used_memory_ids
+        assert len(result.used_memory_ids) == 2
+
+    async def test_agent_filters_hallucinated_memory_ids(self, mock_llm, bank_profile):
+        """Test agent filters out memory IDs that were not in recall results."""
+        valid_memory_id = str(uuid.uuid4())
+        hallucinated_memory_id = str(uuid.uuid4())
+
+        # Mock recall returns only one memory ID
+        mock_recall = AsyncMock(
+            return_value={
+                "query": "test",
+                "count": 1,
+                "memories": [
+                    {"id": valid_memory_id, "text": "Real memory", "type": "experience"},
+                ],
+            }
+        )
+
+        mock_llm.call_with_tools.side_effect = [
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(id="call_0", name="recall", arguments={"query": "test query"})],
+                finish_reason="tool_calls",
+            ),
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(
+                    id="call_1",
+                    name="done",
+                    arguments={"answer": "Based on evidence...", "memory_ids": [valid_memory_id, hallucinated_memory_id]}
+                )],
+                finish_reason="tool_calls",
+            ),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="What do we know?",
+            bank_profile=bank_profile,
+            lookup_fn=AsyncMock(return_value={"count": 0, "models": []}),
+            recall_fn=mock_recall,
+            expand_fn=AsyncMock(return_value={}),
+        )
+
+        # Only the valid memory ID should be in the result
+        assert valid_memory_id in result.used_memory_ids
+        assert hallucinated_memory_id not in result.used_memory_ids
+        assert len(result.used_memory_ids) == 1
+
+    async def test_agent_returns_validated_model_ids(self, mock_llm, bank_profile):
+        """Test agent returns only validated model IDs that were actually looked up."""
+        model_id = "team-structure"
+        hallucinated_model_id = "non-existent-model"
+
+        # Mock lookup returns different results based on input
+        # - None (or no arg): list_mental_models - returns list of models
+        # - model_id: get_mental_model - returns specific model with found=True
+        async def mock_lookup_impl(arg=None):
+            if arg is None:
+                return {"count": 1, "models": [{"id": model_id, "name": "Team Structure", "description": "desc"}]}
+            else:
+                return {"found": True, "model": {"id": model_id, "name": "Team Structure", "summary": "Full summary"}}
+
+        mock_lookup = AsyncMock(side_effect=mock_lookup_impl)
+
+        mock_llm.call_with_tools.side_effect = [
+            LLMToolCallResult(
+                tool_calls=[
+                    LLMToolCall(id="call_0", name="list_mental_models", arguments={}),
+                    LLMToolCall(id="call_1", name="get_mental_model", arguments={"model_id": model_id}),
+                ],
+                finish_reason="tool_calls",
+            ),
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(
+                    id="call_2",
+                    name="done",
+                    arguments={"answer": "Based on team structure...", "model_ids": [model_id, hallucinated_model_id]}
+                )],
+                finish_reason="tool_calls",
+            ),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="How is the team organized?",
+            bank_profile=bank_profile,
+            lookup_fn=mock_lookup,
+            recall_fn=AsyncMock(return_value={"query": "test", "count": 0, "memories": []}),
+            expand_fn=AsyncMock(return_value={}),
+        )
+
+        # Only the valid model ID should be in the result
+        assert model_id in result.used_model_ids
+        assert hallucinated_model_id not in result.used_model_ids
+
+    async def test_agent_plain_text_answer(self, mock_llm, bank_profile, mock_tools):
+        """Test agent with plain text answer format."""
+        mock_llm.call_with_tools.side_effect = [
+            # First: gather evidence via recall (required by guardrail)
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(id="call_0", name="recall", arguments={"query": "answer"})],
+                finish_reason="tool_calls",
+            ),
+            # Then: done with plain text answer
+            LLMToolCallResult(
+                tool_calls=[LLMToolCall(
+                    id="call_1",
+                    name="done",
+                    arguments={"answer": "The answer is simple and direct."}
+                )],
+                finish_reason="tool_calls",
+            ),
+        ]
+
+        result = await run_reflect_agent(
+            llm_config=mock_llm,
+            bank_id="test-bank",
+            query="What's the answer?",
+            bank_profile=bank_profile,
+            **mock_tools,
+        )
+
+        assert result.text == "The answer is simple and direct."
+
+
+@pytest.mark.integration
+class TestReflectIntegration:
+    """Integration tests for reflect with real database.
+
+    These tests require a running database and LLM provider.
+    Skip with: pytest -m "not integration"
+    """
+
+    async def test_reflect_creates_learned_mental_model(self, memory, request_context):
+        """Test that reflect can create a 'learned' mental model via the agent."""
+        bank_id = f"test-reflect-{uuid.uuid4().hex[:8]}"
+
+        # Add some test data
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Alice is the team lead and manages the engineering team."},
+                {"content": "The team has weekly planning meetings on Monday."},
+                {"content": "Alice prefers asynchronous communication via Slack."},
+            ],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Run reflect - this should use the agentic loop
+        result = await memory.reflect_async(
+            bank_id=bank_id,
+            query="What do you know about Alice and how she manages the team?",
+            request_context=request_context,
+        )
+
+        assert result.text is not None
+        assert len(result.text) > 0
+
+        # Check if any mental models were created (may or may not happen depending on LLM)
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        # If models were created, they should be 'learned' subtype
+        for model in models:
+            if model.get("subtype") == "learned":
+                # Learned models are created as placeholders pending generation
+                assert model.get("name") is not None
+                assert model.get("description") is not None
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_reflect_learn_triggers_background_generation(self, memory, request_context):
+        """Test that when reflect calls learn, background generation is triggered.
+
+        This test verifies the full flow:
+        1. Agent decides to learn something important
+        2. learn tool creates a placeholder model
+        3. Background generation is automatically triggered
+        """
+        import asyncio
+
+        bank_id = f"test-reflect-learn-{uuid.uuid4().hex[:8]}"
+
+        # Add rich test data that should prompt the agent to learn something
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "Bob is the CEO and founder of the company."},
+                {"content": "Bob started the company in 2015 after leaving Google."},
+                {"content": "Bob holds weekly all-hands meetings every Friday at 3pm."},
+                {"content": "Bob's management style is very hands-off and trusts his team."},
+                {"content": "Bob prefers face-to-face communication over email."},
+                {"content": "Bob has a strong focus on company culture and team building."},
+            ],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Run reflect with a query that should prompt learning
+        result = await memory.reflect_async(
+            bank_id=bank_id,
+            query="Tell me everything about Bob's leadership style and how he runs the company. "
+            "This is important information I'll need to reference frequently.",
+            request_context=request_context,
+        )
+
+        assert result.text is not None
+        assert len(result.text) > 0
+
+        # Wait for any background tasks to complete
+        await memory.wait_for_background_tasks()
+        # Give a bit more time for async generation
+        await asyncio.sleep(2)
+
+        # Check if learned models were created
+        models = await memory.list_mental_models(
+            bank_id=bank_id,
+            request_context=request_context,
+        )
+
+        learned_models = [m for m in models if m.get("subtype") == "learned"]
+
+        # If learned models were created, verify they have proper structure
+        for model in learned_models:
+            assert model.get("name") is not None
+            assert model.get("description") is not None
+            # After background generation, the model should have been updated
+            # (observations may or may not be populated depending on timing)
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)
+
+    async def test_reflect_excludes_opinions_from_recall(self, memory, request_context):
+        """Test that reflect's recall tool doesn't return opinions."""
+        bank_id = f"test-reflect-no-opinions-{uuid.uuid4().hex[:8]}"
+
+        # Add test data (note: we can't directly add opinions since opinion
+        # extraction was removed, but we can verify recall behavior)
+        await memory.retain_batch_async(
+            bank_id=bank_id,
+            contents=[
+                {"content": "The weather today is sunny and warm."},
+            ],
+            request_context=request_context,
+        )
+        await memory.wait_for_background_tasks()
+
+        # Run recall directly to verify it excludes opinions
+        recall_result = await memory.recall_async(
+            bank_id=bank_id,
+            query="weather",
+            request_context=request_context,
+        )
+
+        # All returned facts should be experience or world, not opinion
+        for fact in recall_result.results:
+            assert fact.fact_type in ["experience", "world"]
+            assert fact.fact_type != "opinion"
+
+        # Cleanup
+        await memory.delete_bank(bank_id, request_context=request_context)

hindsight-api/tests/test_retain.py

@@ -465,7 +465,7 @@ async def test_occurred_dates_not_defaulted(memory, request_context):
             query="Tell me about Alice",
             budget=Budget.LOW,
             max_tokens=500,
-            fact_type=["world", "opinion"],
+            fact_type=["world", "experience"],
             request_context=request_context,
         )
 

hindsight-api/tests/test_server_module.py

@@ -0,0 +1,290 @@
+"""
+Tests for hindsight_api.server module (multi-worker code path).
+
+The server.py module is used when running with multiple workers:
+    uvicorn hindsight_api.server:app --workers 2
+
+This module executes code at import time, creating the app at module level.
+These tests ensure that extensions are properly loaded in this code path,
+which was previously a regression that caused authentication bypass in production.
+"""
+
+import importlib
+import sys
+from unittest.mock import MagicMock, patch
+
+
+def _clean_server_module():
+    """Remove hindsight_api.server from sys.modules for fresh import."""
+    modules_to_remove = [k for k in sys.modules.keys() if k.startswith("hindsight_api.server")]
+    for mod in modules_to_remove:
+        del sys.modules[mod]
+
+
+class TestServerModuleExtensionLoading:
+    """Tests that server.py correctly loads extensions when configured via environment."""
+
+    def test_server_loads_tenant_extension_when_configured(self, monkeypatch):
+        """
+        Verify that server.py loads tenant extension from HINDSIGHT_API_TENANT_EXTENSION.
+
+        This test catches the regression where server.py didn't call load_extension(),
+        causing authentication to be bypassed in multi-worker deployments.
+        """
+        # Set up environment to configure a tenant extension
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_server_module:MockTenantExtension",
+        )
+
+        _clean_server_module()
+
+        # Track what extensions were loaded via load_extension
+        loaded_extensions = {}
+
+        # Get the real load_extension function
+        from hindsight_api.extensions.loader import load_extension as real_load_extension
+
+        def tracking_load_extension(name, base_class):
+            """Track calls to load_extension and delegate to original."""
+            result = real_load_extension(name, base_class)
+            loaded_extensions[name] = result
+            return result
+
+        # Patch at source level BEFORE importing server
+        # Note: We patch the entire hindsight_api module namespace
+        with patch("hindsight_api.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.api.create_app") as mock_create_app, \
+             patch("hindsight_api.config.get_config") as mock_get_config, \
+             patch("hindsight_api.extensions.load_extension", side_effect=tracking_load_extension), \
+             patch("hindsight_api.extensions.DefaultExtensionContext"):
+
+            mock_config = MagicMock()
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+            mock_create_app.return_value = MagicMock()
+
+            # Now import server - this triggers module-level code
+            import hindsight_api.server
+
+        # Verify TENANT extension was loaded
+        assert "TENANT" in loaded_extensions, \
+            "server.py did not call load_extension('TENANT', ...) - extensions not loaded!"
+        assert loaded_extensions["TENANT"] is not None, \
+            "load_extension('TENANT', ...) returned None despite env var being set"
+        assert isinstance(loaded_extensions["TENANT"], MockTenantExtension), \
+            f"Expected MockTenantExtension, got {type(loaded_extensions['TENANT'])}"
+
+    def test_server_loads_operation_validator_when_configured(self, monkeypatch):
+        """
+        Verify that server.py loads operation validator from HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION",
+            "tests.test_server_module:MockOperationValidator",
+        )
+
+        _clean_server_module()
+
+        loaded_extensions = {}
+
+        from hindsight_api.extensions.loader import load_extension as real_load_extension
+
+        def tracking_load_extension(name, base_class):
+            result = real_load_extension(name, base_class)
+            loaded_extensions[name] = result
+            return result
+
+        with patch("hindsight_api.MemoryEngine") as mock_engine, \
+             patch("hindsight_api.api.create_app") as mock_create_app, \
+             patch("hindsight_api.config.get_config") as mock_get_config, \
+             patch("hindsight_api.extensions.load_extension", side_effect=tracking_load_extension), \
+             patch("hindsight_api.extensions.DefaultExtensionContext"):
+
+            mock_config = MagicMock()
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_engine.return_value = MagicMock()
+            mock_create_app.return_value = MagicMock()
+
+            import hindsight_api.server
+
+        assert "OPERATION_VALIDATOR" in loaded_extensions, \
+            "server.py did not call load_extension('OPERATION_VALIDATOR', ...)"
+        assert loaded_extensions["OPERATION_VALIDATOR"] is not None
+        assert isinstance(loaded_extensions["OPERATION_VALIDATOR"], MockOperationValidator)
+
+    def test_server_passes_extensions_to_memory_engine(self, monkeypatch):
+        """
+        Verify that server.py passes loaded extensions to MemoryEngine constructor.
+
+        This is the critical test - even if extensions are loaded, they must be
+        passed to MemoryEngine for authentication to work.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_server_module:MockTenantExtension",
+        )
+
+        _clean_server_module()
+
+        memory_engine_calls = []
+
+        def capture_memory_engine(*args, **kwargs):
+            memory_engine_calls.append({"args": args, "kwargs": kwargs})
+            return MagicMock()
+
+        with patch("hindsight_api.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.api.create_app") as mock_create_app, \
+             patch("hindsight_api.config.get_config") as mock_get_config, \
+             patch("hindsight_api.extensions.DefaultExtensionContext"):
+
+            mock_config = MagicMock()
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            import hindsight_api.server
+
+        # Verify MemoryEngine was called
+        assert len(memory_engine_calls) == 1, "MemoryEngine should be called exactly once"
+
+        call_kwargs = memory_engine_calls[0]["kwargs"]
+
+        # THE CRITICAL ASSERTION: tenant_extension must be passed and not None
+        assert "tenant_extension" in call_kwargs, \
+            "MemoryEngine was not called with tenant_extension parameter!"
+        assert call_kwargs["tenant_extension"] is not None, \
+            "tenant_extension was None - server.py did not pass loaded extension to MemoryEngine!"
+
+    def test_server_sets_extension_context_on_tenant_extension(self, monkeypatch):
+        """
+        Verify that server.py sets the extension context on tenant extension.
+
+        This is required for tenant extensions that need to provision schemas.
+        """
+        monkeypatch.setenv(
+            "HINDSIGHT_API_TENANT_EXTENSION",
+            "tests.test_server_module:MockTenantExtension",
+        )
+
+        _clean_server_module()
+
+        context_set_calls = []
+        captured_tenant_ext = [None]
+
+        def capture_memory_engine(*args, **kwargs):
+            captured_tenant_ext[0] = kwargs.get("tenant_extension")
+            return MagicMock()
+
+        def capture_context(*args, **kwargs):
+            ctx = MagicMock()
+            context_set_calls.append(ctx)
+            return ctx
+
+        with patch("hindsight_api.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.api.create_app") as mock_create_app, \
+             patch("hindsight_api.config.get_config") as mock_get_config, \
+             patch("hindsight_api.extensions.DefaultExtensionContext", side_effect=capture_context):
+
+            mock_config = MagicMock()
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            import hindsight_api.server
+
+        # Verify context was created and set
+        assert len(context_set_calls) == 1, "DefaultExtensionContext should be created"
+        assert captured_tenant_ext[0] is not None, "Tenant extension should be captured"
+        assert captured_tenant_ext[0]._context_set, \
+            "set_context was not called on tenant extension"
+
+    def test_server_works_without_extensions(self, monkeypatch):
+        """
+        Verify that server.py works correctly when no extensions are configured.
+        """
+        # Ensure no extension env vars are set
+        monkeypatch.delenv("HINDSIGHT_API_TENANT_EXTENSION", raising=False)
+        monkeypatch.delenv("HINDSIGHT_API_OPERATION_VALIDATOR_EXTENSION", raising=False)
+
+        _clean_server_module()
+
+        memory_engine_calls = []
+
+        def capture_memory_engine(*args, **kwargs):
+            memory_engine_calls.append({"args": args, "kwargs": kwargs})
+            return MagicMock()
+
+        with patch("hindsight_api.MemoryEngine", side_effect=capture_memory_engine), \
+             patch("hindsight_api.api.create_app") as mock_create_app, \
+             patch("hindsight_api.config.get_config") as mock_get_config:
+
+            mock_config = MagicMock()
+            mock_config.mcp_enabled = False
+            mock_config.run_migrations_on_startup = False
+            mock_config.database_url = "postgresql://test:test@localhost/test"
+            mock_get_config.return_value = mock_config
+            mock_create_app.return_value = MagicMock()
+
+            import hindsight_api.server
+
+        # Should work without extensions
+        assert len(memory_engine_calls) == 1
+        call_kwargs = memory_engine_calls[0]["kwargs"]
+
+        # Extensions should be None when not configured
+        assert call_kwargs.get("tenant_extension") is None
+        assert call_kwargs.get("operation_validator") is None
+
+
+# Mock extensions for testing
+from hindsight_api.extensions import (
+    TenantExtension,
+    TenantContext,
+    RequestContext,
+    OperationValidatorExtension,
+    ValidationResult,
+    RetainContext,
+    RecallContext,
+    ReflectContext,
+)
+
+
+class MockTenantExtension(TenantExtension):
+    """Mock tenant extension for testing server.py extension loading."""
+
+    def __init__(self, config: dict):
+        super().__init__(config)
+        self._context_set = False
+
+    async def authenticate(self, request_context: RequestContext) -> TenantContext:
+        return TenantContext(schema_name="public")
+
+    def set_context(self, context) -> None:
+        self._context_set = True
+
+
+class MockOperationValidator(OperationValidatorExtension):
+    """Mock operation validator for testing server.py extension loading."""
+
+    def __init__(self, config: dict):
+        super().__init__(config)
+
+    async def validate_retain(self, ctx: RetainContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_recall(self, ctx: RecallContext) -> ValidationResult:
+        return ValidationResult.accept()
+
+    async def validate_reflect(self, ctx: ReflectContext) -> ValidationResult:
+        return ValidationResult.accept()

hindsight-api/tests/test_tags_visibility.py

@@ -0,0 +1,886 @@
+"""
+Tests for tags-based visibility scoping.
+
+This module tests the tags feature which allows filtering memories by visibility tags.
+Use cases:
+- Multi-user agent: Agent has a single memory bank, users should only see memories from
+  conversations they participated in
+- Student tracking: Teacher tracks students, students should only see their own data
+
+The tags use OR-based matching: a memory matches if ANY of its tags overlap with the request tags.
+"""
+from datetime import datetime
+
+import httpx
+import pytest
+import pytest_asyncio
+
+from hindsight_api.api import create_app
+from hindsight_api.engine.search.tags import build_tags_where_clause_simple, filter_results_by_tags
+
+# ============================================================================
+# Unit Tests for tags SQL builder
+# ============================================================================
+
+
+class TestTagsWhereClauseBuilder:
+    """Unit tests for the tags WHERE clause SQL builder."""
+
+    def test_no_tags_returns_empty_string(self):
+        """When tags is None, should return empty string (no filtering)."""
+        result = build_tags_where_clause_simple(None, 5)
+        assert result == ""
+
+    def test_empty_tags_list_returns_empty_string(self):
+        """When tags is an empty list, should return empty string (no filtering)."""
+        result = build_tags_where_clause_simple([], 5)
+        assert result == ""
+
+    def test_tags_with_different_param_num(self):
+        """Should use the provided parameter number."""
+        result = build_tags_where_clause_simple(["user_a", "user_b"], 3)
+        # Default is "any" which includes untagged
+        assert "$3" in result
+
+    def test_tags_with_table_alias(self):
+        """Should include table alias when provided."""
+        result = build_tags_where_clause_simple(["user_a"], 5, table_alias="mu.")
+        assert "mu.tags" in result
+
+    # ---- Test "any" mode (OR, includes untagged - default) ----
+
+    def test_tags_match_any_includes_untagged(self):
+        """When match='any', should include untagged memories (NULL or empty)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="any")
+        # Should use OR with NULL/empty check
+        assert "IS NULL" in result
+        assert "= '{}'" in result
+        assert "&&" in result  # overlap operator
+
+    def test_tags_match_any_uses_overlap(self):
+        """When match='any', should use overlap operator (&&)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="any")
+        assert "&&" in result
+
+    # ---- Test "all" mode (AND, includes untagged) ----
+
+    def test_tags_match_all_includes_untagged(self):
+        """When match='all', should include untagged memories (NULL or empty)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="all")
+        # Should use OR with NULL/empty check
+        assert "IS NULL" in result
+        assert "= '{}'" in result
+        assert "@>" in result  # contains operator
+
+    def test_tags_match_all_uses_contains(self):
+        """When match='all', should use contains operator (@>)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="all")
+        assert "@>" in result
+
+    # ---- Test "any_strict" mode (OR, excludes untagged) ----
+
+    def test_tags_match_any_strict_excludes_untagged(self):
+        """When match='any_strict', should exclude untagged memories."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="any_strict")
+        # Should require tags to be NOT NULL and not empty
+        assert "IS NOT NULL" in result
+        assert "!= '{}'" in result
+        assert "&&" in result  # overlap operator
+
+    def test_tags_match_any_strict_uses_overlap(self):
+        """When match='any_strict', should use overlap operator (&&)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="any_strict")
+        assert "&&" in result
+        # Should NOT include untagged
+        assert "IS NULL" not in result or "IS NOT NULL" in result
+
+    # ---- Test "all_strict" mode (AND, excludes untagged) ----
+
+    def test_tags_match_all_strict_excludes_untagged(self):
+        """When match='all_strict', should exclude untagged memories."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="all_strict")
+        # Should require tags to be NOT NULL and not empty
+        assert "IS NOT NULL" in result
+        assert "!= '{}'" in result
+        assert "@>" in result  # contains operator
+
+    def test_tags_match_all_strict_uses_contains(self):
+        """When match='all_strict', should use contains operator (@>)."""
+        result = build_tags_where_clause_simple(["user_a"], 5, match="all_strict")
+        assert "@>" in result
+
+    # ---- Test table alias with all modes ----
+
+    def test_tags_match_any_with_table_alias(self):
+        """Should include table alias with any mode."""
+        result = build_tags_where_clause_simple(["user_a"], 3, table_alias="mu.", match="any")
+        assert "mu.tags" in result
+
+    def test_tags_match_all_strict_with_table_alias(self):
+        """Should include table alias with all_strict mode."""
+        result = build_tags_where_clause_simple(["user_a", "user_b"], 3, table_alias="mu.", match="all_strict")
+        assert "mu.tags" in result
+        assert "@>" in result
+        assert "IS NOT NULL" in result
+
+
+# ============================================================================
+# Unit Tests for filter_results_by_tags (Python-side filtering)
+# ============================================================================
+
+
+class MockResult:
+    """Mock result object for testing filter_results_by_tags."""
+
+    def __init__(self, tags):
+        self.tags = tags
+
+
+class TestFilterResultsByTags:
+    """Unit tests for the Python-side tags filter function."""
+
+    def test_no_tags_returns_all(self):
+        """When tags is None, should return all results."""
+        results = [MockResult(["a"]), MockResult(["b"]), MockResult(None)]
+        filtered = filter_results_by_tags(results, None)
+        assert len(filtered) == 3
+
+    def test_empty_tags_returns_all(self):
+        """When tags is empty list, should return all results."""
+        results = [MockResult(["a"]), MockResult(["b"]), MockResult(None)]
+        filtered = filter_results_by_tags(results, [])
+        assert len(filtered) == 3
+
+    # ---- Test "any" mode (OR, includes untagged) ----
+
+    def test_any_mode_includes_matching_tags(self):
+        """'any' mode should include results with matching tags."""
+        results = [MockResult(["a"]), MockResult(["b"]), MockResult(["c"])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="any")
+        # "a" and "b" match, "c" doesn't match and isn't untagged, so excluded
+        assert len(filtered) == 2
+        tags_found = [r.tags[0] for r in filtered if r.tags]
+        assert "a" in tags_found
+        assert "b" in tags_found
+        assert "c" not in tags_found
+
+    def test_any_mode_includes_untagged(self):
+        """'any' mode should include untagged results."""
+        results = [MockResult(["a"]), MockResult(None), MockResult([])]
+        filtered = filter_results_by_tags(results, ["a"], match="any")
+        assert len(filtered) == 3  # a matches, None is untagged, [] is untagged
+
+    def test_any_mode_includes_partial_overlap(self):
+        """'any' mode should include results with ANY overlapping tag."""
+        results = [MockResult(["a", "x"]), MockResult(["b", "y"])]
+        filtered = filter_results_by_tags(results, ["a"], match="any")
+        # ["a", "x"] matches, ["b", "y"] doesn't, but untagged would be included
+        tags_found = [r.tags for r in filtered]
+        assert ["a", "x"] in tags_found
+
+    # ---- Test "any_strict" mode (OR, excludes untagged) ----
+
+    def test_any_strict_excludes_untagged(self):
+        """'any_strict' mode should exclude untagged results."""
+        results = [MockResult(["a"]), MockResult(None), MockResult([])]
+        filtered = filter_results_by_tags(results, ["a"], match="any_strict")
+        assert len(filtered) == 1  # Only ["a"] matches
+        assert filtered[0].tags == ["a"]
+
+    def test_any_strict_excludes_non_matching(self):
+        """'any_strict' mode should exclude non-matching tagged results."""
+        results = [MockResult(["a"]), MockResult(["b"]), MockResult(["c"])]
+        filtered = filter_results_by_tags(results, ["a"], match="any_strict")
+        assert len(filtered) == 1
+        assert filtered[0].tags == ["a"]
+
+    # ---- Test "all" mode (AND, includes untagged) ----
+
+    def test_all_mode_requires_all_tags(self):
+        """'all' mode should require ALL requested tags to be present."""
+        results = [MockResult(["a", "b"]), MockResult(["a"]), MockResult(["b"])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="all")
+        # Only ["a", "b"] has both tags, but untagged would also be included
+        tags_found = [r.tags for r in filtered]
+        assert ["a", "b"] in tags_found
+
+    def test_all_mode_includes_untagged(self):
+        """'all' mode should include untagged results."""
+        results = [MockResult(["a", "b"]), MockResult(None), MockResult([])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="all")
+        assert len(filtered) == 3  # ["a", "b"] matches, None is untagged, [] is untagged
+
+    # ---- Test "all_strict" mode (AND, excludes untagged) ----
+
+    def test_all_strict_requires_all_tags(self):
+        """'all_strict' mode should require ALL requested tags."""
+        results = [MockResult(["a", "b"]), MockResult(["a"]), MockResult(["b"])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="all_strict")
+        assert len(filtered) == 1
+        assert filtered[0].tags == ["a", "b"]
+
+    def test_all_strict_excludes_untagged(self):
+        """'all_strict' mode should exclude untagged results."""
+        results = [MockResult(["a", "b"]), MockResult(None), MockResult([])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="all_strict")
+        assert len(filtered) == 1
+        assert filtered[0].tags == ["a", "b"]
+
+    def test_all_strict_allows_superset(self):
+        """'all_strict' mode should allow results with MORE tags than requested."""
+        results = [MockResult(["a", "b", "c"]), MockResult(["a"])]
+        filtered = filter_results_by_tags(results, ["a", "b"], match="all_strict")
+        assert len(filtered) == 1
+        assert filtered[0].tags == ["a", "b", "c"]  # Has a, b, AND c
+
+
+# ============================================================================
+# Integration Tests for tags in retain/recall/reflect
+# ============================================================================
+
+
+@pytest_asyncio.fixture
+async def api_client(memory):
+    """Create an async test client for the FastAPI app."""
+    app = create_app(memory, initialize_memory=False)
+    transport = httpx.ASGITransport(app=app)
+    async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
+        yield client
+
+
+@pytest.fixture
+def test_bank_id():
+    """Provide a unique bank ID for this test run."""
+    return f"tags_test_{datetime.now().timestamp()}"
+
+
+@pytest.mark.asyncio
+async def test_retain_with_tags(api_client, test_bank_id):
+    """Test that memories can be stored with tags."""
+    # Store memory with tags
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {
+                    "content": "Alice loves hiking in the mountains.",
+                    "tags": ["user_alice"]
+                }
+            ]
+        }
+    )
+    assert response.status_code == 200
+    result = response.json()
+    assert result["success"] is True
+    assert result["items_count"] == 1
+
+
+@pytest.mark.asyncio
+async def test_retain_with_document_tags(api_client, test_bank_id):
+    """Test that document-level tags are applied to all items."""
+    # Store memories with document-level tags
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "document_tags": ["session_123"],
+            "items": [
+                {"content": "Bob discussed the quarterly report."},
+                {"content": "Charlie mentioned the new product launch."}
+            ]
+        }
+    )
+    assert response.status_code == 200
+    result = response.json()
+    assert result["success"] is True
+    assert result["items_count"] == 2
+
+
+@pytest.mark.asyncio
+async def test_retain_merges_document_and_item_tags(api_client, test_bank_id):
+    """Test that document tags and item tags are merged."""
+    # Store memory with both document and item tags
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "document_tags": ["session_abc"],
+            "items": [
+                {
+                    "content": "Dave talked about machine learning.",
+                    "tags": ["user_dave"]
+                }
+            ]
+        }
+    )
+    assert response.status_code == 200
+    result = response.json()
+    assert result["success"] is True
+
+
+@pytest.mark.asyncio
+async def test_recall_without_tags_returns_all_memories(api_client, test_bank_id):
+    """Test that recall without tags returns all memories (no filtering)."""
+    # Store memories for different users
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Eve works on natural language processing.", "tags": ["user_eve"]},
+                {"content": "Frank specializes in computer vision.", "tags": ["user_frank"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Recall without tags - should return all
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories/recall",
+        json={"query": "Who works on what?", "budget": "low"}
+    )
+    assert response.status_code == 200
+    results = response.json()["results"]
+
+    # Should find both Eve and Frank
+    texts = [r["text"] for r in results]
+    assert any("Eve" in t for t in texts), "Should find Eve"
+    assert any("Frank" in t for t in texts), "Should find Frank"
+
+
+@pytest.mark.asyncio
+async def test_recall_with_tags_filters_memories(api_client, test_bank_id):
+    """Test that recall with tags only returns matching memories."""
+    # Store memories for different users
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Grace is a data scientist at Google.", "tags": ["user_grace"]},
+                {"content": "Henry is a software engineer at Meta.", "tags": ["user_henry"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Recall with user_grace tag - should only return Grace's memory
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories/recall",
+        json={"query": "Who works at which company?", "budget": "low", "tags": ["user_grace"]}
+    )
+    assert response.status_code == 200
+    results = response.json()["results"]
+
+    # Should find Grace but not Henry
+    texts = [r["text"] for r in results]
+    assert any("Grace" in t for t in texts), "Should find Grace with user_grace tag"
+    # Henry should NOT be found since he has user_henry tag
+    assert not any("Henry" in t for t in texts), "Should NOT find Henry (different tag)"
+
+
+@pytest.mark.asyncio
+async def test_recall_with_multiple_tags_uses_or_matching(api_client, test_bank_id):
+    """Test that multiple tags use OR matching (any match returns the memory)."""
+    # Store memories for different users
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Ivan leads the security team.", "tags": ["user_ivan"]},
+                {"content": "Julia manages the design team.", "tags": ["user_julia"]},
+                {"content": "Karl oversees the marketing team.", "tags": ["user_karl"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Recall with user_ivan OR user_julia - should return both Ivan and Julia, but not Karl
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories/recall",
+        json={"query": "Who leads which team?", "budget": "low", "tags": ["user_ivan", "user_julia"]}
+    )
+    assert response.status_code == 200
+    results = response.json()["results"]
+
+    texts = [r["text"] for r in results]
+    assert any("Ivan" in t for t in texts), "Should find Ivan (tag matches)"
+    assert any("Julia" in t for t in texts), "Should find Julia (tag matches)"
+    assert not any("Karl" in t for t in texts), "Should NOT find Karl (tag doesn't match)"
+
+
+@pytest.mark.asyncio
+async def test_recall_returns_memories_with_any_overlapping_tag(api_client, test_bank_id):
+    """Test that memories with multiple tags are returned if ANY tag matches."""
+    # Store memory with multiple tags
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {
+                    "content": "Lisa and Mike discussed the budget in a group chat.",
+                    "tags": ["user_lisa", "user_mike"]  # Memory visible to both
+                },
+                {"content": "Nancy reviewed the budget alone.", "tags": ["user_nancy"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Recall with user_lisa - should return the group chat memory
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories/recall",
+        json={"query": "What was discussed about the budget?", "budget": "low", "tags": ["user_lisa"]}
+    )
+    assert response.status_code == 200
+    results = response.json()["results"]
+
+    texts = [r["text"] for r in results]
+    assert any("Lisa" in t and "Mike" in t for t in texts), "Should find group chat (Lisa is in tags)"
+    assert not any("Nancy" in t for t in texts), "Should NOT find Nancy's memory"
+
+
+@pytest.mark.asyncio
+async def test_reflect_with_tags_filters_memories(api_client, test_bank_id):
+    """Test that reflect with tags only uses matching memories for reasoning."""
+    # Store different memories for different users
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Oscar's favorite color is blue.", "tags": ["user_oscar"]},
+                {"content": "Peter's favorite color is red.", "tags": ["user_peter"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Reflect with user_oscar tag - should only use Oscar's memories
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/reflect",
+        json={
+            "query": "What is the favorite color?",
+            "budget": "low",
+            "tags": ["user_oscar"],
+            "include": {"facts": {}}  # Request facts to verify what was used
+        }
+    )
+    assert response.status_code == 200
+    result = response.json()
+
+    # The response should mention Oscar's color (blue), not Peter's (red)
+    # Note: We can check based_on facts if they're returned
+    if result.get("based_on"):
+        based_on = result["based_on"]
+        memories = based_on.get("memories", []) if isinstance(based_on, dict) else []
+        fact_texts = [f["text"] for f in memories]
+        # Should use Oscar's memory (if facts are included)
+        if fact_texts:
+            assert any("Oscar" in t or "blue" in t for t in fact_texts), "Should use Oscar's memory"
+
+
+@pytest.mark.asyncio
+async def test_recall_with_empty_tags_returns_all(api_client, test_bank_id):
+    """Test that empty tags list behaves same as no tags (returns all)."""
+    # Store memories
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Quinn studies mathematics.", "tags": ["user_quinn"]},
+                {"content": "Rachel studies physics.", "tags": ["user_rachel"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Recall with empty tags list - should return all
+    response = await api_client.post(
+        f"/v1/default/banks/{test_bank_id}/memories/recall",
+        json={"query": "Who studies what?", "budget": "low", "tags": []}
+    )
+    assert response.status_code == 200
+    results = response.json()["results"]
+
+    texts = [r["text"] for r in results]
+    assert any("Quinn" in t for t in texts), "Should find Quinn"
+    assert any("Rachel" in t for t in texts), "Should find Rachel"
+
+
+@pytest.mark.asyncio
+async def test_multi_user_agent_visibility(api_client):
+    """
+    Test multi-user agent visibility scoping.
+
+    Scenario:
+    - Agent has one memory bank
+    - Agent chats with User A (room 1) and User B (room 2) separately
+    - Agent also hosts a group chat with both users (room 3)
+    - User A should only see memories from rooms 1 and 3
+    - User B should only see memories from rooms 2 and 3
+    - Agent (no filter) should see all memories
+    """
+    bank_id = f"multi_user_test_{datetime.now().timestamp()}"
+
+    # Store memories from different chat rooms
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                # Room 1: Agent + User A private chat
+                {"content": "User A said they prefer morning meetings.", "tags": ["user_a"]},
+                # Room 2: Agent + User B private chat
+                {"content": "User B mentioned they like afternoon meetings.", "tags": ["user_b"]},
+                # Room 3: Group chat with both users
+                {"content": "In the group meeting, they agreed to meet at noon.", "tags": ["user_a", "user_b"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # User A queries - should see their private chat and group chat
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories/recall",
+        json={"query": "What meeting time preferences were discussed?", "budget": "low", "tags": ["user_a"]}
+    )
+    assert response.status_code == 200
+    user_a_results = response.json()["results"]
+    user_a_texts = [r["text"] for r in user_a_results]
+
+    assert any("morning" in t for t in user_a_texts), "User A should see their own preference (morning)"
+    assert any("noon" in t for t in user_a_texts), "User A should see group chat (noon)"
+    assert not any("afternoon" in t for t in user_a_texts), "User A should NOT see User B's private preference"
+
+    # User B queries - should see their private chat and group chat
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories/recall",
+        json={"query": "What meeting time preferences were discussed?", "budget": "low", "tags": ["user_b"]}
+    )
+    assert response.status_code == 200
+    user_b_results = response.json()["results"]
+    user_b_texts = [r["text"] for r in user_b_results]
+
+    assert any("afternoon" in t for t in user_b_texts), "User B should see their own preference (afternoon)"
+    assert any("noon" in t for t in user_b_texts), "User B should see group chat (noon)"
+    assert not any("morning" in t for t in user_b_texts), "User B should NOT see User A's private preference"
+
+    # Agent queries (no filter) - should see everything
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories/recall",
+        json={"query": "What meeting time preferences were discussed?", "budget": "low"}  # No tags
+    )
+    assert response.status_code == 200
+    agent_results = response.json()["results"]
+    agent_texts = [r["text"] for r in agent_results]
+
+    assert any("morning" in t for t in agent_texts), "Agent should see User A's preference"
+    assert any("afternoon" in t for t in agent_texts), "Agent should see User B's preference"
+    assert any("noon" in t for t in agent_texts), "Agent should see group chat"
+
+
+@pytest.mark.asyncio
+async def test_student_tracking_visibility(api_client):
+    """
+    Test student tracking visibility scoping.
+
+    Scenario:
+    - Teacher bot has one memory bank
+    - Teacher records observations for Student A, Student B
+    - Student A should only see their own data
+    - Teacher (no filter) should see all student data
+    """
+    bank_id = f"student_test_{datetime.now().timestamp()}"
+
+    # Store memories for different students
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Student A showed improvement in algebra today.", "tags": ["student_a"]},
+                {"content": "Student B struggled with geometry concepts.", "tags": ["student_b"]},
+                {"content": "Student A participated actively in class discussion.", "tags": ["student_a"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # Student A queries - should only see their own data
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories/recall",
+        json={"query": "How am I doing in class?", "budget": "low", "tags": ["student_a"]}
+    )
+    assert response.status_code == 200
+    student_a_results = response.json()["results"]
+    student_a_texts = [r["text"] for r in student_a_results]
+
+    assert any("algebra" in t for t in student_a_texts), "Student A should see their algebra progress"
+    assert any("participated" in t for t in student_a_texts), "Student A should see their participation"
+    assert not any("Student B" in t or "geometry" in t for t in student_a_texts), "Student A should NOT see Student B's data"
+
+    # Teacher queries (no filter) - should see all students
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories/recall",
+        json={"query": "Which students need help?", "budget": "low"}  # No tags
+    )
+    assert response.status_code == 200
+    teacher_results = response.json()["results"]
+    teacher_texts = [r["text"] for r in teacher_results]
+
+    assert any("Student A" in t for t in teacher_texts), "Teacher should see Student A's data"
+    assert any("Student B" in t for t in teacher_texts), "Teacher should see Student B's data"
+
+
+# ============================================================================
+# Tests for list_tags API endpoint
+# ============================================================================
+
+
+@pytest.mark.asyncio
+async def test_list_tags_returns_all_tags(api_client):
+    """Test that list_tags returns all unique tags with counts."""
+    bank_id = f"list_tags_test_{datetime.now().timestamp()}"
+
+    # Store memories with various tags
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Memory 1 for user alice.", "tags": ["user:alice"]},
+                {"content": "Memory 2 for user alice.", "tags": ["user:alice"]},
+                {"content": "Memory 3 for user bob.", "tags": ["user:bob"]},
+                {"content": "Memory 4 in session 123.", "tags": ["session:123"]},
+                {"content": "Memory 5 for alice in session 456.", "tags": ["user:alice", "session:456"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List all tags
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags")
+    assert response.status_code == 200
+    result = response.json()
+
+    # Verify structure
+    assert "items" in result
+    assert "total" in result
+    assert "limit" in result
+    assert "offset" in result
+
+    # Verify tags and counts
+    tags_map = {item["tag"]: item["count"] for item in result["items"]}
+    assert "user:alice" in tags_map
+    assert tags_map["user:alice"] == 3  # 3 memories have this tag
+    assert "user:bob" in tags_map
+    assert tags_map["user:bob"] == 1
+    assert "session:123" in tags_map
+    assert tags_map["session:123"] == 1
+    assert "session:456" in tags_map
+    assert tags_map["session:456"] == 1
+
+    assert result["total"] == 4  # 4 unique tags
+
+
+@pytest.mark.asyncio
+async def test_list_tags_with_wildcard_prefix(api_client):
+    """Test that list_tags filters with prefix wildcard pattern (user:*)."""
+    bank_id = f"list_tags_wildcard_test_{datetime.now().timestamp()}"
+
+    # Store memories with various tags
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Memory for alice who works at tech.", "tags": ["user:alice"]},
+                {"content": "Memory for bob who is an engineer.", "tags": ["user:bob"]},
+                {"content": "Memory for charlie the designer.", "tags": ["user:charlie"]},
+                {"content": "Session memory about the meeting.", "tags": ["session:abc"]},
+                {"content": "Room memory for conference room.", "tags": ["room:123"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List tags with 'user:*' wildcard pattern
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"q": "user:*"})
+    assert response.status_code == 200
+    result = response.json()
+
+    # Should only return user:* tags
+    tags = [item["tag"] for item in result["items"]]
+    assert "user:alice" in tags
+    assert "user:bob" in tags
+    assert "user:charlie" in tags
+    assert "session:abc" not in tags
+    assert "room:123" not in tags
+    assert result["total"] == 3
+
+
+@pytest.mark.asyncio
+async def test_list_tags_with_wildcard_suffix(api_client):
+    """Test that list_tags filters with suffix wildcard pattern (*-admin)."""
+    bank_id = f"list_tags_suffix_test_{datetime.now().timestamp()}"
+
+    # Store memories with various tags - use meaningful content for reliable fact extraction
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "John has the role-admin permission and can manage user accounts.", "tags": ["role-admin"]},
+                {"content": "Sarah has super-admin access and can modify system settings.", "tags": ["super-admin"]},
+                {"content": "Mike is a standard role-user who can only view content.", "tags": ["role-user"]},
+                {"content": "Alice is a role-guest visitor with limited read access.", "tags": ["role-guest"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List tags with '*-admin' wildcard pattern (suffix match)
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"q": "*-admin"})
+    assert response.status_code == 200
+    result = response.json()
+
+    # Should only return *-admin tags
+    tags = [item["tag"] for item in result["items"]]
+    assert "role-admin" in tags
+    assert "super-admin" in tags
+    assert "role-user" not in tags
+    assert "role-guest" not in tags
+    assert result["total"] == 2
+
+
+@pytest.mark.asyncio
+async def test_list_tags_with_wildcard_middle(api_client):
+    """Test that list_tags filters with middle wildcard pattern (env*-prod)."""
+    bank_id = f"list_tags_middle_test_{datetime.now().timestamp()}"
+
+    # Store memories with various tags - use meaningful content for fact extraction
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "The production environment is configured with high availability and uses AWS infrastructure.", "tags": ["env-prod"]},
+                {"content": "The enterprise environment for production runs on dedicated servers with 24/7 monitoring.", "tags": ["environment-prod"]},
+                {"content": "The staging environment mirrors production but uses smaller instance sizes.", "tags": ["env-staging"]},
+                {"content": "The development environment allows developers to test their code locally.", "tags": ["env-dev"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List tags with 'env*-prod' wildcard pattern (middle match)
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"q": "env*-prod"})
+    assert response.status_code == 200
+    result = response.json()
+
+    # Should only return env*-prod tags
+    tags = [item["tag"] for item in result["items"]]
+    assert "env-prod" in tags
+    assert "environment-prod" in tags
+    assert "env-staging" not in tags
+    assert "env-dev" not in tags
+    assert result["total"] == 2
+
+
+@pytest.mark.asyncio
+async def test_list_tags_case_insensitive(api_client):
+    """Test that list_tags wildcard matching is case-insensitive."""
+    bank_id = f"list_tags_case_test_{datetime.now().timestamp()}"
+
+    # Store memories with mixed case tags - use meaningful content
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Alice is a software engineer who specializes in machine learning algorithms.", "tags": ["User:Alice"]},
+                {"content": "Bob works as a data scientist at a large technology company.", "tags": ["user:bob"]},
+                {"content": "Charlie is the lead designer responsible for the user interface.", "tags": ["USER:CHARLIE"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List tags with lowercase pattern - should match all cases
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"q": "user:*"})
+    assert response.status_code == 200
+    result = response.json()
+
+    # Should match all user tags regardless of case
+    tags = [item["tag"] for item in result["items"]]
+    assert len(tags) == 3
+    assert result["total"] == 3
+
+
+@pytest.mark.asyncio
+async def test_list_tags_pagination(api_client):
+    """Test that list_tags supports pagination."""
+    bank_id = f"list_tags_pagination_test_{datetime.now().timestamp()}"
+
+    # Store memories with many tags - use meaningful content for fact extraction
+    names = ["Alice", "Bob", "Charlie", "Diana", "Eve", "Frank", "Grace", "Henry", "Ivan", "Julia"]
+    items = [
+        {"content": f"{name} works as a software engineer at company {i}.", "tags": [f"tag:{i:03d}"]}
+        for i, name in enumerate(names)
+    ]
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={"items": items}
+    )
+    assert response.status_code == 200
+
+    # Get first page (limit 3)
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"limit": 3, "offset": 0})
+    assert response.status_code == 200
+    result = response.json()
+    assert len(result["items"]) == 3
+    assert result["total"] == 10
+    assert result["limit"] == 3
+    assert result["offset"] == 0
+
+    # Get second page
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags", params={"limit": 3, "offset": 3})
+    assert response.status_code == 200
+    result = response.json()
+    assert len(result["items"]) == 3
+    assert result["offset"] == 3
+
+
+@pytest.mark.asyncio
+async def test_list_tags_empty_bank(api_client):
+    """Test that list_tags returns empty for bank with no tags."""
+    bank_id = f"list_tags_empty_test_{datetime.now().timestamp()}"
+
+    # List tags without storing anything
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags")
+    assert response.status_code == 200
+    result = response.json()
+
+    assert result["items"] == []
+    assert result["total"] == 0
+
+
+@pytest.mark.asyncio
+async def test_list_tags_ordered_by_count(api_client):
+    """Test that list_tags returns tags ordered by frequency (most used first)."""
+    bank_id = f"list_tags_order_test_{datetime.now().timestamp()}"
+
+    # Store memories with tags having different frequencies - use meaningful content
+    response = await api_client.post(
+        f"/v1/default/banks/{bank_id}/memories",
+        json={
+            "items": [
+                {"content": "Alice works at a startup company as a developer.", "tags": ["rare"]},
+                {"content": "Bob is a senior engineer at Google.", "tags": ["common"]},
+                {"content": "Charlie manages the marketing team at Microsoft.", "tags": ["common"]},
+                {"content": "Diana leads the design department at Apple.", "tags": ["common"]},
+                {"content": "Eve is a data scientist at Amazon.", "tags": ["medium"]},
+                {"content": "Frank handles customer support at Meta.", "tags": ["medium"]},
+            ]
+        }
+    )
+    assert response.status_code == 200
+
+    # List tags - should be ordered by count descending
+    response = await api_client.get(f"/v1/default/banks/{bank_id}/tags")
+    assert response.status_code == 200
+    result = response.json()
+
+    tags = [item["tag"] for item in result["items"]]
+    # common (3) should come before medium (2) which should come before rare (1)
+    assert tags.index("common") < tags.index("medium")
+    assert tags.index("medium") < tags.index("rare")

hindsight-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "hindsight-cli"
-version = "0.2.1"
+version = "0.3.0"
 edition = "2021"
 authors = ["Hindsight Team"]
 description = "A beautiful CLI for Hindsight - semantic memory system"

hindsight-cli/src/api.rs

@@ -116,6 +116,7 @@ impl ApiClient {
         self.runtime.block_on(async {
             let request = types::CreateBankRequest {
                 name: Some(name.to_string()),
+                mission: None,
                 background: None,
                 disposition: None,
             };

hindsight-cli/src/commands/bank.rs

@@ -201,7 +201,7 @@ pub fn update_background(
         Ok(profile) => {
             if output_format == OutputFormat::Pretty {
                 ui::print_success("Background updated successfully");
-                println!("\n{}", profile.background);
+                println!("\n{}", profile.mission);
 
                 if !no_update_disposition {
                     if let (Some(old_p), Some(new_p)) =

hindsight-cli/src/commands/explore.rs

@@ -5,7 +5,7 @@ use crossterm::{
     execute,
     terminal::{disable_raw_mode, enable_raw_mode, EnterAlternateScreen, LeaveAlternateScreen},
 };
-use hindsight_client::types::{BankListItem, RecallResult, EntityListItem, Budget};
+use hindsight_client::types::{BankListItem, RecallResult, EntityListItem, Budget, TagsMatch};
 use serde_json::{Map, Value};
 use ratatui::{
     backend::{Backend, CrosstermBackend},
@@ -341,6 +341,8 @@ impl App {
                             trace: false,
                             query_timestamp: None,
                             include: None,
+                            tags: None,
+                            tags_match: TagsMatch::Any,
                         };
 
                         let result = client.recall(&bank_id, &request, false)
@@ -357,6 +359,8 @@ impl App {
                             max_tokens: 4096,
                             include: None,
                             response_schema: None,
+                            tags: None,
+                            tags_match: TagsMatch::Any,
                         };
 
                         let result = client.reflect(&bank_id, &request, false)

hindsight-cli/src/commands/memory.rs

@@ -9,7 +9,7 @@ use crate::output::{self, OutputFormat};
 use crate::ui;
 
 // Import types from generated client
-use hindsight_client::types::{Budget, ChunkIncludeOptions, IncludeOptions};
+use hindsight_client::types::{Budget, ChunkIncludeOptions, IncludeOptions, TagsMatch};
 use serde_json;
 
 // Helper function to parse budget string to Budget enum
@@ -60,6 +60,8 @@ pub fn recall(
         trace,
         query_timestamp: None,
         include,
+        tags: None,
+        tags_match: TagsMatch::Any,
     };
 
     let response = client.recall(agent_id, &request, verbose);
@@ -116,6 +118,8 @@ pub fn reflect(
         max_tokens: max_tokens.unwrap_or(4096),
         include: None,
         response_schema,
+        tags: None,
+        tags_match: TagsMatch::Any,
     };
 
     let response = client.reflect(agent_id, &request, verbose);
@@ -162,11 +166,13 @@ pub fn retain(
         timestamp: None,
         document_id: Some(doc_id.clone()),
         entities: None,
+        tags: None,
     };
 
     let request = RetainRequest {
         items: vec![item],
         async_: r#async,
+        document_tags: None,
     };
 
     let response = client.retain(agent_id, &request, r#async, verbose);
@@ -272,6 +278,7 @@ pub fn retain_files(
             timestamp: None,
             document_id: Some(doc_id),
             entities: None,
+            tags: None,
         });
 
         pb.inc(1);
@@ -288,6 +295,7 @@ pub fn retain_files(
     let request = RetainRequest {
         items,
         async_: r#async,
+        document_tags: None,
     };
 
     let response = client.retain(agent_id, &request, r#async, verbose);

hindsight-cli/src/ui.rs

@@ -172,8 +172,11 @@ pub fn print_think_response(response: &ReflectResponse) {
     println!("{}", response.text);
     println!();
 
-    if !response.based_on.is_empty() {
-        println!("{}", dim(&format!("Based on {} memory units", response.based_on.len())));
+    if let Some(based_on) = &response.based_on {
+        let count = based_on.memories.len() + based_on.mental_models.len();
+        if count > 0 {
+            println!("{}", dim(&format!("Based on {} memory units", count)));
+        }
     }
 
     // Display structured output if present
@@ -322,10 +325,10 @@ pub fn print_disposition(profile: &BankProfileResponse) {
     println!("{} {}", dim("Name:"), gradient_start(&profile.name));
     println!();
 
-    // Print background if available
-    if !profile.background.is_empty() {
-        println!("{}", gradient_mid("Background:"));
-        for line in profile.background.lines() {
+    // Print mission if available
+    if !profile.mission.is_empty() {
+        println!("{}", gradient_mid("Mission:"));
+        for line in profile.mission.lines() {
             println!("{}", line);
         }
         println!();

hindsight-clients/python/.openapi-generator/FILES

@@ -4,6 +4,7 @@ hindsight_client_api/api/banks_api.py
 hindsight_client_api/api/documents_api.py
 hindsight_client_api/api/entities_api.py
 hindsight_client_api/api/memory_api.py
+hindsight_client_api/api/mental_models_api.py
 hindsight_client_api/api/monitoring_api.py
 hindsight_client_api/api/operations_api.py
 hindsight_client_api/api_client.py
@@ -12,6 +13,7 @@ hindsight_client_api/configuration.py
 hindsight_client_api/exceptions.py
 hindsight_client_api/models/__init__.py
 hindsight_client_api/models/add_background_request.py
+hindsight_client_api/models/async_operation_submit_response.py
 hindsight_client_api/models/background_response.py
 hindsight_client_api/models/bank_list_item.py
 hindsight_client_api/models/bank_list_response.py
@@ -23,6 +25,8 @@ hindsight_client_api/models/chunk_data.py
 hindsight_client_api/models/chunk_include_options.py
 hindsight_client_api/models/chunk_response.py
 hindsight_client_api/models/create_bank_request.py
+hindsight_client_api/models/create_mental_model_request.py
+hindsight_client_api/models/created_mental_model.py
 hindsight_client_api/models/delete_document_response.py
 hindsight_client_api/models/delete_response.py
 hindsight_client_api/models/disposition_traits.py
@@ -39,19 +43,32 @@ hindsight_client_api/models/http_validation_error.py
 hindsight_client_api/models/include_options.py
 hindsight_client_api/models/list_documents_response.py
 hindsight_client_api/models/list_memory_units_response.py
+hindsight_client_api/models/list_tags_response.py
 hindsight_client_api/models/memory_item.py
+hindsight_client_api/models/mental_model_list_response.py
+hindsight_client_api/models/mental_model_observation_response.py
+hindsight_client_api/models/mental_model_response.py
 hindsight_client_api/models/operation_response.py
+hindsight_client_api/models/operation_status_response.py
 hindsight_client_api/models/operations_list_response.py
 hindsight_client_api/models/recall_request.py
 hindsight_client_api/models/recall_response.py
 hindsight_client_api/models/recall_result.py
+hindsight_client_api/models/reflect_based_on.py
 hindsight_client_api/models/reflect_fact.py
 hindsight_client_api/models/reflect_include_options.py
+hindsight_client_api/models/reflect_llm_call.py
+hindsight_client_api/models/reflect_mental_model.py
 hindsight_client_api/models/reflect_request.py
 hindsight_client_api/models/reflect_response.py
+hindsight_client_api/models/reflect_tool_call.py
+hindsight_client_api/models/reflect_trace.py
+hindsight_client_api/models/refresh_mental_models_request.py
 hindsight_client_api/models/retain_request.py
 hindsight_client_api/models/retain_response.py
+hindsight_client_api/models/tag_item.py
 hindsight_client_api/models/token_usage.py
+hindsight_client_api/models/tool_calls_include_options.py
 hindsight_client_api/models/update_disposition_request.py
 hindsight_client_api/models/validation_error.py
 hindsight_client_api/models/validation_error_loc_inner.py

hindsight-clients/python/hindsight_client_api/__init__.py

@@ -21,6 +21,7 @@ from hindsight_client_api.api.banks_api import BanksApi
 from hindsight_client_api.api.documents_api import DocumentsApi
 from hindsight_client_api.api.entities_api import EntitiesApi
 from hindsight_client_api.api.memory_api import MemoryApi
+from hindsight_client_api.api.mental_models_api import MentalModelsApi
 from hindsight_client_api.api.monitoring_api import MonitoringApi
 from hindsight_client_api.api.operations_api import OperationsApi
 
@@ -37,6 +38,7 @@ from hindsight_client_api.exceptions import ApiException
 
 # import models into sdk package
 from hindsight_client_api.models.add_background_request import AddBackgroundRequest
+from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
 from hindsight_client_api.models.background_response import BackgroundResponse
 from hindsight_client_api.models.bank_list_item import BankListItem
 from hindsight_client_api.models.bank_list_response import BankListResponse
@@ -48,6 +50,8 @@ from hindsight_client_api.models.chunk_data import ChunkData
 from hindsight_client_api.models.chunk_include_options import ChunkIncludeOptions
 from hindsight_client_api.models.chunk_response import ChunkResponse
 from hindsight_client_api.models.create_bank_request import CreateBankRequest
+from hindsight_client_api.models.create_mental_model_request import CreateMentalModelRequest
+from hindsight_client_api.models.created_mental_model import CreatedMentalModel
 from hindsight_client_api.models.delete_document_response import DeleteDocumentResponse
 from hindsight_client_api.models.delete_response import DeleteResponse
 from hindsight_client_api.models.disposition_traits import DispositionTraits
@@ -64,19 +68,32 @@ from hindsight_client_api.models.http_validation_error import HTTPValidationErro
 from hindsight_client_api.models.include_options import IncludeOptions
 from hindsight_client_api.models.list_documents_response import ListDocumentsResponse
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
+from hindsight_client_api.models.list_tags_response import ListTagsResponse
 from hindsight_client_api.models.memory_item import MemoryItem
+from hindsight_client_api.models.mental_model_list_response import MentalModelListResponse
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
 from hindsight_client_api.models.operation_response import OperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 from hindsight_client_api.models.recall_request import RecallRequest
 from hindsight_client_api.models.recall_response import RecallResponse
 from hindsight_client_api.models.recall_result import RecallResult
+from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
 from hindsight_client_api.models.reflect_fact import ReflectFact
 from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
+from hindsight_client_api.models.reflect_llm_call import ReflectLLMCall
+from hindsight_client_api.models.reflect_mental_model import ReflectMentalModel
 from hindsight_client_api.models.reflect_request import ReflectRequest
 from hindsight_client_api.models.reflect_response import ReflectResponse
+from hindsight_client_api.models.reflect_tool_call import ReflectToolCall
+from hindsight_client_api.models.reflect_trace import ReflectTrace
+from hindsight_client_api.models.refresh_mental_models_request import RefreshMentalModelsRequest
 from hindsight_client_api.models.retain_request import RetainRequest
 from hindsight_client_api.models.retain_response import RetainResponse
+from hindsight_client_api.models.tag_item import TagItem
 from hindsight_client_api.models.token_usage import TokenUsage
+from hindsight_client_api.models.tool_calls_include_options import ToolCallsIncludeOptions
 from hindsight_client_api.models.update_disposition_request import UpdateDispositionRequest
 from hindsight_client_api.models.validation_error import ValidationError
 from hindsight_client_api.models.validation_error_loc_inner import ValidationErrorLocInner

hindsight-clients/python/hindsight_client_api/api/__init__.py

@@ -5,6 +5,7 @@ from hindsight_client_api.api.banks_api import BanksApi
 from hindsight_client_api.api.documents_api import DocumentsApi
 from hindsight_client_api.api.entities_api import EntitiesApi
 from hindsight_client_api.api.memory_api import MemoryApi
+from hindsight_client_api.api.mental_models_api import MentalModelsApi
 from hindsight_client_api.api.monitoring_api import MonitoringApi
 from hindsight_client_api.api.operations_api import OperationsApi
 

hindsight-clients/python/hindsight_client_api/api/banks_api.py

@@ -64,9 +64,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> BackgroundResponse:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -95,6 +95,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -140,9 +141,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> ApiResponse[BackgroundResponse]:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -171,6 +172,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -216,9 +218,9 @@ class BanksApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> RESTResponseType:
-        """Add/merge memory bank background
+        """(Deprecated) Add/merge memory bank background (deprecated)
 
-        Add new background information or merge with existing. LLM intelligently resolves conflicts, normalizes to first person, and optionally infers disposition traits.
+        Deprecated: Use PUT /mission instead. This endpoint now updates the mission field.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -247,6 +249,7 @@ class BanksApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/background is deprecated.", DeprecationWarning)
 
         _param = self._add_bank_background_serialize(
             bank_id=bank_id,
@@ -372,7 +375,7 @@ class BanksApi:
     ) -> BankProfileResponse:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -448,7 +451,7 @@ class BanksApi:
     ) -> ApiResponse[BankProfileResponse]:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -524,7 +527,7 @@ class BanksApi:
     ) -> RESTResponseType:
         """Create or update memory bank
 
-        Create a new agent or update existing agent with disposition and background. Auto-fills missing fields with defaults.
+        Create a new agent or update existing agent with disposition and mission. Auto-fills missing fields with defaults.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1233,7 +1236,7 @@ class BanksApi:
     ) -> BankProfileResponse:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1305,7 +1308,7 @@ class BanksApi:
     ) -> ApiResponse[BankProfileResponse]:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1377,7 +1380,7 @@ class BanksApi:
     ) -> RESTResponseType:
         """Get memory bank profile
 
-        Get disposition traits and background for a memory bank. Auto-creates agent with defaults if not exists.
+        Get disposition traits and mission for a memory bank. Auto-creates agent with defaults if not exists.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -1754,6 +1757,312 @@ class BanksApi:
 
 
 
+    @validate_call
+    async def update_bank(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> BankProfileResponse:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def update_bank_with_http_info(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[BankProfileResponse]:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def update_bank_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        create_bank_request: CreateBankRequest,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """Partial update memory bank
+
+        Partially update an agent's profile. Only provided fields will be updated.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param create_bank_request: (required)
+        :type create_bank_request: CreateBankRequest
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._update_bank_serialize(
+            bank_id=bank_id,
+            create_bank_request=create_bank_request,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "BankProfileResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _update_bank_serialize(
+        self,
+        bank_id,
+        create_bank_request,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        # process the query parameters
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+        if create_bank_request is not None:
+            _body_params = create_bank_request
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+        # set the HTTP header `Content-Type`
+        if _content_type:
+            _header_params['Content-Type'] = _content_type
+        else:
+            _default_content_type = (
+                self.api_client.select_header_content_type(
+                    [
+                        'application/json'
+                    ]
+                )
+            )
+            if _default_content_type is not None:
+                _header_params['Content-Type'] = _default_content_type
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='PATCH',
+            resource_path='/v1/default/banks/{bank_id}',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def update_bank_disposition(
         self,

hindsight-clients/python/hindsight_client_api/api/entities_api.py

@@ -664,9 +664,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> EntityDetailResponse:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -695,6 +695,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,
@@ -740,9 +741,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> ApiResponse[EntityDetailResponse]:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -771,6 +772,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,
@@ -816,9 +818,9 @@ class EntitiesApi:
         _headers: Optional[Dict[StrictStr, Any]] = None,
         _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
     ) -> RESTResponseType:
-        """Regenerate entity observations
+        """(Deprecated) Regenerate entity observations (deprecated)
 
-        Regenerate observations for an entity based on all facts mentioning it.
+        This endpoint is deprecated. Entity observations have been replaced by mental models.
 
         :param bank_id: (required)
         :type bank_id: str
@@ -847,6 +849,7 @@ class EntitiesApi:
         :type _host_index: int, optional
         :return: Returns the result object.
         """ # noqa: E501
+        warnings.warn("POST /v1/default/banks/{bank_id}/entities/{entity_id}/regenerate is deprecated.", DeprecationWarning)
 
         _param = self._regenerate_entity_observations_serialize(
             bank_id=bank_id,

hindsight-clients/python/hindsight_client_api/api/memory_api.py

@@ -17,11 +17,12 @@ from typing import Any, Dict, List, Optional, Tuple, Union
 from typing_extensions import Annotated
 
 from pydantic import Field, StrictInt, StrictStr
-from typing import Optional
+from typing import Any, Optional
 from typing_extensions import Annotated
 from hindsight_client_api.models.delete_response import DeleteResponse
 from hindsight_client_api.models.graph_data_response import GraphDataResponse
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
+from hindsight_client_api.models.list_tags_response import ListTagsResponse
 from hindsight_client_api.models.recall_request import RecallRequest
 from hindsight_client_api.models.recall_response import RecallResponse
 from hindsight_client_api.models.reflect_request import ReflectRequest
@@ -654,6 +655,299 @@ class MemoryApi:
 
 
 
+    @validate_call
+    async def get_memory(
+        self,
+        bank_id: StrictStr,
+        memory_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> object:
+        """Get memory unit
+
+        Get a single memory unit by ID with all its metadata including entities and tags.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param memory_id: (required)
+        :type memory_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_memory_serialize(
+            bank_id=bank_id,
+            memory_id=memory_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "object",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def get_memory_with_http_info(
+        self,
+        bank_id: StrictStr,
+        memory_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[object]:
+        """Get memory unit
+
+        Get a single memory unit by ID with all its metadata including entities and tags.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param memory_id: (required)
+        :type memory_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_memory_serialize(
+            bank_id=bank_id,
+            memory_id=memory_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "object",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def get_memory_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        memory_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """Get memory unit
+
+        Get a single memory unit by ID with all its metadata including entities and tags.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param memory_id: (required)
+        :type memory_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_memory_serialize(
+            bank_id=bank_id,
+            memory_id=memory_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "object",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _get_memory_serialize(
+        self,
+        bank_id,
+        memory_id,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        if memory_id is not None:
+            _path_params['memory_id'] = memory_id
+        # process the query parameters
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='GET',
+            resource_path='/v1/default/banks/{bank_id}/memories/{memory_id}',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def list_memories(
         self,
@@ -1000,6 +1294,335 @@ class MemoryApi:
 
 
 
+    @validate_call
+    async def list_tags(
+        self,
+        bank_id: StrictStr,
+        q: Annotated[Optional[StrictStr], Field(description="Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.")] = None,
+        limit: Annotated[Optional[StrictInt], Field(description="Maximum number of tags to return")] = None,
+        offset: Annotated[Optional[StrictInt], Field(description="Offset for pagination")] = None,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ListTagsResponse:
+        """List tags
+
+        List all unique tags in a memory bank with usage counts. Supports wildcard search using '*' (e.g., 'user:*', '*-fred', 'tag*-2'). Case-insensitive.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param q: Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.
+        :type q: str
+        :param limit: Maximum number of tags to return
+        :type limit: int
+        :param offset: Offset for pagination
+        :type offset: int
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._list_tags_serialize(
+            bank_id=bank_id,
+            q=q,
+            limit=limit,
+            offset=offset,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "ListTagsResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def list_tags_with_http_info(
+        self,
+        bank_id: StrictStr,
+        q: Annotated[Optional[StrictStr], Field(description="Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.")] = None,
+        limit: Annotated[Optional[StrictInt], Field(description="Maximum number of tags to return")] = None,
+        offset: Annotated[Optional[StrictInt], Field(description="Offset for pagination")] = None,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[ListTagsResponse]:
+        """List tags
+
+        List all unique tags in a memory bank with usage counts. Supports wildcard search using '*' (e.g., 'user:*', '*-fred', 'tag*-2'). Case-insensitive.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param q: Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.
+        :type q: str
+        :param limit: Maximum number of tags to return
+        :type limit: int
+        :param offset: Offset for pagination
+        :type offset: int
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._list_tags_serialize(
+            bank_id=bank_id,
+            q=q,
+            limit=limit,
+            offset=offset,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "ListTagsResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def list_tags_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        q: Annotated[Optional[StrictStr], Field(description="Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.")] = None,
+        limit: Annotated[Optional[StrictInt], Field(description="Maximum number of tags to return")] = None,
+        offset: Annotated[Optional[StrictInt], Field(description="Offset for pagination")] = None,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """List tags
+
+        List all unique tags in a memory bank with usage counts. Supports wildcard search using '*' (e.g., 'user:*', '*-fred', 'tag*-2'). Case-insensitive.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param q: Wildcard pattern to filter tags (e.g., 'user:*' for user:alice, '*-admin' for role-admin). Use '*' as wildcard. Case-insensitive.
+        :type q: str
+        :param limit: Maximum number of tags to return
+        :type limit: int
+        :param offset: Offset for pagination
+        :type offset: int
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._list_tags_serialize(
+            bank_id=bank_id,
+            q=q,
+            limit=limit,
+            offset=offset,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "ListTagsResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _list_tags_serialize(
+        self,
+        bank_id,
+        q,
+        limit,
+        offset,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        # process the query parameters
+        if q is not None:
+            
+            _query_params.append(('q', q))
+            
+        if limit is not None:
+            
+            _query_params.append(('limit', limit))
+            
+        if offset is not None:
+            
+            _query_params.append(('offset', offset))
+            
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='GET',
+            resource_path='/v1/default/banks/{bank_id}/tags',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def recall_memories(
         self,

hindsight-clients/python/hindsight_client_api/api/operations_api.py

@@ -19,6 +19,7 @@ from typing_extensions import Annotated
 from pydantic import StrictStr
 from typing import Optional
 from hindsight_client_api.models.cancel_operation_response import CancelOperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 
 from hindsight_client_api.api_client import ApiClient, RequestSerialized
@@ -332,6 +333,299 @@ class OperationsApi:
 
 
 
+    @validate_call
+    async def get_operation_status(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> OperationStatusResponse:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        ).data
+
+
+    @validate_call
+    async def get_operation_status_with_http_info(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> ApiResponse[OperationStatusResponse]:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        await response_data.read()
+        return self.api_client.response_deserialize(
+            response_data=response_data,
+            response_types_map=_response_types_map,
+        )
+
+
+    @validate_call
+    async def get_operation_status_without_preload_content(
+        self,
+        bank_id: StrictStr,
+        operation_id: StrictStr,
+        authorization: Optional[StrictStr] = None,
+        _request_timeout: Union[
+            None,
+            Annotated[StrictFloat, Field(gt=0)],
+            Tuple[
+                Annotated[StrictFloat, Field(gt=0)],
+                Annotated[StrictFloat, Field(gt=0)]
+            ]
+        ] = None,
+        _request_auth: Optional[Dict[StrictStr, Any]] = None,
+        _content_type: Optional[StrictStr] = None,
+        _headers: Optional[Dict[StrictStr, Any]] = None,
+        _host_index: Annotated[StrictInt, Field(ge=0, le=0)] = 0,
+    ) -> RESTResponseType:
+        """Get operation status
+
+        Get the status of a specific async operation. Returns 'pending', 'completed', or 'failed'. Completed operations are removed from storage, so 'completed' means the operation finished successfully.
+
+        :param bank_id: (required)
+        :type bank_id: str
+        :param operation_id: (required)
+        :type operation_id: str
+        :param authorization:
+        :type authorization: str
+        :param _request_timeout: timeout setting for this request. If one
+                                 number provided, it will be total request
+                                 timeout. It can also be a pair (tuple) of
+                                 (connection, read) timeouts.
+        :type _request_timeout: int, tuple(int, int), optional
+        :param _request_auth: set to override the auth_settings for an a single
+                              request; this effectively ignores the
+                              authentication in the spec for a single request.
+        :type _request_auth: dict, optional
+        :param _content_type: force content-type for the request.
+        :type _content_type: str, Optional
+        :param _headers: set to override the headers for a single
+                         request; this effectively ignores the headers
+                         in the spec for a single request.
+        :type _headers: dict, optional
+        :param _host_index: set to override the host_index for a single
+                            request; this effectively ignores the host_index
+                            in the spec for a single request.
+        :type _host_index: int, optional
+        :return: Returns the result object.
+        """ # noqa: E501
+
+        _param = self._get_operation_status_serialize(
+            bank_id=bank_id,
+            operation_id=operation_id,
+            authorization=authorization,
+            _request_auth=_request_auth,
+            _content_type=_content_type,
+            _headers=_headers,
+            _host_index=_host_index
+        )
+
+        _response_types_map: Dict[str, Optional[str]] = {
+            '200': "OperationStatusResponse",
+            '422': "HTTPValidationError",
+        }
+        response_data = await self.api_client.call_api(
+            *_param,
+            _request_timeout=_request_timeout
+        )
+        return response_data.response
+
+
+    def _get_operation_status_serialize(
+        self,
+        bank_id,
+        operation_id,
+        authorization,
+        _request_auth,
+        _content_type,
+        _headers,
+        _host_index,
+    ) -> RequestSerialized:
+
+        _host = None
+
+        _collection_formats: Dict[str, str] = {
+        }
+
+        _path_params: Dict[str, str] = {}
+        _query_params: List[Tuple[str, str]] = []
+        _header_params: Dict[str, Optional[str]] = _headers or {}
+        _form_params: List[Tuple[str, str]] = []
+        _files: Dict[
+            str, Union[str, bytes, List[str], List[bytes], List[Tuple[str, bytes]]]
+        ] = {}
+        _body_params: Optional[bytes] = None
+
+        # process the path parameters
+        if bank_id is not None:
+            _path_params['bank_id'] = bank_id
+        if operation_id is not None:
+            _path_params['operation_id'] = operation_id
+        # process the query parameters
+        # process the header parameters
+        if authorization is not None:
+            _header_params['authorization'] = authorization
+        # process the form parameters
+        # process the body parameter
+
+
+        # set the HTTP header `Accept`
+        if 'Accept' not in _header_params:
+            _header_params['Accept'] = self.api_client.select_header_accept(
+                [
+                    'application/json'
+                ]
+            )
+
+
+        # authentication setting
+        _auth_settings: List[str] = [
+        ]
+
+        return self.api_client.param_serialize(
+            method='GET',
+            resource_path='/v1/default/banks/{bank_id}/operations/{operation_id}',
+            path_params=_path_params,
+            query_params=_query_params,
+            header_params=_header_params,
+            body=_body_params,
+            post_params=_form_params,
+            files=_files,
+            auth_settings=_auth_settings,
+            collection_formats=_collection_formats,
+            _host=_host,
+            _request_auth=_request_auth
+        )
+
+
+
+
     @validate_call
     async def list_operations(
         self,

hindsight-clients/python/hindsight_client_api/models/__init__.py

@@ -15,6 +15,7 @@
 
 # import models into model package
 from hindsight_client_api.models.add_background_request import AddBackgroundRequest
+from hindsight_client_api.models.async_operation_submit_response import AsyncOperationSubmitResponse
 from hindsight_client_api.models.background_response import BackgroundResponse
 from hindsight_client_api.models.bank_list_item import BankListItem
 from hindsight_client_api.models.bank_list_response import BankListResponse
@@ -26,6 +27,8 @@ from hindsight_client_api.models.chunk_data import ChunkData
 from hindsight_client_api.models.chunk_include_options import ChunkIncludeOptions
 from hindsight_client_api.models.chunk_response import ChunkResponse
 from hindsight_client_api.models.create_bank_request import CreateBankRequest
+from hindsight_client_api.models.create_mental_model_request import CreateMentalModelRequest
+from hindsight_client_api.models.created_mental_model import CreatedMentalModel
 from hindsight_client_api.models.delete_document_response import DeleteDocumentResponse
 from hindsight_client_api.models.delete_response import DeleteResponse
 from hindsight_client_api.models.disposition_traits import DispositionTraits
@@ -42,19 +45,32 @@ from hindsight_client_api.models.http_validation_error import HTTPValidationErro
 from hindsight_client_api.models.include_options import IncludeOptions
 from hindsight_client_api.models.list_documents_response import ListDocumentsResponse
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
+from hindsight_client_api.models.list_tags_response import ListTagsResponse
 from hindsight_client_api.models.memory_item import MemoryItem
+from hindsight_client_api.models.mental_model_list_response import MentalModelListResponse
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
 from hindsight_client_api.models.operation_response import OperationResponse
+from hindsight_client_api.models.operation_status_response import OperationStatusResponse
 from hindsight_client_api.models.operations_list_response import OperationsListResponse
 from hindsight_client_api.models.recall_request import RecallRequest
 from hindsight_client_api.models.recall_response import RecallResponse
 from hindsight_client_api.models.recall_result import RecallResult
+from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
 from hindsight_client_api.models.reflect_fact import ReflectFact
 from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
+from hindsight_client_api.models.reflect_llm_call import ReflectLLMCall
+from hindsight_client_api.models.reflect_mental_model import ReflectMentalModel
 from hindsight_client_api.models.reflect_request import ReflectRequest
 from hindsight_client_api.models.reflect_response import ReflectResponse
+from hindsight_client_api.models.reflect_tool_call import ReflectToolCall
+from hindsight_client_api.models.reflect_trace import ReflectTrace
+from hindsight_client_api.models.refresh_mental_models_request import RefreshMentalModelsRequest
 from hindsight_client_api.models.retain_request import RetainRequest
 from hindsight_client_api.models.retain_response import RetainResponse
+from hindsight_client_api.models.tag_item import TagItem
 from hindsight_client_api.models.token_usage import TokenUsage
+from hindsight_client_api.models.tool_calls_include_options import ToolCallsIncludeOptions
 from hindsight_client_api.models.update_disposition_request import UpdateDispositionRequest
 from hindsight_client_api.models.validation_error import ValidationError
 from hindsight_client_api.models.validation_error_loc_inner import ValidationErrorLocInner

hindsight-clients/python/hindsight_client_api/models/add_background_request.py

@@ -24,10 +24,10 @@ from typing_extensions import Self
 
 class AddBackgroundRequest(BaseModel):
     """
-    Request model for adding/merging background information.
+    Request model for adding/merging background information. Deprecated: use SetMissionRequest instead.
     """ # noqa: E501
     content: StrictStr = Field(description="New background information to add or merge")
-    update_disposition: Optional[StrictBool] = Field(default=True, description="If true, infer disposition traits from the merged background (default: true)")
+    update_disposition: Optional[StrictBool] = Field(default=True, description="Deprecated - disposition is no longer auto-inferred from mission")
     __properties: ClassVar[List[str]] = ["content", "update_disposition"]
 
     model_config = ConfigDict(

hindsight-clients/python/hindsight_client_api/models/async_operation_submit_response.py

@@ -0,0 +1,89 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class AsyncOperationSubmitResponse(BaseModel):
+    """
+    Response model for submitting an async operation.
+    """ # noqa: E501
+    operation_id: StrictStr
+    status: StrictStr
+    __properties: ClassVar[List[str]] = ["operation_id", "status"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of AsyncOperationSubmitResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of AsyncOperationSubmitResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "operation_id": obj.get("operation_id"),
+            "status": obj.get("status")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/background_response.py

@@ -25,11 +25,12 @@ from typing_extensions import Self
 
 class BackgroundResponse(BaseModel):
     """
-    Response model for background update.
+    Response model for background update. Deprecated: use MissionResponse instead.
     """ # noqa: E501
-    background: StrictStr
+    mission: StrictStr
+    background: Optional[StrictStr] = None
     disposition: Optional[DispositionTraits] = None
-    __properties: ClassVar[List[str]] = ["background", "disposition"]
+    __properties: ClassVar[List[str]] = ["mission", "background", "disposition"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -73,6 +74,11 @@ class BackgroundResponse(BaseModel):
         # override the default output from pydantic by calling `to_dict()` of disposition
         if self.disposition:
             _dict['disposition'] = self.disposition.to_dict()
+        # set to None if background (nullable) is None
+        # and model_fields_set contains the field
+        if self.background is None and "background" in self.model_fields_set:
+            _dict['background'] = None
+
         # set to None if disposition (nullable) is None
         # and model_fields_set contains the field
         if self.disposition is None and "disposition" in self.model_fields_set:
@@ -90,6 +96,7 @@ class BackgroundResponse(BaseModel):
             return cls.model_validate(obj)
 
         _obj = cls.model_validate({
+            "mission": obj.get("mission"),
             "background": obj.get("background"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None
         })

hindsight-clients/python/hindsight_client_api/models/bank_list_item.py

@@ -30,10 +30,10 @@ class BankListItem(BaseModel):
     bank_id: StrictStr
     name: Optional[StrictStr] = None
     disposition: DispositionTraits
-    background: Optional[StrictStr] = None
+    mission: Optional[StrictStr] = None
     created_at: Optional[StrictStr] = None
     updated_at: Optional[StrictStr] = None
-    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "background", "created_at", "updated_at"]
+    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "mission", "created_at", "updated_at"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -82,10 +82,10 @@ class BankListItem(BaseModel):
         if self.name is None and "name" in self.model_fields_set:
             _dict['name'] = None
 
-        # set to None if background (nullable) is None
+        # set to None if mission (nullable) is None
         # and model_fields_set contains the field
-        if self.background is None and "background" in self.model_fields_set:
-            _dict['background'] = None
+        if self.mission is None and "mission" in self.model_fields_set:
+            _dict['mission'] = None
 
         # set to None if created_at (nullable) is None
         # and model_fields_set contains the field
@@ -112,7 +112,7 @@ class BankListItem(BaseModel):
             "bank_id": obj.get("bank_id"),
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
-            "background": obj.get("background"),
+            "mission": obj.get("mission"),
             "created_at": obj.get("created_at"),
             "updated_at": obj.get("updated_at")
         })

hindsight-clients/python/hindsight_client_api/models/bank_profile_response.py

@@ -17,8 +17,8 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictStr
-from typing import Any, ClassVar, Dict, List
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
 from hindsight_client_api.models.disposition_traits import DispositionTraits
 from typing import Optional, Set
 from typing_extensions import Self
@@ -30,8 +30,9 @@ class BankProfileResponse(BaseModel):
     bank_id: StrictStr
     name: StrictStr
     disposition: DispositionTraits
-    background: StrictStr
-    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "background"]
+    mission: StrictStr = Field(description="The agent's mission - who they are and what they're trying to accomplish")
+    background: Optional[StrictStr] = None
+    __properties: ClassVar[List[str]] = ["bank_id", "name", "disposition", "mission", "background"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -75,6 +76,11 @@ class BankProfileResponse(BaseModel):
         # override the default output from pydantic by calling `to_dict()` of disposition
         if self.disposition:
             _dict['disposition'] = self.disposition.to_dict()
+        # set to None if background (nullable) is None
+        # and model_fields_set contains the field
+        if self.background is None and "background" in self.model_fields_set:
+            _dict['background'] = None
+
         return _dict
 
     @classmethod
@@ -90,6 +96,7 @@ class BankProfileResponse(BaseModel):
             "bank_id": obj.get("bank_id"),
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
+            "mission": obj.get("mission"),
             "background": obj.get("background")
         })
         return _obj

hindsight-clients/python/hindsight_client_api/models/create_bank_request.py

@@ -29,8 +29,9 @@ class CreateBankRequest(BaseModel):
     """ # noqa: E501
     name: Optional[StrictStr] = None
     disposition: Optional[DispositionTraits] = None
+    mission: Optional[StrictStr] = None
     background: Optional[StrictStr] = None
-    __properties: ClassVar[List[str]] = ["name", "disposition", "background"]
+    __properties: ClassVar[List[str]] = ["name", "disposition", "mission", "background"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -84,6 +85,11 @@ class CreateBankRequest(BaseModel):
         if self.disposition is None and "disposition" in self.model_fields_set:
             _dict['disposition'] = None
 
+        # set to None if mission (nullable) is None
+        # and model_fields_set contains the field
+        if self.mission is None and "mission" in self.model_fields_set:
+            _dict['mission'] = None
+
         # set to None if background (nullable) is None
         # and model_fields_set contains the field
         if self.background is None and "background" in self.model_fields_set:
@@ -103,6 +109,7 @@ class CreateBankRequest(BaseModel):
         _obj = cls.model_validate({
             "name": obj.get("name"),
             "disposition": DispositionTraits.from_dict(obj["disposition"]) if obj.get("disposition") is not None else None,
+            "mission": obj.get("mission"),
             "background": obj.get("background")
         })
         return _obj

hindsight-clients/python/hindsight_client_api/models/create_mental_model_request.py

@@ -0,0 +1,91 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class CreateMentalModelRequest(BaseModel):
+    """
+    Request model for creating a pinned mental model.
+    """ # noqa: E501
+    name: StrictStr = Field(description="Human-readable name for the mental model")
+    description: StrictStr = Field(description="One-liner description for quick scanning")
+    tags: Optional[List[StrictStr]] = Field(default=None, description="Tags for scoped visibility")
+    __properties: ClassVar[List[str]] = ["name", "description", "tags"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of CreateMentalModelRequest from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of CreateMentalModelRequest from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "name": obj.get("name"),
+            "description": obj.get("description"),
+            "tags": obj.get("tags")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/created_mental_model.py

@@ -0,0 +1,91 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class CreatedMentalModel(BaseModel):
+    """
+    A mental model created during reflection.
+    """ # noqa: E501
+    id: StrictStr = Field(description="Mental model ID")
+    name: StrictStr = Field(description="Human-readable name")
+    description: StrictStr = Field(description="What this model tracks")
+    __properties: ClassVar[List[str]] = ["id", "name", "description"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of CreatedMentalModel from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of CreatedMentalModel from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "id": obj.get("id"),
+            "name": obj.get("name"),
+            "description": obj.get("description")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/document_response.py

@@ -17,7 +17,7 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictInt, StrictStr
+from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr
 from typing import Any, ClassVar, Dict, List, Optional
 from typing import Optional, Set
 from typing_extensions import Self
@@ -33,7 +33,8 @@ class DocumentResponse(BaseModel):
     created_at: StrictStr
     updated_at: StrictStr
     memory_unit_count: StrictInt
-    __properties: ClassVar[List[str]] = ["id", "bank_id", "original_text", "content_hash", "created_at", "updated_at", "memory_unit_count"]
+    tags: Optional[List[StrictStr]] = Field(default=None, description="Tags associated with this document")
+    __properties: ClassVar[List[str]] = ["id", "bank_id", "original_text", "content_hash", "created_at", "updated_at", "memory_unit_count", "tags"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -97,7 +98,8 @@ class DocumentResponse(BaseModel):
             "content_hash": obj.get("content_hash"),
             "created_at": obj.get("created_at"),
             "updated_at": obj.get("updated_at"),
-            "memory_unit_count": obj.get("memory_unit_count")
+            "memory_unit_count": obj.get("memory_unit_count"),
+            "tags": obj.get("tags")
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/list_tags_response.py

@@ -0,0 +1,101 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, StrictInt
+from typing import Any, ClassVar, Dict, List
+from hindsight_client_api.models.tag_item import TagItem
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ListTagsResponse(BaseModel):
+    """
+    Response model for list tags endpoint.
+    """ # noqa: E501
+    items: List[TagItem]
+    total: StrictInt
+    limit: StrictInt
+    offset: StrictInt
+    __properties: ClassVar[List[str]] = ["items", "total", "limit", "offset"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ListTagsResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in items (list)
+        _items = []
+        if self.items:
+            for _item_items in self.items:
+                if _item_items:
+                    _items.append(_item_items.to_dict())
+            _dict['items'] = _items
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ListTagsResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "items": [TagItem.from_dict(_item) for _item in obj["items"]] if obj.get("items") is not None else None,
+            "total": obj.get("total"),
+            "limit": obj.get("limit"),
+            "offset": obj.get("offset")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/memory_item.py

@@ -34,7 +34,8 @@ class MemoryItem(BaseModel):
     metadata: Optional[Dict[str, StrictStr]] = None
     document_id: Optional[StrictStr] = None
     entities: Optional[List[EntityInput]] = None
-    __properties: ClassVar[List[str]] = ["content", "timestamp", "context", "metadata", "document_id", "entities"]
+    tags: Optional[List[StrictStr]] = None
+    __properties: ClassVar[List[str]] = ["content", "timestamp", "context", "metadata", "document_id", "entities", "tags"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -107,6 +108,11 @@ class MemoryItem(BaseModel):
         if self.entities is None and "entities" in self.model_fields_set:
             _dict['entities'] = None
 
+        # set to None if tags (nullable) is None
+        # and model_fields_set contains the field
+        if self.tags is None and "tags" in self.model_fields_set:
+            _dict['tags'] = None
+
         return _dict
 
     @classmethod
@@ -124,7 +130,8 @@ class MemoryItem(BaseModel):
             "context": obj.get("context"),
             "metadata": obj.get("metadata"),
             "document_id": obj.get("document_id"),
-            "entities": [EntityInput.from_dict(_item) for _item in obj["entities"]] if obj.get("entities") is not None else None
+            "entities": [EntityInput.from_dict(_item) for _item in obj["entities"]] if obj.get("entities") is not None else None,
+            "tags": obj.get("tags")
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/mental_model_list_response.py

@@ -0,0 +1,95 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict
+from typing import Any, ClassVar, Dict, List
+from hindsight_client_api.models.mental_model_response import MentalModelResponse
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelListResponse(BaseModel):
+    """
+    Response model for listing mental models.
+    """ # noqa: E501
+    items: List[MentalModelResponse]
+    __properties: ClassVar[List[str]] = ["items"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelListResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in items (list)
+        _items = []
+        if self.items:
+            for _item_items in self.items:
+                if _item_items:
+                    _items.append(_item_items.to_dict())
+            _dict['items'] = _items
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelListResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "items": [MentalModelResponse.from_dict(_item) for _item in obj["items"]] if obj.get("items") is not None else None
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_observation_response.py

@@ -0,0 +1,91 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelObservationResponse(BaseModel):
+    """
+    An observation within a mental model with its supporting memories.
+    """ # noqa: E501
+    title: StrictStr = Field(description="Observation header (empty for intro)")
+    text: StrictStr = Field(description="Observation content")
+    based_on: Optional[List[StrictStr]] = Field(default=None, description="Memory IDs supporting this observation")
+    __properties: ClassVar[List[str]] = ["title", "text", "based_on"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelObservationResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelObservationResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "title": obj.get("title"),
+            "text": obj.get("text"),
+            "based_on": obj.get("based_on")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/mental_model_response.py

@@ -0,0 +1,125 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.mental_model_observation_response import MentalModelObservationResponse
+from typing import Optional, Set
+from typing_extensions import Self
+
+class MentalModelResponse(BaseModel):
+    """
+    Response model for a mental model.
+    """ # noqa: E501
+    id: StrictStr
+    bank_id: StrictStr
+    subtype: StrictStr
+    name: StrictStr
+    description: StrictStr
+    observations: Optional[List[MentalModelObservationResponse]] = Field(default=None, description="Structured observations with per-observation fact attribution")
+    entity_id: Optional[StrictStr] = None
+    links: Optional[List[StrictStr]] = None
+    tags: Optional[List[StrictStr]] = None
+    last_updated: Optional[StrictStr] = None
+    created_at: StrictStr
+    __properties: ClassVar[List[str]] = ["id", "bank_id", "subtype", "name", "description", "observations", "entity_id", "links", "tags", "last_updated", "created_at"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of MentalModelResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in observations (list)
+        _items = []
+        if self.observations:
+            for _item_observations in self.observations:
+                if _item_observations:
+                    _items.append(_item_observations.to_dict())
+            _dict['observations'] = _items
+        # set to None if entity_id (nullable) is None
+        # and model_fields_set contains the field
+        if self.entity_id is None and "entity_id" in self.model_fields_set:
+            _dict['entity_id'] = None
+
+        # set to None if last_updated (nullable) is None
+        # and model_fields_set contains the field
+        if self.last_updated is None and "last_updated" in self.model_fields_set:
+            _dict['last_updated'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of MentalModelResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "id": obj.get("id"),
+            "bank_id": obj.get("bank_id"),
+            "subtype": obj.get("subtype"),
+            "name": obj.get("name"),
+            "description": obj.get("description"),
+            "observations": [MentalModelObservationResponse.from_dict(_item) for _item in obj["observations"]] if obj.get("observations") is not None else None,
+            "entity_id": obj.get("entity_id"),
+            "links": obj.get("links"),
+            "tags": obj.get("tags"),
+            "last_updated": obj.get("last_updated"),
+            "created_at": obj.get("created_at")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/operation_response.py

@@ -29,7 +29,7 @@ class OperationResponse(BaseModel):
     id: StrictStr
     task_type: StrictStr
     items_count: StrictInt
-    document_id: Optional[StrictStr]
+    document_id: Optional[StrictStr] = None
     created_at: StrictStr
     status: StrictStr
     error_message: Optional[StrictStr]

hindsight-clients/python/hindsight_client_api/models/operation_status_response.py

@@ -0,0 +1,131 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, StrictStr, field_validator
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class OperationStatusResponse(BaseModel):
+    """
+    Response model for getting a single operation status.
+    """ # noqa: E501
+    operation_id: StrictStr
+    status: StrictStr
+    operation_type: Optional[StrictStr] = None
+    created_at: Optional[StrictStr] = None
+    updated_at: Optional[StrictStr] = None
+    completed_at: Optional[StrictStr] = None
+    error_message: Optional[StrictStr] = None
+    __properties: ClassVar[List[str]] = ["operation_id", "status", "operation_type", "created_at", "updated_at", "completed_at", "error_message"]
+
+    @field_validator('status')
+    def status_validate_enum(cls, value):
+        """Validates the enum"""
+        if value not in set(['pending', 'completed', 'failed', 'not_found']):
+            raise ValueError("must be one of enum values ('pending', 'completed', 'failed', 'not_found')")
+        return value
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of OperationStatusResponse from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # set to None if operation_type (nullable) is None
+        # and model_fields_set contains the field
+        if self.operation_type is None and "operation_type" in self.model_fields_set:
+            _dict['operation_type'] = None
+
+        # set to None if created_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.created_at is None and "created_at" in self.model_fields_set:
+            _dict['created_at'] = None
+
+        # set to None if updated_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.updated_at is None and "updated_at" in self.model_fields_set:
+            _dict['updated_at'] = None
+
+        # set to None if completed_at (nullable) is None
+        # and model_fields_set contains the field
+        if self.completed_at is None and "completed_at" in self.model_fields_set:
+            _dict['completed_at'] = None
+
+        # set to None if error_message (nullable) is None
+        # and model_fields_set contains the field
+        if self.error_message is None and "error_message" in self.model_fields_set:
+            _dict['error_message'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of OperationStatusResponse from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "operation_id": obj.get("operation_id"),
+            "status": obj.get("status"),
+            "operation_type": obj.get("operation_type"),
+            "created_at": obj.get("created_at"),
+            "updated_at": obj.get("updated_at"),
+            "completed_at": obj.get("completed_at"),
+            "error_message": obj.get("error_message")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/operations_list_response.py

@@ -17,7 +17,7 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictStr
+from pydantic import BaseModel, ConfigDict, StrictInt, StrictStr
 from typing import Any, ClassVar, Dict, List
 from hindsight_client_api.models.operation_response import OperationResponse
 from typing import Optional, Set
@@ -28,8 +28,9 @@ class OperationsListResponse(BaseModel):
     Response model for list operations endpoint.
     """ # noqa: E501
     bank_id: StrictStr
+    total: StrictInt
     operations: List[OperationResponse]
-    __properties: ClassVar[List[str]] = ["bank_id", "operations"]
+    __properties: ClassVar[List[str]] = ["bank_id", "total", "operations"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -90,6 +91,7 @@ class OperationsListResponse(BaseModel):
 
         _obj = cls.model_validate({
             "bank_id": obj.get("bank_id"),
+            "total": obj.get("total"),
             "operations": [OperationResponse.from_dict(_item) for _item in obj["operations"]] if obj.get("operations") is not None else None
         })
         return _obj

hindsight-clients/python/hindsight_client_api/models/recall_request.py

@@ -17,7 +17,7 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictInt, StrictStr
+from pydantic import BaseModel, ConfigDict, Field, StrictBool, StrictInt, StrictStr, field_validator
 from typing import Any, ClassVar, Dict, List, Optional
 from hindsight_client_api.models.budget import Budget
 from hindsight_client_api.models.include_options import IncludeOptions
@@ -35,7 +35,19 @@ class RecallRequest(BaseModel):
     trace: Optional[StrictBool] = False
     query_timestamp: Optional[StrictStr] = None
     include: Optional[IncludeOptions] = Field(default=None, description="Options for including additional data (entities are included by default)")
-    __properties: ClassVar[List[str]] = ["query", "types", "budget", "max_tokens", "trace", "query_timestamp", "include"]
+    tags: Optional[List[StrictStr]] = None
+    tags_match: Optional[StrictStr] = Field(default='any', description="How to match tags: 'any' (OR, includes untagged), 'all' (AND, includes untagged), 'any_strict' (OR, excludes untagged), 'all_strict' (AND, excludes untagged).")
+    __properties: ClassVar[List[str]] = ["query", "types", "budget", "max_tokens", "trace", "query_timestamp", "include", "tags", "tags_match"]
+
+    @field_validator('tags_match')
+    def tags_match_validate_enum(cls, value):
+        """Validates the enum"""
+        if value is None:
+            return value
+
+        if value not in set(['any', 'all', 'any_strict', 'all_strict']):
+            raise ValueError("must be one of enum values ('any', 'all', 'any_strict', 'all_strict')")
+        return value
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -89,6 +101,11 @@ class RecallRequest(BaseModel):
         if self.query_timestamp is None and "query_timestamp" in self.model_fields_set:
             _dict['query_timestamp'] = None
 
+        # set to None if tags (nullable) is None
+        # and model_fields_set contains the field
+        if self.tags is None and "tags" in self.model_fields_set:
+            _dict['tags'] = None
+
         return _dict
 
     @classmethod
@@ -107,7 +124,9 @@ class RecallRequest(BaseModel):
             "max_tokens": obj.get("max_tokens") if obj.get("max_tokens") is not None else 4096,
             "trace": obj.get("trace") if obj.get("trace") is not None else False,
             "query_timestamp": obj.get("query_timestamp"),
-            "include": IncludeOptions.from_dict(obj["include"]) if obj.get("include") is not None else None
+            "include": IncludeOptions.from_dict(obj["include"]) if obj.get("include") is not None else None,
+            "tags": obj.get("tags"),
+            "tags_match": obj.get("tags_match") if obj.get("tags_match") is not None else 'any'
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/recall_result.py

@@ -37,7 +37,8 @@ class RecallResult(BaseModel):
     document_id: Optional[StrictStr] = None
     metadata: Optional[Dict[str, StrictStr]] = None
     chunk_id: Optional[StrictStr] = None
-    __properties: ClassVar[List[str]] = ["id", "text", "type", "entities", "context", "occurred_start", "occurred_end", "mentioned_at", "document_id", "metadata", "chunk_id"]
+    tags: Optional[List[StrictStr]] = None
+    __properties: ClassVar[List[str]] = ["id", "text", "type", "entities", "context", "occurred_start", "occurred_end", "mentioned_at", "document_id", "metadata", "chunk_id", "tags"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -123,6 +124,11 @@ class RecallResult(BaseModel):
         if self.chunk_id is None and "chunk_id" in self.model_fields_set:
             _dict['chunk_id'] = None
 
+        # set to None if tags (nullable) is None
+        # and model_fields_set contains the field
+        if self.tags is None and "tags" in self.model_fields_set:
+            _dict['tags'] = None
+
         return _dict
 
     @classmethod
@@ -145,7 +151,8 @@ class RecallResult(BaseModel):
             "mentioned_at": obj.get("mentioned_at"),
             "document_id": obj.get("document_id"),
             "metadata": obj.get("metadata"),
-            "chunk_id": obj.get("chunk_id")
+            "chunk_id": obj.get("chunk_id"),
+            "tags": obj.get("tags")
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/reflect_based_on.py

@@ -0,0 +1,105 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field
+from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.reflect_fact import ReflectFact
+from hindsight_client_api.models.reflect_mental_model import ReflectMentalModel
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ReflectBasedOn(BaseModel):
+    """
+    Evidence the response is based on: memories and mental models.
+    """ # noqa: E501
+    memories: Optional[List[ReflectFact]] = Field(default=None, description="Memory facts used to generate the response")
+    mental_models: Optional[List[ReflectMentalModel]] = Field(default=None, description="Mental models accessed during reflection")
+    __properties: ClassVar[List[str]] = ["memories", "mental_models"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ReflectBasedOn from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # override the default output from pydantic by calling `to_dict()` of each item in memories (list)
+        _items = []
+        if self.memories:
+            for _item_memories in self.memories:
+                if _item_memories:
+                    _items.append(_item_memories.to_dict())
+            _dict['memories'] = _items
+        # override the default output from pydantic by calling `to_dict()` of each item in mental_models (list)
+        _items = []
+        if self.mental_models:
+            for _item_mental_models in self.mental_models:
+                if _item_mental_models:
+                    _items.append(_item_mental_models.to_dict())
+            _dict['mental_models'] = _items
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ReflectBasedOn from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "memories": [ReflectFact.from_dict(_item) for _item in obj["memories"]] if obj.get("memories") is not None else None,
+            "mental_models": [ReflectMentalModel.from_dict(_item) for _item in obj["mental_models"]] if obj.get("mental_models") is not None else None
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/reflect_include_options.py

@@ -19,6 +19,7 @@ import json
 
 from pydantic import BaseModel, ConfigDict, Field
 from typing import Any, ClassVar, Dict, List, Optional
+from hindsight_client_api.models.tool_calls_include_options import ToolCallsIncludeOptions
 from typing import Optional, Set
 from typing_extensions import Self
 
@@ -27,7 +28,8 @@ class ReflectIncludeOptions(BaseModel):
     Options for including additional data in reflect results.
     """ # noqa: E501
     facts: Optional[Dict[str, Any]] = Field(default=None, description="Options for including facts (based_on) in reflect results.")
-    __properties: ClassVar[List[str]] = ["facts"]
+    tool_calls: Optional[ToolCallsIncludeOptions] = None
+    __properties: ClassVar[List[str]] = ["facts", "tool_calls"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -68,6 +70,14 @@ class ReflectIncludeOptions(BaseModel):
             exclude=excluded_fields,
             exclude_none=True,
         )
+        # override the default output from pydantic by calling `to_dict()` of tool_calls
+        if self.tool_calls:
+            _dict['tool_calls'] = self.tool_calls.to_dict()
+        # set to None if tool_calls (nullable) is None
+        # and model_fields_set contains the field
+        if self.tool_calls is None and "tool_calls" in self.model_fields_set:
+            _dict['tool_calls'] = None
+
         return _dict
 
     @classmethod
@@ -80,7 +90,8 @@ class ReflectIncludeOptions(BaseModel):
             return cls.model_validate(obj)
 
         _obj = cls.model_validate({
-            "facts": obj.get("facts")
+            "facts": obj.get("facts"),
+            "tool_calls": ToolCallsIncludeOptions.from_dict(obj["tool_calls"]) if obj.get("tool_calls") is not None else None
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/reflect_llm_call.py

@@ -0,0 +1,89 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr
+from typing import Any, ClassVar, Dict, List
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ReflectLLMCall(BaseModel):
+    """
+    An LLM call made during reflect agent execution.
+    """ # noqa: E501
+    scope: StrictStr = Field(description="Call scope: agent_1, agent_2, final, etc.")
+    duration_ms: StrictInt = Field(description="Execution time in milliseconds")
+    __properties: ClassVar[List[str]] = ["scope", "duration_ms"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ReflectLLMCall from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ReflectLLMCall from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "scope": obj.get("scope"),
+            "duration_ms": obj.get("duration_ms")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/reflect_mental_model.py

@@ -0,0 +1,102 @@
+# coding: utf-8
+
+"""
+    Hindsight HTTP API
+
+    HTTP API for Hindsight
+
+    The version of the OpenAPI document: 0.1.0
+    Generated by OpenAPI Generator (https://openapi-generator.tech)
+
+    Do not edit the class manually.
+"""  # noqa: E501
+
+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401
+import json
+
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
+from typing import Any, ClassVar, Dict, List, Optional
+from typing import Optional, Set
+from typing_extensions import Self
+
+class ReflectMentalModel(BaseModel):
+    """
+    A mental model accessed during reflect.
+    """ # noqa: E501
+    id: StrictStr = Field(description="Mental model ID")
+    name: StrictStr = Field(description="Mental model name")
+    type: StrictStr = Field(description="Mental model type: entity, concept, event")
+    subtype: StrictStr = Field(description="Mental model subtype: structural, emergent, learned")
+    description: StrictStr = Field(description="Brief description")
+    summary: Optional[StrictStr] = None
+    __properties: ClassVar[List[str]] = ["id", "name", "type", "subtype", "description", "summary"]
+
+    model_config = ConfigDict(
+        populate_by_name=True,
+        validate_assignment=True,
+        protected_namespaces=(),
+    )
+
+
+    def to_str(self) -> str:
+        """Returns the string representation of the model using alias"""
+        return pprint.pformat(self.model_dump(by_alias=True))
+
+    def to_json(self) -> str:
+        """Returns the JSON representation of the model using alias"""
+        # TODO: pydantic v2: use .model_dump_json(by_alias=True, exclude_unset=True) instead
+        return json.dumps(self.to_dict())
+
+    @classmethod
+    def from_json(cls, json_str: str) -> Optional[Self]:
+        """Create an instance of ReflectMentalModel from a JSON string"""
+        return cls.from_dict(json.loads(json_str))
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the dictionary representation of the model using alias.
+
+        This has the following differences from calling pydantic's
+        `self.model_dump(by_alias=True)`:
+
+        * `None` is only added to the output dict for nullable fields that
+          were set at model initialization. Other fields with value `None`
+          are ignored.
+        """
+        excluded_fields: Set[str] = set([
+        ])
+
+        _dict = self.model_dump(
+            by_alias=True,
+            exclude=excluded_fields,
+            exclude_none=True,
+        )
+        # set to None if summary (nullable) is None
+        # and model_fields_set contains the field
+        if self.summary is None and "summary" in self.model_fields_set:
+            _dict['summary'] = None
+
+        return _dict
+
+    @classmethod
+    def from_dict(cls, obj: Optional[Dict[str, Any]]) -> Optional[Self]:
+        """Create an instance of ReflectMentalModel from a dict"""
+        if obj is None:
+            return None
+
+        if not isinstance(obj, dict):
+            return cls.model_validate(obj)
+
+        _obj = cls.model_validate({
+            "id": obj.get("id"),
+            "name": obj.get("name"),
+            "type": obj.get("type"),
+            "subtype": obj.get("subtype"),
+            "description": obj.get("description"),
+            "summary": obj.get("summary")
+        })
+        return _obj
+
+

hindsight-clients/python/hindsight_client_api/models/reflect_request.py

@@ -17,7 +17,7 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr
+from pydantic import BaseModel, ConfigDict, Field, StrictInt, StrictStr, field_validator
 from typing import Any, ClassVar, Dict, List, Optional
 from hindsight_client_api.models.budget import Budget
 from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
@@ -34,7 +34,19 @@ class ReflectRequest(BaseModel):
     max_tokens: Optional[StrictInt] = Field(default=4096, description="Maximum tokens for the response")
     include: Optional[ReflectIncludeOptions] = Field(default=None, description="Options for including additional data (disabled by default)")
     response_schema: Optional[Dict[str, Any]] = None
-    __properties: ClassVar[List[str]] = ["query", "budget", "context", "max_tokens", "include", "response_schema"]
+    tags: Optional[List[StrictStr]] = None
+    tags_match: Optional[StrictStr] = Field(default='any', description="How to match tags: 'any' (OR, includes untagged), 'all' (AND, includes untagged), 'any_strict' (OR, excludes untagged), 'all_strict' (AND, excludes untagged).")
+    __properties: ClassVar[List[str]] = ["query", "budget", "context", "max_tokens", "include", "response_schema", "tags", "tags_match"]
+
+    @field_validator('tags_match')
+    def tags_match_validate_enum(cls, value):
+        """Validates the enum"""
+        if value is None:
+            return value
+
+        if value not in set(['any', 'all', 'any_strict', 'all_strict']):
+            raise ValueError("must be one of enum values ('any', 'all', 'any_strict', 'all_strict')")
+        return value
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -88,6 +100,11 @@ class ReflectRequest(BaseModel):
         if self.response_schema is None and "response_schema" in self.model_fields_set:
             _dict['response_schema'] = None
 
+        # set to None if tags (nullable) is None
+        # and model_fields_set contains the field
+        if self.tags is None and "tags" in self.model_fields_set:
+            _dict['tags'] = None
+
         return _dict
 
     @classmethod
@@ -105,7 +122,9 @@ class ReflectRequest(BaseModel):
             "context": obj.get("context"),
             "max_tokens": obj.get("max_tokens") if obj.get("max_tokens") is not None else 4096,
             "include": ReflectIncludeOptions.from_dict(obj["include"]) if obj.get("include") is not None else None,
-            "response_schema": obj.get("response_schema")
+            "response_schema": obj.get("response_schema"),
+            "tags": obj.get("tags"),
+            "tags_match": obj.get("tags_match") if obj.get("tags_match") is not None else 'any'
         })
         return _obj
 

hindsight-clients/python/hindsight_client_api/models/reflect_response.py

@@ -17,9 +17,11 @@ import pprint
 import re  # noqa: F401
 import json
 
-from pydantic import BaseModel, ConfigDict, StrictStr
+from pydantic import BaseModel, ConfigDict, Field, StrictStr
 from typing import Any, ClassVar, Dict, List, Optional
-from hindsight_client_api.models.reflect_fact import ReflectFact
+from hindsight_client_api.models.created_mental_model import CreatedMentalModel
+from hindsight_client_api.models.reflect_based_on import ReflectBasedOn
+from hindsight_client_api.models.reflect_trace import ReflectTrace
 from hindsight_client_api.models.token_usage import TokenUsage
 from typing import Optional, Set
 from typing_extensions import Self
@@ -29,10 +31,12 @@ class ReflectResponse(BaseModel):
     Response model for think endpoint.
     """ # noqa: E501
     text: StrictStr
-    based_on: Optional[List[ReflectFact]] = None
+    based_on: Optional[ReflectBasedOn] = None
     structured_output: Optional[Dict[str, Any]] = None
     usage: Optional[TokenUsage] = None
-    __properties: ClassVar[List[str]] = ["text", "based_on", "structured_output", "usage"]
+    trace: Optional[ReflectTrace] = None
+    mental_models_created: Optional[List[CreatedMentalModel]] = Field(default=None, description="Mental models created during this reflection (via the learn tool).")
+    __properties: ClassVar[List[str]] = ["text", "based_on", "structured_output", "usage", "trace", "mental_models_created"]
 
     model_config = ConfigDict(
         populate_by_name=True,
@@ -73,16 +77,27 @@ class ReflectResponse(BaseModel):
             exclude=excluded_fields,
             exclude_none=True,
         )
-        # override the default output from pydantic by calling `to_dict()` of each item in based_on (list)
-        _items = []
+        # override the default output from pydantic by calling `to_dict()` of based_on
         if self.based_on:
-            for _item_based_on in self.based_on:
-                if _item_based_on:
-                    _items.append(_item_based_on.to_dict())
-            _dict['based_on'] = _items
+            _dict['based_on'] = self.based_on.to_dict()
         # override the default output from pydantic by calling `to_dict()` of usage
         if self.usage:
             _dict['usage'] = self.usage.to_dict()
+        # override the default output from pydantic by calling `to_dict()` of trace
+        if self.trace:
+            _dict['trace'] = self.trace.to_dict()
+        # override the default output from pydantic by calling `to_dict()` of each item in mental_models_created (list)
+        _items = []
+        if self.mental_models_created:
+            for _item_mental_models_created in self.mental_models_created:
+                if _item_mental_models_created:
+                    _items.append(_item_mental_models_created.to_dict())
+            _dict['mental_models_created'] = _items
+        # set to None if based_on (nullable) is None
+        # and model_fields_set contains the field
+        if self.based_on is None and "based_on" in self.model_fields_set:
+            _dict['based_on'] = None
+
         # set to None if structured_output (nullable) is None
         # and model_fields_set contains the field
         if self.structured_output is None and "structured_output" in self.model_fields_set:
@@ -93,6 +108,11 @@ class ReflectResponse(BaseModel):
         if self.usage is None and "usage" in self.model_fields_set:
             _dict['usage'] = None
 
+        # set to None if trace (nullable) is None
+        # and model_fields_set contains the field
+        if self.trace is None and "trace" in self.model_fields_set:
+            _dict['trace'] = None
+
         return _dict
 
     @classmethod
@@ -106,9 +126,11 @@ class ReflectResponse(BaseModel):
 
         _obj = cls.model_validate({
             "text": obj.get("text"),
-            "based_on": [ReflectFact.from_dict(_item) for _item in obj["based_on"]] if obj.get("based_on") is not None else None,
+            "based_on": ReflectBasedOn.from_dict(obj["based_on"]) if obj.get("based_on") is not None else None,
             "structured_output": obj.get("structured_output"),
-            "usage": TokenUsage.from_dict(obj["usage"]) if obj.get("usage") is not None else None
+            "usage": TokenUsage.from_dict(obj["usage"]) if obj.get("usage") is not None else None,
+            "trace": ReflectTrace.from_dict(obj["trace"]) if obj.get("trace") is not None else None,
+            "mental_models_created": [CreatedMentalModel.from_dict(_item) for _item in obj["mental_models_created"]] if obj.get("mental_models_created") is not None else None
         })
         return _obj