fix: create report.pdf in working directory for retain.sh file upload examples

fix: add include_facts to reflect client, fix retain.sh temp files, fix main-methods based_on access
doc: improve api explanation
2026-02-20 15:40:35 +01:00 · 2026-02-20 15:18:11 +01:00 · 2026-02-20 15:12:22 +01:00 · 2026-02-20 14:37:11 +01:00 · 2026-02-20 14:36:53 +01:00
31 changed files with 1255 additions and 1111 deletions
@@ -429,7 +429,6 @@ class RetainRequest(BaseModel):
                    },
                ],
                "async": False,
-                "document_tags": ["user_a", "user_b"],
            }
        }
    )
@@ -442,7 +441,8 @@ class RetainRequest(BaseModel):
    )
    document_tags: list[str] | None = Field(
        default=None,
-        description="Tags applied to all items in this request. These are merged with any item-level tags.",
+        description="Deprecated. Use item-level tags instead.",
+        deprecated=True,
    )


@@ -4094,9 +4094,6 @@ components:
      description: Request model for retain endpoint.
      example:
        async: false
-        document_tags:
-        - user_a
-        - user_b
        items:
        - content: Alice works at Google
          context: work
@@ -19,6 +19,7 @@ from hindsight_client_api.models import (
    reflect_request,
    retain_request,
 )
+from hindsight_client_api.models.reflect_include_options import ReflectIncludeOptions
 from hindsight_client_api.models.bank_profile_response import BankProfileResponse
 from hindsight_client_api.models.file_retain_response import FileRetainResponse
 from hindsight_client_api.models.list_memory_units_response import ListMemoryUnitsResponse
@@ -322,6 +323,7 @@ class Hindsight:
        response_schema: dict[str, Any] | None = None,
        tags: list[str] | None = None,
        tags_match: Literal["any", "all", "any_strict", "all_strict"] = "any",
+        include_facts: bool = False,
    ) -> ReflectResponse:
        """
        Generate a contextual answer based on bank identity and memories.
@@ -338,11 +340,14 @@ class Hindsight:
            tags: Optional list of tags to filter memories by
            tags_match: How to match tags - "any" (OR, includes untagged), "all" (AND, includes untagged),
                "any_strict" (OR, excludes untagged), "all_strict" (AND, excludes untagged). Default: "any"
+            include_facts: If True, the response will include a 'based_on' field listing
+                the memories, mental models, and directives used to construct the answer.

        Returns:
            ReflectResponse with answer text, optionally facts used, and optionally
            structured_output if response_schema was provided
        """
+        include = ReflectIncludeOptions(facts={}) if include_facts else None
        request_obj = reflect_request.ReflectRequest(
            query=query,
            budget=budget,
@@ -351,6 +356,7 @@ class Hindsight:
            response_schema=response_schema,
            tags=tags,
            tags_match=tags_match,
+            include=include,
        )

        return _run_async(self._memory_api.reflect(bank_id, request_obj, _request_timeout=self._timeout))
@@ -1782,7 +1782,9 @@ export type RetainRequest = {
  /**
   * Document Tags
   *
-   * Tags applied to all items in this request. These are merged with any item-level tags.
+   * Deprecated. Use item-level tags instead.
+   *
+   * @deprecated
   */
  document_tags?: Array<string> | null;
 };
@@ -4,7 +4,9 @@ sidebar_position: 2

 # Recall Memories

-Retrieve memories using multi-strategy recall.
+Retrieve memories from a bank using multi-strategy recall.
+
+When you **recall**, Hindsight runs four retrieval strategies in parallel — semantic similarity, keyword (BM25), graph traversal, and temporal — then fuses and reranks the results into a single ranked list. The response contains structured facts, not raw documents.

 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
@@ -37,34 +39,19 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

-## Recall Parameters
+---

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Natural language query |
-| `types` | list | all | Filter: `world`, `experience`, `observation` |
-| `budget` | string | "mid" | Budget level: `low`, `mid`, `high` |
-| `max_tokens` | int | 4096 | Token budget for memory facts (text only) |
-| `trace` | bool | false | Enable trace output for debugging |
-| `include_chunks` | bool | false | Include raw text chunks that generated the memories |
-| `max_chunk_tokens` | int | 500 | Token budget for chunks (independent of `max_tokens`) |
-| `include_source_facts` | bool | false | Include source facts for observation-type results (see [Source Facts](#source-facts)) |
-| `max_source_facts_tokens` | int | 4096 | Token budget for source facts |
-| `tags` | list | None | Filter memories by tags (see [Tag Filtering](#filter-by-tags)) |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
+## Parameters

-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-with-options" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={recallMjs} section="recall-with-options" language="javascript" />
-</TabItem>
-</Tabs>
+### query

-## Filter by Fact Type
+The natural language question or statement to search for. This is the only required field. The query drives all four retrieval strategies simultaneously: it is embedded for semantic search, tokenized for BM25 keyword search, used to seed graph traversal, and parsed for temporal expressions. After retrieval, the raw query text is also passed to the cross-encoder reranker to re-score every candidate. Queries exceeding 500 tokens are rejected.

-Recall specific memory types:
+### types
+
+Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (consolidated knowledge synthesized over time). When omitted, all three types are searched.
+
+Each type runs the full four-strategy retrieval pipeline independently, so narrowing `types` reduces both the result set and query cost.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -78,58 +65,12 @@ Recall specific memory types:
 </Tabs>

 :::tip About Observations
-Observations are consolidated knowledge synthesized from multiple facts. They capture patterns, preferences, and learnings that the memory bank has built up over time. Observations are automatically created in the background after retain operations.
+Observations are consolidated knowledge synthesized from multiple facts over time — patterns, preferences, and learnings the memory bank has built up. They are created automatically in the background after retain operations.
 :::

-## Source Facts
+### budget

-When recalling `observation`-type memories, you can fetch the underlying facts they were derived from. This is useful when you need to understand or verify the evidence behind a synthesized observation.
-
-Source facts are returned as a top-level `source_facts` dict keyed by fact ID. Each observation result includes a `source_fact_ids` list for cross-referencing. Facts are deduplicated — if two observations share a source fact, it only appears once.
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-source-facts" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={recallMjs} section="recall-source-facts" language="javascript" />
-</TabItem>
-</Tabs>
-
-:::note Source Facts Token Budget
-Source facts are fetched independently of the main `max_tokens` budget, up to `max_source_facts_tokens`. Facts are included in order of first appearance across all observations — once the budget is reached, remaining source facts are omitted.
-:::
-
-## Token Budget Management
-
-Hindsight is built for AI agents, not humans. Traditional retrieval systems return "top-k" results, but agents don't think in terms of result counts—they think in tokens. An agent's context window is measured in tokens, and that's exactly how Hindsight measures results.
-
-The `max_tokens` parameter lets you control how much of your agent's context budget to spend on memories:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-token-budget" language="python" />
-</TabItem>
-</Tabs>
-
-This design means you never have to guess whether 10 results or 50 results will fit your context. Just specify the token budget and Hindsight returns as many relevant memories as will fit.
-
-:::note Chunks are Independent
-When `include_chunks=True`, chunks are fetched **independently** of the `max_tokens` filtering. This means:
- Setting `max_tokens=0` will return **0 memory facts** but can still return **chunks** (up to `max_chunk_tokens`)
- Chunks are based on the top-scored (reranked) results **before** token filtering
- Chunks are fetched in batches (batch size estimated as `(max_chunk_tokens / retain_chunk_size) * 2`) until the token budget is exhausted
- This batching approach handles varying chunk sizes across documents efficiently
- This allows you to retrieve raw source text without memory facts when needed
-:::
-
-## Budget Levels
-
-The `budget` parameter controls graph traversal depth:
-
- **"low"**: Fast, shallow retrieval — good for simple lookups
- **"mid"**: Balanced — default for most queries
- **"high"**: Deep exploration — finds indirect connections
+Controls retrieval depth and breadth. Accepted values are `low`, `mid` (default), and `high`. Use `low` for fast simple lookups, `mid` for balanced everyday queries, and `high` when you need to find indirect connections or exhaustive coverage.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -140,11 +81,59 @@ The `budget` parameter controls graph traversal depth:
 </TabItem>
 </Tabs>

-## Filter by Tags
+### max_tokens

-Tags enable **visibility scoping**—filter memories based on tags assigned during [retain](./retain#tagging-memories). This is essential for multi-user agents where each user should only see their own memories.
+The maximum number of tokens the returned facts can collectively occupy. Defaults to `4096`. Only the `text` field of each fact is counted toward this budget — metadata, tags, entities, and other fields are not included. After reranking, facts are included in relevance order until this budget is exhausted — so you always get the most relevant memories that fit. Hindsight is designed for agents, which think in tokens rather than result counts: set `max_tokens` to however much of your context window you want to allocate to memories.

-### Basic Tag Filtering
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={recallPy} section="recall-token-budget" language="python" />
+</TabItem>
+</Tabs>
+
+### query_timestamp
+
+An ISO 8601 datetime representing when the query is being asked, from the user's perspective. When provided, it is used as the anchor for resolving relative temporal expressions in the query — for example, if the query says "last month" and `query_timestamp` is `2023-05-30`, the temporal search window becomes approximately April 2023. Without it, the server's current time is used as the anchor. This field matters most for replaying historical conversations or building agents that need time-anchored recall.
+
+### include
+
+An optional object controlling supplementary data returned alongside the main facts.
+
+#### chunks
+
+When enabled, the response includes the raw source text chunks from which each fact was extracted. Chunks are fetched before the `max_tokens` filter, so setting `max_tokens=0` returns no facts but can still return chunks. The `max_tokens` sub-option (default `8192`) controls the total chunk token budget independently of the main fact budget. This is useful when agents need surrounding context beyond the extracted fact text.
+
+:::note
+When `include_chunks` is enabled, chunks are fetched based on the top-scored reranked results before token filtering. The last chunk is truncated (not dropped) to fit exactly within the budget, and each chunk carries a `truncated` flag indicating whether it was cut.
+:::
+
+#### source_facts
+
+When enabled and `types` includes `observation`, each observation result is accompanied by the original contributing facts it was synthesized from. Source facts are returned in a top-level `source_facts` dict keyed by fact ID, and each observation result carries a `source_fact_ids` list for cross-referencing. Facts are deduplicated across observations. The `max_tokens` sub-option (default `4096`) limits the total token budget for source facts.
+
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={recallPy} section="recall-source-facts" language="python" />
+</TabItem>
+<TabItem value="node" label="Node.js">
+<CodeSnippet code={recallMjs} section="recall-source-facts" language="javascript" />
+</TabItem>
+</Tabs>
+
+#### entities
+
+Enabled by default. When active, each returned fact includes the canonical names of entities associated with it. Set to `null` to skip the entity JOIN query and reduce response size. The `max_tokens` sub-option (default `500`) is a future-facing guard for entity data.
+
+### tags
+
+Filters recall to only memories that match the specified tags. When omitted, all memories regardless of tags are eligible. Tag filtering is applied at the database level across all four retrieval strategies, not as a post-processing step.
+
+The `tags_match` parameter controls the filtering logic:
+
+- `any` (default) — memory matches if it has at least one of the specified tags, or has no tags at all. Use this for "user-specific + shared global" patterns.
+- `any_strict` — memory matches if it has at least one of the specified tags, and untagged memories are excluded. Use this when you want only explicitly scoped memories.
+- `all` — memory matches if it has every specified tag, or has no tags at all.
+- `all_strict` — memory matches if it has every specified tag, and untagged memories are excluded.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -152,38 +141,96 @@ Tags enable **visibility scoping**—filter memories based on tags assigned duri
 </TabItem>
 </Tabs>

-### Tag Match Modes
-
-The `tags_match` parameter controls how tags are matched:
-
-| Mode | Behavior | Untagged Memories |
-|------|----------|-------------------|
-| `any` | OR: memory has ANY of the specified tags | **Included** |
-| `all` | AND: memory has ALL of the specified tags | **Included** |
-| `any_strict` | OR: memory has ANY of the specified tags | **Excluded** |
-| `all_strict` | AND: memory has ALL of the specified tags | **Excluded** |
-
-**Strict modes** are useful when you want to ensure only tagged memories are returned:
-
 <Tabs>
 <TabItem value="python" label="Python">
 <CodeSnippet code={recallPy} section="recall-tags-strict" language="python" />
 </TabItem>
 </Tabs>

-**AND matching** requires all specified tags to be present:
-
 <Tabs>
 <TabItem value="python" label="Python">
 <CodeSnippet code={recallPy} section="recall-tags-all" language="python" />
 </TabItem>
 </Tabs>

-### Use Cases
+### trace

-| Scenario | Tags | Mode | Result |
-|----------|------|------|--------|
-| User A's memories only | `["user:alice"]` | `any_strict` | Only memories tagged `user:alice` |
-| Support + feedback | `["support", "feedback"]` | `any` | Memories with either tag + untagged |
-| Multi-user room | `["user:alice", "room:general"]` | `all_strict` | Only memories with both tags |
-| Global + user-specific | `["user:alice"]` | `any` | Alice's memories + shared (untagged) |
+When set to `true`, the response includes a detailed debug trace covering the query embedding, entry points, per-strategy retrieval results, RRF fusion candidates, reranked results, temporal constraints detected, and per-phase timings. Has no effect on the retrieval logic itself. Useful for understanding why specific memories were or were not returned.
+
+---
+
+## Response
+
+### results
+
+The main list of recalled facts, ordered by relevance. Relevance is computed by running four retrieval strategies in parallel — semantic similarity, BM25 keyword, graph traversal, and temporal — fusing their rankings with Reciprocal Rank Fusion (RRF), then re-scoring the merged candidates with a cross-encoder reranker against the original query.
+
+Results do not include a numeric score. Raw retrieval scores are not meaningful on an absolute scale — a score of 0.8 from one query tells you nothing useful compared to a score of 0.8 from another. What matters is the relative ordering, which is already reflected in the list order. Agents should consume memories in order and let `max_tokens` determine how many fit, rather than filtering by score.
+
+Each item in `results` has the following fields:
+
+#### id
+
+The unique identifier of this fact. Use it to cross-reference with `source_facts` or for application-level deduplication.
+
+#### text
+
+The extracted fact text as stored in the memory bank.
+
+#### type
+
+The fact category: `world` for objective information, `experience` for events and conversations, or `observation` for consolidated knowledge synthesized over time.
+
+#### context
+
+The context label provided when the fact was retained (e.g., `"team meeting"`, `"slack"`). `null` if none was set.
+
+#### metadata
+
+The key-value string pairs attached when the fact was retained. `null` if none were set.
+
+#### tags
+
+The visibility-scoping tags attached to this fact.
+
+#### entities
+
+A list of canonical entity name strings linked to this fact. Only populated when `include.entities` is enabled (the default). `null` otherwise.
+
+#### occurred_start / occurred_end
+
+ISO 8601 datetimes representing when the described event started and ended. Extracted by the LLM from the content during retain. `null` if the content had no temporal information.
+
+#### mentioned_at
+
+ISO 8601 datetime of when this fact was retained into the bank.
+
+#### document_id
+
+The document ID this fact belongs to, as set during retain.
+
+#### chunk_id
+
+The ID of the source text chunk this fact was extracted from. Used to cross-reference with `chunks` in the response when `include.chunks` is enabled.
+
+#### source_fact_ids
+
+For `observation`-type results only: the IDs of the original facts this observation was synthesized from. Cross-references with `source_facts` in the response. `null` for other types or when `include.source_facts` is not enabled.
+
+---
+
+### source_facts
+
+A dict keyed by fact ID containing full `RecallResult` objects for the source facts that contributed to observation results. Only present when `include.source_facts` is enabled. Facts are deduplicated — if two observations share a source fact, it appears once.
+
+### chunks
+
+A dict keyed by chunk ID containing the raw source text chunks associated with the returned facts. Only present when `include.chunks` is enabled. Each chunk has `id`, `text`, `chunk_index`, and `truncated` (whether the text was cut to fit the token budget).
+
+### entities
+
+A dict keyed by canonical entity name containing entity state objects. Only present when `include.entities` is enabled. Each entry has `entity_id`, `canonical_name`, and `observations`.
+
+### trace
+
+A debug object present only when `trace: true` was set in the request. Contains per-phase timings, retrieval breakdowns, and RRF fusion details.
@@ -4,14 +4,9 @@ sidebar_position: 3

 # Reflect

-Generate disposition-aware responses using an agentic reasoning loop.
+Generate a grounded, disposition-aware response using an agentic reasoning loop.

-When you call **reflect**, Hindsight runs an **agentic loop** that:
-1. **Autonomously searches** for relevant information using multiple tools
-2. **Applies** the bank's disposition traits to shape the reasoning style
-3. **Generates** a grounded answer with citations to the sources used
-
-The agent has access to hierarchical retrieval tools (mental models → observations → raw facts) and decides what information it needs to answer your query.
+When you call **reflect**, Hindsight runs an agentic loop that autonomously searches the memory bank using multiple retrieval tools, applies the bank's disposition traits to shape the reasoning style, and produces a final answer grounded in what it found. Unlike recall — which returns raw facts — reflect returns a synthesized response written by the LLM.

 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
@@ -44,33 +39,17 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

+---
+
 ## Parameters

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Question or prompt |
-| `budget` | string | "low" | Budget level: `low`, `mid`, `high` (see below) |
-| `max_tokens` | int | 4096 | Maximum tokens for the final response |
-| `response_schema` | object | None | JSON Schema for [structured output](#structured-output) |
-| `tags` | list | None | Filter memories by tags during reflection |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
-| `trace` | bool | false | Include detailed agent trace in response |
+### query

-### Budget
+The question or prompt to reflect on. This is the only required field. If you have situational context that should influence the answer, include it directly in the query rather than as a separate field.

-The `budget` parameter controls the research depth — how thoroughly the agent explores before answering:
+### budget

-| Budget | Research Depth | Use Case |
-|--------|----------------|----------|
-| `low` | Shallow | Quick answers, simple lookups. Prioritizes speed over completeness. |
-| `mid` | Moderate | Balanced exploration. Checks multiple sources when warranted. |
-| `high` | Deep | Comprehensive analysis. Explores all knowledge levels, uses multiple query variations. |
-
-Use `high` for complex questions that require synthesizing information from multiple sources or verifying facts across different retrieval levels.
-
-### Max Tokens
-
-The `max_tokens` parameter limits the length of the final generated response. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.
+Controls how thoroughly the agent explores the memory bank before answering. Accepted values are `low` (default), `mid`, and `high`. At `low`, the agent does a shallow search optimized for speed. At `mid`, it checks multiple sources when the question warrants it. At `high`, it performs deep exploration across all knowledge levels and may use multiple query variations to find indirect connections. Use `high` for complex questions that require synthesizing information from many sources.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -81,45 +60,13 @@ The `max_tokens` parameter limits the length of the final generated response. Th
 </TabItem>
 </Tabs>

-## Disposition Influence
+### max_tokens

-The bank's disposition affects reflect responses:
+Limits the length of the final generated response. Defaults to `4096`. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.

-| Trait | Low (1) | High (5) |
-|-------|---------|----------|
-| **Skepticism** | Trusting, accepts claims | Questions and doubts claims |
-| **Literalism** | Flexible interpretation | Exact, literal interpretation |
-| **Empathy** | Detached, fact-focused | Considers emotional context |
+### response_schema

-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={reflectPy} section="reflect-disposition" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={reflectMjs} section="reflect-disposition" language="javascript" />
-</TabItem>
-</Tabs>
-
-## Citations
-
-The response includes a `based_on` field that shows which sources were used:
-
- `based_on.memories` — Memory facts (world, experience) that were retrieved and cited
- `based_on.mental_models` — User-curated mental models that were used
- `based_on.directives` — Directives that were enforced
-
-**Important:** Only IDs that were actually retrieved during the agent loop can be cited. The agent validates citations to prevent hallucinated references.
-
-This enables:
- **Transparency** — users see exactly which sources informed the answer
- **Verification** — check if the response is grounded in actual memories
- **Debugging** — use `trace=True` for detailed tool call logs
-
-## Structured Output
-
-For applications that need to process responses programmatically, you can request structured output by providing a JSON Schema via `response_schema`. When provided, the response includes a `structured_output` field with the LLM response parsed according to the schema. The `text` field will be empty since only a single LLM call is made for efficiency.
-
-The easiest way to define a schema is using **Pydantic models**:
+An optional JSON Schema object. When provided, the LLM generates a response that conforms to the schema and the response includes a `structured_output` field with the result parsed accordingly. The `text` field will be empty since only a single structured LLM call is made. Use this when you need to process the response programmatically rather than display it as prose.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -133,22 +80,9 @@ The easiest way to define a schema is using **Pydantic models**:
 </TabItem>
 </Tabs>

-| Use Case | Why Structured Output Helps |
-|----------|----------------------------|
-| **Decision pipelines** | Parse recommendations into workflow systems |
-| **Dashboards** | Extract confidence scores, risk factors for visualization |
-| **Multi-agent systems** | Pass structured data between agents |
-| **Auditing** | Log structured decisions with clear reasoning |
+### tags

-**Tips:**
- Use Pydantic's `model_json_schema()` for type-safe schema generation
- Use `model_validate()` to parse the response back into your Pydantic model
- Keep schemas focused — extract only what you need
- Use `Optional` fields for data that may not always be available
-
-## Filter by Tags
-
-Like [recall](./recall#filter-by-tags), reflect supports tag filtering to scope which memories are considered during reasoning. This is essential for multi-user scenarios where reflection should only consider memories relevant to a specific user.
+Filters which memories the agent can access during reflection. Works identically to [recall tags](./recall#tags) — only memories matching the specified tags are considered. The `tags_match` parameter controls the matching logic (`any`, `all`, `any_strict`, `all_strict`) with the same semantics as recall.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -156,13 +90,51 @@ Like [recall](./recall#filter-by-tags), reflect supports tag filtering to scope
 </TabItem>
 </Tabs>

-The `tags_match` parameter works the same as in recall:
+### include

-| Mode | Behavior |
-|------|----------|
-| `any` | OR matching, includes untagged memories |
-| `all` | AND matching, includes untagged memories |
-| `any_strict` | OR matching, excludes untagged memories |
-| `all_strict` | AND matching, excludes untagged memories |
+Controls optional supplementary data returned alongside the main response.

-See [Retain API](./retain#tagging-memories) for how to tag memories and [Recall API](./recall#filter-by-tags) for more details on tag matching modes.
+#### include.facts
+
+When enabled, the response includes a `based_on` object listing the memories, mental models, and directives the agent actually used to construct the answer. Only sources retrieved during the agent loop can appear here — citations are validated to prevent hallucinated references. Useful for transparency and verification.
+
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={reflectPy} section="reflect-sources" language="python" />
+</TabItem>
+</Tabs>
+
+#### include.tool_calls
+
+When enabled, the response includes a `trace` object with the full execution log of every tool call and LLM call made during the agentic loop, including inputs, outputs, and durations. Set `output: false` to include only tool inputs for a smaller payload. Useful for debugging why the agent reached a particular conclusion.
+
+---
+
+## Response
+
+### text
+
+The synthesized answer as a well-formatted markdown string. This is the primary output of reflect. Empty when `response_schema` is provided (use `structured_output` instead in that case).
+
+### structured_output
+
+The LLM's response parsed according to the `response_schema` provided in the request. Only present when `response_schema` was set. `null` otherwise.
+
+### based_on
+
+The sources the agent used to construct the answer. Only present when `include.facts` was enabled. Contains three fields:
+
+- `memories` — a list of memory facts (world, experience, observation) that were retrieved and cited. Each item has `id`, `text`, `type`, `context`, `occurred_start`, and `occurred_end`.
+- `mental_models` — a list of mental models that were used. Each item has `id`, `text`, and `context`.
+- `directives` — a list of directives that were enforced during reasoning. Each item has `id`, `name`, and `content`.
+
+### usage
+
+Token usage for all LLM calls made during the agentic loop: `input_tokens`, `output_tokens`, and `total_tokens`. Useful for cost tracking.
+
+### trace
+
+The full execution log of the agentic loop. Only present when `include.tool_calls` was enabled. Contains:
+
+- `tool_calls` — each tool invocation with `tool` name (`lookup`, `recall`, `learn`, `expand`), `input`, `output` (if `output: true`), `duration_ms`, and `iteration` number.
+- `llm_calls` — each LLM call with `scope` (e.g., `"agent_1"`, `"final"`) and `duration_ms`.
@@ -25,7 +25,9 @@ Learn about fact extraction, entity resolution, and graph construction in the [R
 Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
 :::

-## Store a Single Memory
+## Store a Document
+
+A single retain call accepts one or more **items**. Each item is a piece of raw content — a conversation, a document, a note — that Hindsight will analyze and decompose into one or many memories. The content itself is never stored verbatim; what gets stored are the structured facts the LLM extracts from it.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -39,18 +41,40 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

-## The Importance of Context
+### Retaining a Conversation

-The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
+A full conversation should be retained as a single item. The LLM can parse any format — plain text, JSON, Markdown, or any structured representation — as long as it clearly conveys who said what and when. The example below uses a simple `Name (timestamp): text` format.

-**Why context matters:**
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={retainPy} section="retain-conversation" language="python" />
+</TabItem>
+<TabItem value="node" label="Node.js">
+<CodeSnippet code={retainMjs} section="retain-conversation" language="javascript" />
+</TabItem>
+</Tabs>

-## Store with Context and Date
+When the conversation grows — a new message arrives — just retain again with the full updated content and the same `document_id`. Hindsight will delete the previous version and reprocess from scratch, so memories always reflect the latest state of the conversation.

-Always provide context and event dates for optimal memory extraction:
+---
+
+## Parameters
+
+### content
+
+The raw text to store. This is the only required field. Hindsight chunks the content, sends each chunk to the LLM for fact extraction, and stores the resulting structured facts — not the original text. A single `content` value can produce many memories depending on how much information it contains.
+
+### timestamp
+
+When the event described in the content actually occurred. Accepts any ISO 8601 string (e.g., `"2024-01-15T10:30:00Z"`). If omitted, defaults to the current time at ingestion.
+
+The timestamp is injected verbatim into the LLM fact-extraction prompt so the model can resolve relative temporal references in the content — for example, if the content says "last Monday", the model uses the provided timestamp as the anchor to pin down the actual date. It also enables temporal recall queries like "What happened last spring?" to work correctly.
+
+### context
+
+A short label describing the source or situation — for example `"team meeting"`, `"slack"`, or `"support ticket"`. It is injected directly into the LLM prompt, so it actively shapes how facts are extracted. The same sentence can mean something very different depending on context: "the project was terminated" in a `"performance review"` context versus a `"product roadmap"` context produces different memories.
+
+Providing context consistently is one of the highest-leverage things you can do to improve memory quality.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -64,30 +88,47 @@ Always provide context and event dates for optimal memory extraction:
 </TabItem>
 </Tabs>

-The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
+### metadata

-### Response Fields
+Arbitrary key-value string pairs attached to every fact extracted from this item. For example: `{"source": "slack", "channel": "engineering", "thread_id": "T123"}`. The LLM never sees this field — it is passed through as-is and stored on each memory unit. During recall, every returned memory includes its metadata, which lets you do client-side filtering or static enrichment without extra lookups — for example, linking a memory back to its source URL, thread ID, or any application-specific identifier.

-The retain response includes:
+### document_id

-| Field | Type | Description |
-|-------|------|-------------|
-| `success` | bool | Whether the operation succeeded |
-| `bank_id` | string | The memory bank ID |
-| `items_count` | int | Number of items processed |
-| `async` | bool | Whether processed asynchronously |
-| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |
+A caller-supplied string that groups one or more items under a logical document. This field is the key to making retain **idempotent**.

-The `usage` field contains token metrics for cost tracking:
- `input_tokens`: Tokens consumed by prompts
- `output_tokens`: Tokens generated by the LLM
- `total_tokens`: Sum of input and output tokens
+When you provide a `document_id`, Hindsight upserts the document: if a document with that ID already exists in the bank, it and all its associated memories are deleted before the new content is processed and inserted. This means you can safely re-run retain on updated content — for example, a chat thread that grew since last time — without accumulating duplicate memories.

-Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.
+If you omit `document_id`, Hindsight assigns a random UUID per request, so re-ingesting the same content will create duplicate memories.
+
+### entities
+
+A list of entities you want to guarantee are recognized, merged with any entities the LLM extracts automatically. Each entry has a `text` field (the entity name) and an optional `type` (e.g., `"PERSON"`, `"ORG"`, `"CONCEPT"` — defaults to `"CONCEPT"` if omitted).
+
+Use this when you know certain entities are important but the LLM might miss them or refer to them inconsistently across different parts of the content. Providing entities explicitly ensures they are always linked in the knowledge graph.
+
+### tags and document_tags
+
+Tags control **visibility scoping** — which memories are visible during recall. A memory is only returned if its tags intersect with the tags filter provided in the recall request. This makes tags useful when a single memory bank serves multiple users or sessions and each should only see their own memories.
+
+Use consistent naming patterns to keep tag filtering predictable. Common conventions: `user:<id>` for per-user scoping, `session:<id>` for session isolation, `room:<id>` for chat rooms, `topic:<name>` for category filtering. The bank also exposes a list-tags endpoint that returns all tags with their memory counts, useful for UI autocomplete or wildcard expansion.
+
+See [Recall API](./recall#filter-by-tags) for filtering by tags during retrieval.
+
+### Response
+
+The synchronous retain response includes:
+
+- `success` — whether the operation completed without errors
+- `bank_id` — the memory bank that received the content
+- `items_count` — number of items processed
+- `async` — whether processing ran asynchronously
+- `usage` — token usage for the LLM calls (`input_tokens`, `output_tokens`, `total_tokens`), only present for synchronous operations
+
+---

 ## Batch Ingestion

-Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
+Multiple items can be submitted in a single request. Batch ingestion is the recommended approach — it reduces network overhead and lets Hindsight optimize extraction across related content.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -98,11 +139,12 @@ Store multiple items in a single request. **Batch ingestion is the recommended a
 </TabItem>
 </Tabs>

-The `document_id` groups related memories for later management.

-## Store from Files
+---

-Upload files directly — Hindsight automatically converts them to text and extracts memories. File processing always runs asynchronously and returns operation IDs for tracking.
+## Files
+
+Upload files directly — Hindsight converts them to text and extracts memories automatically. File processing always runs asynchronously and returns operation IDs for tracking.

 **Supported formats:** PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, images (JPG, PNG, GIF, etc. — OCR), audio (MP3, WAV, FLAC, etc. — transcription), HTML, and plain text formats (TXT, MD, CSV, JSON, YAML, etc.)

@@ -121,15 +163,7 @@ Upload files directly — Hindsight automatically converts them to text and extr
 </TabItem>
 </Tabs>

-### File Retain Response
-
-The file retain endpoint always returns asynchronously:
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `operation_ids` | string[] | One operation ID per uploaded file. Use `GET /v1/default/banks/{bank_id}/operations` to track progress. |
-
-### Batch File Uploads
+The file retain endpoint always returns asynchronously. The response contains `operation_ids` — one per uploaded file — which you can poll via `GET /v1/default/banks/{bank_id}/operations` to track progress.

 Upload up to 10 files per request (max 100 MB total). Each file becomes a separate document with optional per-file metadata:

@@ -143,10 +177,11 @@ Upload up to 10 files per request (max 100 MB total). Each file becomes a separa
 Uploaded files are stored server-side (PostgreSQL by default, or S3/GCS/Azure for production). Configure storage via `HINDSIGHT_API_FILE_STORAGE_TYPE`. See [Configuration](../configuration#file-processing) for details.
 :::

+---

 ## Async Ingestion

-For large batches, use async ingestion to avoid blocking:
+For large batches, use async ingestion to avoid blocking your application:

 <Tabs>
 <TabItem value="python" label="Python">
@@ -157,6 +192,8 @@ For large batches, use async ingestion to avoid blocking:
 </TabItem>
 </Tabs>

+When `async: true`, the call returns immediately with an `operation_id`. Processing runs in the background via the worker service. No `usage` metrics are returned for async operations.
+
 ### Cut Costs 50% with Provider Batch APIs

 When using async retain, enable the provider Batch API to reduce LLM fact-extraction costs by 50%. OpenAI and Groq both offer this discount in exchange for a processing window of up to 24 hours — a trade-off that's typically invisible when retain already runs in the background.
@@ -170,50 +207,3 @@ Hindsight submits fact extraction calls as a batch job to the provider, polls fo
 :::note
 Batch API cost savings require `async=true` in your retain request and a compatible provider (OpenAI or Groq).
 :::
-
-## Tagging Memories
-
-Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.
-
-### Tag Individual Items
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-with-tags" language="python" />
-</TabItem>
-</Tabs>
-
-### Apply Tags to All Items in a Batch
-
-Use `document_tags` to apply the same tags to all items in a request:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-with-document-tags" language="python" />
-</TabItem>
-</Tabs>
-
-When both `document_tags` and item-level `tags` are provided, they are merged together.
-
-### Tag Naming Conventions
-
-Use consistent naming patterns for tags:
-
-| Pattern | Example | Use Case |
-|---------|---------|----------|
-| `user:<id>` | `user:alice` | Multi-user agent filtering |
-| `session:<id>` | `session:123` | Session-based scoping |
-| `room:<id>` | `room:general` | Chat room isolation |
-| `topic:<name>` | `topic:feedback` | Topic categorization |
-
-### Listing Tags
-
-Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-list-tags" language="python" />
-</TabItem>
-</Tabs>
-
-See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.
@@ -232,61 +232,10 @@ export HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-5-20250929
 # No API key needed - uses claude auth login credentials
 ```

-:::tip OpenAI Codex & Claude Code Setup
-For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro) and **Claude Code** (Claude Pro/Max), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
+:::tip OpenAI Codex, Claude Code & Vertex AI Setup
+For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro), **Claude Code** (Claude Pro/Max), and **Vertex AI** (Google Cloud), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
 :::

-#### Vertex AI Setup
-
-Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK. Hindsight supports two authentication methods:
-
-**Prerequisites:**
- GCP project with Vertex AI API enabled
- IAM role `roles/aiplatform.user` for your credentials
-
-**Environment Variables:**
-
-| Variable | Description | Required |
-|----------|-------------|----------|
-| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
-| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
-| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
-
-**Authentication Methods:**
-
-1. **Application Default Credentials (ADC)** - Recommended for development
-   ```bash
-   # Setup ADC
-   gcloud auth application-default login
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   ```
-
-2. **Service Account Key** - Recommended for production
-   ```bash
-   # Create service account and download key
-   gcloud iam service-accounts create hindsight-api
-   gcloud projects add-iam-policy-binding your-project-id \
-     --member="serviceAccount:[email protected]" \
-     --role="roles/aiplatform.user"
-   gcloud iam service-accounts keys create key.json \
-     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
-   ```
-
-**Notes:**
- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) - it will be stripped automatically
- The native SDK handles token refresh automatically
- Uses service account credentials if provided, otherwise falls back to ADC
-
 ### Per-Operation LLM Configuration

 Different memory operations have different requirements. **Retain** (fact extraction) benefits from models with strong structured output capabilities, while **Reflect** (reasoning/response generation) can use lighter, faster models. Configure separate LLM models for each operation to optimize for cost and performance.
@@ -77,7 +77,7 @@ docker run --rm -it -p 8888:8888 \
  -e HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai \
  -e HINDSIGHT_API_RERANKER_PROVIDER=cohere \
  -e HINDSIGHT_API_COHERE_API_KEY=$COHERE_API_KEY \
-  ghcr.io/vectorize-io/hindsight:slim
+  ghcr.io/vectorize-io/hindsight:latest-slim
 ```
 - ✅ Dramatically smaller image (~95% reduction on AMD64)
 - ✅ Faster pull/deploy times
@@ -114,7 +114,7 @@ See [Configuration](./configuration#embeddings-and-reranking) for all embedding
 ```bash
 # Standalone (API + Control Plane)
 ghcr.io/vectorize-io/hindsight:latest        # Full, latest release
-ghcr.io/vectorize-io/hindsight:slim          # Slim, latest release
+ghcr.io/vectorize-io/hindsight:latest-slim          # Slim, latest release
 ghcr.io/vectorize-io/hindsight:0.4.9         # Full, specific version
 ghcr.io/vectorize-io/hindsight:0.4.9-slim    # Slim, specific version

@@ -137,6 +137,15 @@ export HINDSIGHT_API_LLM_MODEL=llama3
 export HINDSIGHT_API_LLM_PROVIDER=lmstudio
 export HINDSIGHT_API_LLM_BASE_URL=http://localhost:1234/v1
 export HINDSIGHT_API_LLM_MODEL=your-local-model
+
+# Vertex AI (Google Cloud)
+export HINDSIGHT_API_LLM_PROVIDER=vertexai
+export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
+# Optional: region (default: us-central1)
+# export HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
+# Optional: service account key (otherwise uses ADC)
+# export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
 ```

 **Note:** The LLM is the primary bottleneck for retain operations. See [Performance](./performance) for optimization strategies.
@@ -268,6 +277,59 @@ You can use any model supported by Claude Code CLI.
 - For personal development use only (see Claude Terms of Service)


+---
+
+### Vertex AI Setup (Google Cloud)
+
+Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK.
+
+**Prerequisites:**
+- GCP project with Vertex AI API enabled
+- IAM role `roles/aiplatform.user` for your credentials
+
+**Environment Variables:**
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
+| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
+| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
+
+**Authentication Methods:**
+
+1. **Application Default Credentials (ADC)** - Recommended for development
+   ```bash
+   # Setup ADC
+   gcloud auth application-default login
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   ```
+
+2. **Service Account Key** - Recommended for production
+   ```bash
+   # Create service account and download key
+   gcloud iam service-accounts create hindsight-api
+   gcloud projects add-iam-policy-binding your-project-id \
+     --member="serviceAccount:[email protected]" \
+     --role="roles/aiplatform.user"
+   gcloud iam service-accounts keys create key.json \
+     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
+   ```
+
+**Notes:**
+- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) — it will be stripped automatically
+- The native SDK handles token refresh automatically
+- Uses service account credentials if provided, otherwise falls back to ADC
+
 ---

 ## Embedding Model
@@ -100,21 +100,23 @@ for result in results.results:
 # Basic reflect
 response = client.reflect(
    bank_id="my-bank",
-    query="Should we adopt TypeScript for our backend?"
+    query="Should we adopt TypeScript for our backend?",
+    include_facts=True,
 )

 print(response.text)
-print("\nBased on:", len(response.based_on or []), "facts")
+print("\nBased on:", len(response.based_on.memories if response.based_on else []), "facts")

 # Reflect with options
 response = client.reflect(
    bank_id="my-bank",
    query="What are Alice's strengths for the team lead role?",
-    budget="high"  # More thorough reasoning
+    budget="high",  # More thorough reasoning
+    include_facts=True,
 )

 # See which facts influenced the response
-for fact in response.based_on or []:
+for fact in (response.based_on.memories if response.based_on else []):
    print(f"- {fact.text}")
 # [/docs:main-reflect]

@@ -23,9 +23,26 @@ await client.retain('my-bank', 'Bob is a data scientist who works with Alice');

 // [docs:recall-basic]
 const response = await client.recall('my-bank', 'What does Alice do?');
-for (const r of response.results) {
-    console.log(`${r.text} (score: ${r.weight})`);
-}
+
+// response.results is an array of result objects, each with:
+// - id:            fact ID
+// - text:          the extracted fact
+// - type:          "world", "experience", or "observation"
+// - context:       context label set during retain
+// - metadata:      Record<string, string> set during retain
+// - tags:          string[] of tags
+// - entities:      string[] of entity names linked to this fact
+// - occurredStart: ISO datetime of when the event started
+// - occurredEnd:   ISO datetime of when the event ended
+// - mentionedAt:   ISO datetime of when the fact was retained
+// - documentId:    document this fact belongs to
+// - chunkId:       chunk this fact was extracted from
+
+// Example response.results:
+// [
+//   { id: "a1b2...", text: "Alice works at Google as a software engineer", type: "world", context: "career", ... },
+//   { id: "c3d4...", text: "Alice got promoted to senior engineer", type: "experience", occurredStart: "2024-03-15T00:00:00Z", ... },
+// ]
 // [/docs:recall-basic]


@@ -26,8 +26,26 @@ client.retain(bank_id="my-bank", content="Bob is a data scientist who works with

 # [docs:recall-basic]
 response = client.recall(bank_id="my-bank", query="What does Alice do?")
-for r in response.results:
-    print(f"- {r.text}")
+
+# response.results is a list of RecallResult objects, each with:
+# - id:             fact ID
+# - text:           the extracted fact
+# - type:           "world", "experience", or "observation"
+# - context:        context label set during retain
+# - metadata:       dict[str, str] set during retain
+# - tags:           list of tags
+# - entities:       list of entity name strings linked to this fact
+# - occurred_start: ISO datetime of when the event started
+# - occurred_end:   ISO datetime of when the event ended
+# - mentioned_at:   ISO datetime of when the fact was retained
+# - document_id:    document this fact belongs to
+# - chunk_id:       chunk this fact was extracted from
+
+# Example response.results:
+# [
+#   RecallResult(id="a1b2...", text="Alice works at Google as a software engineer", type="world", context="career", ...),
+#   RecallResult(id="c3d4...", text="Alice got promoted to senior engineer", type="experience", occurred_start="2024-03-15T00:00:00Z", ...),
+# ]
 # [/docs:recall-basic]


@@ -32,9 +32,8 @@ client.reflect(bank_id="my-bank", query="What should I know about Alice?")
 # [docs:reflect-with-params]
 response = client.reflect(
    bank_id="my-bank",
-    query="What do you think about remote work?",
+    query="We're considering a hybrid work policy. What do you think about remote work?",
    budget="mid",
-    context="We're considering a hybrid work policy"
 )
 # [/docs:reflect-with-params]

@@ -72,11 +71,16 @@ response = client.reflect(


 # [docs:reflect-sources]
-response = client.reflect(bank_id="my-bank", query="Tell me about Alice")
+# include_facts=True enables the based_on field in the response
+response = client.reflect(
+    bank_id="my-bank",
+    query="Tell me about Alice",
+    include_facts=True,
+)

 print("Response:", response.text)
 print("\nBased on:")
-for fact in response.based_on or []:
+for fact in (response.based_on.memories if response.based_on else []):
    print(f"  - [{fact.type}] {fact.text}")
 # [/docs:reflect-sources]

@@ -21,6 +21,26 @@ await client.retain('my-bank', 'Alice works at Google as a software engineer');
 // [/docs:retain-basic]


+// [docs:retain-conversation]
+// Retain an entire conversation as a single document.
+// Format each message as "Name (timestamp): text" so the LLM can attribute
+// facts to the right person and resolve temporal references across the thread.
+const conversation = [
+    'Alice (2024-03-15T09:00:00Z): Hi Bob! Did you end up going to the doctor last week?',
+    'Bob (2024-03-15T09:01:00Z): Yes, finally. Turns out I have a mild peanut allergy.',
+    'Alice (2024-03-15T09:02:00Z): Oh no! Are you okay?',
+    'Bob (2024-03-15T09:03:00Z): Yeah, nothing serious. Just need to carry an antihistamine.',
+    'Alice (2024-03-15T09:04:00Z): Good to know. We\'ll avoid peanuts at the team lunch.',
+].join('\n');
+
+await client.retain('my-bank', conversation, {
+    context: 'team chat',
+    timestamp: '2024-03-15T09:04:00Z',
+    documentId: 'chat-2024-03-15-alice-bob',
+});
+// [/docs:retain-conversation]
+
+
 // [docs:retain-with-context]
 await client.retain('my-bank', 'Alice got promoted to senior engineer', {
    context: 'career update',
@@ -29,6 +29,28 @@ client.retain(
 # [/docs:retain-basic]


+# [docs:retain-conversation]
+# Retain an entire conversation as a single document.
+# Format each message as "Name (timestamp): text" so the LLM can attribute
+# facts to the right person and resolve temporal references across the thread.
+conversation = "\n".join([
+    "Alice (2024-03-15T09:00:00Z): Hi Bob! Did you end up going to the doctor last week?",
+    "Bob (2024-03-15T09:01:00Z): Yes, finally. Turns out I have a mild peanut allergy.",
+    "Alice (2024-03-15T09:02:00Z): Oh no! Are you okay?",
+    "Bob (2024-03-15T09:03:00Z): Yeah, nothing serious. Just need to carry an antihistamine.",
+    "Alice (2024-03-15T09:04:00Z): Good to know. We'll avoid peanuts at the team lunch.",
+])
+
+client.retain(
+    bank_id="my-bank",
+    content=conversation,
+    context="team chat",
+    timestamp="2024-03-15T09:04:00Z",
+    document_id="chat-2024-03-15-alice-bob",
+)
+# [/docs:retain-conversation]
+
+
 # [docs:retain-with-context]
 client.retain(
    bank_id="my-bank",
@@ -6,6 +6,14 @@ set -e

 HINDSIGHT_URL="${HINDSIGHT_API_URL:-http://localhost:8888}"

+# =============================================================================
+# Setup (not shown in docs)
+# =============================================================================
+# Create placeholder files for file upload examples
+echo "%PDF-1.4 sample document" > report.pdf
+mkdir -p documents
+cp report.pdf documents/report.pdf
+
 # =============================================================================
 # Doc Examples
 # =============================================================================
@@ -33,9 +41,6 @@ hindsight memory retain-files my-bank report.pdf
 # Upload a directory of files
 hindsight memory retain-files my-bank ./documents/

-# Upload and wait for processing to complete (polls until done)
-hindsight memory retain-files my-bank report.pdf
-
 # Queue files for background processing (returns immediately)
 hindsight memory retain-files my-bank ./documents/ --async
 # [/docs:retain-files]
@@ -6378,7 +6378,8 @@
              }
            ],
            "title": "Document Tags",
-            "description": "Tags applied to all items in this request. These are merged with any item-level tags."
+            "description": "Deprecated. Use item-level tags instead.",
+            "deprecated": true
          }
        },
        "type": "object",
@@ -6389,10 +6390,6 @@
        "description": "Request model for retain endpoint.",
        "example": {
          "async": false,
-          "document_tags": [
-            "user_a",
-            "user_b"
-          ],
          "items": [
            {
              "content": "Alice works at Google",
@@ -4,7 +4,9 @@ sidebar_position: 2

 # Recall Memories

-Retrieve memories using multi-strategy recall.
+Retrieve memories from a bank using multi-strategy recall.
+
+When you **recall**, Hindsight runs four retrieval strategies in parallel — semantic similarity, keyword (BM25), graph traversal, and temporal — then fuses and reranks the results into a single ranked list. The response contains structured facts, not raw documents.

 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
@@ -37,34 +39,19 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

-## Recall Parameters
+---

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Natural language query |
-| `types` | list | all | Filter: `world`, `experience`, `observation` |
-| `budget` | string | "mid" | Budget level: `low`, `mid`, `high` |
-| `max_tokens` | int | 4096 | Token budget for memory facts (text only) |
-| `trace` | bool | false | Enable trace output for debugging |
-| `include_chunks` | bool | false | Include raw text chunks that generated the memories |
-| `max_chunk_tokens` | int | 500 | Token budget for chunks (independent of `max_tokens`) |
-| `include_source_facts` | bool | false | Include source facts for observation-type results (see [Source Facts](#source-facts)) |
-| `max_source_facts_tokens` | int | 4096 | Token budget for source facts |
-| `tags` | list | None | Filter memories by tags (see [Tag Filtering](#filter-by-tags)) |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
+## Parameters

-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-with-options" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={recallMjs} section="recall-with-options" language="javascript" />
-</TabItem>
-</Tabs>
+### query

-## Filter by Fact Type
+The natural language question or statement to search for. This is the only required field. The query drives all four retrieval strategies simultaneously: it is embedded for semantic search, tokenized for BM25 keyword search, used to seed graph traversal, and parsed for temporal expressions. After retrieval, the raw query text is also passed to the cross-encoder reranker to re-score every candidate. Queries exceeding 500 tokens are rejected.

-Recall specific memory types:
+### types
+
+Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (consolidated knowledge synthesized over time). When omitted, all three types are searched.
+
+Each type runs the full four-strategy retrieval pipeline independently, so narrowing `types` reduces both the result set and query cost.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -78,58 +65,12 @@ Recall specific memory types:
 </Tabs>

 :::tip About Observations
-Observations are consolidated knowledge synthesized from multiple facts. They capture patterns, preferences, and learnings that the memory bank has built up over time. Observations are automatically created in the background after retain operations.
+Observations are consolidated knowledge synthesized from multiple facts over time — patterns, preferences, and learnings the memory bank has built up. They are created automatically in the background after retain operations.
 :::

-## Source Facts
+### budget

-When recalling `observation`-type memories, you can fetch the underlying facts they were derived from. This is useful when you need to understand or verify the evidence behind a synthesized observation.
-
-Source facts are returned as a top-level `source_facts` dict keyed by fact ID. Each observation result includes a `source_fact_ids` list for cross-referencing. Facts are deduplicated — if two observations share a source fact, it only appears once.
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-source-facts" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={recallMjs} section="recall-source-facts" language="javascript" />
-</TabItem>
-</Tabs>
-
-:::note Source Facts Token Budget
-Source facts are fetched independently of the main `max_tokens` budget, up to `max_source_facts_tokens`. Facts are included in order of first appearance across all observations — once the budget is reached, remaining source facts are omitted.
-:::
-
-## Token Budget Management
-
-Hindsight is built for AI agents, not humans. Traditional retrieval systems return "top-k" results, but agents don't think in terms of result counts—they think in tokens. An agent's context window is measured in tokens, and that's exactly how Hindsight measures results.
-
-The `max_tokens` parameter lets you control how much of your agent's context budget to spend on memories:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={recallPy} section="recall-token-budget" language="python" />
-</TabItem>
-</Tabs>
-
-This design means you never have to guess whether 10 results or 50 results will fit your context. Just specify the token budget and Hindsight returns as many relevant memories as will fit.
-
-:::note Chunks are Independent
-When `include_chunks=True`, chunks are fetched **independently** of the `max_tokens` filtering. This means:
- Setting `max_tokens=0` will return **0 memory facts** but can still return **chunks** (up to `max_chunk_tokens`)
- Chunks are based on the top-scored (reranked) results **before** token filtering
- Chunks are fetched in batches (batch size estimated as `(max_chunk_tokens / retain_chunk_size) * 2`) until the token budget is exhausted
- This batching approach handles varying chunk sizes across documents efficiently
- This allows you to retrieve raw source text without memory facts when needed
-:::
-
-## Budget Levels
-
-The `budget` parameter controls graph traversal depth:
-
- **"low"**: Fast, shallow retrieval — good for simple lookups
- **"mid"**: Balanced — default for most queries
- **"high"**: Deep exploration — finds indirect connections
+Controls retrieval depth and breadth. Accepted values are `low`, `mid` (default), and `high`. Use `low` for fast simple lookups, `mid` for balanced everyday queries, and `high` when you need to find indirect connections or exhaustive coverage.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -140,11 +81,59 @@ The `budget` parameter controls graph traversal depth:
 </TabItem>
 </Tabs>

-## Filter by Tags
+### max_tokens

-Tags enable **visibility scoping**—filter memories based on tags assigned during [retain](./retain#tagging-memories). This is essential for multi-user agents where each user should only see their own memories.
+The maximum number of tokens the returned facts can collectively occupy. Defaults to `4096`. Only the `text` field of each fact is counted toward this budget — metadata, tags, entities, and other fields are not included. After reranking, facts are included in relevance order until this budget is exhausted — so you always get the most relevant memories that fit. Hindsight is designed for agents, which think in tokens rather than result counts: set `max_tokens` to however much of your context window you want to allocate to memories.

-### Basic Tag Filtering
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={recallPy} section="recall-token-budget" language="python" />
+</TabItem>
+</Tabs>
+
+### query_timestamp
+
+An ISO 8601 datetime representing when the query is being asked, from the user's perspective. When provided, it is used as the anchor for resolving relative temporal expressions in the query — for example, if the query says "last month" and `query_timestamp` is `2023-05-30`, the temporal search window becomes approximately April 2023. Without it, the server's current time is used as the anchor. This field matters most for replaying historical conversations or building agents that need time-anchored recall.
+
+### include
+
+An optional object controlling supplementary data returned alongside the main facts.
+
+#### chunks
+
+When enabled, the response includes the raw source text chunks from which each fact was extracted. Chunks are fetched before the `max_tokens` filter, so setting `max_tokens=0` returns no facts but can still return chunks. The `max_tokens` sub-option (default `8192`) controls the total chunk token budget independently of the main fact budget. This is useful when agents need surrounding context beyond the extracted fact text.
+
+:::note
+When `include_chunks` is enabled, chunks are fetched based on the top-scored reranked results before token filtering. The last chunk is truncated (not dropped) to fit exactly within the budget, and each chunk carries a `truncated` flag indicating whether it was cut.
+:::
+
+#### source_facts
+
+When enabled and `types` includes `observation`, each observation result is accompanied by the original contributing facts it was synthesized from. Source facts are returned in a top-level `source_facts` dict keyed by fact ID, and each observation result carries a `source_fact_ids` list for cross-referencing. Facts are deduplicated across observations. The `max_tokens` sub-option (default `4096`) limits the total token budget for source facts.
+
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={recallPy} section="recall-source-facts" language="python" />
+</TabItem>
+<TabItem value="node" label="Node.js">
+<CodeSnippet code={recallMjs} section="recall-source-facts" language="javascript" />
+</TabItem>
+</Tabs>
+
+#### entities
+
+Enabled by default. When active, each returned fact includes the canonical names of entities associated with it. Set to `null` to skip the entity JOIN query and reduce response size. The `max_tokens` sub-option (default `500`) is a future-facing guard for entity data.
+
+### tags
+
+Filters recall to only memories that match the specified tags. When omitted, all memories regardless of tags are eligible. Tag filtering is applied at the database level across all four retrieval strategies, not as a post-processing step.
+
+The `tags_match` parameter controls the filtering logic:
+
+- `any` (default) — memory matches if it has at least one of the specified tags, or has no tags at all. Use this for "user-specific + shared global" patterns.
+- `any_strict` — memory matches if it has at least one of the specified tags, and untagged memories are excluded. Use this when you want only explicitly scoped memories.
+- `all` — memory matches if it has every specified tag, or has no tags at all.
+- `all_strict` — memory matches if it has every specified tag, and untagged memories are excluded.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -152,38 +141,96 @@ Tags enable **visibility scoping**—filter memories based on tags assigned duri
 </TabItem>
 </Tabs>

-### Tag Match Modes
-
-The `tags_match` parameter controls how tags are matched:
-
-| Mode | Behavior | Untagged Memories |
-|------|----------|-------------------|
-| `any` | OR: memory has ANY of the specified tags | **Included** |
-| `all` | AND: memory has ALL of the specified tags | **Included** |
-| `any_strict` | OR: memory has ANY of the specified tags | **Excluded** |
-| `all_strict` | AND: memory has ALL of the specified tags | **Excluded** |
-
-**Strict modes** are useful when you want to ensure only tagged memories are returned:
-
 <Tabs>
 <TabItem value="python" label="Python">
 <CodeSnippet code={recallPy} section="recall-tags-strict" language="python" />
 </TabItem>
 </Tabs>

-**AND matching** requires all specified tags to be present:
-
 <Tabs>
 <TabItem value="python" label="Python">
 <CodeSnippet code={recallPy} section="recall-tags-all" language="python" />
 </TabItem>
 </Tabs>

-### Use Cases
+### trace

-| Scenario | Tags | Mode | Result |
-|----------|------|------|--------|
-| User A's memories only | `["user:alice"]` | `any_strict` | Only memories tagged `user:alice` |
-| Support + feedback | `["support", "feedback"]` | `any` | Memories with either tag + untagged |
-| Multi-user room | `["user:alice", "room:general"]` | `all_strict` | Only memories with both tags |
-| Global + user-specific | `["user:alice"]` | `any` | Alice's memories + shared (untagged) |
+When set to `true`, the response includes a detailed debug trace covering the query embedding, entry points, per-strategy retrieval results, RRF fusion candidates, reranked results, temporal constraints detected, and per-phase timings. Has no effect on the retrieval logic itself. Useful for understanding why specific memories were or were not returned.
+
+---
+
+## Response
+
+### results
+
+The main list of recalled facts, ordered by relevance. Relevance is computed by running four retrieval strategies in parallel — semantic similarity, BM25 keyword, graph traversal, and temporal — fusing their rankings with Reciprocal Rank Fusion (RRF), then re-scoring the merged candidates with a cross-encoder reranker against the original query.
+
+Results do not include a numeric score. Raw retrieval scores are not meaningful on an absolute scale — a score of 0.8 from one query tells you nothing useful compared to a score of 0.8 from another. What matters is the relative ordering, which is already reflected in the list order. Agents should consume memories in order and let `max_tokens` determine how many fit, rather than filtering by score.
+
+Each item in `results` has the following fields:
+
+#### id
+
+The unique identifier of this fact. Use it to cross-reference with `source_facts` or for application-level deduplication.
+
+#### text
+
+The extracted fact text as stored in the memory bank.
+
+#### type
+
+The fact category: `world` for objective information, `experience` for events and conversations, or `observation` for consolidated knowledge synthesized over time.
+
+#### context
+
+The context label provided when the fact was retained (e.g., `"team meeting"`, `"slack"`). `null` if none was set.
+
+#### metadata
+
+The key-value string pairs attached when the fact was retained. `null` if none were set.
+
+#### tags
+
+The visibility-scoping tags attached to this fact.
+
+#### entities
+
+A list of canonical entity name strings linked to this fact. Only populated when `include.entities` is enabled (the default). `null` otherwise.
+
+#### occurred_start / occurred_end
+
+ISO 8601 datetimes representing when the described event started and ended. Extracted by the LLM from the content during retain. `null` if the content had no temporal information.
+
+#### mentioned_at
+
+ISO 8601 datetime of when this fact was retained into the bank.
+
+#### document_id
+
+The document ID this fact belongs to, as set during retain.
+
+#### chunk_id
+
+The ID of the source text chunk this fact was extracted from. Used to cross-reference with `chunks` in the response when `include.chunks` is enabled.
+
+#### source_fact_ids
+
+For `observation`-type results only: the IDs of the original facts this observation was synthesized from. Cross-references with `source_facts` in the response. `null` for other types or when `include.source_facts` is not enabled.
+
+---
+
+### source_facts
+
+A dict keyed by fact ID containing full `RecallResult` objects for the source facts that contributed to observation results. Only present when `include.source_facts` is enabled. Facts are deduplicated — if two observations share a source fact, it appears once.
+
+### chunks
+
+A dict keyed by chunk ID containing the raw source text chunks associated with the returned facts. Only present when `include.chunks` is enabled. Each chunk has `id`, `text`, `chunk_index`, and `truncated` (whether the text was cut to fit the token budget).
+
+### entities
+
+A dict keyed by canonical entity name containing entity state objects. Only present when `include.entities` is enabled. Each entry has `entity_id`, `canonical_name`, and `observations`.
+
+### trace
+
+A debug object present only when `trace: true` was set in the request. Contains per-phase timings, retrieval breakdowns, and RRF fusion details.
@@ -4,14 +4,9 @@ sidebar_position: 3

 # Reflect

-Generate disposition-aware responses using an agentic reasoning loop.
+Generate a grounded, disposition-aware response using an agentic reasoning loop.

-When you call **reflect**, Hindsight runs an **agentic loop** that:
-1. **Autonomously searches** for relevant information using multiple tools
-2. **Applies** the bank's disposition traits to shape the reasoning style
-3. **Generates** a grounded answer with citations to the sources used
-
-The agent has access to hierarchical retrieval tools (mental models → observations → raw facts) and decides what information it needs to answer your query.
+When you call **reflect**, Hindsight runs an agentic loop that autonomously searches the memory bank using multiple retrieval tools, applies the bank's disposition traits to shape the reasoning style, and produces a final answer grounded in what it found. Unlike recall — which returns raw facts — reflect returns a synthesized response written by the LLM.

 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
@@ -44,33 +39,17 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

+---
+
 ## Parameters

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Question or prompt |
-| `budget` | string | "low" | Budget level: `low`, `mid`, `high` (see below) |
-| `max_tokens` | int | 4096 | Maximum tokens for the final response |
-| `response_schema` | object | None | JSON Schema for [structured output](#structured-output) |
-| `tags` | list | None | Filter memories by tags during reflection |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
-| `trace` | bool | false | Include detailed agent trace in response |
+### query

-### Budget
+The question or prompt to reflect on. This is the only required field. If you have situational context that should influence the answer, include it directly in the query rather than as a separate field.

-The `budget` parameter controls the research depth — how thoroughly the agent explores before answering:
+### budget

-| Budget | Research Depth | Use Case |
-|--------|----------------|----------|
-| `low` | Shallow | Quick answers, simple lookups. Prioritizes speed over completeness. |
-| `mid` | Moderate | Balanced exploration. Checks multiple sources when warranted. |
-| `high` | Deep | Comprehensive analysis. Explores all knowledge levels, uses multiple query variations. |
-
-Use `high` for complex questions that require synthesizing information from multiple sources or verifying facts across different retrieval levels.
-
-### Max Tokens
-
-The `max_tokens` parameter limits the length of the final generated response. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.
+Controls how thoroughly the agent explores the memory bank before answering. Accepted values are `low` (default), `mid`, and `high`. At `low`, the agent does a shallow search optimized for speed. At `mid`, it checks multiple sources when the question warrants it. At `high`, it performs deep exploration across all knowledge levels and may use multiple query variations to find indirect connections. Use `high` for complex questions that require synthesizing information from many sources.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -81,45 +60,13 @@ The `max_tokens` parameter limits the length of the final generated response. Th
 </TabItem>
 </Tabs>

-## Disposition Influence
+### max_tokens

-The bank's disposition affects reflect responses:
+Limits the length of the final generated response. Defaults to `4096`. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.

-| Trait | Low (1) | High (5) |
-|-------|---------|----------|
-| **Skepticism** | Trusting, accepts claims | Questions and doubts claims |
-| **Literalism** | Flexible interpretation | Exact, literal interpretation |
-| **Empathy** | Detached, fact-focused | Considers emotional context |
+### response_schema

-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={reflectPy} section="reflect-disposition" language="python" />
-</TabItem>
-<TabItem value="node" label="Node.js">
-<CodeSnippet code={reflectMjs} section="reflect-disposition" language="javascript" />
-</TabItem>
-</Tabs>
-
-## Citations
-
-The response includes a `based_on` field that shows which sources were used:
-
- `based_on.memories` — Memory facts (world, experience) that were retrieved and cited
- `based_on.mental_models` — User-curated mental models that were used
- `based_on.directives` — Directives that were enforced
-
-**Important:** Only IDs that were actually retrieved during the agent loop can be cited. The agent validates citations to prevent hallucinated references.
-
-This enables:
- **Transparency** — users see exactly which sources informed the answer
- **Verification** — check if the response is grounded in actual memories
- **Debugging** — use `trace=True` for detailed tool call logs
-
-## Structured Output
-
-For applications that need to process responses programmatically, you can request structured output by providing a JSON Schema via `response_schema`. When provided, the response includes a `structured_output` field with the LLM response parsed according to the schema. The `text` field will be empty since only a single LLM call is made for efficiency.
-
-The easiest way to define a schema is using **Pydantic models**:
+An optional JSON Schema object. When provided, the LLM generates a response that conforms to the schema and the response includes a `structured_output` field with the result parsed accordingly. The `text` field will be empty since only a single structured LLM call is made. Use this when you need to process the response programmatically rather than display it as prose.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -133,22 +80,9 @@ The easiest way to define a schema is using **Pydantic models**:
 </TabItem>
 </Tabs>

-| Use Case | Why Structured Output Helps |
-|----------|----------------------------|
-| **Decision pipelines** | Parse recommendations into workflow systems |
-| **Dashboards** | Extract confidence scores, risk factors for visualization |
-| **Multi-agent systems** | Pass structured data between agents |
-| **Auditing** | Log structured decisions with clear reasoning |
+### tags

-**Tips:**
- Use Pydantic's `model_json_schema()` for type-safe schema generation
- Use `model_validate()` to parse the response back into your Pydantic model
- Keep schemas focused — extract only what you need
- Use `Optional` fields for data that may not always be available
-
-## Filter by Tags
-
-Like [recall](./recall#filter-by-tags), reflect supports tag filtering to scope which memories are considered during reasoning. This is essential for multi-user scenarios where reflection should only consider memories relevant to a specific user.
+Filters which memories the agent can access during reflection. Works identically to [recall tags](./recall#tags) — only memories matching the specified tags are considered. The `tags_match` parameter controls the matching logic (`any`, `all`, `any_strict`, `all_strict`) with the same semantics as recall.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -156,13 +90,51 @@ Like [recall](./recall#filter-by-tags), reflect supports tag filtering to scope
 </TabItem>
 </Tabs>

-The `tags_match` parameter works the same as in recall:
+### include

-| Mode | Behavior |
-|------|----------|
-| `any` | OR matching, includes untagged memories |
-| `all` | AND matching, includes untagged memories |
-| `any_strict` | OR matching, excludes untagged memories |
-| `all_strict` | AND matching, excludes untagged memories |
+Controls optional supplementary data returned alongside the main response.

-See [Retain API](./retain#tagging-memories) for how to tag memories and [Recall API](./recall#filter-by-tags) for more details on tag matching modes.
+#### include.facts
+
+When enabled, the response includes a `based_on` object listing the memories, mental models, and directives the agent actually used to construct the answer. Only sources retrieved during the agent loop can appear here — citations are validated to prevent hallucinated references. Useful for transparency and verification.
+
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={reflectPy} section="reflect-sources" language="python" />
+</TabItem>
+</Tabs>
+
+#### include.tool_calls
+
+When enabled, the response includes a `trace` object with the full execution log of every tool call and LLM call made during the agentic loop, including inputs, outputs, and durations. Set `output: false` to include only tool inputs for a smaller payload. Useful for debugging why the agent reached a particular conclusion.
+
+---
+
+## Response
+
+### text
+
+The synthesized answer as a well-formatted markdown string. This is the primary output of reflect. Empty when `response_schema` is provided (use `structured_output` instead in that case).
+
+### structured_output
+
+The LLM's response parsed according to the `response_schema` provided in the request. Only present when `response_schema` was set. `null` otherwise.
+
+### based_on
+
+The sources the agent used to construct the answer. Only present when `include.facts` was enabled. Contains three fields:
+
+- `memories` — a list of memory facts (world, experience, observation) that were retrieved and cited. Each item has `id`, `text`, `type`, `context`, `occurred_start`, and `occurred_end`.
+- `mental_models` — a list of mental models that were used. Each item has `id`, `text`, and `context`.
+- `directives` — a list of directives that were enforced during reasoning. Each item has `id`, `name`, and `content`.
+
+### usage
+
+Token usage for all LLM calls made during the agentic loop: `input_tokens`, `output_tokens`, and `total_tokens`. Useful for cost tracking.
+
+### trace
+
+The full execution log of the agentic loop. Only present when `include.tool_calls` was enabled. Contains:
+
+- `tool_calls` — each tool invocation with `tool` name (`lookup`, `recall`, `learn`, `expand`), `input`, `output` (if `output: true`), `duration_ms`, and `iteration` number.
+- `llm_calls` — each LLM call with `scope` (e.g., `"agent_1"`, `"final"`) and `duration_ms`.
@@ -25,7 +25,9 @@ Learn about fact extraction, entity resolution, and graph construction in the [R
 Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
 :::

-## Store a Single Memory
+## Store a Document
+
+A single retain call accepts one or more **items**. Each item is a piece of raw content — a conversation, a document, a note — that Hindsight will analyze and decompose into one or many memories. The content itself is never stored verbatim; what gets stored are the structured facts the LLM extracts from it.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -39,18 +41,40 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client
 </TabItem>
 </Tabs>

-## The Importance of Context
+### Retaining a Conversation

-The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
+A full conversation should be retained as a single item. The LLM can parse any format — plain text, JSON, Markdown, or any structured representation — as long as it clearly conveys who said what and when. The example below uses a simple `Name (timestamp): text` format.

-**Why context matters:**
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
+<Tabs>
+<TabItem value="python" label="Python">
+<CodeSnippet code={retainPy} section="retain-conversation" language="python" />
+</TabItem>
+<TabItem value="node" label="Node.js">
+<CodeSnippet code={retainMjs} section="retain-conversation" language="javascript" />
+</TabItem>
+</Tabs>

-## Store with Context and Date
+When the conversation grows — a new message arrives — just retain again with the full updated content and the same `document_id`. Hindsight will delete the previous version and reprocess from scratch, so memories always reflect the latest state of the conversation.

-Always provide context and event dates for optimal memory extraction:
+---
+
+## Parameters
+
+### content
+
+The raw text to store. This is the only required field. Hindsight chunks the content, sends each chunk to the LLM for fact extraction, and stores the resulting structured facts — not the original text. A single `content` value can produce many memories depending on how much information it contains.
+
+### timestamp
+
+When the event described in the content actually occurred. Accepts any ISO 8601 string (e.g., `"2024-01-15T10:30:00Z"`). If omitted, defaults to the current time at ingestion.
+
+The timestamp is injected verbatim into the LLM fact-extraction prompt so the model can resolve relative temporal references in the content — for example, if the content says "last Monday", the model uses the provided timestamp as the anchor to pin down the actual date. It also enables temporal recall queries like "What happened last spring?" to work correctly.
+
+### context
+
+A short label describing the source or situation — for example `"team meeting"`, `"slack"`, or `"support ticket"`. It is injected directly into the LLM prompt, so it actively shapes how facts are extracted. The same sentence can mean something very different depending on context: "the project was terminated" in a `"performance review"` context versus a `"product roadmap"` context produces different memories.
+
+Providing context consistently is one of the highest-leverage things you can do to improve memory quality.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -64,30 +88,47 @@ Always provide context and event dates for optimal memory extraction:
 </TabItem>
 </Tabs>

-The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
+### metadata

-### Response Fields
+Arbitrary key-value string pairs attached to every fact extracted from this item. For example: `{"source": "slack", "channel": "engineering", "thread_id": "T123"}`. The LLM never sees this field — it is passed through as-is and stored on each memory unit. During recall, every returned memory includes its metadata, which lets you do client-side filtering or static enrichment without extra lookups — for example, linking a memory back to its source URL, thread ID, or any application-specific identifier.

-The retain response includes:
+### document_id

-| Field | Type | Description |
-|-------|------|-------------|
-| `success` | bool | Whether the operation succeeded |
-| `bank_id` | string | The memory bank ID |
-| `items_count` | int | Number of items processed |
-| `async` | bool | Whether processed asynchronously |
-| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |
+A caller-supplied string that groups one or more items under a logical document. This field is the key to making retain **idempotent**.

-The `usage` field contains token metrics for cost tracking:
- `input_tokens`: Tokens consumed by prompts
- `output_tokens`: Tokens generated by the LLM
- `total_tokens`: Sum of input and output tokens
+When you provide a `document_id`, Hindsight upserts the document: if a document with that ID already exists in the bank, it and all its associated memories are deleted before the new content is processed and inserted. This means you can safely re-run retain on updated content — for example, a chat thread that grew since last time — without accumulating duplicate memories.

-Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.
+If you omit `document_id`, Hindsight assigns a random UUID per request, so re-ingesting the same content will create duplicate memories.
+
+### entities
+
+A list of entities you want to guarantee are recognized, merged with any entities the LLM extracts automatically. Each entry has a `text` field (the entity name) and an optional `type` (e.g., `"PERSON"`, `"ORG"`, `"CONCEPT"` — defaults to `"CONCEPT"` if omitted).
+
+Use this when you know certain entities are important but the LLM might miss them or refer to them inconsistently across different parts of the content. Providing entities explicitly ensures they are always linked in the knowledge graph.
+
+### tags and document_tags
+
+Tags control **visibility scoping** — which memories are visible during recall. A memory is only returned if its tags intersect with the tags filter provided in the recall request. This makes tags useful when a single memory bank serves multiple users or sessions and each should only see their own memories.
+
+Use consistent naming patterns to keep tag filtering predictable. Common conventions: `user:<id>` for per-user scoping, `session:<id>` for session isolation, `room:<id>` for chat rooms, `topic:<name>` for category filtering. The bank also exposes a list-tags endpoint that returns all tags with their memory counts, useful for UI autocomplete or wildcard expansion.
+
+See [Recall API](./recall#filter-by-tags) for filtering by tags during retrieval.
+
+### Response
+
+The synchronous retain response includes:
+
+- `success` — whether the operation completed without errors
+- `bank_id` — the memory bank that received the content
+- `items_count` — number of items processed
+- `async` — whether processing ran asynchronously
+- `usage` — token usage for the LLM calls (`input_tokens`, `output_tokens`, `total_tokens`), only present for synchronous operations
+
+---

 ## Batch Ingestion

-Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
+Multiple items can be submitted in a single request. Batch ingestion is the recommended approach — it reduces network overhead and lets Hindsight optimize extraction across related content.

 <Tabs>
 <TabItem value="python" label="Python">
@@ -98,11 +139,12 @@ Store multiple items in a single request. **Batch ingestion is the recommended a
 </TabItem>
 </Tabs>

-The `document_id` groups related memories for later management.

-## Store from Files
+---

-Upload files directly — Hindsight automatically converts them to text and extracts memories. File processing always runs asynchronously and returns operation IDs for tracking.
+## Files
+
+Upload files directly — Hindsight converts them to text and extracts memories automatically. File processing always runs asynchronously and returns operation IDs for tracking.

 **Supported formats:** PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, images (JPG, PNG, GIF, etc. — OCR), audio (MP3, WAV, FLAC, etc. — transcription), HTML, and plain text formats (TXT, MD, CSV, JSON, YAML, etc.)

@@ -121,15 +163,7 @@ Upload files directly — Hindsight automatically converts them to text and extr
 </TabItem>
 </Tabs>

-### File Retain Response
-
-The file retain endpoint always returns asynchronously:
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `operation_ids` | string[] | One operation ID per uploaded file. Use `GET /v1/default/banks/{bank_id}/operations` to track progress. |
-
-### Batch File Uploads
+The file retain endpoint always returns asynchronously. The response contains `operation_ids` — one per uploaded file — which you can poll via `GET /v1/default/banks/{bank_id}/operations` to track progress.

 Upload up to 10 files per request (max 100 MB total). Each file becomes a separate document with optional per-file metadata:

@@ -143,10 +177,11 @@ Upload up to 10 files per request (max 100 MB total). Each file becomes a separa
 Uploaded files are stored server-side (PostgreSQL by default, or S3/GCS/Azure for production). Configure storage via `HINDSIGHT_API_FILE_STORAGE_TYPE`. See [Configuration](../configuration#file-processing) for details.
 :::

+---

 ## Async Ingestion

-For large batches, use async ingestion to avoid blocking:
+For large batches, use async ingestion to avoid blocking your application:

 <Tabs>
 <TabItem value="python" label="Python">
@@ -157,6 +192,8 @@ For large batches, use async ingestion to avoid blocking:
 </TabItem>
 </Tabs>

+When `async: true`, the call returns immediately with an `operation_id`. Processing runs in the background via the worker service. No `usage` metrics are returned for async operations.
+
 ### Cut Costs 50% with Provider Batch APIs

 When using async retain, enable the provider Batch API to reduce LLM fact-extraction costs by 50%. OpenAI and Groq both offer this discount in exchange for a processing window of up to 24 hours — a trade-off that's typically invisible when retain already runs in the background.
@@ -170,50 +207,3 @@ Hindsight submits fact extraction calls as a batch job to the provider, polls fo
 :::note
 Batch API cost savings require `async=true` in your retain request and a compatible provider (OpenAI or Groq).
 :::
-
-## Tagging Memories
-
-Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.
-
-### Tag Individual Items
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-with-tags" language="python" />
-</TabItem>
-</Tabs>
-
-### Apply Tags to All Items in a Batch
-
-Use `document_tags` to apply the same tags to all items in a request:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-with-document-tags" language="python" />
-</TabItem>
-</Tabs>
-
-When both `document_tags` and item-level `tags` are provided, they are merged together.
-
-### Tag Naming Conventions
-
-Use consistent naming patterns for tags:
-
-| Pattern | Example | Use Case |
-|---------|---------|----------|
-| `user:<id>` | `user:alice` | Multi-user agent filtering |
-| `session:<id>` | `session:123` | Session-based scoping |
-| `room:<id>` | `room:general` | Chat room isolation |
-| `topic:<name>` | `topic:feedback` | Topic categorization |
-
-### Listing Tags
-
-Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:
-
-<Tabs>
-<TabItem value="python" label="Python">
-<CodeSnippet code={retainPy} section="retain-list-tags" language="python" />
-</TabItem>
-</Tabs>
-
-See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.
@@ -232,61 +232,10 @@ export HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-5-20250929
 # No API key needed - uses claude auth login credentials
 ```

-:::tip OpenAI Codex & Claude Code Setup
-For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro) and **Claude Code** (Claude Pro/Max), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
+:::tip OpenAI Codex, Claude Code & Vertex AI Setup
+For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro), **Claude Code** (Claude Pro/Max), and **Vertex AI** (Google Cloud), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
 :::

-#### Vertex AI Setup
-
-Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK. Hindsight supports two authentication methods:
-
-**Prerequisites:**
- GCP project with Vertex AI API enabled
- IAM role `roles/aiplatform.user` for your credentials
-
-**Environment Variables:**
-
-| Variable | Description | Required |
-|----------|-------------|----------|
-| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
-| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
-| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
-
-**Authentication Methods:**
-
-1. **Application Default Credentials (ADC)** - Recommended for development
-   ```bash
-   # Setup ADC
-   gcloud auth application-default login
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   ```
-
-2. **Service Account Key** - Recommended for production
-   ```bash
-   # Create service account and download key
-   gcloud iam service-accounts create hindsight-api
-   gcloud projects add-iam-policy-binding your-project-id \
-     --member="serviceAccount:[email protected]" \
-     --role="roles/aiplatform.user"
-   gcloud iam service-accounts keys create key.json \
-     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
-   ```
-
-**Notes:**
- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) - it will be stripped automatically
- The native SDK handles token refresh automatically
- Uses service account credentials if provided, otherwise falls back to ADC
-
 ### Per-Operation LLM Configuration

 Different memory operations have different requirements. **Retain** (fact extraction) benefits from models with strong structured output capabilities, while **Reflect** (reasoning/response generation) can use lighter, faster models. Configure separate LLM models for each operation to optimize for cost and performance.
@@ -77,7 +77,7 @@ docker run --rm -it -p 8888:8888 \
  -e HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai \
  -e HINDSIGHT_API_RERANKER_PROVIDER=cohere \
  -e HINDSIGHT_API_COHERE_API_KEY=$COHERE_API_KEY \
-  ghcr.io/vectorize-io/hindsight:slim
+  ghcr.io/vectorize-io/hindsight:latest-slim
 ```
 - ✅ Dramatically smaller image (~95% reduction on AMD64)
 - ✅ Faster pull/deploy times
@@ -114,7 +114,7 @@ See [Configuration](./configuration#embeddings-and-reranking) for all embedding
 ```bash
 # Standalone (API + Control Plane)
 ghcr.io/vectorize-io/hindsight:latest        # Full, latest release
-ghcr.io/vectorize-io/hindsight:slim          # Slim, latest release
+ghcr.io/vectorize-io/hindsight:latest-slim          # Slim, latest release
 ghcr.io/vectorize-io/hindsight:0.4.9         # Full, specific version
 ghcr.io/vectorize-io/hindsight:0.4.9-slim    # Slim, specific version

@@ -137,6 +137,15 @@ export HINDSIGHT_API_LLM_MODEL=llama3
 export HINDSIGHT_API_LLM_PROVIDER=lmstudio
 export HINDSIGHT_API_LLM_BASE_URL=http://localhost:1234/v1
 export HINDSIGHT_API_LLM_MODEL=your-local-model
+
+# Vertex AI (Google Cloud)
+export HINDSIGHT_API_LLM_PROVIDER=vertexai
+export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
+# Optional: region (default: us-central1)
+# export HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
+# Optional: service account key (otherwise uses ADC)
+# export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
 ```

 **Note:** The LLM is the primary bottleneck for retain operations. See [Performance](./performance) for optimization strategies.
@@ -268,6 +277,59 @@ You can use any model supported by Claude Code CLI.
 - For personal development use only (see Claude Terms of Service)


+---
+
+### Vertex AI Setup (Google Cloud)
+
+Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK.
+
+**Prerequisites:**
+- GCP project with Vertex AI API enabled
+- IAM role `roles/aiplatform.user` for your credentials
+
+**Environment Variables:**
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
+| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
+| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
+
+**Authentication Methods:**
+
+1. **Application Default Credentials (ADC)** - Recommended for development
+   ```bash
+   # Setup ADC
+   gcloud auth application-default login
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   ```
+
+2. **Service Account Key** - Recommended for production
+   ```bash
+   # Create service account and download key
+   gcloud iam service-accounts create hindsight-api
+   gcloud projects add-iam-policy-binding your-project-id \
+     --member="serviceAccount:[email protected]" \
+     --role="roles/aiplatform.user"
+   gcloud iam service-accounts keys create key.json \
+     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
+   ```
+
+**Notes:**
+- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) — it will be stripped automatically
+- The native SDK handles token refresh automatically
+- Uses service account credentials if provided, otherwise falls back to ADC
+
 ---

 ## Embedding Model
@@ -211,7 +211,10 @@
        {
          "type": "doc",
          "id": "sdks/integrations/crewai",
-          "label": "CrewAI"
+          "label": "CrewAI",
+          "customProps": {
+            "icon": "/img/icons/crewai.png"
+          }
        },
        {
          "type": "doc",
@@ -1,7 +1,9 @@

 # Recall Memories

-Retrieve memories using multi-strategy recall.
+Retrieve memories from a bank using multi-strategy recall.
+
+When you **recall**, Hindsight runs four retrieval strategies in parallel — semantic similarity, keyword (BM25), graph traversal, and temporal — then fuses and reranks the results into a single ranked list. The response contains structured facts, not raw documents.

 {/* Import raw source files */}

@@ -16,17 +18,52 @@ Make sure you've completed the [Quick Start](./quickstart) to install the client

 ```python
 response = client.recall(bank_id="my-bank", query="What does Alice do?")
-for r in response.results:
-    print(f"- {r.text}")
+
+# response.results is a list of RecallResult objects, each with:
+# - id:             fact ID
+# - text:           the extracted fact
+# - type:           "world", "experience", or "observation"
+# - context:        context label set during retain
+# - metadata:       dict[str, str] set during retain
+# - tags:           list of tags
+# - entities:       list of entity name strings linked to this fact
+# - occurred_start: ISO datetime of when the event started
+# - occurred_end:   ISO datetime of when the event ended
+# - mentioned_at:   ISO datetime of when the fact was retained
+# - document_id:    document this fact belongs to
+# - chunk_id:       chunk this fact was extracted from
+
+# Example response.results:
+# [
+#   RecallResult(id="a1b2...", text="Alice works at Google as a software engineer", type="world", context="career", ...),
+#   RecallResult(id="c3d4...", text="Alice got promoted to senior engineer", type="experience", occurred_start="2024-03-15T00:00:00Z", ...),
+# ]
 ```

 ### Node.js

 ```javascript
 const response = await client.recall('my-bank', 'What does Alice do?');
-for (const r of response.results) {
-    console.log(`${r.text} (score: ${r.weight})`);
-}
+
+// response.results is an array of result objects, each with:
+// - id:            fact ID
+// - text:          the extracted fact
+// - type:          "world", "experience", or "observation"
+// - context:       context label set during retain
+// - metadata:      Record<string, string> set during retain
+// - tags:          string[] of tags
+// - entities:      string[] of entity names linked to this fact
+// - occurredStart: ISO datetime of when the event started
+// - occurredEnd:   ISO datetime of when the event ended
+// - mentionedAt:   ISO datetime of when the fact was retained
+// - documentId:    document this fact belongs to
+// - chunkId:       chunk this fact was extracted from
+
+// Example response.results:
+// [
+//   { id: "a1b2...", text: "Alice works at Google as a software engineer", type: "world", context: "career", ... },
+//   { id: "c3d4...", text: "Alice got promoted to senior engineer", type: "experience", occurredStart: "2024-03-15T00:00:00Z", ... },
+// ]
 ```

 ### CLI
@@ -35,58 +72,19 @@ for (const r of response.results) {
 hindsight memory recall my-bank "What does Alice do?"
 ```

-## Recall Parameters
+---

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Natural language query |
-| `types` | list | all | Filter: `world`, `experience`, `observation` |
-| `budget` | string | "mid" | Budget level: `low`, `mid`, `high` |
-| `max_tokens` | int | 4096 | Token budget for memory facts (text only) |
-| `trace` | bool | false | Enable trace output for debugging |
-| `include_chunks` | bool | false | Include raw text chunks that generated the memories |
-| `max_chunk_tokens` | int | 500 | Token budget for chunks (independent of `max_tokens`) |
-| `include_source_facts` | bool | false | Include source facts for observation-type results (see [Source Facts](#source-facts)) |
-| `max_source_facts_tokens` | int | 4096 | Token budget for source facts |
-| `tags` | list | None | Filter memories by tags (see [Tag Filtering](#filter-by-tags)) |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
+## Parameters

-### Python
+### query

-```python
-response = client.recall(
-    bank_id="my-bank",
-    query="What does Alice do?",
-    types=["world", "experience"],
-    budget="high",
-    max_tokens=8000,
-    trace=True,
-)
+The natural language question or statement to search for. This is the only required field. The query drives all four retrieval strategies simultaneously: it is embedded for semantic search, tokenized for BM25 keyword search, used to seed graph traversal, and parsed for temporal expressions. After retrieval, the raw query text is also passed to the cross-encoder reranker to re-score every candidate. Queries exceeding 500 tokens are rejected.

-# Access results
-for r in response.results:
-    print(f"- {r.text}")
-```
+### types

-### Node.js
+Controls which categories of memory facts are searched. Accepted values are `world` (objective facts), `experience` (events and conversations), and `observation` (consolidated knowledge synthesized over time). When omitted, all three types are searched.

-```javascript
-const detailedResponse = await client.recall('my-bank', 'What does Alice do?', {
-    types: ['world', 'experience'],
-    budget: 'high',
-    maxTokens: 8000,
-    trace: true
-});
-
-// Access results
-for (const r of detailedResponse.results) {
-    console.log(`${r.text} (score: ${r.weight})`);
-}
-```
-
-## Filter by Fact Type
-
-Recall specific memory types:
+Each type runs the full four-strategy retrieval pipeline independently, so narrowing `types` reduces both the result set and query cost.

 ### Python

@@ -123,12 +121,62 @@ hindsight memory recall my-bank "query" --fact-type world,observation

 > **💡 About Observations**
 > 
-Observations are consolidated knowledge synthesized from multiple facts. They capture patterns, preferences, and learnings that the memory bank has built up over time. Observations are automatically created in the background after retain operations.
-## Source Facts
+Observations are consolidated knowledge synthesized from multiple facts over time — patterns, preferences, and learnings the memory bank has built up. They are created automatically in the background after retain operations.
+### budget

-When recalling `observation`-type memories, you can fetch the underlying facts they were derived from. This is useful when you need to understand or verify the evidence behind a synthesized observation.
+Controls retrieval depth and breadth. Accepted values are `low`, `mid` (default), and `high`. Use `low` for fast simple lookups, `mid` for balanced everyday queries, and `high` when you need to find indirect connections or exhaustive coverage.

-Source facts are returned as a top-level `source_facts` dict keyed by fact ID. Each observation result includes a `source_fact_ids` list for cross-referencing. Facts are deduplicated — if two observations share a source fact, it only appears once.
+### Python
+
+```python
+# Quick lookup
+results = client.recall(bank_id="my-bank", query="Alice's email", budget="low")
+
+# Deep exploration
+results = client.recall(bank_id="my-bank", query="How are Alice and Bob connected?", budget="high")
+```
+
+### Node.js
+
+```javascript
+// Quick lookup
+const quickResults = await client.recall('my-bank', "Alice's email", { budget: 'low' });
+
+// Deep exploration
+const deepResults = await client.recall('my-bank', 'How are Alice and Bob connected?', { budget: 'high' });
+```
+
+### max_tokens
+
+The maximum number of tokens the returned facts can collectively occupy. Defaults to `4096`. Only the `text` field of each fact is counted toward this budget — metadata, tags, entities, and other fields are not included. After reranking, facts are included in relevance order until this budget is exhausted — so you always get the most relevant memories that fit. Hindsight is designed for agents, which think in tokens rather than result counts: set `max_tokens` to however much of your context window you want to allocate to memories.
+
+### Python
+
+```python
+# Fill up to 4K tokens of context with relevant memories
+results = client.recall(bank_id="my-bank", query="What do I know about Alice?", max_tokens=4096)
+
+# Smaller budget for quick lookups
+results = client.recall(bank_id="my-bank", query="Alice's email", max_tokens=500)
+```
+
+### query_timestamp
+
+An ISO 8601 datetime representing when the query is being asked, from the user's perspective. When provided, it is used as the anchor for resolving relative temporal expressions in the query — for example, if the query says "last month" and `query_timestamp` is `2023-05-30`, the temporal search window becomes approximately April 2023. Without it, the server's current time is used as the anchor. This field matters most for replaying historical conversations or building agents that need time-anchored recall.
+
+### include
+
+An optional object controlling supplementary data returned alongside the main facts.
+
+#### chunks
+
+When enabled, the response includes the raw source text chunks from which each fact was extracted. Chunks are fetched before the `max_tokens` filter, so setting `max_tokens=0` returns no facts but can still return chunks. The `max_tokens` sub-option (default `8192`) controls the total chunk token budget independently of the main fact budget. This is useful when agents need surrounding context beyond the extracted fact text.
+
+:::note
+When `include_chunks` is enabled, chunks are fetched based on the top-scored reranked results before token filtering. The last chunk is truncated (not dropped) to fit exactly within the budget, and each chunk carries a `truncated` flag indicating whether it was cut.
+#### source_facts
+
+When enabled and `types` includes `observation`, each observation result is accompanied by the original contributing facts it was synthesized from. Source facts are returned in a top-level `source_facts` dict keyed by fact ID, and each observation result carries a `source_fact_ids` list for cross-referencing. Facts are deduplicated across observations. The `max_tokens` sub-option (default `4096`) limits the total token budget for source facts.

 ### Python

@@ -174,68 +222,20 @@ for (const obs of obsResponse.results) {
 }
 ```

-> **📝 Source Facts Token Budget**
-> 
-Source facts are fetched independently of the main `max_tokens` budget, up to `max_source_facts_tokens`. Facts are included in order of first appearance across all observations — once the budget is reached, remaining source facts are omitted.
-## Token Budget Management
+#### entities

-Hindsight is built for AI agents, not humans. Traditional retrieval systems return "top-k" results, but agents don't think in terms of result counts—they think in tokens. An agent's context window is measured in tokens, and that's exactly how Hindsight measures results.
+Enabled by default. When active, each returned fact includes the canonical names of entities associated with it. Set to `null` to skip the entity JOIN query and reduce response size. The `max_tokens` sub-option (default `500`) is a future-facing guard for entity data.

-The `max_tokens` parameter lets you control how much of your agent's context budget to spend on memories:
+### tags

-### Python
+Filters recall to only memories that match the specified tags. When omitted, all memories regardless of tags are eligible. Tag filtering is applied at the database level across all four retrieval strategies, not as a post-processing step.

-```python
-# Fill up to 4K tokens of context with relevant memories
-results = client.recall(bank_id="my-bank", query="What do I know about Alice?", max_tokens=4096)
+The `tags_match` parameter controls the filtering logic:

-# Smaller budget for quick lookups
-results = client.recall(bank_id="my-bank", query="Alice's email", max_tokens=500)
-```
-
-This design means you never have to guess whether 10 results or 50 results will fit your context. Just specify the token budget and Hindsight returns as many relevant memories as will fit.
-
-> **📝 Chunks are Independent**
-> 
-When `include_chunks=True`, chunks are fetched **independently** of the `max_tokens` filtering. This means:
- Setting `max_tokens=0` will return **0 memory facts** but can still return **chunks** (up to `max_chunk_tokens`)
- Chunks are based on the top-scored (reranked) results **before** token filtering
- Chunks are fetched in batches (batch size estimated as `(max_chunk_tokens / retain_chunk_size) * 2`) until the token budget is exhausted
- This batching approach handles varying chunk sizes across documents efficiently
- This allows you to retrieve raw source text without memory facts when needed
-## Budget Levels
-
-The `budget` parameter controls graph traversal depth:
-
- **"low"**: Fast, shallow retrieval — good for simple lookups
- **"mid"**: Balanced — default for most queries
- **"high"**: Deep exploration — finds indirect connections
-
-### Python
-
-```python
-# Quick lookup
-results = client.recall(bank_id="my-bank", query="Alice's email", budget="low")
-
-# Deep exploration
-results = client.recall(bank_id="my-bank", query="How are Alice and Bob connected?", budget="high")
-```
-
-### Node.js
-
-```javascript
-// Quick lookup
-const quickResults = await client.recall('my-bank', "Alice's email", { budget: 'low' });
-
-// Deep exploration
-const deepResults = await client.recall('my-bank', 'How are Alice and Bob connected?', { budget: 'high' });
-```
-
-## Filter by Tags
-
-Tags enable **visibility scoping**—filter memories based on tags assigned during [retain](./retain#tagging-memories). This is essential for multi-user agents where each user should only see their own memories.
-
-### Basic Tag Filtering
+- `any` (default) — memory matches if it has at least one of the specified tags, or has no tags at all. Use this for "user-specific + shared global" patterns.
+- `any_strict` — memory matches if it has at least one of the specified tags, and untagged memories are excluded. Use this when you want only explicitly scoped memories.
+- `all` — memory matches if it has every specified tag, or has no tags at all.
+- `all_strict` — memory matches if it has every specified tag, and untagged memories are excluded.

 ### Python

@@ -249,19 +249,6 @@ response = client.recall(
 )
 ```

-### Tag Match Modes
-
-The `tags_match` parameter controls how tags are matched:
-
-| Mode | Behavior | Untagged Memories |
-|------|----------|-------------------|
-| `any` | OR: memory has ANY of the specified tags | **Included** |
-| `all` | AND: memory has ALL of the specified tags | **Included** |
-| `any_strict` | OR: memory has ANY of the specified tags | **Excluded** |
-| `all_strict` | AND: memory has ALL of the specified tags | **Excluded** |
-
-**Strict modes** are useful when you want to ensure only tagged memories are returned:
-
 ### Python

 ```python
@@ -274,8 +261,6 @@ response = client.recall(
 )
 ```

-**AND matching** requires all specified tags to be present:
-
 ### Python

 ```python
@@ -288,11 +273,84 @@ response = client.recall(
 )
 ```

-### Use Cases
+### trace

-| Scenario | Tags | Mode | Result |
-|----------|------|------|--------|
-| User A's memories only | `["user:alice"]` | `any_strict` | Only memories tagged `user:alice` |
-| Support + feedback | `["support", "feedback"]` | `any` | Memories with either tag + untagged |
-| Multi-user room | `["user:alice", "room:general"]` | `all_strict` | Only memories with both tags |
-| Global + user-specific | `["user:alice"]` | `any` | Alice's memories + shared (untagged) |
+When set to `true`, the response includes a detailed debug trace covering the query embedding, entry points, per-strategy retrieval results, RRF fusion candidates, reranked results, temporal constraints detected, and per-phase timings. Has no effect on the retrieval logic itself. Useful for understanding why specific memories were or were not returned.
+
+---
+
+## Response
+
+### results
+
+The main list of recalled facts, ordered by relevance. Relevance is computed by running four retrieval strategies in parallel — semantic similarity, BM25 keyword, graph traversal, and temporal — fusing their rankings with Reciprocal Rank Fusion (RRF), then re-scoring the merged candidates with a cross-encoder reranker against the original query.
+
+Results do not include a numeric score. Raw retrieval scores are not meaningful on an absolute scale — a score of 0.8 from one query tells you nothing useful compared to a score of 0.8 from another. What matters is the relative ordering, which is already reflected in the list order. Agents should consume memories in order and let `max_tokens` determine how many fit, rather than filtering by score.
+
+Each item in `results` has the following fields:
+
+#### id
+
+The unique identifier of this fact. Use it to cross-reference with `source_facts` or for application-level deduplication.
+
+#### text
+
+The extracted fact text as stored in the memory bank.
+
+#### type
+
+The fact category: `world` for objective information, `experience` for events and conversations, or `observation` for consolidated knowledge synthesized over time.
+
+#### context
+
+The context label provided when the fact was retained (e.g., `"team meeting"`, `"slack"`). `null` if none was set.
+
+#### metadata
+
+The key-value string pairs attached when the fact was retained. `null` if none were set.
+
+#### tags
+
+The visibility-scoping tags attached to this fact.
+
+#### entities
+
+A list of canonical entity name strings linked to this fact. Only populated when `include.entities` is enabled (the default). `null` otherwise.
+
+#### occurred_start / occurred_end
+
+ISO 8601 datetimes representing when the described event started and ended. Extracted by the LLM from the content during retain. `null` if the content had no temporal information.
+
+#### mentioned_at
+
+ISO 8601 datetime of when this fact was retained into the bank.
+
+#### document_id
+
+The document ID this fact belongs to, as set during retain.
+
+#### chunk_id
+
+The ID of the source text chunk this fact was extracted from. Used to cross-reference with `chunks` in the response when `include.chunks` is enabled.
+
+#### source_fact_ids
+
+For `observation`-type results only: the IDs of the original facts this observation was synthesized from. Cross-references with `source_facts` in the response. `null` for other types or when `include.source_facts` is not enabled.
+
+---
+
+### source_facts
+
+A dict keyed by fact ID containing full `RecallResult` objects for the source facts that contributed to observation results. Only present when `include.source_facts` is enabled. Facts are deduplicated — if two observations share a source fact, it appears once.
+
+### chunks
+
+A dict keyed by chunk ID containing the raw source text chunks associated with the returned facts. Only present when `include.chunks` is enabled. Each chunk has `id`, `text`, `chunk_index`, and `truncated` (whether the text was cut to fit the token budget).
+
+### entities
+
+A dict keyed by canonical entity name containing entity state objects. Only present when `include.entities` is enabled. Each entry has `entity_id`, `canonical_name`, and `observations`.
+
+### trace
+
+A debug object present only when `trace: true` was set in the request. Contains per-phase timings, retrieval breakdowns, and RRF fusion details.
@@ -1,14 +1,9 @@

 # Reflect

-Generate disposition-aware responses using an agentic reasoning loop.
+Generate a grounded, disposition-aware response using an agentic reasoning loop.

-When you call **reflect**, Hindsight runs an **agentic loop** that:
-1. **Autonomously searches** for relevant information using multiple tools
-2. **Applies** the bank's disposition traits to shape the reasoning style
-3. **Generates** a grounded answer with citations to the sources used
-
-The agent has access to hierarchical retrieval tools (mental models → observations → raw facts) and decides what information it needs to answer your query.
+When you call **reflect**, Hindsight runs an agentic loop that autonomously searches the memory bank using multiple retrieval tools, applies the bank's disposition traits to shape the reasoning style, and produces a final answer grounded in what it found. Unlike recall — which returns raw facts — reflect returns a synthesized response written by the LLM.

 {/* Import raw source files */}

@@ -37,42 +32,25 @@ await client.reflect('my-bank', 'What should I know about Alice?');
 hindsight memory reflect my-bank "What do you know about Alice?"
 ```

+---
+
 ## Parameters

-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `query` | string | required | Question or prompt |
-| `budget` | string | "low" | Budget level: `low`, `mid`, `high` (see below) |
-| `max_tokens` | int | 4096 | Maximum tokens for the final response |
-| `response_schema` | object | None | JSON Schema for [structured output](#structured-output) |
-| `tags` | list | None | Filter memories by tags during reflection |
-| `tags_match` | string | "any" | How to match tags: `any`, `all`, `any_strict`, `all_strict` |
-| `trace` | bool | false | Include detailed agent trace in response |
+### query

-### Budget
+The question or prompt to reflect on. This is the only required field. If you have situational context that should influence the answer, include it directly in the query rather than as a separate field.

-The `budget` parameter controls the research depth — how thoroughly the agent explores before answering:
+### budget

-| Budget | Research Depth | Use Case |
-|--------|----------------|----------|
-| `low` | Shallow | Quick answers, simple lookups. Prioritizes speed over completeness. |
-| `mid` | Moderate | Balanced exploration. Checks multiple sources when warranted. |
-| `high` | Deep | Comprehensive analysis. Explores all knowledge levels, uses multiple query variations. |
-
-Use `high` for complex questions that require synthesizing information from multiple sources or verifying facts across different retrieval levels.
-
-### Max Tokens
-
-The `max_tokens` parameter limits the length of the final generated response. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.
+Controls how thoroughly the agent explores the memory bank before answering. Accepted values are `low` (default), `mid`, and `high`. At `low`, the agent does a shallow search optimized for speed. At `mid`, it checks multiple sources when the question warrants it. At `high`, it performs deep exploration across all knowledge levels and may use multiple query variations to find indirect connections. Use `high` for complex questions that require synthesizing information from many sources.

 ### Python

 ```python
 response = client.reflect(
    bank_id="my-bank",
-    query="What do you think about remote work?",
+    query="We're considering a hybrid work policy. What do you think about remote work?",
    budget="mid",
-    context="We're considering a hybrid work policy"
 )
 ```

@@ -85,77 +63,13 @@ const response = await client.reflect('my-bank', 'What do you think about remote
 });
 ```

-## Disposition Influence
+### max_tokens

-The bank's disposition affects reflect responses:
+Limits the length of the final generated response. Defaults to `4096`. This does not affect how much the agent can retrieve during the agentic loop — only the final answer length.

-| Trait | Low (1) | High (5) |
-|-------|---------|----------|
-| **Skepticism** | Trusting, accepts claims | Questions and doubts claims |
-| **Literalism** | Flexible interpretation | Exact, literal interpretation |
-| **Empathy** | Detached, fact-focused | Considers emotional context |
+### response_schema

-### Python
-
-```python
-# Create a bank with specific disposition
-client.create_bank(
-    bank_id="cautious-advisor",
-    name="Cautious Advisor",
-    mission="I am a risk-aware financial advisor",
-    disposition={
-        "skepticism": 5,   # Very skeptical of claims
-        "literalism": 4,   # Focuses on exact requirements
-        "empathy": 2       # Prioritizes facts over feelings
-    }
-)
-
-# Reflect responses will reflect this disposition
-response = client.reflect(
-    bank_id="cautious-advisor",
-    query="Should I invest in crypto?"
-)
-# Response will likely emphasize risks and caution
-```
-
-### Node.js
-
-```javascript
-// Create a bank with specific disposition
-await client.createBank('cautious-advisor', {
-    name: 'Cautious Advisor',
-    background: 'I am a risk-aware financial advisor',
-    disposition: {
-        skepticism: 5,
-        literalism: 4,
-        empathy: 2
-    }
-});
-
-// Reflect responses will reflect this disposition
-const advisorResponse = await client.reflect('cautious-advisor', 'Should I invest in crypto?');
-```
-
-## Citations
-
-The response includes a `based_on` field that shows which sources were used:
-
- `based_on.memories` — Memory facts (world, experience) that were retrieved and cited
- `based_on.mental_models` — User-curated mental models that were used
- `based_on.directives` — Directives that were enforced
-
-**Important:** Only IDs that were actually retrieved during the agent loop can be cited. The agent validates citations to prevent hallucinated references.
-
-This enables:
- **Transparency** — users see exactly which sources informed the answer
- **Verification** — check if the response is grounded in actual memories
- **Debugging** — use `trace=True` for detailed tool call logs
-
-## Structured Output
-
-For applications that need to process responses programmatically, you can request structured output by providing a JSON Schema via `response_schema`. When provided, the response includes a `structured_output` field with the LLM response parsed according to the schema. The `text` field will be empty since only a single LLM call is made for efficiency.
-
-The easiest way to define a schema is using **Pydantic models**:
+An optional JSON Schema object. When provided, the LLM generates a response that conforms to the schema and the response includes a `structured_output` field with the result parsed accordingly. The `text` field will be empty since only a single structured LLM call is made. Use this when you need to process the response programmatically rather than display it as prose.

 ### Python

@@ -233,22 +147,9 @@ hindsight memory reflect hiring-team \
 rm -f schema.json
 ```

-| Use Case | Why Structured Output Helps |
-|----------|----------------------------|
-| **Decision pipelines** | Parse recommendations into workflow systems |
-| **Dashboards** | Extract confidence scores, risk factors for visualization |
-| **Multi-agent systems** | Pass structured data between agents |
-| **Auditing** | Log structured decisions with clear reasoning |
+### tags

-**Tips:**
- Use Pydantic's `model_json_schema()` for type-safe schema generation
- Use `model_validate()` to parse the response back into your Pydantic model
- Keep schemas focused — extract only what you need
- Use `Optional` fields for data that may not always be available
-
-## Filter by Tags
-
-Like [recall](./recall#filter-by-tags), reflect supports tag filtering to scope which memories are considered during reasoning. This is essential for multi-user scenarios where reflection should only consider memories relevant to a specific user.
+Filters which memories the agent can access during reflection. Works identically to [recall tags](./recall#tags) — only memories matching the specified tags are considered. The `tags_match` parameter controls the matching logic (`any`, `all`, `any_strict`, `all_strict`) with the same semantics as recall.

 ### Python

@@ -262,13 +163,61 @@ response = client.reflect(
 )
 ```

-The `tags_match` parameter works the same as in recall:
+### include

-| Mode | Behavior |
-|------|----------|
-| `any` | OR matching, includes untagged memories |
-| `all` | AND matching, includes untagged memories |
-| `any_strict` | OR matching, excludes untagged memories |
-| `all_strict` | AND matching, excludes untagged memories |
+Controls optional supplementary data returned alongside the main response.

-See [Retain API](./retain#tagging-memories) for how to tag memories and [Recall API](./recall#filter-by-tags) for more details on tag matching modes.
+#### include.facts
+
+When enabled, the response includes a `based_on` object listing the memories, mental models, and directives the agent actually used to construct the answer. Only sources retrieved during the agent loop can appear here — citations are validated to prevent hallucinated references. Useful for transparency and verification.
+
+### Python
+
+```python
+# include_facts=True enables the based_on field in the response
+response = client.reflect(
+    bank_id="my-bank",
+    query="Tell me about Alice",
+    include_facts=True,
+)
+
+print("Response:", response.text)
+print("\nBased on:")
+for fact in (response.based_on.memories if response.based_on else []):
+    print(f"  - [{fact.type}] {fact.text}")
+```
+
+#### include.tool_calls
+
+When enabled, the response includes a `trace` object with the full execution log of every tool call and LLM call made during the agentic loop, including inputs, outputs, and durations. Set `output: false` to include only tool inputs for a smaller payload. Useful for debugging why the agent reached a particular conclusion.
+
+---
+
+## Response
+
+### text
+
+The synthesized answer as a well-formatted markdown string. This is the primary output of reflect. Empty when `response_schema` is provided (use `structured_output` instead in that case).
+
+### structured_output
+
+The LLM's response parsed according to the `response_schema` provided in the request. Only present when `response_schema` was set. `null` otherwise.
+
+### based_on
+
+The sources the agent used to construct the answer. Only present when `include.facts` was enabled. Contains three fields:
+
+- `memories` — a list of memory facts (world, experience, observation) that were retrieved and cited. Each item has `id`, `text`, `type`, `context`, `occurred_start`, and `occurred_end`.
+- `mental_models` — a list of mental models that were used. Each item has `id`, `text`, and `context`.
+- `directives` — a list of directives that were enforced during reasoning. Each item has `id`, `name`, and `content`.
+
+### usage
+
+Token usage for all LLM calls made during the agentic loop: `input_tokens`, `output_tokens`, and `total_tokens`. Useful for cost tracking.
+
+### trace
+
+The full execution log of the agentic loop. Only present when `include.tool_calls` was enabled. Contains:
+
+- `tool_calls` — each tool invocation with `tool` name (`lookup`, `recall`, `learn`, `expand`), `input`, `output` (if `output: true`), `duration_ms`, and `iteration` number.
+- `llm_calls` — each LLM call with `scope` (e.g., `"agent_1"`, `"final"`) and `duration_ms`.
@@ -12,7 +12,9 @@ Learn about fact extraction, entity resolution, and graph construction in the [R
 > **💡 Prerequisites**
 > 
 Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
-## Store a Single Memory
+## Store a Document
+
+A single retain call accepts one or more **items**. Each item is a piece of raw content — a conversation, a document, a note — that Hindsight will analyze and decompose into one or many memories. The content itself is never stored verbatim; what gets stored are the structured facts the LLM extracts from it.

 ### Python

@@ -35,18 +37,75 @@ await client.retain('my-bank', 'Alice works at Google as a software engineer');
 hindsight memory retain my-bank "Alice works at Google as a software engineer"
 ```

-## The Importance of Context
+### Retaining a Conversation

-The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
+A full conversation should be retained as a single item. The LLM can parse any format — plain text, JSON, Markdown, or any structured representation — as long as it clearly conveys who said what and when. The example below uses a simple `Name (timestamp): text` format.

-**Why context matters:**
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
+### Python

-## Store with Context and Date
+```python
+# Retain an entire conversation as a single document.
+# Format each message as "Name (timestamp): text" so the LLM can attribute
+# facts to the right person and resolve temporal references across the thread.
+conversation = "\n".join([
+    "Alice (2024-03-15T09:00:00Z): Hi Bob! Did you end up going to the doctor last week?",
+    "Bob (2024-03-15T09:01:00Z): Yes, finally. Turns out I have a mild peanut allergy.",
+    "Alice (2024-03-15T09:02:00Z): Oh no! Are you okay?",
+    "Bob (2024-03-15T09:03:00Z): Yeah, nothing serious. Just need to carry an antihistamine.",
+    "Alice (2024-03-15T09:04:00Z): Good to know. We'll avoid peanuts at the team lunch.",
+])

-Always provide context and event dates for optimal memory extraction:
+client.retain(
+    bank_id="my-bank",
+    content=conversation,
+    context="team chat",
+    timestamp="2024-03-15T09:04:00Z",
+    document_id="chat-2024-03-15-alice-bob",
+)
+```
+
+### Node.js
+
+```javascript
+// Retain an entire conversation as a single document.
+// Format each message as "Name (timestamp): text" so the LLM can attribute
+// facts to the right person and resolve temporal references across the thread.
+const conversation = [
+    'Alice (2024-03-15T09:00:00Z): Hi Bob! Did you end up going to the doctor last week?',
+    'Bob (2024-03-15T09:01:00Z): Yes, finally. Turns out I have a mild peanut allergy.',
+    'Alice (2024-03-15T09:02:00Z): Oh no! Are you okay?',
+    'Bob (2024-03-15T09:03:00Z): Yeah, nothing serious. Just need to carry an antihistamine.',
+    'Alice (2024-03-15T09:04:00Z): Good to know. We\'ll avoid peanuts at the team lunch.',
+].join('\n');
+
+await client.retain('my-bank', conversation, {
+    context: 'team chat',
+    timestamp: '2024-03-15T09:04:00Z',
+    documentId: 'chat-2024-03-15-alice-bob',
+});
+```
+
+When the conversation grows — a new message arrives — just retain again with the full updated content and the same `document_id`. Hindsight will delete the previous version and reprocess from scratch, so memories always reflect the latest state of the conversation.
+
+---
+
+## Parameters
+
+### content
+
+The raw text to store. This is the only required field. Hindsight chunks the content, sends each chunk to the LLM for fact extraction, and stores the resulting structured facts — not the original text. A single `content` value can produce many memories depending on how much information it contains.
+
+### timestamp
+
+When the event described in the content actually occurred. Accepts any ISO 8601 string (e.g., `"2024-01-15T10:30:00Z"`). If omitted, defaults to the current time at ingestion.
+
+The timestamp is injected verbatim into the LLM fact-extraction prompt so the model can resolve relative temporal references in the content — for example, if the content says "last Monday", the model uses the provided timestamp as the anchor to pin down the actual date. It also enables temporal recall queries like "What happened last spring?" to work correctly.
+
+### context
+
+A short label describing the source or situation — for example `"team meeting"`, `"slack"`, or `"support ticket"`. It is injected directly into the LLM prompt, so it actively shapes how facts are extracted. The same sentence can mean something very different depending on context: "the project was terminated" in a `"performance review"` context versus a `"product roadmap"` context produces different memories.
+
+Providing context consistently is one of the highest-leverage things you can do to improve memory quality.

 ### Python

@@ -75,30 +134,47 @@ hindsight memory retain my-bank "Alice got promoted" \
    --context "career update"
 ```

-The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
+### metadata

-### Response Fields
+Arbitrary key-value string pairs attached to every fact extracted from this item. For example: `{"source": "slack", "channel": "engineering", "thread_id": "T123"}`. The LLM never sees this field — it is passed through as-is and stored on each memory unit. During recall, every returned memory includes its metadata, which lets you do client-side filtering or static enrichment without extra lookups — for example, linking a memory back to its source URL, thread ID, or any application-specific identifier.

-The retain response includes:
+### document_id

-| Field | Type | Description |
-|-------|------|-------------|
-| `success` | bool | Whether the operation succeeded |
-| `bank_id` | string | The memory bank ID |
-| `items_count` | int | Number of items processed |
-| `async` | bool | Whether processed asynchronously |
-| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |
+A caller-supplied string that groups one or more items under a logical document. This field is the key to making retain **idempotent**.

-The `usage` field contains token metrics for cost tracking:
- `input_tokens`: Tokens consumed by prompts
- `output_tokens`: Tokens generated by the LLM
- `total_tokens`: Sum of input and output tokens
+When you provide a `document_id`, Hindsight upserts the document: if a document with that ID already exists in the bank, it and all its associated memories are deleted before the new content is processed and inserted. This means you can safely re-run retain on updated content — for example, a chat thread that grew since last time — without accumulating duplicate memories.

-Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.
+If you omit `document_id`, Hindsight assigns a random UUID per request, so re-ingesting the same content will create duplicate memories.
+
+### entities
+
+A list of entities you want to guarantee are recognized, merged with any entities the LLM extracts automatically. Each entry has a `text` field (the entity name) and an optional `type` (e.g., `"PERSON"`, `"ORG"`, `"CONCEPT"` — defaults to `"CONCEPT"` if omitted).
+
+Use this when you know certain entities are important but the LLM might miss them or refer to them inconsistently across different parts of the content. Providing entities explicitly ensures they are always linked in the knowledge graph.
+
+### tags and document_tags
+
+Tags control **visibility scoping** — which memories are visible during recall. A memory is only returned if its tags intersect with the tags filter provided in the recall request. This makes tags useful when a single memory bank serves multiple users or sessions and each should only see their own memories.
+
+Use consistent naming patterns to keep tag filtering predictable. Common conventions: `user:<id>` for per-user scoping, `session:<id>` for session isolation, `room:<id>` for chat rooms, `topic:<name>` for category filtering. The bank also exposes a list-tags endpoint that returns all tags with their memory counts, useful for UI autocomplete or wildcard expansion.
+
+See [Recall API](./recall#filter-by-tags) for filtering by tags during retrieval.
+
+### Response
+
+The synchronous retain response includes:
+
+- `success` — whether the operation completed without errors
+- `bank_id` — the memory bank that received the content
+- `items_count` — number of items processed
+- `async` — whether processing ran asynchronously
+- `usage` — token usage for the LLM calls (`input_tokens`, `output_tokens`, `total_tokens`), only present for synchronous operations
+
+---

 ## Batch Ingestion

-Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
+Multiple items can be submitted in a single request. Batch ingestion is the recommended approach — it reduces network overhead and lets Hindsight optimize extraction across related content.

 ### Python

@@ -123,11 +199,11 @@ await client.retainBatch('my-bank', [
 ]);
 ```

-The `document_id` groups related memories for later management.
+---

-## Store from Files
+## Files

-Upload files directly — Hindsight automatically converts them to text and extracts memories. File processing always runs asynchronously and returns operation IDs for tracking.
+Upload files directly — Hindsight converts them to text and extracts memories automatically. File processing always runs asynchronously and returns operation IDs for tracking.

 **Supported formats:** PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, images (JPG, PNG, GIF, etc. — OCR), audio (MP3, WAV, FLAC, etc. — transcription), HTML, and plain text formats (TXT, MD, CSV, JSON, YAML, etc.)

@@ -187,15 +263,7 @@ const result = await client.retainFiles('my-bank', [
 console.log(result.operation_ids);  // Track processing via the operations endpoint
 ```

-### File Retain Response
-
-The file retain endpoint always returns asynchronously:
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `operation_ids` | string[] | One operation ID per uploaded file. Use `GET /v1/default/banks/{bank_id}/operations` to track progress. |
-
-### Batch File Uploads
+The file retain endpoint always returns asynchronously. The response contains `operation_ids` — one per uploaded file — which you can poll via `GET /v1/default/banks/{bank_id}/operations` to track progress.

 Upload up to 10 files per request (max 100 MB total). Each file becomes a separate document with optional per-file metadata:

@@ -220,9 +288,11 @@ print(result.operation_ids)  # One operation ID per file

 :::info File Storage
 Uploaded files are stored server-side (PostgreSQL by default, or S3/GCS/Azure for production). Configure storage via `HINDSIGHT_API_FILE_STORAGE_TYPE`. See [Configuration](../configuration#file-processing) for details.
+---
+
 ## Async Ingestion

-For large batches, use async ingestion to avoid blocking:
+For large batches, use async ingestion to avoid blocking your application:

 ### Python

@@ -253,6 +323,8 @@ await client.retainBatch('my-bank', [
 });
 ```

+When `async: true`, the call returns immediately with an `operation_id`. Processing runs in the background via the worker service. No `usage` metrics are returned for async operations.
+
 ### Cut Costs 50% with Provider Batch APIs

 When using async retain, enable the provider Batch API to reduce LLM fact-extraction costs by 50%. OpenAI and Groq both offer this discount in exchange for a processing window of up to 24 hours — a trade-off that's typically invisible when retain already runs in the background.
@@ -265,82 +337,3 @@ Hindsight submits fact extraction calls as a batch job to the provider, polls fo

 :::note
 Batch API cost savings require `async=true` in your retain request and a compatible provider (OpenAI or Groq).
-## Tagging Memories
-
-Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.
-
-### Tag Individual Items
-
-### Python
-
-```python
-# Tag individual items for visibility scoping
-client.retain_batch(
-    bank_id="my-bank",
-    items=[
-        {
-            "content": "User Alice said she loves the new dashboard",
-            "tags": ["user:alice", "feedback"],
-            "document_id": "user_feedback_001"
-        },
-        {
-            "content": "User Bob reported a bug in the search feature",
-            "tags": ["user:bob", "bug-report"],
-            "document_id": "user_feedback_002"
-        }
-    ]
-)
-```
-
-### Apply Tags to All Items in a Batch
-
-Use `document_tags` to apply the same tags to all items in a request:
-
-### Python
-
-```python
-# Apply tags to all items in a batch
-client.retain_batch(
-    bank_id="my-bank",
-    items=[
-        {"content": "Alice mentioned she prefers dark mode", "document_id": "support_session_123_msg_1"},
-        {"content": "Bob asked about keyboard shortcuts", "document_id": "support_session_123_msg_2"}
-    ],
-    document_tags=["session:123", "support"]  # Applied to all items
-)
-```
-
-When both `document_tags` and item-level `tags` are provided, they are merged together.
-
-### Tag Naming Conventions
-
-Use consistent naming patterns for tags:
-
-| Pattern | Example | Use Case |
-|---------|---------|----------|
-| `user:<id>` | `user:alice` | Multi-user agent filtering |
-| `session:<id>` | `session:123` | Session-based scoping |
-| `room:<id>` | `room:general` | Chat room isolation |
-| `topic:<name>` | `topic:feedback` | Topic categorization |
-
-### Listing Tags
-
-Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:
-
-### Python
-
-```python
-# List all tags in a bank
-response = requests.get(f"{HINDSIGHT_URL}/v1/default/banks/my-bank/tags")
-tags = response.json()
-for tag in tags["items"]:
-    print(f"{tag['tag']}: {tag['count']} memories")
-
-# Search with wildcards (* matches any characters)
-response = requests.get(f"{HINDSIGHT_URL}/v1/default/banks/my-bank/tags", params={"q": "user:*"})
-user_tags = response.json()
-response = requests.get(f"{HINDSIGHT_URL}/v1/default/banks/my-bank/tags", params={"q": "*-admin"})
-admin_tags = response.json()
-```
-
-See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.
@@ -232,61 +232,10 @@ export HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-5-20250929
 # No API key needed - uses claude auth login credentials
 ```

-:::tip OpenAI Codex & Claude Code Setup
-For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro) and **Claude Code** (Claude Pro/Max), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
+:::tip OpenAI Codex, Claude Code & Vertex AI Setup
+For detailed setup instructions for **OpenAI Codex** (ChatGPT Plus/Pro), **Claude Code** (Claude Pro/Max), and **Vertex AI** (Google Cloud), see the [Models documentation](./models#openai-codex-setup-chatgpt-pluspro).
 :::

-#### Vertex AI Setup
-
-Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK. Hindsight supports two authentication methods:
-
-**Prerequisites:**
- GCP project with Vertex AI API enabled
- IAM role `roles/aiplatform.user` for your credentials
-
-**Environment Variables:**
-
-| Variable | Description | Required |
-|----------|-------------|----------|
-| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
-| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
-| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
-
-**Authentication Methods:**
-
-1. **Application Default Credentials (ADC)** - Recommended for development
-   ```bash
-   # Setup ADC
-   gcloud auth application-default login
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   ```
-
-2. **Service Account Key** - Recommended for production
-   ```bash
-   # Create service account and download key
-   gcloud iam service-accounts create hindsight-api
-   gcloud projects add-iam-policy-binding your-project-id \
-     --member="serviceAccount:[email protected]" \
-     --role="roles/aiplatform.user"
-   gcloud iam service-accounts keys create key.json \
-     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
-
-   # Configure Hindsight
-   export HINDSIGHT_API_LLM_PROVIDER=vertexai
-   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
-   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
-   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
-   ```
-
-**Notes:**
- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) - it will be stripped automatically
- The native SDK handles token refresh automatically
- Uses service account credentials if provided, otherwise falls back to ADC
-
 ### Per-Operation LLM Configuration

 Different memory operations have different requirements. **Retain** (fact extraction) benefits from models with strong structured output capabilities, while **Reflect** (reasoning/response generation) can use lighter, faster models. Configure separate LLM models for each operation to optimize for cost and performance.
@@ -77,7 +77,7 @@ docker run --rm -it -p 8888:8888 \
  -e HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai \
  -e HINDSIGHT_API_RERANKER_PROVIDER=cohere \
  -e HINDSIGHT_API_COHERE_API_KEY=$COHERE_API_KEY \
-  ghcr.io/vectorize-io/hindsight:slim
+  ghcr.io/vectorize-io/hindsight:latest-slim
 ```
 - ✅ Dramatically smaller image (~95% reduction on AMD64)
 - ✅ Faster pull/deploy times
@@ -114,7 +114,7 @@ See [Configuration](./configuration#embeddings-and-reranking) for all embedding
 ```bash
 # Standalone (API + Control Plane)
 ghcr.io/vectorize-io/hindsight:latest        # Full, latest release
-ghcr.io/vectorize-io/hindsight:slim          # Slim, latest release
+ghcr.io/vectorize-io/hindsight:latest-slim          # Slim, latest release
 ghcr.io/vectorize-io/hindsight:0.4.9         # Full, specific version
 ghcr.io/vectorize-io/hindsight:0.4.9-slim    # Slim, specific version

@@ -137,6 +137,15 @@ export HINDSIGHT_API_LLM_MODEL=llama3
 export HINDSIGHT_API_LLM_PROVIDER=lmstudio
 export HINDSIGHT_API_LLM_BASE_URL=http://localhost:1234/v1
 export HINDSIGHT_API_LLM_MODEL=your-local-model
+
+# Vertex AI (Google Cloud)
+export HINDSIGHT_API_LLM_PROVIDER=vertexai
+export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
+# Optional: region (default: us-central1)
+# export HINDSIGHT_API_LLM_VERTEXAI_REGION=us-central1
+# Optional: service account key (otherwise uses ADC)
+# export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
 ```

 **Note:** The LLM is the primary bottleneck for retain operations. See [Performance](./performance) for optimization strategies.
@@ -268,6 +277,59 @@ You can use any model supported by Claude Code CLI.
 - For personal development use only (see Claude Terms of Service)


+---
+
+### Vertex AI Setup (Google Cloud)
+
+Google Cloud's Vertex AI provides access to Gemini models via the native Google GenAI SDK.
+
+**Prerequisites:**
+- GCP project with Vertex AI API enabled
+- IAM role `roles/aiplatform.user` for your credentials
+
+**Environment Variables:**
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID` | Your GCP project ID | Yes |
+| `HINDSIGHT_API_LLM_VERTEXAI_REGION` | GCP region (e.g., `us-central1`) | No (default: `us-central1`) |
+| `HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY` | Path to service account JSON key file | No (uses ADC if not set) |
+
+**Authentication Methods:**
+
+1. **Application Default Credentials (ADC)** - Recommended for development
+   ```bash
+   # Setup ADC
+   gcloud auth application-default login
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   ```
+
+2. **Service Account Key** - Recommended for production
+   ```bash
+   # Create service account and download key
+   gcloud iam service-accounts create hindsight-api
+   gcloud projects add-iam-policy-binding your-project-id \
+     --member="serviceAccount:[email protected]" \
+     --role="roles/aiplatform.user"
+   gcloud iam service-accounts keys create key.json \
+     --iam-account=hindsight-api@your-project-id.iam.gserviceaccount.com
+
+   # Configure Hindsight
+   export HINDSIGHT_API_LLM_PROVIDER=vertexai
+   export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash-001
+   export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-project-id
+   export HINDSIGHT_API_LLM_VERTEXAI_SERVICE_ACCOUNT_KEY=/path/to/key.json
+   ```
+
+**Notes:**
+- Model names can optionally include the `google/` prefix (e.g., `google/gemini-2.0-flash-001`) — it will be stripped automatically
+- The native SDK handles token refresh automatically
+- Uses service account credentials if provided, otherwise falls back to ADC
+
 ---

 ## Embedding Model
Author	SHA1	Message	Date
Nicolò Boschi	efbecb6423	fix: create report.pdf in working directory for retain.sh file upload examples	2026-02-20 15:40:35 +01:00
Nicolò Boschi	4b5b580410	fix: add include_facts to reflect client, fix retain.sh temp files, fix main-methods based_on access	2026-02-20 15:18:11 +01:00
Nicolò Boschi	f8eb0c84c2	doc: improve api explanation	2026-02-20 15:12:22 +01:00
Nicolò Boschi	23165c244c	doc: improve api explanation	2026-02-20 14:37:11 +01:00
Nicolò Boschi	7074893f70	doc: improve api explanation	2026-02-20 14:36:53 +01:00