fix(ci): align config field count and CLI consolidation call with #1746

PR #1746 added enable_auto_consolidation to _CONFIGURABLE_FIELDS and
introduced a ConsolidationRequest body on the /consolidate endpoint, but
didn't update test_hierarchical_fields_categorization (still expects 35
fields) or the CLI's trigger_consolidation wrapper (still calls the
generated client with 2 args), so CI on this branch breaks on test-api,
test-rust-cli, test-embed-windows, and test-doc-examples (cli).

Bump the expected count to 36, add enable_auto_consolidation to the
explicit assertions, and pass a default ConsolidationRequest to the
generated client so the no-scope CLI invocation keeps consolidating all
unconsolidated memories.

docker/docker-compose/pgroonga/Dockerfile

@@ -0,0 +1,23 @@
+# PostgreSQL with pgvector and pgroonga extensions.
+#
+# pgroonga is a multilingual full-text search extension built on Groonga.
+# It works out of the box for CJK (Chinese, Japanese, Korean) and other
+# non-whitespace-segmented languages via the TokenBigram tokenizer.
+FROM groonga/pgroonga:latest-debian-pg17
+
+# Install pgvector on top of the pgroonga base image (which already provides
+# pgroonga and the Groonga library).
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    git \
+    postgresql-server-dev-17 \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN cd /tmp && \
+    git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git && \
+    cd pgvector && \
+    make && \
+    make install
+
+RUN rm -rf /tmp/pgvector && \
+    apt-get purge -y --auto-remove build-essential git postgresql-server-dev-17

docker/docker-compose/pgroonga/docker-compose.yaml

@@ -0,0 +1,91 @@
+name: hindsight
+# Docker Compose file for Hindsight with PostgreSQL and pgroonga
+#
+# pgroonga provides multilingual BM25 indexing that works out of the box for
+# CJK (Chinese, Japanese, Korean) and other non-whitespace-segmented languages.
+# Use this recipe if your bank content is not English/European.
+#
+# docker compose -f docker/docker-compose/pgroonga/docker-compose.yaml down && \
+#   sleep 2 && \
+#   docker compose -f docker/docker-compose/pgroonga/docker-compose.yaml up -d
+#
+# Optional environment variables with defaults:
+# - HINDSIGHT_VERSION: Hindsight application version (default: latest)
+# - HINDSIGHT_DB_USER: PostgreSQL user (default: hindsight_user)
+# - HINDSIGHT_DB_NAME: PostgreSQL database name (default: hindsight_db)
+# - HINDSIGHT_DB_PASSWORD: PostgreSQL password (default: hindsight_password)
+
+services:
+  db:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: hindsight-db
+    restart: always
+    ports:
+      - "5439:5432"
+    environment:
+      POSTGRES_USER: ${HINDSIGHT_DB_USER:-hindsight_user}
+      POSTGRES_PASSWORD: ${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+      POSTGRES_DB: ${HINDSIGHT_DB_NAME:-hindsight_db}
+    volumes:
+      - pg_data:/var/lib/postgresql/data
+    networks:
+      - hindsight-net
+
+  pgroonga-init:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    depends_on:
+      - db
+    environment:
+      - PGPASSWORD=${HINDSIGHT_DB_PASSWORD:-hindsight_password}
+    command: >
+      bash -c "
+        echo 'Waiting for PostgreSQL to be ready...';
+        until pg_isready -h hindsight-db -p 5432 -U hindsight_user; do
+          echo 'PostgreSQL is unavailable - sleeping';
+          sleep 2;
+        done;
+        echo 'PostgreSQL is ready - creating hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -c 'CREATE DATABASE hindsight_db;' 2>/dev/null || echo 'Database already exists';
+        echo 'Creating extensions in hindsight_db database';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS vector CASCADE;';
+        psql -h hindsight-db -p 5432 -U hindsight_user -d hindsight_db -c 'CREATE EXTENSION IF NOT EXISTS pgroonga CASCADE;';
+        echo 'Database and extensions created successfully';
+      "
+    restart: "no"
+    networks:
+      - hindsight-net
+
+  hindsight:
+    image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest}
+    container_name: hindsight-app
+    ports:
+      - "8888:8888"
+      - "9999:9999"
+    environment:
+      # LLM Configuration
+      HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
+      HINDSIGHT_API_LLM_API_KEY: ${OPENAI_API_KEY:-your-api-key}
+
+      # Database Configuration
+      HINDSIGHT_API_DATABASE_URL: postgresql://${HINDSIGHT_DB_USER:-hindsight_user}:${HINDSIGHT_DB_PASSWORD:-hindsight_password}@db:5432/${HINDSIGHT_DB_NAME:-hindsight_db}
+
+      # Vector and Text Search Extensions
+      HINDSIGHT_API_VECTOR_EXTENSION: pgvector
+      HINDSIGHT_API_TEXT_SEARCH_EXTENSION: pgroonga
+
+    depends_on:
+      - db
+    networks:
+      - hindsight-net
+
+
+networks:
+  hindsight-net:
+    driver: bridge
+
+volumes:
+  pg_data:

hindsight-api-slim/hindsight_api/alembic/versions/5a366d414dce_initial_schema.py

@@ -91,9 +91,14 @@ def _vector_index_using_clause(ext: str) -> str:
 
 def _detect_text_search_extension() -> str:
     """
-    Detect or validate text search extension: 'native', 'vchord', or 'pg_textsearch'.
-    Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var.
-    Creates the extension if needed.
+    Detect or validate text search extension: 'native', 'vchord', 'pg_textsearch',
+    or 'pgroonga'. Respects HINDSIGHT_API_TEXT_SEARCH_EXTENSION env var. Creates
+    the extension if needed.
+
+    pgroonga is treated as native here so the initial schema still creates valid
+    tsvector columns. ensure_text_search_extension() at startup converts the
+    schema to pgroonga structures (drops the tsvector column, builds a pgroonga
+    index on the base text column).
     """
     text_search_extension = os.getenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "native").lower()
 
@@ -123,9 +128,14 @@ def _detect_text_search_extension() -> str:
         return "pg_textsearch"
     elif text_search_extension == "native":
         return "native"
+    elif text_search_extension == "pgroonga":
+        # ensure_text_search_extension() at runtime converts to pgroonga.
+        # Treat as native here so the initial schema still creates valid columns.
+        return "native"
     else:
         raise ValueError(
-            f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. Must be 'native', 'vchord', or 'pg_textsearch'"
+            f"Invalid HINDSIGHT_API_TEXT_SEARCH_EXTENSION: {text_search_extension}. "
+            "Must be 'native', 'vchord', 'pg_textsearch', or 'pgroonga'"
         )
 
 

hindsight-api-slim/hindsight_api/alembic/versions/p4q5r6s7t8u9_configurable_bm25_language.py

@@ -0,0 +1,170 @@
+"""Drop GENERATED expression on tsvector search_vector columns.
+
+The search_vector tsvector column was originally GENERATED ALWAYS with a
+hardcoded ``to_tsvector('english', ...)`` expression. To support configurable
+``HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE``, we convert it to a
+regular tsvector column that the application populates at INSERT time via
+``to_tsvector($lang, ...)``.
+
+Existing rows retain their English-derived lexemes — switching the configured
+language only affects newly-written rows. Users who need to backfill existing
+rows in a different language can run an admin UPDATE after this migration.
+
+Only the ``native`` text-search backend is affected. ``vchord``, ``pg_textsearch``,
+and ``pgroonga`` use other column types or no column at all.
+
+Revision ID: p4q5r6s7t8u9
+Revises: 86f7a033d372
+Create Date: 2026-05-08
+"""
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+
+from alembic import context, op
+from sqlalchemy import Connection, text
+
+from hindsight_api.alembic._dialect import run_for_dialect
+
+revision: str = "p4q5r6s7t8u9"
+down_revision: str | Sequence[str] | None = "86f7a033d372"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+@dataclass(frozen=True)
+class _TsvectorTableSpec:
+    """Native-backend tsvector table targeted by this migration.
+
+    ``upgrade`` is a one-way DROP EXPRESSION; ``downgrade`` re-attaches the
+    original GENERATED expression so the schema returns to the state created
+    by the initial migration (and a2b3c4d5e6f7_add_text_signals_column for
+    memory_units).
+    """
+
+    table: str
+    generated_expression: str
+
+
+# Tables that may have a GENERATED tsvector ``search_vector`` column under the
+# native backend. Note: the ``learnings`` table was dropped in
+# p1k2l3m4n5o6_new_knowledge_architecture and ``pinned_reflections`` was renamed
+# to ``reflections`` in the same migration.
+_NATIVE_TSVECTOR_TABLES: tuple[_TsvectorTableSpec, ...] = (
+    _TsvectorTableSpec(
+        table="memory_units",
+        generated_expression=(
+            "to_tsvector('english', COALESCE(text, '') || ' ' || "
+            "COALESCE(context, '') || ' ' || COALESCE(text_signals, ''))"
+        ),
+    ),
+    _TsvectorTableSpec(
+        table="reflections",
+        generated_expression="to_tsvector('english', COALESCE(name, '') || ' ' || content)",
+    ),
+)
+
+
+def _schema_prefix() -> str:
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def _is_generated_tsvector(conn: Connection, schema: str, table: str) -> bool:
+    """Return True iff ``schema.table.search_vector`` is a GENERATED tsvector column."""
+    row = conn.execute(
+        text(
+            """
+            SELECT is_generated, udt_name
+            FROM information_schema.columns
+            WHERE table_schema = :schema
+              AND table_name = :table
+              AND column_name = 'search_vector'
+            """
+        ),
+        {"schema": schema, "table": table},
+    ).fetchone()
+    if not row:
+        return False
+    is_generated, udt_name = row[0], row[1]
+    return is_generated == "ALWAYS" and udt_name == "tsvector"
+
+
+def _is_regular_tsvector(conn: Connection, schema: str, table: str) -> bool:
+    """Return True iff ``schema.table.search_vector`` is a non-generated tsvector column."""
+    row = conn.execute(
+        text(
+            """
+            SELECT is_generated, udt_name
+            FROM information_schema.columns
+            WHERE table_schema = :schema
+              AND table_name = :table
+              AND column_name = 'search_vector'
+            """
+        ),
+        {"schema": schema, "table": table},
+    ).fetchone()
+    if not row:
+        return False
+    is_generated, udt_name = row[0], row[1]
+    return udt_name == "tsvector" and is_generated != "ALWAYS"
+
+
+def _table_exists(conn: Connection, schema: str, table: str) -> bool:
+    return bool(
+        conn.execute(
+            text(
+                """
+                SELECT 1 FROM information_schema.tables
+                WHERE table_schema = :schema AND table_name = :table
+                """
+            ),
+            {"schema": schema, "table": table},
+        ).fetchone()
+    )
+
+
+def _pg_upgrade() -> None:
+    schema_prefix = _schema_prefix()
+    schema_name = (context.config.get_main_option("target_schema") or "public").strip('"')
+    conn = op.get_bind()
+
+    for spec in _NATIVE_TSVECTOR_TABLES:
+        if not _table_exists(conn, schema_name, spec.table):
+            continue
+        if not _is_generated_tsvector(conn, schema_name, spec.table):
+            # Either the column doesn't exist (non-native backend) or it's
+            # already a regular tsvector — nothing to do.
+            continue
+        op.execute(f"ALTER TABLE {schema_prefix}{spec.table} ALTER COLUMN search_vector DROP EXPRESSION")
+
+
+def _pg_downgrade() -> None:
+    schema_prefix = _schema_prefix()
+    schema_name = (context.config.get_main_option("target_schema") or "public").strip('"')
+    conn = op.get_bind()
+
+    for spec in _NATIVE_TSVECTOR_TABLES:
+        if not _table_exists(conn, schema_name, spec.table):
+            continue
+        # Only restore the GENERATED expression if a non-generated tsvector
+        # column exists — otherwise the table is on a different backend.
+        if not _is_regular_tsvector(conn, schema_name, spec.table):
+            continue
+        # Drop and recreate to re-attach the GENERATED expression. Index will be
+        # recreated by re-running ensure_text_search_extension on next startup.
+        op.execute(f"DROP INDEX IF EXISTS {schema_prefix}idx_{spec.table}_text_search")
+        op.execute(f"ALTER TABLE {schema_prefix}{spec.table} DROP COLUMN search_vector")
+        op.execute(
+            f"ALTER TABLE {schema_prefix}{spec.table} "
+            f"ADD COLUMN search_vector tsvector GENERATED ALWAYS AS ({spec.generated_expression}) STORED"
+        )
+        op.execute(f"CREATE INDEX idx_{spec.table}_text_search ON {schema_prefix}{spec.table} USING gin(search_vector)")
+
+
+def upgrade() -> None:
+    run_for_dialect(pg=_pg_upgrade)
+
+
+def downgrade() -> None:
+    run_for_dialect(pg=_pg_downgrade)

hindsight-api-slim/hindsight_api/config.py

@@ -7,6 +7,7 @@ All environment variables and their defaults are defined here.
 import json
 import logging
 import os
+import re
 import sys
 from dataclasses import dataclass, field, fields
 from datetime import datetime, timezone
@@ -290,6 +291,8 @@ ENV_RERANKER_GOOGLE_SERVICE_ACCOUNT_KEY = "HINDSIGHT_API_RERANKER_GOOGLE_SERVICE
 
 ENV_VECTOR_EXTENSION = "HINDSIGHT_API_VECTOR_EXTENSION"
 ENV_TEXT_SEARCH_EXTENSION = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION"
+ENV_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE = "HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE"
+ENV_LLM_OUTPUT_LANGUAGE = "HINDSIGHT_API_LLM_OUTPUT_LANGUAGE"
 
 ENV_HOST = "HINDSIGHT_API_HOST"
 ENV_PORT = "HINDSIGHT_API_PORT"
@@ -562,8 +565,14 @@ DEFAULT_RERANKER_GOOGLE_MODEL = "semantic-ranker-default-004"
 # Vector extension (pgvector, vchord, pgvectorscale, or AlloyDB ScaNN)
 DEFAULT_VECTOR_EXTENSION = "pgvector"  # Options: "pgvector", "vchord", "pgvectorscale", "scann"
 
-# Text search extension (native PostgreSQL, vchord BM25, or Timescale pg_textsearch)
-DEFAULT_TEXT_SEARCH_EXTENSION = "native"  # Options: "native", "vchord", "pg_textsearch"
+# Text search extension (native PostgreSQL, vchord BM25, Timescale pg_textsearch, or pgroonga)
+DEFAULT_TEXT_SEARCH_EXTENSION = "native"  # Options: "native", "vchord", "pg_textsearch", "pgroonga"
+
+# PostgreSQL text search dictionary used by the native tsvector backend. Only
+# affects text_search_extension == "native"; other backends use their own
+# tokenizers (vchord: llmlingua2, pg_textsearch: hardcoded english,
+# pgroonga: TokenBigram polyglot).
+DEFAULT_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE = "english"
 
 # LiteLLM defaults
 DEFAULT_LITELLM_API_BASE = "http://localhost:4000"
@@ -890,7 +899,15 @@ class HindsightConfig:
     migration_database_url: str | None
     database_schema: str
     vector_extension: str  # "pgvector", "vchord", "pgvectorscale", or "scann"
-    text_search_extension: str  # "native" or "vchord"
+    text_search_extension: str  # "native", "vchord", "pg_textsearch", or "pgroonga"
+    # PostgreSQL text search dictionary for the "native" backend (ignored by
+    # other backends). Only the "native" backend reads this field; pgroonga
+    # uses TokenBigram, vchord uses llmlingua2, pg_textsearch hardcodes english.
+    text_search_extension_native_language: str
+    # When set, every LLM-generated artifact (retain facts, consolidation
+    # observations, reflect responses) is forced into this language regardless
+    # of the source content. Unset preserves source language.
+    llm_output_language: str | None
 
     # LLM (default, used as fallback for per-operation config)
     llm_provider: str
@@ -1355,12 +1372,26 @@ class HindsightConfig:
         validate_extension(self.vector_extension)
 
         # Validate text_search_extension
-        valid_text_search = ("native", "vchord", "pg_textsearch")
+        valid_text_search = ("native", "vchord", "pg_textsearch", "pgroonga")
         if self.text_search_extension not in valid_text_search:
             raise ValueError(
                 f"Invalid text_search_extension: {self.text_search_extension}. Must be one of: {', '.join(valid_text_search)}"
             )
 
+        # Validate text_search_extension_native_language as a PG identifier.
+        # Embedded directly into raw SQL via to_tsvector('<lang>', ...), so we
+        # reject anything that isn't a plain identifier to prevent injection.
+        # Intentionally permissive about which dictionaries exist — users may
+        # install custom ones like zhparser; we only check shape here. PG
+        # raises a clear error at query time if the dictionary is missing.
+        if not re.fullmatch(r"[a-zA-Z_][a-zA-Z0-9_]*", self.text_search_extension_native_language):
+            raise ValueError(
+                f"Invalid text_search_extension_native_language: "
+                f"{self.text_search_extension_native_language!r}. Must be a valid PostgreSQL identifier "
+                f"(letters, digits, underscores; not starting with a digit). Examples: 'english', "
+                f"'french', 'simple', 'zhparser'."
+            )
+
         # When LLM provider is "none", force chunks-only mode and disable LLM-dependent features
         if self.llm_provider == "none":
             self.retain_extraction_mode = "chunks"
@@ -1437,6 +1468,11 @@ class HindsightConfig:
             database_schema=os.getenv(ENV_DATABASE_SCHEMA, DEFAULT_DATABASE_SCHEMA),
             vector_extension=os.getenv(ENV_VECTOR_EXTENSION, DEFAULT_VECTOR_EXTENSION).lower(),
             text_search_extension=os.getenv(ENV_TEXT_SEARCH_EXTENSION, DEFAULT_TEXT_SEARCH_EXTENSION).lower(),
+            text_search_extension_native_language=os.getenv(
+                ENV_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
+                DEFAULT_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE,
+            ).lower(),
+            llm_output_language=(os.getenv(ENV_LLM_OUTPUT_LANGUAGE) or None),
             # LLM
             llm_provider=llm_provider,
             llm_api_key=os.getenv(ENV_LLM_API_KEY),

hindsight-api-slim/hindsight_api/engine/consolidation/consolidator.py

@@ -1299,7 +1299,11 @@ async def _consolidate_batch_with_llm(
                 f"(out of {max_observations_per_scope}). Prefer UPDATE over CREATE when possible."
             )
 
-    prompt_template = build_batch_consolidation_prompt(config.observations_mission, observation_capacity_note)
+    prompt_template = build_batch_consolidation_prompt(
+        config.observations_mission,
+        observation_capacity_note,
+        llm_output_language=getattr(config, "llm_output_language", None),
+    )
     prompt = prompt_template.format(
         facts_text=facts_lines,
         observations_text=observations_text,

hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py

@@ -1,6 +1,6 @@
 """Prompts for the consolidation engine."""
 
-from hindsight_api.engine.prompt_utils import escape_for_prompt
+from hindsight_api.engine.prompt_utils import escape_for_prompt, output_language_directive
 
 # Default mission when no bank-specific mission is set
 _DEFAULT_MISSION = "Track every detail: names, numbers, dates, places, and relationships. Prefer specifics over abstractions, never generalise."
@@ -85,12 +85,14 @@ Rules:
 def build_batch_consolidation_prompt(
     observations_mission: str | None = None,
     observation_capacity_note: str | None = None,
+    llm_output_language: str | None = None,
 ) -> str:
     """
     Build the consolidation prompt for batch mode (multiple facts per LLM call).
 
     The mission defines *what* to track (customisable per bank).
     Processing rules and output format are always present regardless of mission.
+    When ``llm_output_language`` is set, observations are emitted in that language.
     """
     mission = escape_for_prompt(observations_mission or _DEFAULT_MISSION)
 
@@ -102,5 +104,8 @@ def build_batch_consolidation_prompt(
         "You are a memory consolidation system. Synthesize facts into observations "
         "and merge with existing observations when appropriate.\n\n"
         f"## MISSION\n{mission}{capacity_section}\n\n"
-        f"{_PROCESSING_RULES}" + _BATCH_DATA_SECTION + _BATCH_OUTPUT_FORMAT
+        f"{_PROCESSING_RULES}"
+        + _BATCH_DATA_SECTION
+        + _BATCH_OUTPUT_FORMAT
+        + output_language_directive(llm_output_language)
     )

hindsight-api-slim/hindsight_api/engine/db/ops_postgresql.py

@@ -104,7 +104,46 @@ class PostgreSQLOps(DataAccessOps):
                 FROM input_data
                 RETURNING id
             """
+        elif config.text_search_extension == "native":
+            # search_vector is a regular tsvector column populated here using the
+            # configured native dictionary. It used to be GENERATED ALWAYS with
+            # a hardcoded 'english', which prevented per-deployment language
+            # configuration. text_search_extension_native_language is validated
+            # in HindsightConfig.validate() as a PG identifier, so embedding it
+            # as a SQL literal is safe.
+            query = f"""
+                WITH input_data AS (
+                    SELECT * FROM unnest(
+                        $2::text[], $3::vector[], $4::timestamptz[], $5::timestamptz[], $6::timestamptz[], $7::timestamptz[],
+                        $8::text[], $9::text[], $10::jsonb[], $11::text[], $12::text[], $13::jsonb[], $14::jsonb[], $15::text[]
+                    ) AS t(text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                           context, fact_type, metadata, chunk_id, document_id, tags_json,
+                           observation_scopes_json, text_signals)
+                )
+                INSERT INTO {table} (bank_id, text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                                     context, fact_type, metadata, chunk_id, document_id, tags,
+                                     observation_scopes, text_signals, search_vector)
+                SELECT
+                    $1,
+                    text, embedding, event_date, occurred_start, occurred_end, mentioned_at,
+                    context, fact_type, metadata, chunk_id, document_id,
+                    COALESCE(
+                        (SELECT array_agg(elem) FROM jsonb_array_elements_text(tags_json) AS elem),
+                        '{{}}'::varchar[]
+                    ),
+                    observation_scopes_json,
+                    text_signals,
+                    to_tsvector(
+                        '{config.text_search_extension_native_language}'::regconfig,
+                        COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, '')
+                    )
+                FROM input_data
+                RETURNING id
+            """
         else:
+            # pg_textsearch and pgroonga: search_vector is a dummy TEXT column;
+            # the actual full-text index operates on the base text columns
+            # directly, so we don't populate search_vector at insert time.
             query = f"""
                 WITH input_data AS (
                     SELECT * FROM unnest(

hindsight-api-slim/hindsight_api/engine/memory_engine.py

@@ -6396,6 +6396,7 @@ class MemoryEngine(MemoryEngineInterface):
                         include_recall=include_recall,
                         budget=effective_budget,
                         max_context_tokens=max_context_tokens,
+                        llm_output_language=getattr(resolved_reflect_config, "llm_output_language", None),
                     ),
                     timeout=wall_timeout,
                 )

hindsight-api-slim/hindsight_api/engine/prompt_utils.py

@@ -21,3 +21,21 @@ def escape_for_prompt(text: str) -> str:
     text = _LONE_OPEN_BRACE.sub("{{", text)
     text = _LONE_CLOSE_BRACE.sub("}}", text)
     return text
+
+
+def output_language_directive(language: str | None) -> str:
+    """Return an LLM directive forcing all output into ``language``.
+
+    Used by retain (fact extraction), consolidation (observations), and reflect
+    (response synthesis) so HINDSIGHT_API_LLM_OUTPUT_LANGUAGE applies uniformly
+    across every LLM-generated artifact. Returns an empty string when
+    ``language`` is unset so the calling prompt stays unchanged.
+    """
+    if not language:
+        return ""
+    return (
+        f"\n\nIMPORTANT: Respond exclusively in {language}. "
+        f"Translate any source content into {language}. "
+        f"All output text — including fact text, observations, entity names, "
+        f"and the final response — must be in {language}."
+    )

hindsight-api-slim/hindsight_api/engine/reflect/agent.py

@@ -321,6 +321,7 @@ async def run_reflect_agent(
     include_recall: bool = True,
     budget: str | None = None,
     max_context_tokens: int = 100_000,
+    llm_output_language: str | None = None,
 ) -> ReflectAgentResult:
     """
     Execute the reflect agent loop using native tool calling.
@@ -452,7 +453,10 @@ async def run_reflect_agent(
             llm_start = time.time()
             response, usage = await llm_config.call(
                 messages=[
-                    {"role": "system", "content": build_final_system_prompt(bank_profile.get("mission"))},
+                    {
+                        "role": "system",
+                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                    },
                     {"role": "user", "content": prompt},
                 ],
                 scope="reflect",
@@ -509,7 +513,10 @@ async def run_reflect_agent(
             llm_start = time.time()
             response, usage = await llm_config.call(
                 messages=[
-                    {"role": "system", "content": build_final_system_prompt(bank_profile.get("mission"))},
+                    {
+                        "role": "system",
+                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                    },
                     {"role": "user", "content": prompt},
                 ],
                 scope="reflect",
@@ -612,7 +619,10 @@ async def run_reflect_agent(
             llm_start = time.time()
             response, usage = await llm_config.call(
                 messages=[
-                    {"role": "system", "content": build_final_system_prompt(bank_profile.get("mission"))},
+                    {
+                        "role": "system",
+                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                    },
                     {"role": "user", "content": prompt},
                 ],
                 scope="reflect",
@@ -733,7 +743,10 @@ async def run_reflect_agent(
             llm_start = time.time()
             response, usage = await llm_config.call(
                 messages=[
-                    {"role": "system", "content": build_final_system_prompt(bank_profile.get("mission"))},
+                    {
+                        "role": "system",
+                        "content": build_final_system_prompt(bank_profile.get("mission"), llm_output_language),
+                    },
                     {"role": "user", "content": prompt},
                 ],
                 scope="reflect",

hindsight-api-slim/hindsight_api/engine/reflect/prompts.py

@@ -560,12 +560,16 @@ Just provide the direct answer with proper markdown formatting.
 CRITICAL: This is a NON-CONVERSATIONAL system. NEVER ask follow-up questions, offer to search again, suggest alternatives, or end with anything like "Would you like me to..." or "Let me know if...". The user cannot reply. Your answer must be complete and self-contained."""
 
 
-def build_final_system_prompt(mission: str | None = None) -> str:
-    """Build the final synthesis system prompt, using mission as role when set."""
-    from hindsight_api.engine.prompt_utils import escape_for_prompt
+def build_final_system_prompt(mission: str | None = None, llm_output_language: str | None = None) -> str:
+    """Build the final synthesis system prompt, using mission as role when set.
+
+    When ``llm_output_language`` is set, the response is forced into that
+    language regardless of the query/source language.
+    """
+    from hindsight_api.engine.prompt_utils import escape_for_prompt, output_language_directive
 
     role_section = escape_for_prompt(mission.strip()) if mission else _DEFAULT_FINAL_ROLE
-    return _FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section)
+    return _FINAL_SYSTEM_PROMPT_BASE.format(role_section=role_section) + output_language_directive(llm_output_language)
 
 
 # Backward-compatible constant for non-identity missions

hindsight-api-slim/hindsight_api/engine/retain/fact_extraction.py

@@ -950,6 +950,16 @@ def _build_extraction_prompt_and_schema(config) -> tuple[str, type]:
     if labels_section:
         prompt = prompt + labels_section
 
+    # Force the LLM to emit fact text in the configured language, regardless of
+    # the source content's language. Same directive is applied to consolidation
+    # and reflect so HINDSIGHT_API_LLM_OUTPUT_LANGUAGE has a uniform effect
+    # across the pipeline. This is independent of the BM25 indexing language
+    # (HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE) by design — search
+    # tokenization and LLM output language are separate concerns.
+    from ..prompt_utils import output_language_directive
+
+    prompt = prompt + output_language_directive(getattr(config, "llm_output_language", None))
+
     response_schema = base_response_class
 
     if labels_cfg and labels_cfg.attributes:

hindsight-api-slim/hindsight_api/engine/search/retrieval.py

@@ -225,6 +225,7 @@ async def retrieve_semantic_bm25_combined(
                     groups_clause=groups_clause,
                     arm_index=i,
                     text_search_extension=text_ext,
+                    bm25_language=config.text_search_extension_native_language,
                     extra_where=created_range_clause,
                 )
             )

hindsight-api-slim/hindsight_api/engine/sql/base.py

@@ -407,6 +407,7 @@ class SQLDialect(ABC):
         groups_clause: str = "",
         arm_index: int = 0,
         text_search_extension: str = "native",
+        bm25_language: str = "english",
         extra_where: str = "",
     ) -> str:
         """Build a BM25/full-text search subquery arm.
@@ -426,7 +427,9 @@ class SQLDialect(ABC):
             arm_index: Index of this arm in the UNION ALL (used by Oracle for
                        unique SCORE labels).
             text_search_extension: Full-text search backend ("native", "vchord",
-                                   "pg_textsearch"). Only relevant for PostgreSQL.
+                                   "pg_textsearch", "pgroonga"). Only relevant for PostgreSQL.
+            bm25_language: PostgreSQL text search dictionary used by the native
+                           backend (e.g. "english", "french"). Ignored by other backends.
             extra_where: Optional additional WHERE clause fragment (e.g. time range filter).
         """
         ...

hindsight-api-slim/hindsight_api/engine/sql/oracle.py

@@ -270,6 +270,7 @@ class OracleDialect(SQLDialect):
         groups_clause: str = "",
         arm_index: int = 0,
         text_search_extension: str = "native",
+        bm25_language: str = "english",
         extra_where: str = "",
     ) -> str:
         # Oracle Text: CONTAINS() / SCORE() with the CTXSYS.CONTEXT index.

hindsight-api-slim/hindsight_api/engine/sql/postgresql.py

@@ -182,6 +182,7 @@ class PostgreSQLDialect(SQLDialect):
         groups_clause: str = "",
         arm_index: int = 0,
         text_search_extension: str = "native",
+        bm25_language: str = "english",
         extra_where: str = "",
     ) -> str:
         if text_search_extension == "vchord":
@@ -193,10 +194,21 @@ class PostgreSQLDialect(SQLDialect):
             bm25_score_expr = f"-({text_param} <@> to_bm25query({text_param}, 'idx_memory_units_text_search'))"
             bm25_order_by = f"text <@> to_bm25query({text_param}, 'idx_memory_units_text_search') ASC"
             bm25_where_filter = ""
-        else:  # native tsvector
-            bm25_score_expr = f"ts_rank_cd(search_vector, to_tsquery('english', {text_param}))"
+        elif text_search_extension == "pgroonga":
+            # &@~ accepts pgroonga's query syntax (raw query text). pgroonga_score
+            # returns a non-negative relevance score (higher = better).
+            bm25_score_expr = "pgroonga_score(tableoid, ctid)"
             bm25_order_by = f"{bm25_score_expr} DESC"
-            bm25_where_filter = f"AND search_vector @@ to_tsquery('english', {text_param})"
+            bm25_where_filter = (
+                f"AND (COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, '')) "
+                f"&@~ {text_param}"
+            )
+        else:  # native tsvector
+            # bm25_language is validated as a PG identifier in HindsightConfig.validate(),
+            # so embedding it as a SQL literal here is safe.
+            bm25_score_expr = f"ts_rank_cd(search_vector, to_tsquery('{bm25_language}', {text_param}))"
+            bm25_order_by = f"{bm25_score_expr} DESC"
+            bm25_where_filter = f"AND search_vector @@ to_tsquery('{bm25_language}', {text_param})"
 
         return (
             f"(SELECT {cols},"
@@ -221,7 +233,7 @@ class PostgreSQLDialect(SQLDialect):
         *,
         text_search_extension: str = "native",
     ) -> str:
-        if text_search_extension in ("vchord", "pg_textsearch"):
+        if text_search_extension in ("vchord", "pg_textsearch", "pgroonga"):
             return query_text
         # native tsvector: join tokens with OR operator
         return " | ".join(tokens)

hindsight-api-slim/hindsight_api/migrations.py

@@ -815,7 +815,8 @@ def ensure_text_search_extension(
 
     Args:
         database_url: SQLAlchemy database URL
-        text_search_extension: Configured text search extension ("native" or "vchord")
+        text_search_extension: Configured text search extension — one of
+            "native", "vchord", "pg_textsearch", or "pgroonga"
         schema: Target PostgreSQL schema name (None for public)
 
     Raises:
@@ -838,6 +839,12 @@ def ensure_text_search_extension(
         elif text_search_extension == "pg_textsearch":
             target_column_type = "text"
             target_index_type = "bm25"
+        elif text_search_extension == "pgroonga":
+            # pgroonga indexes the base text column directly. We keep a dummy
+            # TEXT column named search_vector for symmetry with pg_textsearch
+            # and so the column-type mismatch detection above keeps working.
+            target_column_type = "text"
+            target_index_type = "pgroonga"
         else:  # native
             target_column_type = "tsvector"
             target_index_type = "gin"
@@ -925,12 +932,17 @@ def ensure_text_search_extension(
         # If there's data in any mismatched table, raise error
         if tables_with_data:
             table_list = ", ".join([f"{table}({count} rows)" for table, count in tables_with_data])
-            # Detect current extension from column type
+            # Detect current extension from column type + index type. tsvector is
+            # unambiguous; text could be either pg_textsearch or pgroonga, so we
+            # disambiguate via the index type.
             current_col_type = mismatched_tables[0][1]
+            current_idx_type = mismatched_tables[0][2]
             if current_col_type == "tsvector":
                 current_ext = "native"
             elif current_col_type == "bm25vector":
                 current_ext = "vchord"
+            elif current_col_type == "text" and current_idx_type == "pgroonga":
+                current_ext = "pgroonga"
             elif current_col_type == "text":
                 current_ext = "pg_textsearch"
             else:
@@ -1000,21 +1012,50 @@ def ensure_text_search_extension(
                         WITH (text_config='english')
                     """)
                 )
-            else:  # native
-                logger.info(f"Creating tsvector column on {table_name}")
-                # Different GENERATED expression for each table
-                if table_name == "memory_units":
-                    generated_expr = "to_tsvector('english', COALESCE(text, '') || ' ' || COALESCE(context, ''))"
-                else:  # reflections
-                    generated_expr = "to_tsvector('english', COALESCE(name, '') || ' ' || content)"
+            elif text_search_extension == "pgroonga":
+                # Ensure pgroonga extension is available
+                try:
+                    conn.execute(text("CREATE EXTENSION IF NOT EXISTS pgroonga CASCADE"))
+                except Exception:
+                    # Extension might already exist or user lacks permissions — verify
+                    has_ext = conn.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'pgroonga'")).fetchone()
+                    if not has_ext:
+                        raise
 
+                logger.info(f"Creating dummy TEXT search_vector on {table_name} for pgroonga")
+                # pgroonga indexes the base text column directly, but we keep a
+                # dummy search_vector column for symmetry with pg_textsearch and
+                # so the column-type mismatch detection above keeps working.
+                conn.execute(text(f"ALTER TABLE {schema_name}.{table_name} ADD COLUMN search_vector TEXT"))
+
+                # pgroonga index expression mirrors pg_textsearch
+                if table_name == "memory_units":
+                    index_expr = (
+                        "(COALESCE(text, '') || ' ' || COALESCE(context, '') || ' ' || COALESCE(text_signals, ''))"
+                    )
+                else:  # reflections
+                    index_expr = "(COALESCE(name, '') || ' ' || content)"
+
+                logger.info(f"Creating pgroonga index on {table_name}")
+                # TokenBigram is the polyglot default — falls back to whitespace
+                # tokenization for space-separated languages and bigram for CJK.
+                # NormalizerNFKC150 handles Unicode normalization (full/half-width,
+                # case folding, etc.) which materially improves Japanese recall.
                 conn.execute(
                     text(f"""
-                        ALTER TABLE {schema_name}.{table_name}
-                        ADD COLUMN search_vector tsvector
-                        GENERATED ALWAYS AS ({generated_expr}) STORED
+                        CREATE INDEX idx_{table_name.replace(".", "_")}_text_search
+                        ON {schema_name}.{table_name}
+                        USING pgroonga ({index_expr})
+                        WITH (tokenizer='TokenBigram', normalizer='NormalizerNFKC150')
                     """)
                 )
+            else:  # native
+                logger.info(f"Creating tsvector column on {table_name}")
+                # Plain tsvector column. The application populates search_vector
+                # at INSERT time via to_tsvector($lang, ...) using the configured
+                # HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE — see
+                # ops_postgresql.insert_facts_batch.
+                conn.execute(text(f"ALTER TABLE {schema_name}.{table_name} ADD COLUMN search_vector tsvector"))
 
                 # Create GIN index
                 logger.info(f"Creating GIN index on {table_name}")

hindsight-api-slim/tests/test_config_validation.py

@@ -204,3 +204,104 @@ def test_log_config_masks_read_database_url(monkeypatch, caplog):
 #
 # The config validation tests above ensure users get early feedback
 # about invalid configurations before runtime errors occur.
+
+
+# ---------------------------------------------------------------------------
+# Multilingual BM25 configuration
+# ---------------------------------------------------------------------------
+
+
+def test_native_language_defaults_to_english(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.delenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE", raising=False)
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.text_search_extension_native_language == "english"
+
+
+def test_native_language_loaded_from_env(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE", "french")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.text_search_extension_native_language == "french"
+
+
+def test_native_language_lowercased(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE", "Spanish")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.text_search_extension_native_language == "spanish"
+
+
+@pytest.mark.parametrize(
+    "bad_value",
+    ["en glish", "english;DROP TABLE", "english'", "1english", "english-extra", ""],
+)
+def test_native_language_rejects_invalid_identifiers(monkeypatch, bad_value):
+    """text_search_extension_native_language is embedded into raw SQL — non-identifiers must be rejected."""
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE", bad_value)
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    with pytest.raises(ValueError, match="Invalid text_search_extension_native_language"):
+        HindsightConfig.from_env()
+
+
+def test_text_search_extension_accepts_pgroonga(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "pgroonga")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.text_search_extension == "pgroonga"
+
+
+def test_text_search_extension_rejects_unknown(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_TEXT_SEARCH_EXTENSION", "bogus")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    with pytest.raises(ValueError, match="Invalid text_search_extension"):
+        HindsightConfig.from_env()
+
+
+def test_llm_output_language_defaults_to_none(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.delenv("HINDSIGHT_API_LLM_OUTPUT_LANGUAGE", raising=False)
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.llm_output_language is None
+
+
+def test_llm_output_language_loaded_from_env(monkeypatch):
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_LLM_OUTPUT_LANGUAGE", "Japanese")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.llm_output_language == "Japanese"
+
+
+def test_llm_output_language_empty_string_is_unset(monkeypatch):
+    """Empty env var (e.g. from Helm) should be treated as unset, not literal ''."""
+    from hindsight_api.config import HindsightConfig
+
+    monkeypatch.setenv("HINDSIGHT_API_LLM_OUTPUT_LANGUAGE", "")
+    monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "mock")
+
+    config = HindsightConfig.from_env()
+    assert config.llm_output_language is None

hindsight-api-slim/tests/test_db_abstraction.py

@@ -210,6 +210,18 @@ class TestPostgreSQLDialect:
         assert "to_tsquery" in arm
         assert "'bm25' AS source" in arm
         assert "LIMIT $3" in arm
+        # Default language is english when bm25_language is not specified
+        assert "to_tsquery('english', $4)" in arm
+
+    def test_build_bm25_arm_native_uses_configured_language(self, d):
+        arm = d.build_bm25_arm(
+            table="schema.memory_units", cols="id, text", fact_type="world",
+            bank_id_param="$2", limit_param="$3", text_param="$4",
+            bm25_language="french",
+        )
+        # Both the score and the WHERE filter must use the configured dictionary
+        assert "to_tsquery('french', $4)" in arm
+        assert "to_tsquery('english'" not in arm
 
     def test_build_bm25_arm_vchord(self, d):
         arm = d.build_bm25_arm(
@@ -220,6 +232,29 @@ class TestPostgreSQLDialect:
         assert "to_bm25query" in arm
         assert "tokenize" in arm
 
+    def test_build_bm25_arm_pgroonga(self, d):
+        arm = d.build_bm25_arm(
+            table="schema.memory_units", cols="id, text", fact_type="world",
+            bank_id_param="$2", limit_param="$3", text_param="$4",
+            text_search_extension="pgroonga",
+        )
+        # pgroonga uses the &@~ operator + pgroonga_score for ranking. The
+        # configured bm25_language is intentionally NOT used here — pgroonga's
+        # tokenizer is set at index creation, not query time.
+        assert "&@~ $4" in arm
+        assert "pgroonga_score(tableoid, ctid)" in arm
+        assert "to_tsquery" not in arm
+
+    def test_build_bm25_arm_pgroonga_ignores_bm25_language(self, d):
+        """pgroonga's tokenizer is fixed at index creation; bm25_language must not leak in."""
+        arm = d.build_bm25_arm(
+            table="t", cols="id", fact_type="world",
+            bank_id_param="$2", limit_param="$3", text_param="$4",
+            text_search_extension="pgroonga",
+            bm25_language="french",
+        )
+        assert "french" not in arm
+
     def test_prepare_bm25_text_native(self, d):
         result = d.prepare_bm25_text(["hello", "world"], "hello world")
         assert result == "hello | world"
@@ -228,6 +263,12 @@ class TestPostgreSQLDialect:
         result = d.prepare_bm25_text(["hello", "world"], "hello world", text_search_extension="vchord")
         assert result == "hello world"
 
+    def test_prepare_bm25_text_pgroonga(self, d):
+        # pgroonga accepts raw query text via &@~ and parses it with its own
+        # query syntax; we pass the original query through unchanged.
+        result = d.prepare_bm25_text(["hello", "world"], "hello world", text_search_extension="pgroonga")
+        assert result == "hello world"
+
 
 # ---------------------------------------------------------------------------
 # OracleDialect tests (no oracledb dependency needed)

hindsight-api-slim/tests/test_hierarchical_config.py

@@ -124,9 +124,10 @@ async def test_hierarchical_fields_categorization():
     assert "llm_gemini_safety_settings" in configurable
     assert "mcp_enabled_tools" in configurable
     assert "retain_chunk_batch_size" in configurable
+    assert "enable_auto_consolidation" in configurable
 
     # Verify count is correct
-    assert len(configurable) == 35
+    assert len(configurable) == 36
 
     # Verify credential fields (NEVER exposed)
     assert "llm_api_key" in credentials

hindsight-api-slim/tests/test_multilingual_bm25.py

@@ -0,0 +1,167 @@
+"""Tests for multilingual BM25 + LLM output language wiring.
+
+Covers:
+- ``HINDSIGHT_API_LLM_OUTPUT_LANGUAGE`` directive injection across all three
+  LLM-generating pipelines: retain (fact extraction), consolidation
+  (observations), and reflect (response synthesis).
+- The new alembic migration's structural shape (chains off the right head).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from hindsight_api.engine.consolidation.prompts import build_batch_consolidation_prompt
+from hindsight_api.engine.prompt_utils import output_language_directive
+from hindsight_api.engine.reflect.prompts import build_final_system_prompt
+from hindsight_api.engine.retain.fact_extraction import _build_extraction_prompt_and_schema
+
+
+def _baseline_config() -> MagicMock:
+    """Mock config with the minimal fields needed by _build_extraction_prompt_and_schema."""
+    config = MagicMock()
+    config.entity_labels = None
+    config.entities_allow_free_form = True
+    config.retain_extraction_mode = "concise"
+    config.retain_extract_causal_links = False
+    config.retain_mission = None
+    config.retain_custom_instructions = None
+    config.llm_output_language = None
+    return config
+
+
+# ---------------------------------------------------------------------------
+# Shared directive helper
+# ---------------------------------------------------------------------------
+
+
+def test_output_language_directive_empty_when_unset():
+    assert output_language_directive(None) == ""
+    assert output_language_directive("") == ""
+
+
+def test_output_language_directive_mentions_language_three_times():
+    directive = output_language_directive("Japanese")
+    # All three references are needed so the LLM applies the constraint to
+    # source translation, fact text, and the final response equally.
+    assert directive.count("Japanese") == 3
+    assert "Respond exclusively in Japanese" in directive
+    assert "Translate any source content into Japanese" in directive
+
+
+# ---------------------------------------------------------------------------
+# Retain (fact extraction)
+# ---------------------------------------------------------------------------
+
+
+def test_retain_unset_does_not_inject_directive():
+    config = _baseline_config()
+    config.llm_output_language = None
+
+    prompt, _ = _build_extraction_prompt_and_schema(config)
+
+    assert "Respond exclusively in" not in prompt
+    assert "Translate any source content" not in prompt
+
+
+def test_retain_injects_directive():
+    config = _baseline_config()
+    config.llm_output_language = "Japanese"
+
+    prompt, _ = _build_extraction_prompt_and_schema(config)
+
+    assert "Respond exclusively in Japanese" in prompt
+    assert "Translate any source content into Japanese" in prompt
+
+
+def test_retain_directive_appears_after_base_prompt():
+    """The directive is appended at the end so mode-specific guidelines are
+    still respected — the LLM reads them, then applies the language constraint."""
+    config = _baseline_config()
+    config.llm_output_language = "Spanish"
+
+    prompt, _ = _build_extraction_prompt_and_schema(config)
+
+    directive_idx = prompt.find("Respond exclusively in Spanish")
+    assert directive_idx > 0
+    # A non-trivial extraction prompt body precedes the directive.
+    assert directive_idx > 100
+
+
+def test_retain_works_with_custom_mode():
+    """Custom extraction mode + llm_output_language: directive must still appear."""
+    config = _baseline_config()
+    config.retain_extraction_mode = "custom"
+    config.retain_custom_instructions = "Extract only product mentions."
+    config.llm_output_language = "French"
+
+    prompt, _ = _build_extraction_prompt_and_schema(config)
+
+    assert "Extract only product mentions." in prompt
+    assert "Respond exclusively in French" in prompt
+
+
+# ---------------------------------------------------------------------------
+# Consolidation (observations)
+# ---------------------------------------------------------------------------
+
+
+def test_consolidation_unset_does_not_inject_directive():
+    prompt = build_batch_consolidation_prompt(llm_output_language=None)
+    assert "Respond exclusively in" not in prompt
+
+
+def test_consolidation_injects_directive():
+    prompt = build_batch_consolidation_prompt(llm_output_language="Chinese")
+    assert "Respond exclusively in Chinese" in prompt
+    assert "Translate any source content into Chinese" in prompt
+
+
+def test_consolidation_directive_does_not_break_format_placeholders():
+    """The consolidation prompt is later passed through str.format(facts_text=..., observations_text=...).
+    The appended directive must not introduce stray { / } that would raise KeyError."""
+    prompt = build_batch_consolidation_prompt(llm_output_language="Japanese")
+    # str.format must succeed with the expected placeholders.
+    prompt.format(facts_text="X", observations_text="Y")
+
+
+# ---------------------------------------------------------------------------
+# Reflect (response synthesis)
+# ---------------------------------------------------------------------------
+
+
+def test_reflect_unset_does_not_inject_directive():
+    prompt = build_final_system_prompt(mission=None, llm_output_language=None)
+    assert "Respond exclusively in" not in prompt
+
+
+def test_reflect_injects_directive():
+    prompt = build_final_system_prompt(mission=None, llm_output_language="Korean")
+    assert "Respond exclusively in Korean" in prompt
+
+
+def test_reflect_preserves_mission_alongside_directive():
+    prompt = build_final_system_prompt(mission="Act as a financial analyst.", llm_output_language="Spanish")
+    assert "financial analyst" in prompt
+    assert "Respond exclusively in Spanish" in prompt
+
+
+# ---------------------------------------------------------------------------
+# Migration shape regression test
+# ---------------------------------------------------------------------------
+
+
+def test_configurable_bm25_language_migration_chains_off_head():
+    """The new migration must descend from the head it was authored against.
+
+    Tests that re-pointing the migration's down_revision wouldn't go
+    unnoticed — it would silently break the chain on a fresh DB.
+    """
+    versions_dir = Path(__file__).resolve().parent.parent / "hindsight_api" / "alembic" / "versions"
+    target = versions_dir / "p4q5r6s7t8u9_configurable_bm25_language.py"
+    assert target.exists(), "configurable_bm25_language migration file is missing"
+
+    src = target.read_text()
+    assert 'revision: str = "p4q5r6s7t8u9"' in src
+    assert 'down_revision: str | Sequence[str] | None = "86f7a033d372"' in src

hindsight-cli/src/api.rs

@@ -953,7 +953,11 @@ impl ApiClient {
         _verbose: bool,
     ) -> Result<types::ConsolidationResponse> {
         self.runtime.block_on(async {
-            let response = self.client.trigger_consolidation(bank_id, None).await?;
+            let body = types::ConsolidationRequest::default();
+            let response = self
+                .client
+                .trigger_consolidation(bank_id, None, &body)
+                .await?;
             Ok(response.into_inner())
         })
     }

hindsight-docs/docs/developer/configuration.md

@@ -140,37 +140,19 @@ If you need to switch from one extension to another:
 
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native`, `vchord`, or `pg_textsearch` | `native` |
+| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native`, `vchord`, `pg_textsearch`, or `pgroonga` | `native` |
+| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE` | PostgreSQL text search dictionary used by the `native` backend (e.g. `english`, `french`, `simple`, `zhparser`) | `english` |
+| `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` | When set, forces every LLM-generated artifact (retain facts, consolidation observations, reflect responses) into this language. Free-form (e.g. `Spanish`, `Japanese`). | unset |
 
-Hindsight supports three text search backends for BM25 keyword retrieval:
-- **native**: PostgreSQL's built-in full-text search (`tsvector` + GIN indexes)
-- **vchord**: VectorChord BM25 (`bm25vector` + BM25 indexes) - requires `vchord_bm25` extension
-- **pg_textsearch**: Timescale BM25 (text columns + BM25 indexes) - requires `pg_textsearch` extension
+Hindsight supports four backends for BM25 keyword retrieval:
+- **native** — PostgreSQL's built-in full-text search (`tsvector` + GIN). Language configurable.
+- **vchord** — VectorChord BM25 (uses the `llmlingua2` multilingual tokenizer).
+- **pg_textsearch** — Timescale's pg_textsearch extension. English-only.
+- **pgroonga** — pgroonga full-text search. Multilingual / CJK out of the box.
 
-**When to use native:**
-- Standard PostgreSQL deployment (no extra extensions)
-- Simpler setup and wider compatibility
-- Works well for most use cases
+To switch backends: set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION`. With existing data, you'll get an error and migration instructions; with an empty database the columns/indexes are recreated automatically on startup.
 
-**When to use vchord:**
-- Already using vchord for vector search (good integration)
-- Want better BM25 ranking performance
-- Need advanced tokenization (uses `llmlingua2` tokenizer)
-
-**When to use pg_textsearch:**
-- Want industry-standard BM25 ranking with better relevance than native PostgreSQL
-- Need efficient top-K queries with Block-Max WAND optimization
-- Prefer lower memory footprint compared to vchord
-- Already using Timescale or have `pg_textsearch` available
-
-**Switching backends:**
-
-To switch between backends:
-1. Set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` to your desired backend (`native`, `vchord`, or `pg_textsearch`)
-2. If your database has existing data, you'll get an error with migration instructions
-3. For empty databases, the columns/indexes will be automatically recreated on startup
-
-**Note:** VectorChord uses the `llmlingua2` tokenizer for multilingual support, while native and pg_textsearch use PostgreSQL's English tokenizer.
+For non-English banks (especially CJK) and the language/extraction-language tradeoffs, see the [Multilingual Support](./multilingual) page.
 
 ### LLM Provider
 

hindsight-docs/docs/developer/multilingual.md

@@ -153,7 +153,7 @@ Most modern LLMs (GPT-4, Claude, Gemini, Llama 3, etc.) support dozens of langua
 
 ## Configuring for Multilingual Use
 
-For optimal multilingual performance, you should configure all three components of the pipeline:
+For optimal multilingual performance, configure all four components of the pipeline:
 
 ### 1. LLM (Required)
 Your LLM must support the target languages. Most modern LLMs do, but verify with your specific model.
@@ -187,6 +187,57 @@ HINDSIGHT_API_RERANKER_LOCAL_MODEL=BAAI/bge-reranker-v2-m3
 | `BAAI/bge-reranker-v2-m3` | 100+ | Best multilingual reranking |
 | `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 14 | Lighter alternative |
 
+### 4. BM25 / Full-Text Search Backend
+
+The semantic (embedding) arm covers cross-lingual matches by meaning. Hindsight runs a BM25 keyword arm in parallel, and **BM25 is inherently within-language** — it's character/token matching against a tokenizer's lexemes. The default `native` backend uses PostgreSQL's English dictionary, which produces poor results for non-English content (and no useful tokenization at all for Chinese / Japanese / Korean, which lack whitespace word boundaries).
+
+There are two knobs that interact:
+
+- `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` — selects the backend (`native`, `vchord`, `pg_textsearch`, or `pgroonga`).
+- `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE` — selects the PostgreSQL dictionary used by the `native` backend (default: `english`).
+
+Pick the backend based on the languages your bank stores:
+
+| Backend | Multilingual / CJK | Notes |
+|---------|--------------------|-------|
+| `native` | European languages only (English, French, German, Spanish, Italian, Portuguese, Russian, Dutch, Swedish, Norwegian, Danish, Finnish, Hungarian, Turkish, Arabic, plus `simple`). CJK requires a third-party dictionary like `zhparser`. | Stock PostgreSQL — no extra extensions. Configure the language via `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE`. |
+| `vchord` | Multilingual via `llmlingua2` tokenizer. | Best when you're already using vchord for vector search. |
+| `pg_textsearch` | English only (hardcoded). | Industry-standard BM25 ranking + Block-Max WAND. |
+| `pgroonga` | **Yes — out of the box.** Single index handles English, CJK, and mixed-script content via the `TokenBigram` polyglot tokenizer + `NormalizerNFKC150` Unicode normalization. | Recommended for non-English / mixed-language banks. Requires the `pgroonga` extension. See `docker/docker-compose/pgroonga/`. |
+
+**Choosing for a single-language bank** (e.g. all Spanish content):
+```bash
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION=native
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE=spanish
+```
+
+**Choosing for a CJK or mixed-language bank**:
+```bash
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pgroonga
+```
+
+The `native` and `pgroonga` knobs do not apply to each other — `pgroonga`'s tokenizer is set at index creation and ignores `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE`.
+
+#### Forcing the LLM Output Language
+
+Independent from the BM25 backend, `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` forces every LLM-generated artifact into a single language regardless of the source content. This applies uniformly to:
+
+- **Retain** — fact text, context, and entity names extracted from source documents.
+- **Consolidation** — observations / mental models synthesized from those facts.
+- **Reflect** — the final natural-language response returned by the reflect API.
+
+```bash
+# Every LLM call (retain, consolidation, reflect) emits Spanish regardless of source language.
+HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=Spanish
+```
+
+Common patterns:
+- **Aligned, single-language bank**: `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE=spanish` + `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=Spanish` — store, index, and respond in Spanish even when sources are mixed.
+- **Mixed-language bank with multilingual indexing**: `HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pgroonga` + leave `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` unset — preserve source-language facts; pgroonga handles all of them in one index; reflect responds in the query's language.
+- **Cross-lingual unification**: `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=English` — every fact, observation, and reflect response in English regardless of source. Useful when the consumer (an English-only LLM, dashboard, or downstream pipeline) needs uniform output.
+
+Leave `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` unset to preserve the source/query language across the pipeline (the default).
+
 ---
 
 ## Best Practices

skills/hindsight-docs/references/developer/configuration.md

@@ -140,37 +140,19 @@ If you need to switch from one extension to another:
 
 | Variable | Description | Default |
 |----------|-------------|---------|
-| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native`, `vchord`, or `pg_textsearch` | `native` |
+| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` | Text search backend: `native`, `vchord`, `pg_textsearch`, or `pgroonga` | `native` |
+| `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE` | PostgreSQL text search dictionary used by the `native` backend (e.g. `english`, `french`, `simple`, `zhparser`) | `english` |
+| `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` | When set, forces every LLM-generated artifact (retain facts, consolidation observations, reflect responses) into this language. Free-form (e.g. `Spanish`, `Japanese`). | unset |
 
-Hindsight supports three text search backends for BM25 keyword retrieval:
-- **native**: PostgreSQL's built-in full-text search (`tsvector` + GIN indexes)
-- **vchord**: VectorChord BM25 (`bm25vector` + BM25 indexes) - requires `vchord_bm25` extension
-- **pg_textsearch**: Timescale BM25 (text columns + BM25 indexes) - requires `pg_textsearch` extension
+Hindsight supports four backends for BM25 keyword retrieval:
+- **native** — PostgreSQL's built-in full-text search (`tsvector` + GIN). Language configurable.
+- **vchord** — VectorChord BM25 (uses the `llmlingua2` multilingual tokenizer).
+- **pg_textsearch** — Timescale's pg_textsearch extension. English-only.
+- **pgroonga** — pgroonga full-text search. Multilingual / CJK out of the box.
 
-**When to use native:**
-- Standard PostgreSQL deployment (no extra extensions)
-- Simpler setup and wider compatibility
-- Works well for most use cases
+To switch backends: set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION`. With existing data, you'll get an error and migration instructions; with an empty database the columns/indexes are recreated automatically on startup.
 
-**When to use vchord:**
-- Already using vchord for vector search (good integration)
-- Want better BM25 ranking performance
-- Need advanced tokenization (uses `llmlingua2` tokenizer)
-
-**When to use pg_textsearch:**
-- Want industry-standard BM25 ranking with better relevance than native PostgreSQL
-- Need efficient top-K queries with Block-Max WAND optimization
-- Prefer lower memory footprint compared to vchord
-- Already using Timescale or have `pg_textsearch` available
-
-**Switching backends:**
-
-To switch between backends:
-1. Set `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` to your desired backend (`native`, `vchord`, or `pg_textsearch`)
-2. If your database has existing data, you'll get an error with migration instructions
-3. For empty databases, the columns/indexes will be automatically recreated on startup
-
-**Note:** VectorChord uses the `llmlingua2` tokenizer for multilingual support, while native and pg_textsearch use PostgreSQL's English tokenizer.
+For non-English banks (especially CJK) and the language/extraction-language tradeoffs, see the [Multilingual Support](./multilingual) page.
 
 ### LLM Provider
 

skills/hindsight-docs/references/developer/multilingual.md

@@ -153,7 +153,7 @@ Most modern LLMs (GPT-4, Claude, Gemini, Llama 3, etc.) support dozens of langua
 
 ## Configuring for Multilingual Use
 
-For optimal multilingual performance, you should configure all three components of the pipeline:
+For optimal multilingual performance, configure all four components of the pipeline:
 
 ### 1. LLM (Required)
 Your LLM must support the target languages. Most modern LLMs do, but verify with your specific model.
@@ -187,6 +187,57 @@ HINDSIGHT_API_RERANKER_LOCAL_MODEL=BAAI/bge-reranker-v2-m3
 | `BAAI/bge-reranker-v2-m3` | 100+ | Best multilingual reranking |
 | `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | 14 | Lighter alternative |
 
+### 4. BM25 / Full-Text Search Backend
+
+The semantic (embedding) arm covers cross-lingual matches by meaning. Hindsight runs a BM25 keyword arm in parallel, and **BM25 is inherently within-language** — it's character/token matching against a tokenizer's lexemes. The default `native` backend uses PostgreSQL's English dictionary, which produces poor results for non-English content (and no useful tokenization at all for Chinese / Japanese / Korean, which lack whitespace word boundaries).
+
+There are two knobs that interact:
+
+- `HINDSIGHT_API_TEXT_SEARCH_EXTENSION` — selects the backend (`native`, `vchord`, `pg_textsearch`, or `pgroonga`).
+- `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE` — selects the PostgreSQL dictionary used by the `native` backend (default: `english`).
+
+Pick the backend based on the languages your bank stores:
+
+| Backend | Multilingual / CJK | Notes |
+|---------|--------------------|-------|
+| `native` | European languages only (English, French, German, Spanish, Italian, Portuguese, Russian, Dutch, Swedish, Norwegian, Danish, Finnish, Hungarian, Turkish, Arabic, plus `simple`). CJK requires a third-party dictionary like `zhparser`. | Stock PostgreSQL — no extra extensions. Configure the language via `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE`. |
+| `vchord` | Multilingual via `llmlingua2` tokenizer. | Best when you're already using vchord for vector search. |
+| `pg_textsearch` | English only (hardcoded). | Industry-standard BM25 ranking + Block-Max WAND. |
+| `pgroonga` | **Yes — out of the box.** Single index handles English, CJK, and mixed-script content via the `TokenBigram` polyglot tokenizer + `NormalizerNFKC150` Unicode normalization. | Recommended for non-English / mixed-language banks. Requires the `pgroonga` extension. See `docker/docker-compose/pgroonga/`. |
+
+**Choosing for a single-language bank** (e.g. all Spanish content):
+```bash
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION=native
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE=spanish
+```
+
+**Choosing for a CJK or mixed-language bank**:
+```bash
+HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pgroonga
+```
+
+The `native` and `pgroonga` knobs do not apply to each other — `pgroonga`'s tokenizer is set at index creation and ignores `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE`.
+
+#### Forcing the LLM Output Language
+
+Independent from the BM25 backend, `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` forces every LLM-generated artifact into a single language regardless of the source content. This applies uniformly to:
+
+- **Retain** — fact text, context, and entity names extracted from source documents.
+- **Consolidation** — observations / mental models synthesized from those facts.
+- **Reflect** — the final natural-language response returned by the reflect API.
+
+```bash
+# Every LLM call (retain, consolidation, reflect) emits Spanish regardless of source language.
+HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=Spanish
+```
+
+Common patterns:
+- **Aligned, single-language bank**: `HINDSIGHT_API_TEXT_SEARCH_EXTENSION_NATIVE_LANGUAGE=spanish` + `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=Spanish` — store, index, and respond in Spanish even when sources are mixed.
+- **Mixed-language bank with multilingual indexing**: `HINDSIGHT_API_TEXT_SEARCH_EXTENSION=pgroonga` + leave `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` unset — preserve source-language facts; pgroonga handles all of them in one index; reflect responds in the query's language.
+- **Cross-lingual unification**: `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE=English` — every fact, observation, and reflect response in English regardless of source. Useful when the consumer (an English-only LLM, dashboard, or downstream pipeline) needs uniform output.
+
+Leave `HINDSIGHT_API_LLM_OUTPUT_LANGUAGE` unset to preserve the source/query language across the pipeline (the default).
+
 ---
 
 ## Best Practices