Duke/hindsight
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(examples): add TEI embeddings + reranker docker-compose example (#3465)

Browse Source 
* docs(examples): add TEI embeddings + reranker docker-compose example

Adds docker/docker-compose/tei/ — a runnable Compose stack that serves
embeddings and reranking from two HuggingFace Text Embeddings Inference
(TEI) sidecars, with the slim Hindsight image talking to them via
HINDSIGHT_API_{EMBEDDINGS,RERANKER}_PROVIDER=tei. Serves Hindsight's
default models so it's a drop-in 'move embeddings/reranking onto TEI'
demo. Links it from the TEI section of the models docs.

Verified end-to-end: retain + recall return the expected memory with
both TEI semantic and reranker scores populated.

* docs(examples): prod-like TEI tuning — bge-reranker-base + throughput flags

Swap the reranker to BAAI/bge-reranker-base (the cross-encoder commonly
paired with bge-small embeddings on dedicated inference servers) and carry
prod-like TEI throughput flags on both services (--max-concurrent-requests,
--max-batch-tokens, --max-client-batch-size) instead of TEI's bare defaults,
so the example doubles as a starting point for real deployments.
This commit is contained in:
Nicolò Boschi
2026-08-13 17:25:30 +02:00
committed by GitHub
parent 39ee97a61c
commit 77bdda7042
4 changed files with 224 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/docker-compose/tei/README.md
+107
View File
@@ -0,0 +1,107 @@
# Hindsight with TEI embeddings + reranker
Example Docker Compose setup that serves **embeddings and reranking from two
[HuggingFace Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference)
sidecars** instead of the in-process local models.
Because embeddings and reranking run outside the API, Hindsight itself needs
no baked-in models, so this uses the **slim** image
(`ghcr.io/vectorize-io/hindsight:latest-slim`). Only the LLM — used for
retain/recall/reflect — still needs a provider and API key.
## When to use this
- You want embeddings/reranking on a dedicated, independently scalable
inference server (e.g. a GPU node) rather than in the API process.
- You run the **slim** image and pull embeddings/reranking from an external
service.
- You want a self-hosted, offline-capable alternative to a cloud embeddings
provider (OpenAI, Cohere, ...).
If you just want local models in-process, use the default full image — no
sidecars required.
## What it runs
| Service | Image | Model |
| --------------- | ------------------------------------------------------ | ---------------------------------------- |
| `tei-embedding` | `ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.3` | `BAAI/bge-small-en-v1.5` (384-dim) |
| `tei-reranker` | `ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.3` | `BAAI/bge-reranker-base` |
| `hindsight` | `ghcr.io/vectorize-io/hindsight:latest-slim` | — (slim; talks to the sidecars) |
This is a prod-like configuration: the embedding model is Hindsight's default
(`bge-small-en-v1.5`), the reranker is the `bge-reranker-base` cross-encoder
commonly paired with it on dedicated inference servers, and both services carry
throughput flags (`--max-concurrent-requests`, `--max-batch-tokens`,
`--max-client-batch-size`) tuned for sustained multi-client load instead of
TEI's bare defaults. The API points at the sidecars with:
```
HINDSIGHT_API_EMBEDDINGS_PROVIDER=tei
HINDSIGHT_API_EMBEDDINGS_TEI_URL=http://tei-embedding:80
HINDSIGHT_API_RERANKER_PROVIDER=tei
HINDSIGHT_API_RERANKER_TEI_URL=http://tei-reranker:80
```
## Quick start
```bash
export HINDSIGHT_API_LLM_API_KEY=sk-xxx
docker compose -f docker/docker-compose/tei/docker-compose.yaml up
```
- API: http://localhost:8888
- Control Plane: http://localhost:9999
- TEI embedding server: http://localhost:8080 (exposed for debugging)
- TEI reranker server: http://localhost:8081 (exposed for debugging)
`hindsight` waits (via `depends_on: service_healthy`) until both TEI servers
report healthy, so the first boot pauses while each model downloads into its
`tei_*_cache` volume. Subsequent boots reuse the cached models.
To use an LLM provider other than the default `openai`:
```bash
export HINDSIGHT_API_LLM_PROVIDER=gemini
export HINDSIGHT_API_LLM_API_KEY=...
docker compose -f docker/docker-compose/tei/docker-compose.yaml up
```
## Using your own models
Change the `--model-id` in each service's `command` to any TEI-supported
model. The embedding dimension is **auto-detected from the server** and the
pgvector schema is adjusted to match on first boot — no dimension env var to
set. (If you switch the embedding model after data already exists, start from
a fresh `pg_data` volume, since the stored vectors were built for the old
dimension.)
## Verifying the servers
```bash
# Health
curl 127.0.0.1:8080/health && curl 127.0.0.1:8081/health
# Embedding (returns a 384-length vector for the default model)
curl 127.0.0.1:8080/embed -H 'content-type: application/json' \
-d '{"inputs":"hello world"}'
# Rerank
curl 127.0.0.1:8081/rerank -H 'content-type: application/json' \
-d '{"query":"what is the capital of France?","texts":["Paris is the capital of France.","Bananas are yellow."]}'
```
## Apple Silicon / arm64
The `cpu-1.8.3` TEI images are published for `linux/amd64` only. On an
Apple Silicon Mac, run under emulation:
```bash
export DOCKER_DEFAULT_PLATFORM=linux/amd64
docker compose -f docker/docker-compose/tei/docker-compose.yaml up
```
Emulated startup is slow (model load takes a few minutes). For production,
run on `amd64` hosts — or a GPU node with the CUDA-tagged TEI image and a GPU
reservation.
docker/docker-compose/tei/docker-compose.yaml
+113
View File
@@ -0,0 +1,113 @@
name: hindsight-tei
# Example: run Hindsight with embeddings and reranking served by two
# HuggingFace Text Embeddings Inference (TEI) sidecars instead of the
# in-process local models.
#
# Because embeddings and reranking are external, Hindsight itself needs no
# baked-in models — this uses the **slim** image
# (`ghcr.io/vectorize-io/hindsight:latest-slim`). Only the LLM (used for
# retain/recall/reflect) still needs a provider + API key.
#
# The two TEI services here run a prod-like configuration:
# `BAAI/bge-small-en-v1.5` embeddings (384-dim, Hindsight's default) and the
# `BAAI/bge-reranker-base` cross-encoder, with the batching/concurrency flags
# tuned for sustained multi-client load rather than TEI's bare defaults. Swap
# the `--model-id` args to serve any TEI-supported model — the embedding
# dimension is auto-detected from the server, and the pgvector schema is
# adjusted to match on first boot.
#
# Quick start:
# export HINDSIGHT_API_LLM_API_KEY=sk-xxx
# docker compose -f docker/docker-compose/tei/docker-compose.yaml up
#
# First boot downloads the two models into the `tei_*_cache` volumes;
# subsequent boots reuse them.
services:
tei-embedding:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.3
container_name: hindsight-tei-embedding
# Prod-like tuning: high request concurrency with bounded batch sizes.
command:
[
"--model-id", "BAAI/bge-small-en-v1.5",
"--max-concurrent-requests", "512",
"--max-batch-tokens", "16384",
"--max-client-batch-size", "32",
"--auto-truncate",
]
environment:
# TEI listens on port 80 inside the container by default.
PORT: "80"
ports:
# Exposed on the host so you can curl the server directly, e.g.
# curl 127.0.0.1:8080/embed -H 'content-type: application/json' \
# -d '{"inputs":"hello world"}'
- "8080:80"
volumes:
- tei_embedding_cache:/data
healthcheck:
# The TEI image ships curl; hit its /health endpoint so Hindsight only
# starts once the model is loaded and serving.
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 10s
timeout: 5s
retries: 60
start_period: 30s
tei-reranker:
image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.8.3
container_name: hindsight-tei-reranker
# Prod-like tuning: reranking batches are larger than embedding batches
# (rerank inputs are query+document pairs scored in bulk during recall).
command:
[
"--model-id", "BAAI/bge-reranker-base",
"--max-concurrent-requests", "512",
"--max-batch-tokens", "32768",
"--max-client-batch-size", "128",
"--auto-truncate",
]
environment:
PORT: "80"
ports:
- "8081:80"
volumes:
- tei_reranker_cache:/data
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:80/health"]
interval: 10s
timeout: 5s
retries: 60
start_period: 30s
hindsight:
image: ghcr.io/vectorize-io/hindsight:${HINDSIGHT_VERSION:-latest-slim}
container_name: hindsight-tei
depends_on:
tei-embedding:
condition: service_healthy
tei-reranker:
condition: service_healthy
ports:
- "8888:8888"
- "9999:9999"
environment:
# LLM still runs through a provider — bring your own key. Pair with
# HINDSIGHT_API_LLM_PROVIDER to use a provider other than openai.
HINDSIGHT_API_LLM_PROVIDER: ${HINDSIGHT_API_LLM_PROVIDER:-openai}
HINDSIGHT_API_LLM_API_KEY: ${HINDSIGHT_API_LLM_API_KEY:-}
# Embeddings + reranking served by the TEI sidecars above. Use the
# in-cluster service DNS names, not localhost.
HINDSIGHT_API_EMBEDDINGS_PROVIDER: tei
HINDSIGHT_API_EMBEDDINGS_TEI_URL: http://tei-embedding:80
HINDSIGHT_API_RERANKER_PROVIDER: tei
HINDSIGHT_API_RERANKER_TEI_URL: http://tei-reranker:80
volumes:
- pg_data:/home/hindsight/.pg0
volumes:
pg_data:
tei_embedding_cache:
tei_reranker_cache:
hindsight-docs/docs/developer/models.mdx
+2
View File
@@ -811,6 +811,8 @@ export HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MODEL=openai/text-embedding-3-small
See [Configuration](./configuration#embeddings) for all options including Azure OpenAI and custom endpoints.
For a runnable Docker Compose setup that serves both embeddings and reranking from self-hosted TEI sidecars, see [`docker/docker-compose/tei/`](https://github.com/vectorize-io/hindsight/tree/main/docker/docker-compose/tei).
---
## Cross-Encoder (Reranker)
skills/hindsight-docs/references/developer/models.md
+2
View File
@@ -869,6 +869,8 @@ export HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MODEL=openai/text-embedding-3-small
See [Configuration](./configuration#embeddings) for all options including Azure OpenAI and custom endpoints.
For a runnable Docker Compose setup that serves both embeddings and reranking from self-hosted TEI sidecars, see [`docker/docker-compose/tei/`](https://github.com/vectorize-io/hindsight/tree/main/docker/docker-compose/tei).
---
## Cross-Encoder (Reranker)