Semantic Search

Overview

When the agent calls memory_recall, MemU performs a multi-stage search that combines full-text matching with semantic similarity to find the most relevant memories.

Search Pipeline

Query → FTS5 Search → Embedding Similarity → Score Fusion → Rerank → Results

Stage 1: FTS5 Full-Text Search

The query is run against the FTS5 index. This finds memories that contain matching terms:

SELECT * FROM memory_fts WHERE memory_fts MATCH 'infrastructure specs'
ORDER BY rank
LIMIT 50

FTS5 provides a relevance rank based on term frequency, position, and document length.

Stage 2: Embedding Similarity

The query is embedded into a vector and compared against stored memory embeddings using cosine similarity:

similarity = dot(query_embedding, memory_embedding) / (|query_embedding| * |memory_embedding|)

This catches semantically related memories that may not share exact words with the query. For example, "computer setup" would match a memory about "infrastructure specs."

Stage 3: Score Fusion

FTS5 scores and embedding similarity scores are combined:

final_score = (fts5_score * fts5_weight) + (similarity² * embedding_weight) + significance_boost

Key details:

• Similarity is squared (sim²) to amplify strong matches and suppress weak ones
• Significance adds a boost based on the memory's importance score
• Memory recall queries get a +0.25 score boost for model routing (ensures at least Haiku tier)

Stage 4: Reranking

Results are reranked with additional signals:

• Recency: More recent memories get a slight boost
• Reinforcement: Memories that have been recalled frequently (log2 reinforcement) rank higher
• Entity matching: Memories mentioning entities in the query get a boost

If the reranker encounters errors, it falls back safely to the fusion score ordering.

Query Optimization

Best Practices

Good: "Jason infrastructure"          # Specific keywords
Good: "telegram OR discord setup"     # OR for alternatives
Good: "meeting notes February"        # Scoped by time context

Bad:  "what did we talk about"        # Too vague
Bad:  "the thing from yesterday"      # No searchable terms

FTS5 Syntax

Result Format

Search results are returned to the agent as a structured list:

[
  {
    "id": 42,
    "content": "Jason runs 2x NVIDIA DGX Spark with 256GB unified memory",
    "significance": 8,
    "score": 0.92,
    "created_at": "2026-01-15T10:30:00Z"
  }
]