When to use vector-based semantic search vs. keyword matching in AI agent memory systems, with real performance comparisons.
I have watched too many AI builders pick vector search as their default memory retrieval strategy because it sounds more advanced. Semantic search. Embeddings. Cosine similarity. The terminology alone makes it feel like the right choice.
Then they run into cases where a simple keyword query returns perfect results in 2 milliseconds while their vector search returns vaguely related results in 200 milliseconds. And they start wondering whether they overcomplicated things.
Both approaches have a place. The mistake is treating this as a binary choice. After building a six-layer memory system that uses both, I can tell you exactly where each one excels and where it fails.
Quick primer for context. Skip this section if you already know the mechanics.
Keyword search matches literal strings. You search for “invoice automation” and it returns documents that contain those words. It can handle variations (plurals, different word forms) and boolean logic (documents containing “invoice” AND “automation” but NOT “manual”). The results are precise: if the words are there, you find them. If the words are not there, you do not.
Semantic search matches meaning. Your query “invoice automation” gets converted to a numerical vector (an embedding) that represents its meaning in a high-dimensional space. The search finds documents whose embeddings are close to your query embedding. This means a document about “automating billing processes” can match even though it does not contain the words “invoice” or “automation.”
The trade-off is precision vs. recall. Keyword search has high precision (what it finds is relevant) but lower recall (it misses relevant documents that use different words). Semantic search has higher recall (it finds conceptually related content) but sometimes lower precision (it can return content that is related but not what you wanted).
The exact architecture, memory layers, and delegation patterns I use to run 50 agents across two businesses.
Get the AI Agent Blueprint →Cross-terminology retrieval. My agents use different vocabulary for the same concepts. The research agent might store a finding about “customer churn reduction strategies.” The content agent searches for “keeping clients from leaving.” Keyword search returns nothing. Semantic search connects these because the meaning is similar.
This matters more than it seems. In a multi-agent system, different agents are built by different prompts, at different times, for different purposes. They do not use a shared vocabulary. Semantic search bridges these vocabulary gaps naturally.
Exploratory queries. When an agent does not know exactly what it is looking for, semantic search is better. “What do I know about this client’s priorities?” is an exploratory query. The relevant memories might be scattered across different conversations, tagged with different labels, using different phrases. Semantic search retrieves a cluster of related content. Keyword search would require knowing the exact phrases used in each memory.
Conceptual similarity. I store lessons learned after production incidents. When a new incident occurs, I want the agent to find similar past incidents. But similarity here is conceptual: two incidents might share the same root cause (a timing issue, a resource contention problem) while having completely different symptoms and descriptions. Semantic search handles this well because it matches on the underlying concept, not the surface-level description.
Exact lookups. When you know exactly what you are looking for, keyword search is faster and more precise. “Find the meeting notes from March 15 with Client X.” This is a lookup, not a search. The relevant fields are the date and the client name. Keyword matching (or structured queries) will find this instantly. Semantic search might return meeting notes from adjacent dates with similar clients because the embeddings are close.
Code and configuration retrieval. Technical content does not embed well. A cron job configuration, a database query, a script name: these are precise strings where meaning is literal. Searching for “the cron job that runs memory consolidation” with semantic search might return conceptually related content about memory management. Keyword search for “memory_consolidation” finds the exact job.
Boolean precision. “Find all memories tagged with client X that mention pricing but not the old pricing model.” This query has specific boolean requirements that keyword search handles natively. Semantic search does not support boolean logic in the same way. You would need to run the semantic search and then filter the results, which adds complexity and latency.
Speed at scale. For my dataset (a few hundred thousand entries), both approaches are fast. But keyword search on indexed fields is consistently faster: 1 to 5 milliseconds versus 20 to 80 milliseconds for vector search. At larger scales, this gap widens. If you are running hundreds of memory queries per hour across multiple agents, the latency difference compounds.
I do not choose one or the other. I use both, with clear rules about when each fires.
Rule 1: Structured queries first. If the query includes specific identifiers (dates, names, tags, IDs), it goes to keyword/structured search. These queries have a right answer, and keyword search finds it faster.
Rule 2: Semantic search for open-ended queries. If the query is conceptual (“what do I know about X’s priorities?” or “what happened in situations similar to Y?”), it goes to vector search. These queries do not have a single right answer, and semantic matching produces better results.
Rule 3: Parallel search for high-stakes decisions. For decisions where missing relevant context could be costly, I run both searches in parallel and merge the results. This is slower (the total latency is the slower of the two) but more thorough. The orchestrator uses this for task routing decisions where an overlooked piece of context could mean assigning work to the wrong agent.
Rule 4: Keyword search as a fallback for semantic misses. If semantic search returns results with low confidence scores (below a threshold I tuned empirically), the system automatically falls back to keyword search. Sometimes the relevant content is too literal or too short for meaningful embeddings. A three-word note like “Client X: paused” has almost no semantic content, but keyword search finds it instantly.
The practical implementation uses PostgreSQL for both search types, which simplifies the infrastructure.
Keyword search uses standard Postgres full-text search with GIN indexes. The queries are fast, the indexing is automatic, and the boolean operators work out of the box.
Semantic search uses pgvector with HNSW indexes. The embeddings are generated by a local model (nomic-embed-text) to avoid API calls and latency. Generating an embedding takes about 10 milliseconds locally versus 100 to 300 milliseconds for an API call. Over thousands of queries, this difference matters.
Both search types query the same database tables. A memory entry has both a text column (for keyword search) and a vector column (for semantic search). This means I do not need to keep two data stores in sync. One insert populates both search paths.
The routing logic (which search type to use for which query) lives in a Python function that the agent calls before any memory query. The function inspects the query for structured identifiers, evaluates the query type, and returns the appropriate search method. It is about 40 lines of code and has been stable since week two.
Do not default to semantic search because it is more sophisticated. Start with keyword search. It is faster to implement, easier to debug, and sufficient for many use cases.
Add semantic search when you hit specific limitations: vocabulary mismatch between agents, exploratory queries returning empty results, or conceptual similarity needs that keyword matching cannot serve.
Build both on the same database. The operational simplicity of one data store outweighs the theoretical performance gains of specialized systems.
And measure everything. Track which search type your agents use most frequently, what the average latency is, and how often the results are useful. The data will tell you where to invest.
The blueprint includes the hybrid search implementation, the routing logic, and the Postgres configuration for running both search types on a single database.
The debate between semantic and keyword search is a false binary. Production systems need both. The skill is knowing which one to reach for in which situation, and building infrastructure that makes switching between them invisible to the agents that depend on it.
Subscribe to The Signal
One issue per week. The biggest AI build of the week, the top five videos worth your time, and an operator's read on what actually matters. For builders who ship, not dreamers who scroll.
No spam. Unsubscribe anytime. Just The Signal.