Semantic Caching for LLM Applications on AWS - Embedding-Based Cache Design with OpenSearch and Similarity Thresholds

First Published: 2026-07-14
Last Updated: 2026-07-14

Two users ask a support assistant the same thing in different words — "How do I reset my password?" and "I forgot my login, how do I get back in?" — and a naive application invokes the model twice, paying full latency and full token cost for what is, semantically, one question answered once. A semantic cache closes that gap: it recognizes that the second request means the same thing as the first, and serves the stored answer instead of calling the model again. This article is an implementation-level guide to building that cache on AWS, with Amazon OpenSearch Service and Amazon MemoryDB as the vector stores, embedding models on Amazon Bedrock as the encoder, and a similarity threshold as the single dial that decides what counts as "the same question."

A deliberate constraint runs through the whole article: there are no prices here. No dollar figures, no per-token rates, no percentage savings quoted as fact. Rates change, and the value of a cache is easiest to reason about in the units that stay stable — hit rate, false-hit rate, latency, and the request path — with links to the official pricing pages where an exact number matters. And a second, more important constraint: a semantic cache trades a small, non-zero risk of a wrong answer for speed. Set the threshold too loosely and the cache will occasionally return one question's answer to a different question — in a multi-tenant system, potentially one user's answer to another. "Add a cache and go faster" is only half the story; the other half is calibrating and bounding that risk, which is why the similarity threshold gets a section of its own.

This guide is the sibling of an existing article on Anthropic Claude API Prompt Caching and Token Efficiency. That article is about prompt caching — reusing the token processing of a byte-identical prefix. This one is about semantic caching — reusing the entire response for a semantically equivalent request. They attack different costs and compose well; Section 7 draws the line between them precisely. Every AWS and model fact below was verified against official AWS documentation; feature names, model IDs, and parameters are current as of writing.

1. Introduction

Most production LLM traffic is more repetitive than it looks. A customer-support assistant answers the same two dozen "how do I…" questions thousands of times a day, phrased a thousand different ways. A documentation copilot fields the same conceptual questions across an entire engineering org. An agent that decomposes a task into sub-steps re-derives the same lookups on every run. In all of these, a large fraction of requests are not new questions — they are paraphrases of questions the system has already answered well.

Caching is the standard tool for repetition, but the classic cache — a key-value store keyed by the exact request — barely helps here, because natural language almost never repeats byte-for-byte. Two users asking the same thing will phrase it differently, reorder the words, add or drop punctuation, switch languages. A cache keyed on the literal string sees two different keys and misses both times.

A semantic cache changes the key. Instead of keying on the text, it keys on the meaning of the text, represented as an embedding vector. It stores each answered query as (embedding, response) and, for every new request, embeds the query and searches for a stored embedding that is close enough in vector space. If a sufficiently similar prior query exists, it returns that query's stored response without calling the model. If not, it calls the model, returns the answer, and writes the new (embedding, response) pair back for next time.

This article covers the design end to end:

Why exact-match caching fails for natural language, and what "close enough in vector space" replaces it with (Section 2).
The reference path of a semantic cache — embed, search, threshold, return-or-generate, write — with copy-paste code on Bedrock and OpenSearch (Section 3).
Choosing the vector store, contrasting OpenSearch (rich filtering and scale) with in-memory stores like MemoryDB and ElastiCache (lowest latency, native TTL) (Section 4).
Similarity threshold design — the core trade-off between hit rate and false-hit rate, and how to calibrate it for a domain (Section 5).
Invalidation, freshness, and scoping — TTLs, invalidating on source updates, and the responses you must never cache (Section 6).
Semantic caching versus prompt caching, and how to run both (Section 7).
Measuring effectiveness and diagnosing the three ways a semantic cache goes wrong (Sections 8 and 9).

Full retrieval-augmented generation, where the store holds source documents rather than prior answers, is a different pattern; see the Production RAG Architecture on Amazon Bedrock guide for that, and Section 2 for why the two are not the same thing. Choosing the vector store itself — index algorithms, capacity, operational trade-offs — is covered in depth in the companion article on Vector Database Selection on AWS; this article treats the store as a component and focuses on the caching logic around it. Terms such as embedding, k-NN, and cosine similarity are defined in the Amazon Bedrock Glossary.

2. Why Exact-Match Caching Fails for Natural Language

To see why a semantic cache is necessary, it helps to watch a conventional cache fail. The natural first attempt is a key-value store — Amazon DynamoDB, or a plain hash in a Redis- or Valkey-compatible store — keyed on the request string:

import hashlib

def cache_key(query: str) -> str:
    return hashlib.sha256(query.encode("utf-8")).hexdigest()

# "How do I reset my password?"            -> 3a7f...e91
# "how do i reset my password"             -> b204...77c   (different key!)
# "I forgot my password, how do I log in?" -> f18c...2ad   (different key!)

Every surface variation produces a different key, so the cache misses on all but a literal repeat. The obvious next move is normalization: lowercase the text, collapse whitespace, strip punctuation, maybe apply stemming or lemmatization so that "resetting" and "reset" fold together.

import re

def normalize(query: str) -> str:
    q = query.lower().strip()
    q = re.sub(r"[^\w\s]", "", q)   # drop punctuation
    q = re.sub(r"\s+", " ", q)      # collapse whitespace
    return q

# "How do I reset my password?"   -> "how do i reset my password"
# "how do i   reset my Password!" -> "how do i reset my password"  (now they match)

Normalization genuinely helps with trivial variation, and it is cheap, deterministic, and worth keeping as a first layer. But it operates on characters and word forms, and meaning does not live there. It cannot see that these three requests are the same question:

"How do I reset my password?"
"I forgot my password, how do I get back in?"
"パスワードを忘れました。ログインするにはどうすればいいですか？"

There is almost no lexical overlap between the first and the second, and none at all with the third. No amount of lowercasing, punctuation-stripping, or stemming will collapse them to one key, because the words are simply different. Conversely, normalization can over-collapse: "how to enable MFA" and "how to disable MFA" differ by one token that inverts the meaning, yet stemming leaves them nearly identical. Lexical matching has no model of meaning, so it fails in both directions — missing true matches and, at the extreme, risking false ones.

An embedding model is exactly a model of meaning. It maps text to a fixed-length vector such that texts with similar meaning land near each other in vector space, regardless of wording or language. The three password questions above map to three nearby vectors; "enable MFA" and "disable MFA" map to vectors that a good model keeps apart. A semantic cache therefore replaces the exact key with an approximate nearest-neighbor (k-NN) search over these vectors: instead of asking "is there a stored query with this exact key?", it asks "is there a stored query whose meaning vector is within a small distance of this one?" AWS states the distinction plainly in its own guidance: unlike a traditional cache that relies on exact string matches, a semantic cache retrieves data based on semantic similarity, comparing the vector embedding of each new query against cached vectors of prior queries (see References).

That single change — from string equality to vector proximity — is the whole idea. Everything else in this article is about implementing it correctly and, above all, about choosing how close "close enough" has to be.

3. Semantic Cache Architecture

The semantic cache sits in front of the model as a lookup with a fallback. Every request follows the same path; the only branch is whether the lookup finds a close-enough neighbor.

Semantic cache reference path with hit and miss branches

The steps are:

Embed the query. Convert the incoming request text to a vector with an embedding model.
Search the cache. Run a k-NN search over the stored query embeddings for the nearest neighbor.
Apply the threshold. If the nearest neighbor's similarity meets the threshold, it is a hit; otherwise a miss.
On a hit, return the neighbor's stored response. No model call.
On a miss, invoke the model, return its answer, and write the new (embedding, response) pair into the cache so the next equivalent query hits.

3.1 Generating the Query Embedding

On Amazon Bedrock, Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) is a natural default: it accepts up to 8,192 input tokens, supports English (with multilingual coverage of 100+ languages in preview), and lets you choose the output vector size (1,024 by default, or 512 or 256) and whether the output is unit-normalized. Normalization matters for a cache, because it makes cosine similarity and dot product equivalent and keeps the similarity scale stable across entries. For cross-lingual caches — where "the same question in another language" should hit — Cohere Embed Multilingual (cohere.embed-multilingual-v3, 1,024 dimensions, 100+ languages) is a strong alternative, and Cohere's newer Embed v4 is a unified multimodal option; consult the Bedrock model catalog for the current lineup rather than hard-coding a choice.

import json
import boto3

REGION = "us-east-1"
EMBED_MODEL_ID = "amazon.titan-embed-text-v2:0"   # 1024-dim, 100+ languages, normalize option
EMBED_DIM = 1024

bedrock = boto3.client("bedrock-runtime", region_name=REGION)

def embed(text: str) -> list[float]:
    """Return a unit-normalized embedding for `text`."""
    body = json.dumps({
        "inputText": text,
        "dimensions": EMBED_DIM,
        "normalize": True,
    })
    resp = bedrock.invoke_model(modelId=EMBED_MODEL_ID, body=body)
    return json.loads(resp["body"].read())["embedding"]

Two operational notes. First, embedding models on Bedrock are throttled by requests per minute (RPM), not tokens per minute — when you plan capacity for a cache that embeds every incoming query, size the RPM quota, not a token budget. Second, the embedding call is now on the critical path of every request, including hits; keep it fast (it is a single, small, latency-optimized invocation) and never swap the embedding model or output dimension under a live cache, because existing vectors were written in the old space and will no longer be comparable.

3.2 Storing and Searching Vectors in OpenSearch

Amazon OpenSearch Service provides k-NN search backed by the FAISS and Lucene libraries (a third engine, NMSLIB, has been deprecated in favor of the other two); FAISS and Lucene both implement HNSW (Hierarchical Navigable Small World) graphs, and FAISS additionally offers IVF. A cache index needs the query embedding as a knn_vector field, the stored response, and a few fields for filtering and freshness (tenant, source version, timestamp):

PUT semantic-cache
{
  "settings": { "index": { "knn": true } },
  "mappings": {
    "properties": {
      "query_text":     { "type": "text" },
      "response":       { "type": "text", "index": false },
      "tenant":         { "type": "keyword" },
      "source_version": { "type": "keyword" },
      "created_at":     { "type": "date" },
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "space_type": "cosinesimil",
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "parameters": { "ef_construction": 256, "m": 16 }
        }
      }
    }
  }
}

space_type is the distance function (cosinesimil, l2, or innerproduct); with unit-normalized embeddings, cosine similarity is the natural choice. The HNSW parameters m and ef_construction govern the recall/latency/memory trade-off of the graph and are covered in depth in the Vector Database Selection on AWS article; the defaults above are reasonable for a cache.

The lookup is where the threshold enters. OpenSearch supports radial search via min_score on the knn query: rather than "give me the k nearest neighbors" (which always returns something, however far away), it returns only neighbors whose score is at or above a floor. That floor is the similarity threshold, and a metadata filter scopes the search to the right tenant in the same query:

GET semantic-cache/_search
{
  "size": 1,
  "query": {
    "knn": {
      "embedding": {
        "vector": [ /* query embedding */ ],
        "min_score": 0.95,
        "filter": { "term": { "tenant": "acme" } }
      }
    }
  }
}

An empty hits array means no stored query was close enough — a clean, unambiguous cache miss. The exact numeric scale of _score depends on the configured space_type, so 0.95 is not a universal constant; it is a value you calibrate per embedding model and space, which is the subject of Section 5.

Wiring the two calls together, the cache is a small module:

from opensearchpy import OpenSearch

SIMILARITY_THRESHOLD = 0.95          # calibrated per model + space (see Section 5)
GENERATION_MODEL_ID = "us.anthropic.claude-sonnet-5"  # e.g. a Claude inference profile; use the exact current ID from the Bedrock catalog

os_client = OpenSearch(...)          # domain endpoint + SigV4 auth

def lookup(query: str, tenant: str) -> str | None:
    vec = embed(query)
    resp = os_client.search(index="semantic-cache", body={
        "size": 1,
        "query": {"knn": {"embedding": {
            "vector": vec,
            "min_score": SIMILARITY_THRESHOLD,
            "filter": {"term": {"tenant": tenant}},
        }}},
    })
    hits = resp["hits"]["hits"]
    return hits[0]["_source"]["response"] if hits else None

def store(query: str, tenant: str, response: str, source_version: str) -> None:
    os_client.index(index="semantic-cache", body={
        "query_text": query,
        "response": response,
        "tenant": tenant,
        "source_version": source_version,
        "created_at": "now",                 # set at index time
        "embedding": embed(query),
    })

def answer(query: str, tenant: str, source_version: str) -> str:
    cached = lookup(query, tenant)
    if cached is not None:
        return cached                        # cache hit: no model call
    generated = invoke_llm(query)            # cache miss: call the model
    store(query, tenant, generated, source_version)
    return generated

invoke_llm is any Bedrock generation call — for example, a Converse request to your chosen model. It is deliberately the least interesting part of the design: the entire point of the cache is to call it as rarely as correctness allows. Note that store embeds the query a second time; in production you would reuse the vector already computed in lookup rather than pay for a second embedding.

3.3 The Same Path in an In-Memory Store

The identical logic runs on Amazon MemoryDB, whose vector search AWS explicitly documents as a way to use the database as a buffer memory for the foundation model — returning previously answered questions from the buffer instead of going through retrieval and generation again. You create an index with FT.CREATE, declaring the vector field's algorithm (HNSW or FLAT), type, dimension, and distance metric:

FT.CREATE semantic-cache
  ON HASH
  PREFIX 1 "cache:"
  SCHEMA
    tenant         TAG
    source_version TAG
    response       TEXT
    embedding      VECTOR HNSW 10 TYPE FLOAT32 DIM 1024 DISTANCE_METRIC COSINE M 16 EF_CONSTRUCTION 256

A cache entry is then a hash written with HSET cache:<id> response "..." tenant "acme" embedding <float32-bytes>, and the lookup is a k-NN query expression that pre-filters by tenant and returns the nearest neighbor plus its distance score:

FT.SEARCH semantic-cache
  "(@tenant:{acme})=>[KNN 1 @embedding $query_vec AS score]"
  PARAMS 2 query_vec "<float32-bytes>"
  RETURN 2 response score
  DIALECT 2

Here score is the configured distance (for COSINE, 0 means identical), so you convert it to a similarity and compare against the calibrated threshold in the client — the same decision as OpenSearch's min_score, applied one layer up. The in-memory path buys single-digit-millisecond lookups and native per-key TTL (Section 6); it gives up some of OpenSearch's richer query surface. Section 4 makes that trade-off explicit.

4. Choosing the Vector Store for a Cache

Any vector store can back a semantic cache, but two families dominate on AWS, and they optimize for different things. This section frames the choice specifically for the cache workload; the general vector-database decision — index internals, capacity planning, cost model — is the subject of the dedicated Vector Database Selection on AWS guide, to which the full analysis is delegated.

The cache workload has a distinctive shape: read-heavy (a lookup on every request), latency-critical (the lookup is on the hot path, and its whole justification is being faster than the model), continuously written (every miss adds an entry), and needing per-entry expiry and tenant isolation. Against that shape:
* You can sort the table by clicking on the column name.

Concern	Amazon OpenSearch Service	Amazon MemoryDB / ElastiCache
Lookup latency	Low (milliseconds), disk- and memory-backed	Lowest (single-digit ms / microseconds), fully in memory
Index algorithms	HNSW (FAISS/Lucene) and IVF (FAISS)	HNSW and FLAT
Filtering	Rich: full query DSL, pre- and post-filtering, hybrid lexical + vector	Tag and numeric filters in the query expression
Per-entry TTL	No native per-document TTL; use a timestamp filter plus an ISM policy or scheduled delete	Native per-key TTL and eviction policies
Best fit	Large caches, complex scoping, hybrid retrieval, when the store is shared with search/RAG	Lowest-latency caches, native expiry, when the store is shared with session/state caching

As a rule of thumb: reach for OpenSearch when the cache is large, when scoping is complex (many tenants, metadata dimensions, or a wish to combine vector similarity with lexical filters), or when you are already running OpenSearch for search or RAG and want one store. Reach for an in-memory store when the lookup latency budget is tight, when you want native per-key TTL and eviction to manage cache size for you, or when you already run MemoryDB or ElastiCache for session and application caching. MemoryDB and ElastiCache both expose vector search over a Valkey/Redis-compatible API; MemoryDB adds Multi-AZ durability, while ElastiCache for Valkey is AWS's most explicitly documented semantic-caching path — its guidance calls out configurable similarity thresholds, tag and numeric metadata filters, per-key TTLs, and real-time index updates as first-class cache concerns.

The deeper point is that this is not a decision to make from latency numbers alone; it depends on what else the store does in your architecture and on the operational model you want. Make it with the full picture in the vector-database selection guide, and treat the cache logic in this article as portable across whichever store you land on.

5. Similarity Threshold Design

This is the section the whole design turns on. The similarity threshold is the one parameter that decides, for every request, whether "close enough" is close enough — and it is a genuine trade-off with no free setting.

Calibrating the similarity threshold: trading off false-hit rate against hit rate

Move the threshold in either direction and you get a real cost:

Set it too low (loose). More requests clear the bar, so the hit rate rises — but so does the false-hit rate, the fraction of hits where the stored answer does not actually match the new question. A false hit is not a slow response or a blank; it is a confidently wrong response, served instantly, that looks exactly like a good one. "How do I enable MFA?" collides with a stored answer for "How do I disable MFA?" and the user is told to do the opposite of what they asked.
Set it too high (strict). Almost nothing clears the bar, so false hits vanish — but the hit rate collapses toward zero, and the cache stops earning its keep because nearly every request falls through to the model anyway.

The right threshold is the highest value at which you still capture genuine paraphrases, chosen so the residual false-hit rate is acceptable for your domain's tolerance for a wrong answer. That tolerance is not universal, which is why there is no universal number.

5.1 There Is No Universal Threshold

A threshold value is meaningful only relative to three things, and copying a number from a blog post that fixed all three differently is how false hits get shipped:

The embedding model. Different models place paraphrases at different distances; a threshold tuned for Titan Text Embeddings V2 does not transfer to Cohere Embed.
The distance space. Cosine, dot product, and Euclidean produce different score scales, and OpenSearch's _score is a further transformation of the raw distance. The same "closeness" is a different number in each.
The domain. In a low-stakes FAQ, a rare false hit is a mild annoyance and a higher hit rate is worth it. In a medical, legal, or financial assistant, a single confidently-wrong answer can be unacceptable, and the threshold should be conservative even at the cost of hit rate.

So a threshold is not a constant you look up; it is a value you measure for your specific model, space, and risk tolerance.

5.2 Calibrating with a Labeled Probe Set

Calibration is an offline procedure, and it is worth doing before the cache ever serves traffic. Build a small labeled probe set of query pairs from your actual domain:

Should-hit pairs — genuine paraphrases that mean the same thing and should return the same cached answer ("reset my password" / "I forgot my login").
Should-miss pairs — questions that are lexically close or topically adjacent but mean different things and must not share an answer ("enable MFA" / "disable MFA"; "cancel my subscription" / "change my subscription").

Then sweep the threshold and, at each value, measure two rates over the probe set:

def evaluate(threshold, should_hit_pairs, should_miss_pairs):
    # hit rate: of the true paraphrases, how many clear the bar (higher is better)
    true_hits = sum(similarity(a, b) >= threshold for a, b in should_hit_pairs)
    hit_rate = true_hits / len(should_hit_pairs)

    # false-hit rate: of the distinct questions, how many wrongly clear the bar (must be near zero)
    false_hits = sum(similarity(a, b) >= threshold for a, b in should_miss_pairs)
    false_hit_rate = false_hits / len(should_miss_pairs)

    return hit_rate, false_hit_rate

# similarity(a, b) = cosine of embed(a) and embed(b)
for t in [round(0.80 + 0.01 * i, 3) for i in range(20)]:
    print(t, *evaluate(t, should_hit_pairs, should_miss_pairs))

As the threshold climbs, both rates fall — the question is which falls faster. Pick the threshold at the lowest false-hit rate your domain requires (often effectively zero for high-stakes domains), and accept whatever hit rate that yields. It is the false-hit rate, not the hit rate, that is the safety constraint; the hit rate is only the efficiency you get to keep after the safety constraint is satisfied. If the resulting hit rate is disappointing, the fix is a better embedding model for your domain or tighter query scoping — not lowering the threshold past your risk tolerance.

Two refinements once the basics work. First, calibration is not one-and-done: query distributions drift, and a threshold that was safe last quarter should be re-checked, which is why Section 8 puts false-hit rate on a live dashboard. Second, for the highest-stakes tier of traffic you can add a cheap verification step on a hit — a lightweight check that the stored answer is still on-topic for the new query — trading a little latency back for extra safety. Neither changes the core discipline: measure the false-hit rate, set the threshold from it, and never treat a copied number as calibrated.

6. Invalidation, Freshness, and Scoping

A cache that never forgets is a liability. Stored answers go stale when the underlying facts change, and a semantic cache adds a second hazard on top of ordinary staleness: because a hit reuses a whole prior response, an entry that should never have been cached — or should have been scoped to one user — can be served to the wrong request entirely. Three mechanisms keep a cache correct: expiry, source-driven invalidation, and scoping.

6.1 Time-Based Expiry (TTL)

The simplest freshness control is a time-to-live: entries expire after a fixed age, so even in the absence of any explicit invalidation the cache cannot serve an answer older than the TTL. How you implement it differs by store, and this is a concrete reason the store choice in Section 4 matters:

In-memory stores (MemoryDB, ElastiCache) offer native per-key TTL. Set an expiry on each cache key at write time (EXPIRE cache:<id> 86400), and the store evicts it automatically. TTL plus a configurable eviction policy also caps total cache size for free.
OpenSearch has no per-document TTL. Enforce freshness by writing a created_at timestamp on every entry and adding a range filter to the lookup so stale documents are never returned, then reclaim their storage with an Index State Management (ISM) policy or a scheduled delete_by_query. The query-time filter is what guarantees correctness; the deletion job is housekeeping.

Choose the TTL from how fast the underlying answers change: hours for volatile operational data, days or weeks for stable how-to content.

6.2 Invalidating on Source Updates

TTL bounds staleness but does not eliminate it — an answer can go wrong the moment its source changes, well before the TTL expires. When a cached response is derived from a knowledge source (a documentation set, a policy database, a RAG corpus), a change to that source should invalidate the answers built from it. The mechanism is to tag each entry with the version of the source it was generated from — the source_version field in the schemas above — and, when the source updates, purge every entry carrying the old version:

# after re-indexing the knowledge source to a new version
os_client.delete_by_query(index="semantic-cache", body={
    "query": {"term": {"source_version": "2026-07-01"}}
})

This keeps the cache coherent with its source without flushing everything: entries generated from the still-current version survive, only the affected ones are evicted, and the next request for an affected question falls through to the model and is re-cached against the new version. For finer granularity, tag entries with the specific source document IDs they drew on and invalidate only the entries touching a changed document.

6.3 Scoping: The Responses You Must Not Cache

The most important rule in this section is a prohibition: do not cache a response whose correct value depends on who is asking. A semantic cache keys on the meaning of the question, not on the identity, permissions, or context of the asker. Two users can ask a semantically identical question and legitimately deserve different answers:

"What is my account balance?" — same question, different answer per user.
"What are my open tickets?" — same question, different answer per user.
"Am I allowed to access this dataset?" — same question, entitlement-dependent answer.

Cache any of these globally and a false hit becomes a data leak: one user's balance, tickets, or entitlements served to another. This is the concrete failure mode behind the honesty caveat at the top of the article — the risk a semantic cache trades away speed for is not merely a slightly-off answer, it is potentially one user's private answer returned to someone else.

Two defenses, applied together:

Never cache personalized, entitlement-gated, or PII-bearing responses at all. Route those requests straight to the model and mark them non-cacheable. Only responses that are correct independent of the asker — general how-to, product facts, public documentation — belong in a shared cache.
Partition the cache by boundary. For anything cached, isolate by tenant (and, where relevant, by user or role) using a metadata filter on every lookup (the tenant filter in Section 3), a key prefix in an in-memory store, or a separate index per tenant. Partitioning ensures a lookup can only ever hit an entry from the same boundary, so even a loose threshold cannot cross tenants.

Because safety failures here read as data-protection incidents rather than mere cache misses, treat the "is this response asker-dependent?" question as a design-time gate on every response type, not a runtime afterthought. When in doubt, do not cache.

7. Semantic Caching vs Prompt Caching

Semantic caching is easy to confuse with prompt caching, and the two are often discussed together because both reduce the cost of LLM calls. They operate at completely different layers, and understanding the difference is what lets you use both at once.
* You can sort the table by clicking on the column name.

	Prompt caching	Semantic caching
What it reuses	The token processing of a repeated prompt prefix	The entire response to an equivalent query
Match rule	Byte-identical prefix (exact)	Embedding similarity above a threshold (approximate)
What it saves	Re-processing input tokens on a call that still happens	The model call itself — no invocation on a hit
Where it lives	Inside the model platform (Bedrock / Claude API)	In your own vector store, in front of the model
Failure mode	A cache miss — full but correct processing	A false hit — a fast but possibly wrong answer

Prompt caching, on Amazon Bedrock, marks a static prefix of a prompt with cache checkpoints so that a later request with the same prefix skips re-processing those tokens. It is a prefix match — any change before the checkpoint invalidates it — with a per-model minimum token count per checkpoint and a short time-to-live that resets on each hit, available through the Converse and InvokeModel APIs. It always still makes the call; it just makes the input side of that call cheaper. The mechanics, breakpoint placement, and economics are covered in the sibling article on Anthropic Claude API Prompt Caching and Token Efficiency, to which the details are delegated — note that the sibling article covers the direct Anthropic Claude API surface (cache_control breakpoints and TTLs), and Bedrock's Converse/InvokeModel cache-checkpoint specifics such as per-model minimum token counts differ in detail.

Semantic caching operates one level up: on a hit, there is no call at all. That is why its failure mode is more dangerous — prompt caching can only ever be correct-but-full-price on a miss, whereas semantic caching can be fast-but-wrong on a false hit — and why the threshold discipline of Section 5 has no analogue in prompt caching.

Because they act at different layers, the two compose, and the best designs run both:

A request arrives; the semantic cache checks for an equivalent prior answer. On a hit, return it — no model call, no tokens.
On a miss, the request goes to the model. That call carries a large, stable system prefix (instructions, tools, retrieved context), and prompt caching reuses the token processing of that prefix across every miss.
The response is returned and written back into the semantic cache for next time.

The semantic cache eliminates the repeated calls; prompt caching makes the remaining calls cheaper on the input side. One reduces call volume, the other reduces per-call input cost, and they stack without interfering. Where each of those remaining calls should run — which model, which region, which inference profile — is an orthogonal decision covered in Amazon Bedrock Inference Throughput and Latency Optimization.

8. Measuring Effectiveness

A semantic cache is only as good as the metrics you keep on it, and one of those metrics is a safety metric, not an efficiency one. Instrument four quantities from day one; the fourth is the one most teams forget until it hurts.

Hit rate — the fraction of requests served from the cache. This is the headline efficiency number: it is the share of traffic that avoided a model call. A very low hit rate means the cache is not earning its place (threshold too strict, query diversity too high, or cache too cold); a suspiciously high one is worth double-checking against the false-hit rate before celebrating.
False-hit rate — the fraction of hits whose stored answer does not actually match the request. This is the safety metric, and the whole design exists to keep it inside your domain's tolerance. You cannot read it directly from traffic the way you read hit rate; sample hits and evaluate them offline (human review for a high-stakes system, or an automated relevance check), and re-run the calibration probe set of Section 5 periodically as query distributions drift.
Latency — compare the p50/p95 of hit-path responses (embed + search) against miss-path responses (embed + search + model call). The gap is the latency the cache buys; if the hit path is not dramatically faster than the miss path, the embedding or search step is too slow and the cache is not delivering its main benefit.
Cache size and growth — track entry count and store size over time. Unbounded growth signals missing TTL or eviction, or that the cache is storing near-unique queries that will never be hit again (Section 9).

Emit these as custom metrics — for example via CloudWatch Embedded Metric Format (EMF) from the function that fronts the cache — recording, per request, whether it was a hit or a miss and the similarity score of the nearest neighbor. The distribution of similarity scores is itself diagnostic: a pile-up of scores just below the threshold means genuine paraphrases are being missed by a hair (consider a small threshold reduction within your safety tolerance), while a pile-up just above it is where false hits are most likely to hide (sample those aggressively). Full distributed tracing and dashboarding of a Bedrock application is its own topic and beyond this article's scope; here the point is narrower — a semantic cache without a live false-hit signal is operating blind on exactly the risk that matters most.

9. Diagnostics

Three symptoms cover almost every semantic-cache problem in production. Each has a small set of likely causes and a direct remedy.
* You can sort the table by clicking on the column name.

Symptom	Likely cause	Remedy
Low hit rate	Threshold set too strict	Re-run calibration (Section 5); lower the threshold only within the false-hit tolerance
	High genuine query diversity	Expected — not every workload is cacheable; measure the cacheable share before investing
	Cold cache	Hit rate ramps as the cache fills; pre-warm with known frequent queries
	Embedding model weak for the domain	Paraphrases land far apart; evaluate a stronger or domain-appropriate embedding model
False hits	Threshold set too loose	Raise it; the false-hit rate is the binding constraint, not the hit rate
	Poor embedding separation for near-opposites	"Enable/disable" style collisions; use a better model, or add a verification step on hits
	Missing tenant/user scoping	A cross-boundary hit is a data leak; enforce partitioning and never cache asker-dependent responses (Section 6.3)
Cache bloat	No TTL or eviction	Set per-key TTL (in-memory) or a timestamp filter plus ISM/delete job (OpenSearch)
	Caching near-unique queries	Entries that will never be hit again; only write responses likely to recur
	Storing oversized payloads	Cache the response, not the full context/history that produced it

The through-line: low hit rate is an efficiency problem you can afford to tune slowly; false hits are a correctness problem you fix immediately. When the two pull against each other — a higher threshold cuts false hits but also hit rate — resolve it in favor of correctness every time, and recover hit rate by improving the embedding model or scoping rather than by loosening the threshold past your tolerance.

10. Frequently Asked Questions

What is semantic caching, and how is it different from a normal cache?

A normal cache keys on the exact request and returns a stored value only on a byte-identical match. A semantic cache keys on the meaning of the request, represented as an embedding vector, and returns a stored response when a prior query's vector is within a calibrated distance — so it can serve "I forgot my password" from an entry created for "how do I reset my password," which an exact-match cache would miss. The trade-off is that the match is approximate: a threshold that is too loose can return a stored answer for a question that only seems similar.

Is semantic caching the same as RAG?

No. Retrieval-augmented generation retrieves source documents to ground a new generation — the model still runs on every request. A semantic cache retrieves a prior answer keyed by query similarity and, on a hit, skips generation entirely. They use the same underlying vector-search machinery and compose well (a cache in front of a RAG pipeline can skip the whole retrieve-and-generate cycle for repeated questions), but they solve different problems. See the Production RAG Architecture on Amazon Bedrock guide for RAG.

How is this different from Amazon Bedrock prompt caching?

Prompt caching reuses the token processing of a byte-identical prompt prefix; the model call still happens, just cheaper on the input side. Semantic caching reuses the entire response for an equivalent query; on a hit, there is no model call at all. Prompt caching's worst case is a full-price-but-correct miss; semantic caching's worst case is a fast-but-wrong false hit. They operate at different layers and are best used together (Section 7).

What similarity threshold should I start with?

There is no universal number — a threshold is only meaningful relative to the embedding model, the distance space, and your domain's tolerance for a wrong answer. Do not copy a value from an article that fixed those differently. Build a small labeled probe set of should-hit paraphrases and should-miss near-misses, sweep the threshold, and pick the value that keeps the false-hit rate inside your tolerance (Section 5). Err toward strict for high-stakes domains.

Can I cache responses that contain user-specific data?

No. A semantic cache keys on the question, not the asker, so caching an asker-dependent response (account balances, personal tickets, entitlement-gated data, anything with PII) turns a false hit into a data leak — one user's answer served to another. Route such requests straight to the model, mark them non-cacheable, and partition anything you do cache by tenant and, where relevant, by user (Section 6.3).

Should I use OpenSearch or an in-memory store like MemoryDB?

Both work; they optimize differently. OpenSearch fits large caches, complex scoping, and hybrid lexical-plus-vector filtering, especially when you already run it for search or RAG. In-memory stores (MemoryDB, ElastiCache) give the lowest latency and native per-key TTL, and fit when the latency budget is tight or you already run them for session caching. The full decision — index internals, capacity, cost model — is delegated to the Vector Database Selection on AWS guide.

11. Summary

A semantic cache is a lookup-with-fallback in front of the model: embed the incoming query, run a k-NN search over the embeddings of previously answered queries, and — if the nearest neighbor clears a similarity threshold — return its stored response instead of calling the model. It works where exact-match caching cannot, because natural language repeats in meaning far more often than in characters, and an embedding model is precisely a model of meaning.

The engineering reduces to a handful of decisions:

Replace the key, not the cache. Key on the embedding and search by vector proximity; keep cheap normalization as a first layer but do not expect it to catch paraphrase or cross-language equivalence (Sections 2, 3).
The threshold is the design. It trades hit rate against false-hit rate, and there is no universal value — calibrate it for your embedding model, distance space, and domain risk with a labeled probe set, and treat the false-hit rate as the binding constraint (Section 5).
Keep the cache correct over time. Use TTL for baseline freshness, source-version tags to invalidate on updates, and hard scoping so asker-dependent responses are never cached and everything cached is partitioned by tenant (Section 6).
Pick the store for the workload. OpenSearch for scale and rich filtering, in-memory stores for lowest latency and native TTL — with the full comparison delegated to the vector-database selection guide (Section 4).
Measure the safety metric, not just the efficiency one. Track hit rate, latency, and size, but put the false-hit rate on a live signal, because it is the risk the cache trades speed for (Section 8).
Compose with prompt caching. The semantic cache removes repeated calls; prompt caching makes the remaining calls cheaper on the input side; they stack (Section 7).

The honest framing is the one to keep: a semantic cache does not simply make an application faster and cheaper for free. It makes it faster and cheaper in exchange for a bounded risk of returning the wrong answer — a risk you control with the threshold, contain with scoping, and watch with the false-hit rate. Build it with those three in place and the trade is a good one; skip any of them and the cache can quietly serve confidently wrong answers, which is a far worse outcome than a slow correct one. For the adjacent pieces, see the sibling guide on prompt caching and token efficiency, the vector database selection guide for the store, and the RAG architecture guide for the retrieval pattern this is often confused with.

12. References

AWS official documentation and blogs

hidekazu-konishi.com