Semantic Caching: Slashing LLM Latency and Costs with Redis

Building and scaling Large Language Model (LLM) applications brings two immediate challenges to the forefront: cost and latency. While a single prompt might cost fractions of a cent, at scale, these costs compound rapidly. Furthermore, waiting several seconds for a model to generate a response—even with streaming—can degrade the user experience significantly.

As engineers, we usually reach for caching to solve these problems. However, traditional exact-match caching is remarkably ineffective for natural language. If a user asks "How do I reset my password?" and another asks "What is the process for a password reset?", a standard key-value store like Redis would treat these as two distinct misses. This is where semantic caching enters the picture.

The Problem with Traditional Caching in the LLM Era

In a typical web application, we cache data based on deterministic keys—a user ID, a product SKU, or a specific URL. The input is exact, and the output is predictable.

LLMs operate in a non-deterministic, high-dimensional space. Users rarely ask the same question in the exact same way. Small variations in punctuation, casing, or word choice change the string hash, rendering a traditional cache useless. Yet, from a business logic perspective, the intent and the required response are identical.

To solve this, we need a cache that understands meaning rather than just characters. We need to move from string matching to vector similarity.

Anatomy of a Semantic Cache

Semantic caching replaces the exact string key with a vector embedding. An embedding is a numerical representation of the text's meaning, typically a high-dimensional vector.

Here is the high-level workflow of a semantic cache request:

1. Input: The user sends a natural language query.
2. Embedding: We convert that query into a vector using an embedding model (like OpenAI’s text-embedding-3-small or an open-source model via HuggingFace).
3. Vector Search: We query a vector database (Redis) to find previously cached embeddings that are "close" to the new query's vector.
4. Similarity Check: If the closest match exceeds a predefined similarity threshold (e.g., 0.95 cosine similarity), we return the cached response.
5. LLM Execution (on miss): If no match is found, we call the LLM, return the result to the user, and store the new query-response pair in the cache.

Why Redis for Semantic Caching?

While there are many vector databases available, Redis is uniquely positioned for semantic caching for several reasons:

• Speed: Redis operates in-memory, providing the sub-millisecond latency required for a caching layer.
• Integrated Vector Search: With the RediSearch module (available in Redis Stack and Redis Cloud), Redis supports vector indexing and K-Nearest Neighbor (KNN) search.
• Existing Infrastructure: Most enterprise stacks already use Redis for session management or traditional caching, reducing the overhead of introducing a new tool.
• TTL and Eviction: Redis handles data expiration natively, which is critical for maintaining a fresh cache.

Implementing Semantic Caching: A Practical Guide

Let’s look at how to implement this using Python and the redisvl (Redis Vector Library) client, which simplifies the interaction with Redis as a vector store.

1. Defining the Schema

First, we need to define how our cache will be structured. We need a field for the original prompt, the generated response, and the vector embedding of the prompt.

from redisvl.schema import IndexSchema

schema = IndexSchema.from_dict({
    "index": {
        "name": "llm_cache",
        "prefix": "cache_entry",
    },
    "fields": [
        {"name": "prompt", "type": "text"},
        {"name": "response", "type": "text"},
        {
            "name": "prompt_embeddings",
            "type": "vector",
            "attrs": {
                "dims": 1536, # For OpenAI embeddings
                "algorithm": "hnsw",
                "distance_metric": "cosine"
            }
        }
    ]
})

2. The Search and Store Logic

The core logic involves calculating the embedding of the incoming query and performing a range query or a KNN search in Redis.

import openai
from redisvl.index import SearchIndex

index = SearchIndex(schema, redis_url="redis://localhost:6379")
index.create(overwrite=True)

def get_embedding(text):
    response = openai.Embedding.create(input=text, model="text-embedding-3-small")
    return response['data'][0]['embedding']

def query_cache(user_query, threshold=0.92):
    # 1. Embed the query
    query_embedding = get_embedding(user_query)
    
    # 2. Search Redis
    # We look for the single most similar entry
    results = index.search(query_embedding, k=1)
    
    if results and results[0].vector_distance < (1 - threshold):
        return results[0].response
    
    return None

3. The Similarity Threshold: The Golden Variable

The threshold parameter is the most critical part of your semantic cache.

• Too high (0.99): You will experience many cache misses for queries that are essentially the same, reducing the cost-saving benefits.
• Too low (0.80): You risk returning a cached response that doesn't actually answer the user's specific question (a "false hit").

In practice, a threshold between 0.90 and 0.95 is usually the sweet spot for general-purpose Q&A, but this should be tuned based on your specific domain and the sensitivity of the information.

Advanced Considerations for Production

Handling Dynamic Data and TTLs

One of the biggest risks in caching is staleness. If your LLM provides information about real-time stock prices or system statuses, a semantic cache could be dangerous.

Use Redis's built-in TTL (Time-To-Live) to expire entries. For semantic caches, you can also implement "Metadata Filtering." For example, if your cache stores documentation for a software product, include a version tag in your schema. When searching the cache, filter for the current version to ensure users don't get outdated instructions.

Privacy and PII

Caching LLM responses means storing potentially sensitive data. If User A asks about their specific account balance, you absolutely do not want that response cached and served to User B.

Solution: Include a user_id or tenant_id field in your vector index schema and use it as a pre-filter in your search query. This ensures that a user only hits the cache for their own previous interactions.

The Cost of Embedding

It is important to remember that generating an embedding is not free. However, the cost difference is massive. As of mid-2024, an OpenAI embedding call for a typical sentence is roughly 100x to 1000x cheaper than a GPT-4o completion call. Furthermore, embedding models are significantly faster, with latencies often under 50ms, compared to the seconds required for LLM generation.

Quantifying the Impact

Let’s look at a hypothetical scenario for a customer support bot receiving 100,000 queries per month.

• Without Caching: 100,000 LLM calls. At an average cost of $0.01 per call, that's $1,000/month. Average latency: 2.5 seconds.
• With 30% Semantic Cache Hit Rate: 70,000 LLM calls + 100,000 embedding calls.
  • LLM Cost: $700
  • Embedding Cost: ~$5
  • Total Cost: $705 (30% savings)
  • Average Latency: (0.3 * 0.1s) + (0.7 * 2.5s) = 1.78 seconds (28% improvement)

In high-volume environments, these savings represent significant budget that can be reallocated to improving the model or increasing throughput.

Conclusion

Semantic caching with Redis is no longer an experimental optimization; it is a requirement for building production-grade LLM applications that are both economically viable and performant. By shifting from exact string matching to vector similarity, we can capitalize on the inherent redundancy in human language.

Actionable Next Steps:

1. Audit your logs: Identify the percentage of redundant or highly similar queries in your current LLM traffic.
2. Prototype with RedisVL: Set up a local Redis Stack instance and implement a basic similarity search using your existing embedding provider.
3. Start with a high threshold: Begin with a conservative threshold (0.95) and monitor for false hits before loosening it to capture more cache hits.