Scaling LLMs: Semantic Caching with Redis Vector Search

As Large Language Model (LLM) applications move from experimental notebooks to production environments, developers invariably hit two major roadblocks: high latency and unpredictable token costs. In a Retrieval-Augmented Generation (RAG) system, every user query triggers a chain of expensive operations—embedding generation, vector database retrieval, and several calls to a model like GPT-4.

When multiple users ask variations of the same question, recalculating the answer every time is a waste of resources. Traditional caching (exact string matching) fails here because natural language is fluid. If one user asks, "How do I reset my password?" and another asks, "What is the process for password resets?", a traditional cache misses.

This is where Semantic Caching comes in. By leveraging Redis and Vector Similarity Search, we can identify semantically equivalent queries and serve cached responses, reducing latency from seconds to milliseconds and cutting token consumption by up to 90%.

The Architecture of Semantic Caching

In a standard RAG workflow, the application flow looks like this:

1. User sends a query.
2. Application generates an embedding for the query.
3. Application searches a vector database for context.
4. Application sends context + query to the LLM.
5. LLM returns a response.

With a semantic cache, we insert a check between step 2 and step 3. Instead of going straight to the vector database and LLM, we check a high-speed cache (Redis) to see if we have already answered a similar question.

The Workflow with Semantic Caching:

1. User Query: "How do I change my login credentials?"
2. Embedding: Generate a vector (e.g., using text-embedding-3-small).
3. Vector Search: Query Redis to find vectors within a specific similarity threshold (e.g., Cosine Similarity > 0.96).
4. Cache Hit: If a match is found, return the cached text response immediately.
5. Cache Miss: If no match is found, proceed with the full RAG/LLM pipeline, then store the result and the query vector in Redis for future use.

Why Redis for Semantic Caching?

While there are many vector databases available, Redis is uniquely suited for semantic caching for several reasons:

• Performance: As an in-memory data store, Redis provides the sub-millisecond latency required for a cache layer to be effective.
• Integrated Vector Search: Redis Search and Query (formerly RediSearch) allows you to store vectors alongside metadata and perform K-Nearest Neighbor (KNN) searches.
• TTL and Persistence: You can set Time-To-Live (TTL) on cached items, ensuring that answers to time-sensitive questions (e.g., "What is the current stock price?") don't stay in the cache forever.
• Operational Simplicity: Most teams already use Redis for session management or traditional caching, reducing the overhead of adding a new tool to the stack.

Implementing the Cache: A Technical Deep Dive

To implement this, we need to define a schema in Redis that can handle both the vector data and the associated metadata (the original query and the LLM's response).

1. Setting up the Redis Index

Using the redis-py library, we can initialize an index designed for vector similarity search. We'll use HNSW (Hierarchical Navigable Small World) for the indexing algorithm because it offers a great balance between search speed and accuracy.

import redis
from redis.commands.search.field import VectorField, TextField
from redis.commands.search.index_definition import IndexDefinition, IndexType

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Define the schema
schema = (
    TextField("query"),
    TextField("response"),
    VectorField("embedding", "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": 1536,  # OpenAI embedding dimension
        "DISTANCE_METRIC": "COSINE"
    })
)

# Create the index
try:
    r.ft("idx:cache").create_index(
        fields=schema, 
        definition=IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH)
    )
except Exception as e:
    print("Index already exists or error:", e)

2. The Lookup Logic

When a query comes in, we convert it to an embedding and search the index. The key here is the distance threshold. If the distance is too high, we might serve an irrelevant answer (a "false hit"). If it's too low, we miss opportunities to save tokens.

import numpy as np
from redis.commands.search.query import Query

def get_cached_response(query_embedding, threshold=0.1):
    # Redis returns distance, so for COSINE, 0 is identical, 1 is orthogonal
    # We want distance < threshold
    
    # Construct the KNN query
    q = Query("*=>[KNN 1 @embedding $vec AS score]") \
        .sort_by("score") \
        .return_fields("query", "response", "score") \
        .dialect(2)
    
    params = {"vec": np.array(query_embedding, dtype=np.float32).tobytes()}
    results = r.ft("idx:cache").search(q, params).docs
    
    if results:
        score = float(results[0].score)
        if score <= threshold:
            return results[0].response
    
    return None

The Challenge of the Similarity Threshold

Choosing the right threshold is the most critical part of implementing semantic caching. It is a classic trade-off between Precision and Recall.

• High Threshold (Strict): You only serve a cached response if the queries are nearly identical. This ensures high accuracy but reduces the "hit rate" and the overall cost savings.
• Low Threshold (Lax): You serve cached responses even for vaguely related queries. This maximizes savings but risks the LLM providing an answer that doesn't actually address the user's specific nuance.

In production, we often start with a very conservative threshold (e.g., 0.05 for Cosine Distance) and monitor the results. Using a "feedback loop" where users can upvote/downvote responses can help you fine-tune this threshold over time.

Handling Dynamic Content and Stale Data

One risk of semantic caching is serving outdated information. If your RAG system is querying a database of product prices that change daily, a cached response from yesterday is a liability.

Strategies for Cache Invalidation:

1. Time-To-Live (TTL): Set an expiration time on your Redis keys based on the volatility of your data.
2. Explicit Invalidation: If the underlying data source for your RAG system is updated (e.g., a documentation page is edited), find and delete the associated cache entries. This is difficult with semantic search, so a common pattern is to flush the entire cache or a specific namespace when major updates occur.
3. Metadata Filtering: Store a timestamp or version ID in the Redis hash. When querying, filter the results to only include entries created after a certain date.

Security and Privacy Considerations

Caching LLM responses introduces security risks, particularly regarding Multi-tenancy and Personally Identifiable Information (PII).

Multi-tenancy: You must ensure that User A cannot see a cached response meant for User B if that response contains private data. To solve this, include a tenant_id or user_id in your Redis schema and include it as a filter in your vector search query.

# Filtering by tenant_id in Redis
q = Query("(@tenant_id:{123})=>[KNN 1 @embedding $vec AS score]") ...

PII Scrubbing: Before storing a query or response in the cache, run it through a PII detection layer (like Microsoft Presidio) to ensure sensitive data is not persisted in the cache.

Cost-Benefit Analysis: Is it Worth It?

Let's look at the numbers for a typical high-traffic RAG application using GPT-4o.

• Average RAG Cost per Query: $0.01 (Input tokens + Retrieval + Output tokens)
• Average Latency: 2.5 seconds
• Queries per Month: 100,000
• Total Monthly Cost: $1,000

If you implement semantic caching and achieve a 30% hit rate (common for customer support bots):

• Monthly Savings: $300
• Latency Improvement: 30,000 queries now return in < 50ms instead of 2,500ms.
• User Experience: Significant reduction in the perceived "slowness" of the AI.

For enterprise-scale applications with millions of queries, the savings scale linearly, often paying for the Redis infrastructure many times over within the first month.

Conclusion: Actionable Next Steps

Semantic caching is no longer an optional optimization for production-grade LLM applications; it is a requirement for operational efficiency. By shifting the burden from expensive inference to high-speed vector retrieval, you create a more resilient and cost-effective system.

To get started:

1. Audit your logs: Identify how often users ask similar questions. If the overlap is >15%, semantic caching will provide immediate ROI.
2. Prototype with Redis: Use the redis-py library to set up a basic HNSW index.
3. Benchmark your threshold: Run a set of known similar queries through your embedding model and calculate their distance to find your "Goldilocks" threshold.
4. Implement multi-tenancy: Ensure your cache logic respects user data boundaries from day one.

By treating LLM responses as a cacheable resource rather than a purely generative one, you bridge the gap between experimental AI and robust, scalable software engineering.