Optimizing LLM Cost and Latency with Redis Semantic Caching

If you have moved an LLM-powered application into production, you have likely encountered the two primary bottlenecks of the modern AI stack: high latency and unpredictable costs. While OpenAI, Anthropic, and Google are racing to make their models faster and cheaper, the reality remains that every token has a price, and every request involves a multi-second round trip to a remote API.

As engineers, we usually reach for a cache to solve these problems. However, traditional key-value caching is remarkably ineffective for Large Language Models. If a user asks "How do I reset my password?" and another asks "What is the process for a password reset?", a standard string-matching cache will fail. To the machine, these strings are different; to the human, they are identical in intent.

This is where Semantic Caching comes in. By leveraging vector embeddings and Redis, we can build a cache that understands the meaning behind a query, allowing us to serve cached responses for semantically similar questions, drastically reducing our dependency on external LLM providers.

The Architecture of a Semantic Cache

In a traditional caching layer, you hash the input string and use it as a key. In a semantic cache, we transform the input string into a high-dimensional vector (an embedding) that represents its meaning. We then store this vector in a database capable of performing vector similarity searches.

The Workflow

1. User Query: The user sends a prompt to the application.
2. Embedding Generation: The application converts the prompt into a vector using an embedding model (like text-embedding-3-small).
3. Vector Search: The application queries Redis to find any existing vectors that are "close" to the new query's vector.
4. Similarity Threshold: If a result is found with a similarity score above a predefined threshold (e.g., 0.95), we consider it a "cache hit" and return the stored response.
5. LLM Fallback: If no similar vector is found (a "cache miss"), we send the query to the LLM.
6. Cache Update: The new query, its vector, and the LLM response are stored in Redis for future use.

Why Redis for Semantic Caching?

While there are many vector databases on the market, Redis has become the preferred choice for semantic caching for several reasons:

1. Speed: As an in-memory data store, Redis offers sub-millisecond search latencies, which is critical when the goal is to bypass a 2-second LLM call.
2. RedisVL: The Redis Vector Library (RedisVL) provides a high-level Python client specifically designed for these use cases, simplifying the integration of vector search into existing stacks.
3. Versatility: Most teams already use Redis for session management or standard caching. Adding vector capabilities doesn't require a new piece of infrastructure.
4. Tiered Storage: Redis supports various indexing methods (FLAT, HNSW) that allow you to balance search accuracy with memory consumption.

Implementing the Solution

Let's walk through a practical implementation using Python and Redis. We will use the redisvl library to handle the heavy lifting of index management and similarity searching.

1. Setting Up the Schema

First, we need to define how our cache will be structured in Redis. We need a field for the original prompt, the generated response, and the vector embedding of the prompt.

from redisvl.schema import IndexSchema

schema = IndexSchema.from_dict({
    "index": {
        "name": "llm_cache",
        "prefix": "cache_entry",
    },
    "fields": [
        {"name": "prompt", "type": "text"},
        {"name": "response", "type": "text"},
        {
            "name": "prompt_embeddings",
            "type": "vector",
            "attrs": {
                "dims": 1536, # OpenAI embedding dimensions
                "distance_metric": "cosine",
                "algorithm": "flat",
                "datatype": "float32"
            }
        }
    ]
})

2. The Semantic Cache Logic

Now, we implement the logic that checks the cache before calling the LLM. We'll use a threshold to determine if a cached result is "good enough."

from redisvl.index import SearchIndex
from openai import OpenAI

client = OpenAI()
index = SearchIndex(schema, redis_url="redis://localhost:6379")
index.create(overwrite=True)

def get_llm_response(user_query, threshold=0.92):
    # 1. Generate embedding for the new query
    query_embedding = client.embeddings.create(
        input=user_query,
        model="text-embedding-3-small"
    ).data[0].embedding

    # 2. Search Redis for similar vectors
    results = index.query(
        vector=query_embedding,
        vector_field_name="prompt_embeddings",
        return_fields=["prompt", "response"],
        num_results=1
    )

    # 3. Check similarity
    if results and results[0]['vector_distance'] < (1 - threshold):
        print("--- Cache Hit ---")
        return results[0]['response']

    # 4. Cache Miss: Call LLM
    print("--- Cache Miss ---")
    response = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{"role": "user", "content": user_query}]
    ).choices[0].message.content

    # 5. Store in Redis for next time
    index.load([{
        "prompt": user_query,
        "response": response,
        "prompt_embeddings": query_embedding
    }])

    return response

Tuning the Similarity Threshold

One of the most critical aspects of semantic caching is the similarity threshold. If the threshold is too high (e.g., 0.99), you will rarely get a cache hit, defeating the purpose. If it's too low (e.g., 0.80), you risk "semantic drift," where the cache returns an answer to a question that is fundamentally different from what the user asked.

In my experience, the sweet spot usually lies between 0.90 and 0.95 for general conversational agents. However, this depends heavily on your embedding model. Models like text-embedding-3-small are highly sensitive; a difference of 0.05 in cosine similarity can represent a significant shift in intent.

Measuring Success

To optimize this, you should log your vector_distance values. If you see many cache hits at 0.98 that users are happy with, but hits at 0.88 are resulting in user corrections, you've found your boundary.

Advanced Strategies for Production

1. Metadata Filtering

Semantic caching can be dangerous if you are dealing with multi-tenant applications. You don't want User A to receive a cached response that contains User B's private data. To solve this, include a user_id or org_id in your Redis schema and use a filter in your vector search to ensure the cache is scoped correctly.

2. TTL (Time to Live)

LLM responses aren't evergreen. If your LLM provides information about a fast-moving product or real-time data, set a TTL on your Redis keys. A semantic cache hit for "What is the current stock price?" is useless if the cached response is three days old.

3. Prompt Normalization

Before generating an embedding, normalize your prompts. Strip out whitespace, convert to lowercase, and perhaps even use a cheaper LLM (like GPT-3.5) to "summarize" the intent of a very long prompt. This makes the embedding more representative of the core request.

The Economic Impact

Let's look at the numbers. Consider a production app handling 100,000 queries a day with an average prompt/response length of 500 tokens.

• Without Caching: 100,000 queries * 500 tokens = 50M tokens. At $10 per million tokens (GPT-4o average), that's $500/day.
• With 30% Semantic Cache Hit Rate: You save 15M tokens daily. That's $150/day saved, or roughly $4,500/month.

Beyond the dollar amount, the latency improvement is transformative. A cache hit returns in ~10ms, while the LLM takes ~2000ms. For those 30,000 users, the app feels instantaneous.

Potential Pitfalls

Despite the benefits, semantic caching isn't a silver bullet. You must be aware of:

• Stale Information: If your underlying data changes but your cache remains, the LLM will appear to "hallucinate" old data.
• Embedding Costs: While much cheaper than chat completions, generating embeddings still costs money and takes time. For very short, simple queries, the overhead might not be worth it.
• The "Refinement" Problem: If a user asks a question and then follows up with "Wait, I meant...", a semantic cache might mistakenly serve the response to the first question if it's too aggressive.

Actionable Conclusion

Implementing semantic caching with Redis is one of the highest-ROI optimizations you can make to an LLM-powered application. It directly addresses the two biggest pain points of AI development: cost and speed.

To get started:

1. Identify your repeat queries: Analyze your logs to see if users are asking similar questions.
2. Select an embedding model: Start with OpenAI's text-embedding-3-small for a balance of cost and performance.
3. Deploy Redis with Vector Search: Use Redis Cloud or the redis/redis-stack Docker image to get vector capabilities immediately.
4. Start with a high threshold: Set your similarity threshold to 0.95 and gradually lower it as you monitor the accuracy of the cached responses.

By moving the heavy lifting from the LLM to your data layer, you build a system that is not only cheaper and faster but also more resilient to API outages and rate limits.