Optimizing LLM Cost and Latency with Semantic Caching

Large Language Models (LLMs) have transitioned from experimental novelties to core infrastructure components. However, as any engineer who has moved an LLM-powered feature into production knows, the 'LLM tax' is real. You are hit with two primary challenges: unpredictable latency and compounding costs.

Every time a user asks a question like "How do I reset my password?", your system sends tokens to a provider (like OpenAI or Anthropic), waits several seconds for a response, and pays for the privilege. If a thousand users ask the same question phrased slightly differently, you pay a thousand times for the same computation. Traditional exact-match caching fails here because natural language is fluid. This is where semantic caching comes in.

The Limitations of Traditional Caching

In a standard web architecture, we use Redis to cache database queries or API responses using a unique key—usually a hash of the request parameters. If the input is exactly the same, we get a cache hit.

In the context of LLMs, consider these three queries:

1. "How do I change my password?"
2. "Password reset instructions."
3. "I forgot my password, what do I do?"

To a traditional cache, these are three distinct strings with three different hashes. To a human—and to an LLM—they represent the exact same intent. Traditional caching would result in three separate model calls, costing you money and making your users wait.

What is Semantic Caching?

Semantic caching identifies the intent of a query rather than its literal string representation. Instead of hashing the text, we transform the text into a vector embedding—a high-dimensional numerical representation of meaning.

By storing these embeddings in a vector database like Redis, we can perform a similarity search. When a new query comes in, we don't look for an exact match; we look for a "close enough" match. If a previous query was semantically similar and we have its response cached, we serve that response instantly.

The Performance Impact

• Latency: A typical LLM call takes 1–5 seconds. A Redis vector search takes 5–20 milliseconds.
• Cost: A cache hit costs effectively $0. An LLM call costs fractions of a cent, which scales significantly at volume.

The Architecture of a Semantic Cache

To implement this, you need three core components:

1. An Embedding Model: A model (like text-embedding-3-small or an open-source alternative like all-MiniLM-L6-v2) to convert text to vectors.
2. A Vector Store: Redis with the Search and Query capability (formerly RediSearch) is ideal here because of its speed and familiar operational profile.
3. A Similarity Threshold: A mathematical boundary that determines if two queries are "similar enough" to share a response.

The Workflow

1. Input: User sends a query.
2. Embedding: Convert the query into a vector.
3. Search: Query Redis for the nearest neighbor to this vector.
4. Evaluation:
   • If the distance is below your threshold (e.g., Cosine Distance < 0.1), return the cached result.
   • If no close match is found, call the LLM.
5. Write: Store the new query vector and the LLM response in Redis for future use.

Implementing with Redis

Redis is often chosen for this task because it allows you to store both the vector data and the metadata (the actual response text) in the same place, utilizing the HNSW (Hierarchical Navigable Small World) index for lightning-fast approximate nearest neighbor searches.

Here is a conceptual implementation using Python and the redis-py library:

import redis
from redis.commands.search.field import VectorField, TextField
from redis.commands.search.query import Query
import numpy as np

# Configuration
INDEX_NAME = "semantic_cache"
VECTOR_DIM = 1536 # Dimension for OpenAI embeddings
DISTANCE_THRESHOLD = 0.1 

client = redis.Redis(host='localhost', port=6379)

# Create the index if it doesn't exist
try:
    client.ft(INDEX_NAME).create_index((
        VectorField("embedding", "HNSW", {
            "TYPE": "FLOAT32", 
            "DIM": VECTOR_DIM, 
            "DISTANCE_METRIC": "COSINE"
        }),
        TextField("response")
    ))
except:
    pass # Index already exists

def get_cached_response(query_vector):
    # Prepare the query
    q = Query(f"*=>[KNN 1 @embedding $vec as score]")\
        .sort_by("score")\
        .return_fields("response", "score")\
        .dialect(2)
    
    query_params = {"vec": np.array(query_vector).astype(np.float32).tobytes()}
    results = client.ft(INDEX_NAME).search(q, query_params)
    
    if results.docs:
        score = float(results.docs[0].score)
        if score < DISTANCE_THRESHOLD:
            return results.docs[0].response
    return None

Critical Considerations: The "Threshold" Problem

The most difficult part of semantic caching isn't the infrastructure; it's the threshold.

• Too Strict (Threshold 0.05): You get very few cache hits. You're paying for the embedding call but still hitting the LLM most of the time.
• Too Loose (Threshold 0.30): You get "false hits." A user asks "How do I delete my account?" and the cache returns the answer for "How do I update my account?" because they are semantically related in the vector space.

Finding the sweet spot requires testing with your specific dataset. I recommend logging the "distance" of your hits during a beta period to visualize the distribution of similarity scores.

Handling Dynamic Content and TTLs

Semantic caching is not a "set it and forget it" solution for all data. If your LLM provides real-time data (e.g., "What is the current stock price of Apple?"), a semantic cache can be dangerous.

To mitigate this, implement a Time-To-Live (TTL) strategy:

1. Standard TTL: Expire cache entries after 24 hours to ensure the model's logic stays relatively fresh.
2. Namespace Partitioning: Use different Redis indexes for different types of queries. Static FAQ-style queries can have long TTLs; account-specific queries should have very short TTLs or not be cached at all.
3. Metadata Filtering: Store tags in Redis alongside the vector. If a user is in a specific context (e.g., version=v2), ensure the search only looks for cached responses with that same tag.

Security and Privacy

Caching LLM responses introduces a risk of PII (Personally Identifiable Information) leakage. If User A asks a question containing their email address and the LLM includes that email in the response, User B might receive that response if their query is semantically similar.

Best Practices:

• Anonymize before caching: Scrub PII from both the query (before embedding) and the response (before storing).
• User-Scoped Caching: If the data is sensitive, include a user_id in the Redis search filter so users only hit their own cache.

Cost-Benefit Analysis

Let's look at the math.

• OpenAI GPT-4o cost (approx): $0.01 per 1k tokens.
• Embedding cost (approx): $0.00002 per 1k tokens.
• Redis Latency: ~5ms.
• LLM Latency: ~2000ms.

If you achieve a 30% cache hit rate on 100,000 queries:

• You save the cost of 30,000 LLM calls.
• You reduce total system latency by 60,000 seconds (1,000 minutes) for your users.
• The cost of the embedding calls and Redis hosting is negligible compared to the GPT-4o savings.

Conclusion: The Path Forward

Semantic caching is one of the highest-leverage optimizations you can implement in an AI-native application. It transforms your LLM from a slow, expensive bottleneck into a responsive system that learns from its previous interactions.

Actionable Steps:

1. Audit your logs: Identify repeated query patterns to estimate your potential cache hit rate.
2. Start with a high threshold: Be conservative. It's better to miss a cache hit than to serve a wrong answer.
3. Use Redis Stack: Leverage the built-in vector search capabilities rather than trying to bolt a vector library onto a standard relational database.
4. Monitor and iterate: Treat your similarity threshold as a hyperparameter that needs tuning as your user base grows.