Semantic Cache-Aside: Reducing LLM Costs with Upstash and Redis

Large Language Models (LLMs) have fundamentally changed how we build software, but they come with two significant 'taxes': high latency and high cost. Every time a user asks a question, we pay several cents in API tokens and wait several seconds for a response. In a production environment, this is often unsustainable, especially when users frequently ask questions that are semantically identical.

Traditional caching strategies, like simple key-value pairs in Redis, fail in the world of natural language. If one user asks, "How do I reset my password?" and another asks, "What is the process for password recovery?", a standard string-match cache will treat these as two distinct misses.

This is where Semantic Cache-Aside comes in. By combining the speed of Redis with the intelligence of vector embeddings via Upstash Vector, we can build a caching layer that understands intent, drastically reducing our reliance on expensive LLM providers.

The Problem with Exact-Match Caching

Standard caching relies on deterministic keys. In a typical REST API, the URL and query parameters form the key. In an LLM context, the 'key' is the prompt. However, human language is messy. Variations in phrasing, punctuation, and capitalization mean that the exact same intent can be expressed in infinite ways.

If we use the prompt as a key in Redis, our cache hit rate will be abysmally low. We need a way to say: "If the incoming query is close enough to a query we’ve seen before, return the cached answer."

Understanding Semantic Cache-Aside

The Cache-Aside (or Lazy Loading) pattern is a staple of backend engineering. When data is requested, the application first checks the cache. If the data is found (a hit), it’s returned. If not (a miss), the data is fetched from the source, stored in the cache for next time, and returned to the user.

Semantic Cache-Aside follows this flow but replaces the 'exact match' check with a 'similarity search.'

1. Vectorization: Convert the incoming prompt into a numerical vector (embedding).
2. Similarity Search: Use a vector database to find the nearest neighbor to that embedding.
3. Thresholding: If the distance between the new prompt and the nearest neighbor is below a certain threshold, we consider it a semantic hit.
4. Retrieval: Fetch the pre-computed response associated with that neighbor.

The Architecture: Upstash Vector + Redis

While some vector databases allow storing large payloads, a professional-grade architecture often separates the Index from the Data Store.

• Upstash Vector: Acts as our semantic index. It stores the embeddings of previous prompts and a reference ID (the Redis key).
• Redis: Acts as our high-speed data store. It stores the actual text of the LLM response, metadata, and TTL (Time-to-Live) logic.

Why Upstash?

For this pattern, serverless tools are ideal. Upstash provides both Redis and Vector databases with a pay-as-you-go model, meaning you don't pay for idle compute while your cache is growing. Their low-latency HTTP API is also perfect for serverless functions (like Vercel or AWS Lambda).

Implementing the Workflow

Let’s walk through a conceptual implementation using TypeScript.

1. Generating the Embedding

Before we can check the cache, we must transform the user's string into a vector. Using OpenAI’s text-embedding-3-small is a cost-effective choice for this.

async function getEmbedding(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return response.data[0].embedding;
}

2. Checking the Semantic Cache

We query Upstash Vector to find the most similar prompt. We are looking for a high similarity score (e.g., > 0.95).

const embedding = await getEmbedding(userPrompt);
const [hit] = await vectorIndex.query({
  vector: embedding,
  topK: 1,
  includeMetadata: true,
});

if (hit && hit.score > 0.95) {
  // We found a semantic match!
  const cachedResponse = await redis.get(hit.metadata.redisKey);
  if (cachedResponse) return cachedResponse;
}

3. Handling a Cache Miss

If no similar prompt is found, we proceed to the LLM, then populate both our index and our data store.

const llmResponse = await callLLM(userPrompt);

// 1. Store the full response in Redis with an expiration
const cacheKey = `response:${uuid()}`;
await redis.set(cacheKey, llmResponse, { ex: 3600 * 24 }); // 24h TTL

// 2. Index the prompt in Upstash Vector for future similarity searches
await vectorIndex.upsert({
  id: uuid(),
  vector: embedding,
  metadata: { redisKey: cacheKey, originalPrompt: userPrompt },
});

The Critical Factor: Similarity Thresholds

Setting the threshold is the most nuanced part of semantic caching.

• Too High (e.g., 0.99): You only catch near-identical strings. Your hit rate remains low, but accuracy is high.
• Too Low (e.g., 0.80): You might return an answer for "How do I delete my account?" when the user asked "How do I update my account?" because both involve account management.

I recommend starting with 0.93 to 0.96. You should also log the "hits" that fall into a 'grey zone' (0.90 - 0.93) to manually review if the responses were appropriate. This allows you to tune the threshold based on real-world usage.

Real-World Benefits

Latency Reduction

A typical LLM call can take anywhere from 2 to 30 seconds depending on the model and output length. A Redis + Upstash Vector lookup typically completes in under 100ms. For a documentation bot, this transforms the user experience from a sluggish "waiting for type-out" to an instant response.

Cost Savings

Consider a support bot handling 10,000 queries a day. If 30% of those are repetitive questions, and each LLM call costs $0.01, you are saving $30/day or $900/month. The cost of running Upstash to support this cache would be a fraction of that, likely under $20/month.

Advanced Considerations

1. Handling Stale Data

LLM knowledge changes. If your documentation updates, your cache might provide outdated information. Use Redis TTLs to ensure data expires, or implement a purge mechanism that clears specific vector clusters when your source data changes.

2. User-Specific Context

If your LLM responses are personalized (e.g., "Hello [Name], your balance is [Balance]"), you cannot use a global semantic cache. In this case, include a userId in your vector metadata and filter your search by that ID. This ensures users only hit their own cached responses.

3. Security and PII

Be cautious about storing Personally Identifiable Information (PII) in your cache. If a user asks a question containing an email address, that embedding and the subsequent cached response will exist in your database. Ensure your caching layer complies with your data privacy policies, perhaps by scrubbing PII before embedding.

Evaluation and Monitoring

To treat this as a first-class engineering component, you need metrics. Track the following:

• Cache Hit Rate: What percentage of queries are handled semantically?
• Distance Distribution: What is the average similarity score of a hit?
• P99 Latency: Compare the latency of a cache hit vs. an LLM miss.
• Cost Avoidance: Calculate (Hits * Average LLM Cost).

Conclusion

Implementing a semantic cache-aside layer is no longer an experimental 'AI' task; it is a standard optimization for production-grade LLM applications. By using Upstash Vector for intent discovery and Redis for rapid data retrieval, you can bridge the gap between the fluid nature of human language and the rigid requirements of high-performance software.

Actionable Next Steps:

1. Identify your most frequent LLM prompts via existing logs.
2. Set up an Upstash Vector index with the same dimensions as your embedding model (e.g., 1536 for OpenAI).
3. Implement the semantic check with a conservative threshold of 0.95.
4. Monitor your hit rate and adjust the threshold based on user feedback.