Automating RAG Evaluation: Using DeepEval and Pytest for CI/CD

Building a Retrieval-Augmented Generation (RAG) system is deceptively simple. You hook up a vector database, write a few lines of LangChain code, and suddenly your LLM is answering questions about your proprietary documentation. The honeymoon phase usually ends during the first week of production when a user reports that the bot is confidently hallucinating facts that don't exist in your source data.

Traditionally, developers have relied on the 'vibe check'—manually testing a dozen prompts and deciding if the output looks 'good enough.' As senior engineers, we know this doesn't scale. If you change a prompt, update your embedding model, or tweak your chunking strategy, you need a deterministic way to measure if your system got better or worse.

This article explores how to bridge the gap between non-deterministic LLM outputs and deterministic CI/CD pipelines using DeepEval and Pytest.

The Problem: The RAG Triad and Hallucinations

In a RAG architecture, there are three main failure points:

1. Retrieval Failure: The system fails to find the relevant documents.
2. Augmentation Failure: The retrieved documents are relevant, but the context is lost or corrupted when passed to the LLM.
3. Generation Failure: The LLM has the right context but ignores it, hallucinating an answer based on its training data instead.

To solve this, we need to move away from binary 'pass/fail' tests and toward unit testing for LLMs. This involves quantifying specific metrics like Faithfulness, Answer Relevancy, and Contextual Precision.

Introducing DeepEval: The Unit Testing Framework for LLMs

DeepEval is an open-source framework that brings the familiarity of Pytest to LLM evaluation. It allows you to define 'test cases' for your LLM outputs and run them against specific metrics.

What makes DeepEval powerful is its use of LLM-as-a-judge. It uses a more capable model (like GPT-4o) to evaluate the output of your production model (like GPT-3.5 or Llama 3). It breaks down the evaluation into atomic claims and verifies them against the retrieved context, providing a score between 0 and 1.

Setting Up the Environment

First, install the necessary dependencies:

pip install deepeval pytest

You will also need an OpenAI API key (or access to another LLM provider) to act as the evaluator.

Quantifying Faithfulness and Relevancy

Let’s look at the two most critical metrics for any RAG pipeline: Faithfulness and Answer Relevancy.

1. Faithfulness (The Hallucination Killer)

Faithfulness measures whether the LLM's response is derived entirely from the retrieved context. If the LLM adds information not present in the context, the faithfulness score drops.

2. Answer Relevancy

This measures how well the response addresses the user's prompt. A response can be 100% faithful to a document but completely fail to answer the user's actual question.

Practical Example: Writing the Test

Imagine we are building a support bot for a FinTech app. Here is how we would write a test case in a file named test_rag.py:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

def test_customer_support_response():
    # 1. Setup the data (In a real scenario, this comes from your RAG pipeline)
    query = "What is the daily withdrawal limit for the Gold card?"
    retrieved_context = [
        "Gold card members have a daily ATM withdrawal limit of $2,000.",
        "Standard card limits are set to $500 per day."
    ]
    actual_output = "The daily withdrawal limit for Gold card holders is $2,000."

    # 2. Define the metrics
    faithfulness_metric = FaithfulnessMetric(threshold=0.7)
    relevancy_metric = AnswerRelevancyMetric(threshold=0.7)

    # 3. Create the test case
    test_case = LLMTestCase(
        input=query,
        actual_output=actual_output,
        retrieval_context=retrieved_context
    )

    # 4. Execute the assertion
    assert_test(test_case, [faithfulness_metric, relevancy_metric])

When you run pytest test_rag.py, DeepEval will:

1. Extract claims from the actual_output.
2. Cross-reference those claims with the retrieval_context.
3. Calculate a score.
4. If the score is below 0.7, the test fails, and DeepEval provides a detailed 'Reason' for the failure.

Integrating into CI/CD Pipelines

Testing LLMs locally is a start, but the real value comes from automating this in your deployment pipeline. By integrating these tests into GitHub Actions or GitLab CI, you can prevent 'regression hallucinations'—where a change to the system prompt fixes one issue but breaks five others.

Step 1: The Batch Evaluation Script

In a production environment, you don't just test one prompt. You test a 'Golden Dataset'—a collection of curated inputs and expected contexts.

# eval_suite.py
from deepeval.dataset import EvaluationDataset
# ... imports and setup ...

def test_rag_batch():
    dataset = EvaluationDataset()
    dataset.pull("FinTech-Support-Dataset") # Or load from a local JSON
    
    results = dataset.run_test(metrics=[faithfulness_metric])

Step 2: GitHub Actions Configuration

Here is a snippet of a .github/workflows/rag-eval.yml file that runs these tests on every Pull Request:

name: RAG Evaluation

on: [pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install deepeval pytest

      - name: Run DeepEval Tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest test_rag.py

If the LLM's faithfulness score drops due to a prompt change in the PR, the build fails, protecting your production environment from degraded performance.

Challenges and Senior-Level Considerations

While automated evaluation is powerful, it is not a silver bullet. As a senior engineer, you should be aware of these nuances:

The 'Judge' Model Matters

You cannot reliably evaluate a GPT-4o output using a GPT-3.5 judge. The evaluator must be at least as sophisticated as the model being tested. Ideally, use the most capable model available (currently GPT-4o or Claude 3.5 Sonnet) for your evaluation suite, even if your production model is smaller/cheaper.

Cost and Latency

Running an evaluation suite with 100 test cases using GPT-4o as a judge can cost several dollars and take several minutes. This is why we treat these as Integration Tests, not unit tests. Run them on PRs or nightly builds, not on every single commit.

Synthetic Data Generation

One of the hardest parts of LLM evaluation is building the 'Golden Dataset.' DeepEval offers features to generate synthetic test cases from your knowledge base. It takes your documents, generates potential user questions, and identifies the 'ground truth' context. This is an excellent way to bootstrap your testing pipeline if you don't have historical user logs.

Beyond Simple Metrics: Custom Scrutiny

Sometimes, standard metrics aren't enough. You might need to enforce brand voice, prevent PII leakage, or ensure specific technical jargon is used correctly. DeepEval allows for G-Eval, a framework where you define custom evaluation criteria in plain English.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

professionalism_metric = GEval(
    name="Professionalism",
    criteria="Determine if the response is professional, avoids slang, and is helpful.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.8
)

This level of flexibility allows the engineering team to codify 'subjective' requirements into the CI/CD pipeline.

Actionable Conclusion

Stop relying on manual testing for your RAG applications. The transition from a prototype to a production-grade AI agent requires a shift in mindset: treat LLM outputs as code that needs testing.

To get started:

1. Identify your Golden Dataset: Collect 20-50 diverse user queries and their corresponding 'correct' contexts.
2. Implement Faithfulness Metrics: Use DeepEval to ensure your LLM stays within the bounds of the provided data.
3. Automate: Integrate Pytest into your CI/CD to make evaluation a non-negotiable part of your deployment workflow.

By quantifying hallucinations, you transform AI development from a series of 'vibes' into a disciplined engineering practice.