Automating RAG Evaluation: Using DeepEval and Pytest for CI/CD

Building a Retrieval-Augmented Generation (RAG) prototype is easy; getting it into production with high confidence is hard. Most engineering teams start their LLM journey with 'vibe checks'—manually prompting the model, reading the output, and nodding if it looks correct. However, as your knowledge base grows and your system architecture evolves, manual evaluation becomes a bottleneck that prevents rapid iteration.

To build production-grade AI, we need to treat LLM outputs like any other critical software component: with automated, unit-testable metrics. This article explores how to implement a robust evaluation pipeline using DeepEval and Pytest, focusing on quantifying hallucinations and faithfulness within a CI/CD workflow.

The Problem with LLM 'Vibe Checks'

In traditional software engineering, a test has a binary outcome: it passes or it fails. With LLMs, the output is probabilistic. A prompt that worked yesterday might produce a hallucination today because of a slight change in the retrieved context or a model update.

When we talk about RAG, we are specifically concerned with three primary failure modes:

1. Hallucination: The model invents facts not present in the retrieved documents.
2. Low Faithfulness: The model's answer contradicts the retrieved context.
3. Irrelevancy: The model answers the question, but ignores the specific constraints or context provided.

To solve this, we need a way to turn qualitative 'vibes' into quantitative scores. This is where LLM-as-a-judge and framework-based evaluation come in.

Introducing the Toolset: DeepEval and Pytest

DeepEval is an open-source testing framework for LLMs that builds on top of Pytest. It allows you to define metrics (like faithfulness, relevancy, and bias) and run them as standard Python tests.

Why use DeepEval instead of writing custom evaluation scripts?

• Integration: Since it's built on Pytest, it integrates natively with GitHub Actions, GitLab CI, and Jenkins.
• Metrics: It provides pre-built, research-backed metrics that use 'LLM-as-a-judge' logic (typically using GPT-4o or Claude 3.5 Sonnet) to evaluate smaller or task-specific models.
• Traceability: It offers a dashboard to track how your model’s performance trends over time.

Understanding the Core Metrics

Before we dive into the code, we must understand the 'RAG Triad' of metrics we'll be implementing.

1. Faithfulness

Faithfulness measures how much the LLM's response aligns with the retrieved context. DeepEval calculates this by deconstructing the output into individual claims and cross-referencing each claim against the retrieval nodes. If the model claims "The API limit is 500 requests" but the context says "The limit is 200," the faithfulness score drops.

2. Answer Relevancy

This measures how well the answer addresses the original prompt. It ignores the context for a moment and focuses on the user's intent. A high faithfulness score with a low relevancy score usually means the model is being too cautious or 'parroting' the context without answering the specific question.

3. Hallucination Metric

While similar to faithfulness, the hallucination metric specifically checks for 'contradictions.' It uses a formal verification process to ensure that no part of the generated output explicitly denies a fact provided in the context.

Implementation: Coding the Evaluation Suite

Let’s walk through a practical implementation. We will assume you have a RAG system that takes a query and returns an answer and a list of retrieval_context strings.

Step 1: Environment Setup

First, install the necessary packages:

pip install deepeval pytest

Step 2: Defining the Test Case

In DeepEval, we wrap our RAG output in an LLMTestCase. Here is a basic example of how to structure a test file (test_rag.py):

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

def test_customer_support_rag():
    # 1. Simulate your RAG pipeline output
    user_input = "What is the refund policy for electronics?"
    retrieved_context = [
        "Electronics can be returned within 30 days of purchase.",
        "Items must be in original packaging with a receipt."
    ]
    actual_output = "You can return electronics within 30 days if you have the receipt and original packaging."

    # 2. Initialize the metrics
    # We set a threshold of 0.7. Anything below this fails the test.
    faithfulness_metric = FaithfulnessMetric(threshold=0.7)
    relevancy_metric = AnswerRelevancyMetric(threshold=0.7)

    # 3. Create the test case
    test_case = LLMTestCase(
        input=user_input,
        actual_output=actual_output,
        retrieval_context=retrieved_context
    )

    # 4. Assert the test
    assert_test(test_case, [faithfulness_metric, relevancy_metric])

Step 3: Running the Tests

You run these tests just like any other Pytest suite:

deepeval test run test_rag.py

DeepEval will invoke an LLM (defaulting to GPT-4) to act as the judge. It will analyze the claims, compare them to the context, and output a score between 0 and 1 for each metric.

Integrating with CI/CD Pipelines

The real power of this approach is realized when it's integrated into your deployment pipeline. By running these evaluations on every Pull Request, you can prevent regressions in model quality.

GitHub Actions Configuration

Here is a snippet for a .github/workflows/llm-eval.yml file:

name: LLM Evaluation

on:
  pull_request:
    branches: [ main ]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install deepeval

      - name: Run DeepEval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_rag.py

In this setup, if your RAG system's faithfulness score drops below 0.7 due to a prompt change or a new data chunking strategy, the CI build will fail, preventing the degraded model from reaching production.

Advanced Strategies: Gold Datasets and Synthetic Data

Testing against a single hardcoded string is a start, but production systems need a "Gold Dataset"—a collection of high-quality (Input, Context, Expected Output) tuples that represent the edge cases of your business logic.

Synthetic Data Generation

If you don't have a curated dataset yet, DeepEval provides a Synthesizer. It can ingest your documentation and automatically generate pairs of questions and ground-truth contexts. This allows you to bootstrap an evaluation suite in minutes rather than weeks.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()
synthesizer.generate_gold_dataset(
    documents=["./docs/refund_policy.pdf"],
    max_gold_samples=20
)

G-Eval: Custom Metrics

Sometimes standard metrics aren't enough. If you need to evaluate something specific, like "Does the response sound professional?" or "Is the tone empathetic?", you can use GEval. This allows you to define a metric using natural language instructions.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

professionalism_metric = GEval(
    name="Professionalism",
    criteria="Determine if the response is professional and avoids slang.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.8
)

Managing Costs and Latency

Running GPT-4o as a judge for every unit test can become expensive and slow. To optimize your pipeline:

1. Sample Your Evals: Don't run the full 500-test suite on every commit. Run a 'smoke test' of 10 critical cases on every commit and the full suite nightly.
2. Use Smaller Judges: For simpler metrics like relevancy, models like GPT-4o-mini or locally hosted Llama 3 (via Ollama) are often sufficient and significantly cheaper.
3. Cache Results: If the prompt and context haven't changed, use cached evaluation scores.

Conclusion: Making LLM Quality Measurable

Moving from manual testing to automated evaluation pipelines is the single most important step in maturing an AI engineering practice. By using DeepEval and Pytest, you bridge the gap between non-deterministic AI outputs and deterministic software requirements.

Actionable Next Steps:

1. Identify 10 critical queries your RAG system must answer correctly.
2. Write a DeepEval test for these queries using the FaithfulnessMetric.
3. Integrate the test into your CI/CD pipeline with a fail-threshold.
4. Iterate on your prompts only after you have a baseline score to compare against.

Stop guessing if your LLM is getting better. Start measuring it.