Deterministic AI Testing: Quantifying LLM Regression in CI/CD

As software engineers, we are accustomed to the predictability of deterministic systems. We write a function, define unit tests, and expect the same input to yield the same output every single time. However, the integration of Large Language Models (LLMs) into our tech stacks has shattered this paradigm. LLMs are inherently probabilistic, meaning the same prompt can yield different results across different versions, temperatures, or even consecutive calls.

In many organizations, the current standard for testing AI features is the 'vibe check'—a developer manually inputs a few queries, glances at the output, and decides it looks 'good enough.' This approach is a ticking time bomb for production environments. To build production-grade AI applications, we must treat prompts like code and apply the same rigor of automated testing and regression detection that we use for our backend APIs.

This article explores how to implement deterministic AI testing using Promptfoo, an open-source CLI tool designed for evaluating LLM outputs, and how to integrate it into your GitHub Actions pipelines to quantify regression.

The Problem: The Fragility of the Prompt

Prompt engineering is often more akin to alchemy than chemistry. A minor change in phrasing—or even updating the underlying model version (e.g., moving from GPT-4 to GPT-4o)—can cause unexpected regressions. These regressions aren't always obvious; a model might become slightly more verbose, lose its ability to format JSON correctly, or start leaking sensitive information.

Without a baseline and automated quantification, you cannot answer the fundamental question: Is this version better than the last?

Enter Promptfoo: Jest for LLMs

Promptfoo fills the gap between manual testing and expensive, custom-built evaluation frameworks. It allows you to define test cases in a declarative YAML format, run evaluations against multiple providers, and assert specific qualities about the output.

What makes Promptfoo powerful is its ability to handle both hard assertions (like Regex or JSON schema validation) and soft assertions (using LLMs to grade other LLMs). This combination allows us to measure both the structure and the semantic quality of the output.

Setting Up a Test Suite

To get started, let’s look at a typical configuration file, promptfooconfig.yaml. Imagine we are building a support bot that needs to summarize technical tickets.

prompts:
  - "Summarize the following technical support ticket in 3 sentences: {{ticket_content}}"
  - "Provide a concise summary of this issue for a senior engineer: {{ticket_content}}"

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-3-5-sonnet-20240620

tests:
  - vars:
      ticket_content: "The database connection is timing out with Error 504 after the recent migration to the v2 API. Logs show high latency in the auth middleware."
    assert:
      - type: contains
        value: "504"
      - type: llm-rubric
        value: "The summary should mention both the database and the auth middleware."
      - type: latency
        threshold: 2000 # milliseconds

  - vars:
      ticket_content: "I can't log in. It just says 'system error'. Help!"
    assert:
      - type: javascript
        value: output.length < 150
      - type: cost
        threshold: 0.01 # dollars

In this configuration, we are testing two different prompts against two different models. We've defined specific variables and assertions that check for keyword presence, semantic meaning (via llm-rubric), and even operational metrics like latency and cost.

Quantifying Quality with Assertions

To move away from vibe checks, we need metrics. Promptfoo provides several assertion types that allow us to quantify performance:

1. Deterministic Assertions

These are the bread and butter of traditional testing.

• contains / not-contains: Simple string matching.
• is-json: Ensures the model output is valid JSON, which is critical for downstream processing.
• javascript: Allows you to write custom logic to validate the output length, format, or content.

2. Model-Graded Evaluations (LLM-as-a-Judge)

This is where we solve the problem of semantic nuance. Using the llm-rubric or similar assertion types, you can use a stronger model (like GPT-4) to grade the output of your production model.

For example, if you want to ensure your model doesn't sound condescending, you can use a rubric: "The output should be professional and empathetic, avoiding any patronizing language."

3. Red Teaming and Safety

You can also use assertions to ensure your model doesn't hallucinate or provide dangerous information. By including test cases that attempt to 'jailbreak' the prompt, you can set assertions like not-contains: "password" or use specific toxicity filters.

Integrating into CI/CD with GitHub Actions

Automating these tests is what transforms them from a utility to a gatekeeper. By integrating Promptfoo into GitHub Actions, you can block a pull request if a prompt change causes a drop in accuracy or an increase in cost.

Here is a practical workflow file (.github/workflows/ai-testing.yml):

name: LLM Regression Testing

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm install -g promptfoo

      - name: Run Promptfoo Evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          promptfoo eval --output output.json

      - name: Comment on PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('output.json', 'utf8'));
            const summary = `### LLM Eval Results
            ✅ Passed: ${results.stats.successes}
            ❌ Failed: ${results.stats.failures}
            📊 Score: ${results.stats.tokenUsage.total}`;
            
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: summary
            });

Why this matters for your workflow

When a developer modifies a prompt, the action triggers. It runs the evaluation and posts a comment directly on the PR. If a test fails—for instance, if the new prompt fails the llm-rubric—the CI build fails. This forces the developer to either refine the prompt or update the baseline expectations, making every change intentional.

Advanced Pattern: Comparing Model Versions

One of the most frequent challenges in AI development is upgrading to a newer, cheaper, or faster model without losing quality. Promptfoo makes this trivial. By listing both the current production model and the candidate model in the providers section, you can generate a side-by-side comparison matrix.

This matrix allows you to see exactly where the new model differs. Does GPT-4o-mini fail on edge cases where GPT-4o succeeded? With deterministic testing, you have the data to back up your migration decisions, rather than relying on anecdotal evidence.

Strategy: Handling Non-Determinism in Tests

Since LLMs are probabilistic, a test might fail occasionally due to 'flakiness' rather than a bad prompt. To mitigate this, consider the following strategies:

1. Set Temperature to 0: For testing, always set your model temperature to 0. This reduces variance and makes the output as deterministic as possible.
2. Thresholding: Instead of binary pass/fail, use the threshold setting in Promptfoo. For example, a semantic similarity test might pass if the score is > 0.85.
3. Multiple Iterations: Run the same test case 3 times and ensure it passes at least 2 out of 3 times. Promptfoo supports this via the repeat configuration.

The Cost of Quality

Running LLM evaluations in CI/CD does incur costs. Every PR check will consume tokens. However, the cost of a hallucination in production—leading to lost customer trust, data leaks, or incorrect business decisions—is infinitely higher. To optimize costs:

• Only run evaluations on files that affect the AI logic.
• Use smaller, faster models (like GPT-4o-mini or Claude Haiku) for basic structural assertions.
• Reserve larger models (like GPT-4o) for the llm-rubric grading of the final output.

Conclusion: Moving Toward LLMOps Maturity

Transitioning from manual evaluation to automated, deterministic testing is a requirement for any team serious about deploying AI. By using Promptfoo and GitHub Actions, you transform your prompts from 'black box' scripts into versioned, tested, and quantifiable assets.

Actionable next steps:

1. Audit your current prompts: Identify the most critical prompts in your application.
2. Create a baseline: Write 5-10 test cases for each prompt using promptfooconfig.yaml.
3. Automate: Integrate the evaluation into your CI/CD pipeline to prevent regressions.
4. Iterate: Use the generated metrics to justify model upgrades and prompt optimizations.

By quantifying your AI’s performance, you move from the uncertainty of 'vibe checks' to the confidence of engineering.