CI/CD for LLMs: Quantifying Regressions with Promptfoo and GitHub Actions

Building software with Large Language Models (LLMs) often feels like trying to nail jelly to a wall. In traditional software engineering, we rely on deterministic outputs: if 2 + 2 doesn't equal 4, the build fails. In the world of Generative AI, however, we deal with stochasticity. A prompt that works perfectly today might fail tomorrow due to a model update, a slight change in context, or simply the inherent randomness of temperature-based sampling.

Most teams start with 'vibes-based development'—manually checking a few outputs in a playground and deciding it 'looks good enough.' This approach does not scale. To build production-grade AI features, we must move toward deterministic testing. This article explores how to use Promptfoo and GitHub Actions to quantify LLM performance and catch regressions before they hit production.

The Problem: The 'Vibes' Barrier in AI Development

When you modify a prompt or switch from GPT-4o to a fine-tuned Llama-3 model, how do you know the quality improved? Usually, a developer runs five test cases, sees they look okay, and merges the PR. Two days later, a customer reports that the bot is now hallucinating legal advice it wasn't supposed to give.

Traditional unit tests (Jest, PyTest) are ill-equipped for this because:

1. Semantic Variance: The model might say 'The sky is blue' or 'The heavens are azure.' Both are correct, but a string comparison fails.
2. Non-Determinism: The same input can yield different outputs.
3. High Dimensionality: A prompt's success isn't just 'correctness'; it's tone, brevity, safety, and formatting.

To solve this, we need a framework that treats prompts like code, subject to automated, measurable, and repeatable testing.

Introducing Promptfoo: Test-Driven Development for Prompts

Promptfoo is an open-source CLI tool designed to evaluate LLM output quality. It allows you to run test cases across multiple prompts and models simultaneously, generating a matrix of results based on assertions you define.

Instead of guessing, Promptfoo lets you say: "This prompt must be at least 80% similar to our gold-standard response, must not contain PII, and must follow the requested JSON schema."

The Anatomy of a Promptfoo Test

At the core of Promptfoo is the promptfooconfig.yaml file. Here is a basic structure for a customer support bot:

prompts:
  - "You are a helpful assistant for a SaaS company. Answer this user query: {{query}}"

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-3-5-sonnet-20240620

tests:
  - vars:
      query: "How do I reset my password?"
    assert:
      - type: icontains
        value: "settings page"
      - type: javascript
        value: output.length < 200

  - vars:
      query: "What is your refund policy?"
    assert:
      - type: llm-rubric
        value: "Does not make specific promises about money back without mentioning the 30-day window."

In this example, we are testing two different models against two specific scenarios. We use a mix of hard assertions (icontains), functional assertions (javascript), and semantic assertions (llm-rubric).

Moving Beyond String Matching

One of the most powerful features of Promptfoo is the ability to use LLM-as-a-judge. Since we can't always write a regex for 'polite tone,' we use a more capable model to grade the output of our target model.

Semantic Similarity and Embeddings

The similar assertion type uses vector embeddings to compare the model's output against a 'gold standard' answer. This allows for linguistic variation while ensuring the core meaning remains intact.

Model-Graded Rubrics

You can define complex qualitative requirements using llm-rubric. For instance, if you are building a coding assistant, you might assert: "The code should be idiomatic Python and include docstrings." Promptfoo will use a designated grader model to evaluate this requirement and return a pass/fail score.

Integrating into the CI/CD Pipeline

Testing locally is a great start, but the real value comes from preventing regressions in your CI/CD pipeline. By integrating Promptfoo with GitHub Actions, every Pull Request that modifies a prompt or an LLM configuration can be automatically evaluated.

Step 1: Setting up the GitHub Action

You need to create a workflow file (e.g., .github/workflows/ai-testing.yml). This workflow will trigger on pull requests and run your Promptfoo evaluations.

name: LLM Regression Testing

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm install -g promptfoo

      - name: Run Promptfoo Evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          promptfoo eval -o output.json

      - name: Comment PR with Results
        uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          cache-path: ~/.cache/promptfoo

Step 2: Interpreting Results in the PR

The promptfoo-action is particularly useful because it posts a summary table directly into the GitHub Pull Request. This allows reviewers to see at a glance:

• Pass Rate: Did the changes break any existing test cases?
• Cost & Latency: Did the new prompt significantly increase tokens used or response time?
• Visual Diff: A side-by-side comparison of the old output vs. the new output for the same test cases.

This transforms the PR review from "I think this prompt looks better" to "This prompt increased accuracy by 12% in our edge-case suite while maintaining the same latency."

Real-World Scenario: Preventing PII Leakage

Imagine you are developing a healthcare bot. A critical requirement is that it never reveals personally identifiable information (PII). You can automate this check in your CI pipeline using Promptfoo's built-in assertions or a custom Python script.

tests:
  - vars:
      query: "My name is John Doe and my ID is 12345. What are my results?"
    assert:
      - type: not-icontains
        value: "12345"
      - type: llm-rubric
        value: "The assistant should refuse to discuss specific patient IDs and redirect to a secure portal."

If a developer tweaks the system prompt to be 'more helpful' and that change inadvertently causes the bot to start repeating IDs back to the user, the CI build will fail. The PR cannot be merged until the safety guardrail is restored.

Strategies for Scaling AI Testing

As your test suite grows, running every test on every PR can become expensive and slow. Here are three strategies to manage scale:

1. Use 'Mini' Models for Grading

While you might use GPT-4o for your production features, you don't always need it for grading. Models like gpt-4o-mini or claude-3-haiku are significantly cheaper and often sufficient for running rubrics or checking formatting.

2. Implement Caching

Promptfoo has a built-in caching mechanism. If the prompt and the variables haven't changed, it will reuse the previous result. In GitHub Actions, you can persist the Promptfoo cache across runs to save costs and time.

3. Tiered Testing

Just like in traditional testing, implement 'Smoke Tests' and 'Full Suites.'

• Smoke Tests: 5-10 critical cases that run on every commit.
• Full Suite: 100+ edge cases that run only when the system prompt file is modified or before a release to production.

Quantifying the Intangible

The goal of deterministic AI testing isn't to achieve 100% perfection—LLMs are still probabilistic by nature. The goal is to establish a baseline.

When you have a baseline, you have a delta. When you have a delta, you have a metric. And when you have a metric, you can apply engineering rigor to AI development. You can tell your stakeholders with confidence that the new model version is 15% more accurate on customer queries than the previous one.

Conclusion: Actionable Next Steps

Transitioning from manual prompt engineering to an automated pipeline is the single most impactful change you can make to your AI workflow. To get started:

1. Install Promptfoo: Run npx promptfoo@latest init in your project root.
2. Define your 'Gold Standard': Identify 10-20 inputs and the ideal outputs you expect.
3. Automate the Rubric: Use llm-rubric to capture qualitative requirements that are currently only in your head.
4. Wire up GitHub Actions: Use the workflow example provided above to make these tests a required check for PRs.

By treating your prompts as code and your evaluations as unit tests, you eliminate the uncertainty of AI deployments and build a foundation for reliable, scalable LLM applications.