Accelerating LLM Inference with Speculative Decoding and vLLM

In the world of Large Language Models (LLMs), the gap between training a model and serving it efficiently in production is a chasm that many engineering teams struggle to cross. While much of the industry's attention focuses on parameter counts and context windows, the day-to-day reality for software engineers is often defined by a single metric: tokens per second (TPS).

Standard autoregressive decoding is inherently slow because it generates tokens one by one, with each step requiring a full pass through the model's weights. For a 70B parameter model, this means moving massive amounts of data from VRAM to the GPU cores for every single word generated. Speculative decoding offers a way to break this linear bottleneck. By using a smaller, faster "draft" model to predict multiple future tokens and then verifying them in parallel with the larger "target" model, we can achieve significant speedups without sacrificing accuracy.

This guide explores how to implement speculative decoding using vLLM, the industry-standard serving engine, to optimize self-hosted LLM deployments.

The Bottleneck: Why LLM Inference is Slow

To understand why speculative decoding works, we must first understand why standard inference is inefficient. Most LLM inference tasks are memory-bandwidth bound, not compute-bound.

When you run a model like Llama-3-70B, the GPU spends most of its time moving model weights from Global Memory (VRAM) to the registers. For a single token generation step, the compute required is relatively small, but the entire model must be loaded into the GPU's processing units. Because the GPU is waiting on memory transfers, its arithmetic logic units (ALUs) are often underutilized, idling while data travels across the bus.

In a standard setup, if you want to generate 10 tokens, you must load the model weights 10 times. This is the fundamental limit of autoregressive generation. Speculative decoding aims to change this ratio by loading the large model weights once to verify several tokens at once.

Enter Speculative Decoding: The Concept

Speculative decoding (also known as assisted generation) introduces a two-step process: Drafting and Verification.

The Draft-and-Verify Cycle

1. The Draft Model: We use a much smaller, computationally inexpensive model (the "draft model") to generate a sequence of $K$ candidate tokens. Because the draft model is small (e.g., a 1B or 7B model), it can generate these tokens very quickly.
2. The Target Model: We take those $K$ candidate tokens and pass them to the large "target" model in a single batch.
3. Parallel Verification: The target model performs a single forward pass on the entire sequence. Because of the way Transformers work, the target model can check the validity of all $K$ tokens simultaneously using its internal attention mechanism.
4. Acceptance: If the target model's predicted probability for a token matches the draft model's choice (within a certain sampling threshold), the token is accepted. If the target model disagrees at token $i$, we keep the tokens up to $i-1$, accept the target model's correction for token $i$, and discard the rest of the draft.

In the best-case scenario, where the draft model is highly accurate, we can generate $K+1$ tokens in the time it usually takes to generate one. Even in the worst-case scenario, we still get at least one token per pass, ensuring that the output quality remains identical to the target model alone.

Selecting the Right Draft Model

Choosing a draft model is a balancing act. If the model is too small, its predictions will be inaccurate, leading to low acceptance rates and minimal speedup. If it is too large, the overhead of running the draft model will negate the gains from parallel verification.

Architecture Alignment

For the best results, the draft model should ideally share the same tokenizer and architectural quirks as the target model. If you are serving Llama-3-70B, using Llama-3-8B or a distilled version like TinyLlama (if compatible) is a common choice. If the tokenizers differ, you have to perform costly re-encoding between steps, which usually kills the performance gains.

N-Gram and Medusa Approaches

Beyond using a second Transformer model, there are other drafting strategies:

• N-Gram Predictors: vLLM supports using a simple N-gram lookup as a draft "model." This is surprisingly effective for repetitive text like code or structured JSON.
• Medusa/Eagle: These are "multi-head" approaches where the target model itself is modified with extra output heads to predict future tokens. While highly efficient, they require specific model architectures and weights that support these heads.

Implementing with vLLM

vLLM has become the go-to framework for LLM serving because of its PagedAttention algorithm, which manages KV cache memory efficiently. Recently, it has integrated robust support for speculative decoding.

Basic Configuration via CLI

To start a vLLM instance with speculative decoding, you specify the --speculative-model flag. For example, if you want to serve Llama-3-70B (target) using Llama-3-8B (draft), your command would look like this:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B \
    --speculative-model meta-llama/Meta-Llama-3-8B \
    --num-speculative-tokens 5 \
    --tensor-parallel-size 4

In this setup:

• --num-speculative-tokens 5: This tells the draft model to look 5 tokens ahead.
• --tensor-parallel-size 4: This distributes the 70B model across 4 GPUs.

Python API Implementation

If you are integrating vLLM directly into your application code, you can configure it via the SamplingParams and LLM engine classes:

from vllm import LLM, SamplingParams

# Initialize the engine with speculation
llm = LLM(
    model="meta-llama/Meta-Llama-3-70B",
    speculative_model="meta-llama/Meta-Llama-3-8B",
    num_speculative_tokens=5,
    tensor_parallel_size=4
)

prompt = "Write a Python function to calculate the Fibonacci sequence."
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=100)

outputs = llm.generate([prompt], sampling_params)

for output in outputs:
    print(f"Generated text: {output.outputs[0].text}")

Production Performance: What to Expect

Implementing speculative decoding isn't a guaranteed 2x speedup. The actual gains depend on the Acceptance Rate—the percentage of draft tokens that the target model accepts.

1. Deterministic Tasks (Code, Math): These usually see the highest gains. Because there is often a "correct" next token, the draft model is more likely to match the target. We often see 2.0x to 2.5x speedups here.
2. Creative Writing: High temperature sampling (e.g., temperature=1.2) reduces the acceptance rate because the "correct" next token is less predictable. Speedups may drop to 1.2x or 1.5x.
3. Batch Size: Speculative decoding is most effective at low batch sizes (e.g., 1-4 concurrent requests). As batch size increases, the GPU becomes more compute-bound and less memory-bound, meaning the overhead of the draft model starts to outweigh the benefits of speculation.

Operational Challenges and Trade-offs

As with any optimization, there is no free lunch. Deploying speculative decoding in production introduces several complexities.

VRAM Overhead

You must fit both models in memory. If your target model already consumes 90% of your available VRAM, adding a draft model might trigger Out-Of-Memory (OOM) errors or force you to reduce your KV cache size, which in turn reduces your maximum concurrency.

The Acceptance Rate Trap

If your draft model is poorly matched to your target model, the acceptance rate might fall below 10-20%. In this scenario, you are paying the computational cost of running the draft model for almost no gain. It is critical to monitor the vllm:spec_decode_acceptance_rate metric in your Prometheus/Grafana dashboards. If it drops too low, you are better off disabling speculation to save compute resources.

Latency vs. Throughput

Speculative decoding is primarily a latency optimization. It reduces the Time Per Output Token (TPOT). However, it can actually decrease total system throughput in high-load scenarios. Because the draft model consumes GPU cycles, a system running at 100% utilization might process fewer total requests per minute with speculative decoding enabled than without it.

Decide based on your product requirements: Do your users need the first word faster, or do you need to serve the maximum number of users on a single cluster?

Practical Recommendations for Engineers

If you are ready to implement this in your stack, follow this checklist:

1. Baseline First: Measure your current TPOT and throughput without speculation.
2. Pair Wisely: Use a draft model from the same family. For Llama-3, use a smaller Llama-3. For Mistral, use a smaller Mistral.
3. Start Small: Set num_speculative_tokens to 4 or 5. Going higher (e.g., 10) often leads to diminishing returns as the probability of a draft error increases exponentially with sequence length.
4. Monitor Metrics: Keep a close eye on the ratio of accepted tokens. If you see it dipping during specific types of queries, consider using vLLM's conditional logic to enable/disable speculation based on the request type.
5. Test at Scale: Don't just benchmark with one user. Run a load test to see how the extra VRAM and compute requirements affect your system under pressure.

Conclusion

Speculative decoding is one of the most effective techniques for reducing LLM inference latency in production environments. By leveraging vLLM's implementation, you can significantly improve the user experience of your AI applications, making them feel more responsive and "real-time."

However, it is not a silver bullet. Its success depends heavily on the alignment between your draft and target models and the nature of your workload. Start by benchmarking your specific use case, focus on the acceptance rate, and ensure your hardware has the VRAM headroom to support the additional model. When tuned correctly, speculative decoding can turn a sluggish 70B model into a snappy, production-ready powerhouse.