Edge-Side LLM Inference: Running Local Models with WebLLM and WebGPU

For the past two years, the standard architecture for integrating Large Language Models (LLMs) into web applications has been straightforward: a client-side frontend sends a request to a server-side API (like OpenAI or Anthropic), which then returns a response. While this works, it introduces three significant friction points: high recurring API costs, latency issues, and data privacy concerns.

With the stabilization of WebGPU in modern browsers, we are seeing a paradigm shift. We can now move the inference engine from the cloud directly to the user’s device. By leveraging WebLLM and MLC-LLM, developers can run models like Llama 3, Mistral, and Phi-3 locally in the browser with near-native performance.

This article explores the technical implementation of edge-side LLM inference and why it’s a viable strategy for modern software architecture.

Why Move Inference to the Edge?

Before diving into the code, it’s important to understand the "why." Moving inference to the edge isn't just a technical novelty; it solves several structural problems:

1. Zero Inference Costs: Once the model weights are downloaded to the client's cache, you stop paying for every token generated. For applications with high volume, this is a massive cost reduction.
2. Privacy by Default: Data never leaves the user's machine. This is critical for applications dealing with sensitive medical, legal, or personal data where GDPR or HIPAA compliance is a factor.
3. Offline Functionality: Applications can provide AI features even when the user has no internet connection.
4. Reduced Latency: While initial model loading takes time, the round-trip time to a centralized server is eliminated, allowing for faster token-to-token generation once the pipeline is primed.

The Technical Foundation: WebGPU and MLC-LLM

To run LLMs in the browser, we need a way to talk to the GPU without the overhead of older APIs like WebGL. WebGPU is the successor to WebGL, providing low-level access to the GPU’s compute capabilities. Unlike WebGL, which was designed for graphics, WebGPU is built with general-purpose compute (GPGPU) in mind, making it ideal for the massive parallel matrix multiplications required by transformers.

MLC-LLM (Machine Learning Compilation for LLMs) is the backbone of this ecosystem. It uses the Apache TVM Unity compiler to take models from PyTorch or HuggingFace and compile them into optimized kernels for specific hardware backends, including WebGPU.

WebLLM is the high-level JavaScript/TypeScript library that provides a clean API for interacting with these compiled models. It handles the complexities of memory management, weight sharding, and communication with the WebGPU device.

Setting Up the Implementation

To implement WebLLM, you need a modern browser with WebGPU support (Chrome 113+, Edge 113+, or Safari 17.4+).

1. Installation

First, install the WebLLM package via npm:

npm install @mlc-ai/web-llm

2. Initializing the Engine

In a real-world application, you shouldn't run the LLM on the main UI thread. Even with WebGPU, the initialization and heavy computation can cause frame drops. We will use a Web Worker pattern, but let’s first look at the core logic.

import { CreateMLCEngine, MLCEngine } from "@mlc-ai/web-llm";

async function initializeLLM() {
  // Define the model you want to use
  const selectedModel = "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC";

  // Callback to monitor download progress
  const initProgressCallback = (report) => {
    console.log("Progress:", report.text);
  };

  const engine = await CreateMLCEngine(
    selectedModel,
    { initProgressCallback }
  );

  return engine;
}

3. Handling Quantization

In the example above, notice the suffix q4f16_1. This indicates 4-bit quantization. A standard Llama 3 8B model in 16-bit precision is roughly 15GB—far too large for a browser download. By using 4-bit quantization, the model size is reduced to approximately 4-5GB, making it feasible to store in the browser's IndexedDB cache. This is the sweet spot for edge inference: balancing model intelligence with manageable download sizes.

Architecting for the Web: The Worker Pattern

To keep your application responsive, wrap the WebLLM engine in a Web Worker. WebLLM provides a built-in WebWorkerMLCEngine to simplify this.

worker.ts:

import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => {
  handler.onmessage(msg);
};

main.ts:

import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm";

async function main() {
  const engine = await CreateWebWorkerMLCEngine(
    new Worker(new URL("./worker.ts", import.meta.url), { type: "module" }),
    "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC"
  );

  const chunks = await engine.chat.completions.create({
    messages: [{ role: "user", content: "Explain WebGPU in one sentence." }],
    stream: true,
  });

  let reply = "";
  for await (const chunk of chunks) {
    reply += chunk.choices[0]?.delta?.content || "";
    console.log(reply);
  }
}

Practical Considerations for Production

Memory Management and VRAM

WebGPU doesn't have unlimited access to system RAM. It uses VRAM (Video RAM). On many integrated GPUs (like Apple Silicon M-series), VRAM is shared with system RAM, but the browser still imposes limits.

If you attempt to load a model that exceeds the browser's maxStorageBufferBindingSize or the system's available VRAM, the engine will crash. Always check the device.limits via the WebGPU API before attempting to load larger models. For 8B models, 8GB of system RAM is usually the bare minimum, with 16GB being the recommended baseline.

Persistence and Caching

WebLLM automatically uses the browser's Cache API to store model weights. This means the 4GB download only happens once. However, you should implement a UI that clearly communicates this initial "installation" phase to the user. Treat it like a large game asset download rather than a simple script load.

Model Selection

Not every task requires a Llama 3 8B model. For many browser-based tasks, smaller models are significantly more efficient:

• Phi-3 (3.8B): Excellent for reasoning and logic, very fast on mid-range hardware.
• Gemma-2B: Ideal for simple classification or text summarization with a tiny footprint.
• Mistral-7B: A great all-rounder if the user has a dedicated GPU.

Benchmarking Performance

Performance on the edge varies wildly based on hardware. On an Apple M2 Pro, you can expect Llama-3-8B (4-bit) to generate roughly 20-30 tokens per second. On a mid-range Windows laptop with an integrated Intel GPU, this might drop to 5-10 tokens per second.

While this is slower than a top-tier H100-backed API, it is more than fast enough for human reading speeds. The "Time to First Token" (TTFT) is often lower than API-based solutions because there is no network overhead once the model is loaded.

Security Implications

Running models locally mitigates many security risks but introduces others. You must still sanitize inputs and outputs. Even though the model is local, "prompt injection" can still occur, potentially tricking the application into performing unintended actions if the LLM is connected to local tools or file system APIs (via the File System Access API).

Furthermore, since the model weights are public (downloaded to the client), you cannot hide "system prompts" or proprietary logic inside the model's instructions. If your business logic depends on a secret system prompt, edge-side inference is not the right choice.

Conclusion: The Actionable Path Forward

Edge-side LLM inference is no longer an experimental curiosity; it is a viable architectural choice for privacy-conscious or cost-sensitive applications. To get started:

1. Audit your use cases: Identify features where latency or privacy is more important than using the absolute largest model (like GPT-4o).
2. Start Small: Implement a prototype using the Phi-3 or Gemma-2B models via WebLLM to understand the VRAM constraints of your target audience.
3. Optimize UX: Build a robust loading state that manages the initial weight download and caches it effectively using IndexedDB.
4. Monitor Hardware: Use the WebGPU capability checks to gracefully fall back to a cloud API if the user's device lacks the necessary compute power.

By moving the heavy lifting to the client, you reclaim control over your infrastructure costs and provide a faster, more private experience for your users.