Choosing hardware for local or on-prem models

What to look at

Running a model on your own hardware keeps drawings, quotes and contracts inside the building. The trade-off is that you choose and pay for the machine, and the right one depends on which model you run and what you ask of it.

Decide on the model and the longest document people will send it, then size the hardware. Three specs matter:

• Memory decides which models fit. If a model doesn't fit, it won't run, or it runs very slowly.
• Memory bandwidth decides how fast answers are written.
• Compute decides how long people wait before the answer starts, which matters most for long documents.

Try it below: pick a machine, a model, a task and how many people will use it at once, then press Run. Compare another setup puts two machines side by side. Type your own prompt and the chosen model answers it, played back at the speed that machine would write it.

Types of hardware

Workstation GPUs (NVIDIA RTX 4090, 5090, PRO 6000) are the fastest option for models that fit. Most have 24 to 32 GB of memory, enough for small and mid-sized models. The RTX PRO 6000 has 96 GB and can run large models like GPT-OSS 120B.

Unified-memory computers (Mac Studio, AMD Strix Halo, NVIDIA DGX Spark) have 128 to 512 GB of memory, so they fit much larger models, but they run them more slowly. Macs let the GPU use about 75% of memory, so a 128 GB Mac has about 96 GB for the model.

Datacenter GPUs (NVIDIA H100, H200) are included for comparison with cloud AI services.

What affects the choice

Model design. Some large models, called mixture-of-experts, only use a small part of themselves for each word. They need a lot of memory but answer quickly, which suits unified-memory computers: a Mac Studio M5 Ultra writes GPT-OSS 120B at a measured 97 to 121 tokens per second. Dense models use all of themselves for every word, so they slow down as they grow. Qwen 3.6 27B is estimated at 5 to 15 tokens per second on Strix Halo.

Long documents. Before answering, the model reads the whole prompt. For the 12,500-token supply contract example, GPT-OSS 120B takes about 17 seconds on Strix Halo, 12.5 seconds on a Mac Studio M5 Ultra and 1 second on an RTX PRO 6000. If your team will regularly send contracts, specs or long quote packages, weigh this heavily.

Several people at once. One machine can serve a team by handling everyone's requests together. Total output goes up, but each person's answer arrives more slowly, and each conversation takes extra memory. With 8 people each sending a 1,800-token quote review to GPT-OSS 120B, the simulator projects 64 tokens per second each on an RTX PRO 6000, against 37 on a Mac Studio M5 Ultra, with the answer starting after 0.8 and 7.4 seconds.

Models that don't quite fit. A model slightly too big for the GPU can run with part of it in the computer's regular memory, but every word slows down. A smaller version of the same model that fits (see quantization in the glossary) is usually the better choice, and the simulator suggests one.

How the numbers work

Each estimate is labelled by how it was produced. Measured means benchmarked on that exact hardware, model and quantization. Derived means estimated from benchmarks of similar setups. Projected means calculated from hardware specs alone.

Without a benchmark, the simulator calculates:

• Writing speed from memory bandwidth and the share of the model used per token.
• Reading speed for the prompt from the GPU's compute (FP16 TFLOPS).
• An efficiency factor for each software platform, calibrated on the benchmarks, with a slowdown for longer conversations.
• Models that don't fit from GPU speed for the part on the GPU and regular memory speed for the rest.
• Several people by starting from the single-person speed and adding the extra memory reads and calculations each additional person needs. These are always projected and assume server software that handles requests together, such as vLLM or llama-server.
• Memory fit from the model, each person's KV cache and a small overhead, against 75% of memory on Macs or 93% on other GPUs.

Open How we calculated this under the simulator to see each step. Derived and projected ranges can be wide, so lean on measured numbers before buying.

Glossary

• GPU: graphics processing unit. A chip that does many calculations at once, which makes models run much faster than on a regular processor (CPU).
• Memory (VRAM): the GPU's own memory, where the model sits while it runs.
• Unified memory: one pool of memory shared by the processor and GPU, as on Macs. It allows much larger models, usually at lower speed.
• Memory bandwidth: how quickly the GPU can read from its memory, in GB per second. The main limit on how fast answers appear.
• Compute: how many calculations the GPU can do per second, in TFLOPS. The main limit on how quickly a long prompt is read.
• Model: the AI that answers questions, such as Llama or GPT-OSS. Open models like these can be downloaded and run on your own hardware.
• Parameters: the numbers a model learned during training, counted in billions (B). More parameters usually means a more capable model that needs more memory.
• Quantization: storing a model's numbers with less precision so it takes less memory and runs faster, at a small cost in quality. Q4 keeps about 4 bits per number; FP16 keeps 16. Q4_K_M is a common balance of size and quality.
• Token: the unit a model reads and writes, about three quarters of a word.
• Tokens per second (tok/s): how fast the answer appears. About 20 is faster than most people read.
• Prompt: the question or document sent to the model.
• Context: everything the model is holding in mind: the prompt plus the answer so far.
• Time to first token (TTFT): the wait between sending a prompt and seeing the first word, mostly spent reading the prompt.
• KV cache: the model's working memory for one conversation. It grows with the length of the context, and each person needs their own.
• Dense model: a model that uses all its parameters for every token.
• Mixture-of-experts (MoE): a model split into specialist parts, of which only a few are used for each token. It needs memory for the whole model but runs at the speed of a much smaller one.
• Partial offload: running a model that doesn't fit by keeping part of it in the computer's regular memory, which is much slower.
• Batching: serving several people in the same step, so the hardware does more total work while each person's answer comes a little slower.
• Backend: the software layer that runs the model on a given chip: CUDA for NVIDIA, Metal for Apple, ROCm for AMD, Vulkan for most others.

Sources

Data

• llmspeed: the speed model and dataset this tool adapts, under the MIT License.
• r/LocalLLaMA, LocalScore and the llama.cpp discussions: community benchmarks behind the measured figures.

Generation and prompt speed

• Ma and Patterson, Challenges and Research Directions for Large Language Model Inference Hardware, 2026: generation is limited by memory and interconnect more than compute.
• Chen, Memory-Bound but Not Bandwidth-Limited, 2026: single-user generation falls short of peak bandwidth because of fixed per-step overhead.
• Usami et al., Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators, 2026: GPUs lead on prompt processing, which is compute-bound.
• Davies et al., LIMINAL: Exploring the Frontiers of LLM Decode Performance, 2025: an analytical model of generation limits from hardware specs.
• Pope et al., Efficiently Scaling Transformer Inference, 2022: the KV cache and the split between memory-bound generation and compute-bound prompt processing.

Hardware

• Knoop and Holtmann, Private LLM Inference on Consumer Blackwell GPUs, 2026: RTX 5060 Ti, 5070 Ti and 5090 benchmarked for on-prem use in small and mid-sized companies.
• Rajesh et al., Production-Grade Local LLM Inference on Apple Silicon, 2025: MLX, llama.cpp, Ollama and others compared on a Mac Studio.
• Apple, recommendedMaxWorkingSetSize: the share of unified memory the GPU can use.

Mixture-of-experts

• Oncescu et al., Opportunistic Expert Activation, 2025: generation latency follows the number of activated experts, which grows with the number of people in a batch.
• Jiang et al., Mixtral of Experts, 2024: total versus active parameters.

Several people at once

• Kwon et al., PagedAttention, 2023: KV cache memory per request in vLLM.
• Agrawal et al., Sarathi-Serve, 2024: how prompt processing delays other people's first token.
• Agrawal et al., On Evaluating Performance of LLM Inference Serving Systems, 2025: common flaws in published serving benchmarks.