Practical engineering

How much GPU memory does a 70B model really need?

A weights-only calculation is a useful first filter. A deployment decision needs context length, concurrency and a measured memory budget.

2026-09-11 · Arvica Cloud · Analysis & buying guide

Start with an explicit calculation

For a hypothetical model with exactly 70 billion parameters, storing every weight at two bytes takes 140 billion bytes: 140 GB in decimal units, approximately 130.4 GiB. At an ideal packed four bits per weight, the raw payload is 35 GB, approximately 32.6 GiB. These are arithmetic estimates for weights only, not required device capacity. Quantisation metadata, non-quantised tensors and the particular checkpoint format change the actual allocation. Do not describe a card as suitable solely because its advertised memory exceeds the smaller number.

Why a successful start is not a capacity test

vLLM documents how KV-cache pressure and scheduling settings affect serving; insufficient cache can cause requests to be preempted and recomputed. [1] Our procurement recommendation is to test the busiest realistic request mix after loading the model. Include long prompts, generated output and simultaneous users. Reserve a separate line in the budget for runtime allocations. A single short demonstration request says little about a document assistant receiving several large documents together.

Use a small test matrix

Choose a short and a long prompt length from your own traffic, then test low and expected peak concurrency at each length. Keep model revision, output limit and precision fixed. Record startup peak memory, steady-state memory, request failures, time to first token and output speed. Repeat after changing one setting. This is a proposed test plan, not an Arvica benchmark: no configuration or speed is certified by the arithmetic above.

What to send with your GPU enquiry

Send the exact model repository and revision, quantisation format, longest supported input, expected concurrent requests and response-time target. If you already have an out-of-memory log, include a redacted excerpt with the software version. Ask for an exact configuration and a trial acceptance criterion, rather than asking only for a GPU model. If quality changes under quantisation, compare business-task accuracy before accepting the cheaper memory footprint.

Sources

Reviewed: 2026-09-11

vLLM — Optimization and Tuning ↗

Apply this to your workload

Explore the next step →Request a compute quote →