A local LLM benchmark is a repeatable test that measures what matters for a particular machine, model, and workload. The strongest answer is to test several dimensions: answer quality, time to first token, generation speed, peak memory use, context capacity, stability, energy use, and operating cost. Hardware specifications alone cannot predict these results, because quantization, model architecture, runtime, prompt length, batch size, and backend all affect performance. A 16 GB Mac mini may be adequate for a 7B–8B model at moderate speed, while a 24 GB or 32 GB configuration offers more room for 12B–14B models. On an RTX 5090, a smaller quantized model can produce tokens much faster, but it does not automatically answer better. The purpose of benchmarking is therefore not to identify one universally best setup, but to establish which model is the best fit for the tasks you actually need.", "quick_facts_note": "", "faq_note": "", "faq": [ { "q": "What is a good tokens-per-second target for a local LLM?", "a": "For interactive single-user use, 20–30 tokens per second is generally comfortable because the result begins appearing quickly. Faster is useful for code generation and long documents, but quality, context handling, and reliability matter more than reaching a maximum speed. Speed claims should be reported separately for prompt processing and token generation." }, { "q": "Is a larger local LLM always better than a smaller one?", "a": "No. A larger model may perform better on reasoning, coding, multilingual work, or long-context tasks, but it usually needs more memory and produces answers more slowly. A well-matched 7B or 8B model can outperform an oversized model on a constrained device, especially when the larger model is poorly quantized or configured." }, { "q": "How much RAM or VRAM does a local LLM need?", "a": "A 7B–8B model commonly runs in 16 GB of unified memory or 12–16 GB of VRAM with 4-bit quantization, although context length and the runtime add overhead. A 12B–14B model typically benefits from 24 GB or more. Exact requirements depend on parameter count, quantization format, and how aggressively the runtime keeps the model in memory." }, { "q": "Should I use Q4, Q5, Q8, or FP16 quantization?", "a": "Q4 is usually the practical starting point when memory is limited, while Q5 or Q6 often gives a better quality-to-memory balance on machines with spare capacity. Q8 preserves more of the original model but requires substantially more memory. FP16 or BF16 is preferable when testing a model’s highest-quality behavior and hardware can accommodate it." }, { "q": "Can one local LLM benchmark predict translation quality?", "a": "Not reliably. Translation evaluation should include terminology accuracy, fluency, omissions, additions, formatting, and cultural adaptation in the target language. A model with strong general benchmark scores may still mishandle a specific glossary, language pair, or document format, so a small domain-specific evaluation is necessary." } ], "sources": [ "https://huggingface.co/docs/transformers/llm", "https://ollama.com/", "https://github.com/ggerganov/llama.cpp", "https://simonwillison.net/2025/Jun/16/quantization/" ], "follow_up_keyword": "local LLM benchmarks", "question_placeholder": "", "answer_placeholder": "", "quick_facts": [ { "label": "Category", "value": "Local LLM evaluation" }, { "label": "Timeline", "value": "A serious benchmark usually takes 2–8 hours after setup" }, { "label": "Cost", "value": "Software is often free; hardware ranges from used laptops to multi-GPU workstations" }, { "label": "Best for", "value": "Users choosing a model for privacy, offline work, coding, or translation" } ] }{ "question": "How Do You Benchmark a Local LLM for Speed, Quality, Memory, and Cost in 2026?", "answer": "A local LLM benchmark is a repeatable test that measures what matters for a particular machine, model, and workload. The strongest answer is to test several dimensions: answer quality, time to first token, generation speed, peak memory use, context capacity, stability, energy use, and operating cost. Hardware specifications alone cannot predict these results, because quantization, model architecture, runtime, prompt length, batch size, and backend all affect performance. A 16 GB Mac mini may be adequate for a 7B–8B model at moderate speed, while a 24 GB or 32 GB configuration offers more room for 12B–14B models. On an RTX 5090, a smaller quantized model can produce tokens much faster, but it does not automatically answer better. The purpose of benchmarking is therefore not to identify one universally best setup, but to establish which model is the best fit for the tasks you actually need.
What Should a Local LLM Benchmark Measure?
Also worth reading: How Do You Build a Reliable Translation Quality Benchmark in 2026? · How Do You Build a Local LLM Memory System That Actually Works? · How Does Human-Reviewed AI Translation Improve Quality Without Adding Too Much Cost?
The direct answer is to benchmark the complete system rather than the model in isolation. Start with task quality, because a fast model that gives incorrect or unusable answers is not a good bargain. Use a fixed set of representative prompts, such as factual questions, coding exercises, summarization, extraction, translation, and long-context retrieval. Record the answer alongside the model version, quantization, runtime, prompt template, and system instructions. For generative work, include both correctness and format compliance; for example, a JSON-only task should be tested for valid syntax, required fields, and values that match the source material. A single score from a public benchmark is useful for comparison, but it is not a substitute for testing your own documents and vocabulary.
Measure performance in several ways rather than focusing on one number. Time to first token reflects perceived responsiveness, while generation speed matters when producing long answers. Peak memory use tells you whether the model fits comfortably or spills into slower system memory. Context testing should increase gradually, using 8K, 16K, 32K, and the model’s advertised maximum where applicable. Reliability testing should repeat important prompts at least 10 times, because local runtimes can vary with thermal pressure and concurrent processes. A benchmark that reports only tokens per second is incomplete, and one that reports only a reasoning score says little about whether the model can run on your equipment.
| Feature | Small model, such as 7B–8B | Larger model, such as 12B–14B or more |
|---|---|---|
| Memory demand | Often fits in 16 GB unified memory or 12–16 GB VRAM when quantized | Commonly needs 24 GB or more, depending on format and context |
| Typical strength | Fast responses, private chat, simple extraction, light coding | More complex reasoning, coding, translation, and long documents |
| Main weakness | May lose detail or reasoning accuracy on difficult tasks | Higher latency, cost, and hardware requirements |
| Best benchmark use | Interactive work and constrained devices | Quality comparisons where spare memory is available |
| Practical recommendation | Begin here if hardware is limited | Test only after the small model meets the basic workload |
Create a controlled environment before collecting results. Update the runtime, close unrelated applications, use the same power mode, and ensure the machine has adequate ventilation. If the device supports it, run the benchmark on AC power and avoid simultaneously compiling, rendering, or downloading large files. Choose one established runtime initially, such as llama.cpp through a compatible interface, rather than changing servers between tests. Ollama is convenient for quick local experiments, while a lower-level runtime offers more control over quantization and performance settings. Keep the prompt identical across models, including whitespace, examples, and stop tokens, because even small differences can change the output or the number of generated tokens.
Select prompts before seeing results. A useful test set might contain 20–50 prompts, divided between easy, medium, and difficult examples. Include a few known-answer questions that reveal hallucinations, several coding problems with test cases, and at least five translation or localization cases with terminology rules. Add long-context prompts that place a fact in different positions, since a model may retrieve early information better than information near the end. Use a fixed generation temperature unless you are specifically testing sampling behavior. At medium and high reasoning settings, compare both quality and time rather than assuming that a “thinking” mode is always worth its latency.
The benchmark should also distinguish prompt processing from generation. A model may load a long document quickly but generate the response slowly, or it may begin generating promptly while consuming excessive memory. Record load time separately from inference time. Run at least three repetitions for each major configuration and report the median as the representative result, with the slowest and fastest observations retained for context. On a laptop, thermal throttling can make the first run slower or faster than later runs. The date of the test matters too: as of 26 September 2026, model releases, runtime optimizations, and driver behavior may change, so save the exact versions used in the report.
How to Measure Speed and Responsiveness
Tokens per second is the most familiar speed measure, but it needs context. Report input prompt processing speed, output generation speed, and time to first token separately. For ordinary chat, output speeds around 20–30 tokens per second feel responsive because users can begin reading before the answer is complete. For coding, 40–80 tokens per second can be noticeably more comfortable, while real-time voice or batch processing may require a different threshold. These are practical targets, not guarantees. A 70B model running at 8 tokens per second may be perfectly acceptable for careful analysis, while a 3B model at 100 tokens per second may still fail at the task you care about.
Do not compare a model at 512 tokens with another at 8,192 tokens and call the result a hardware ranking. Context length changes computation, memory pressure, and sometimes attention behavior. Test the same output limit and use realistic prompts, ideally between 500 and 2,000 input tokens for a standard workload. For long documents, compare 4K, 8K, 16K, and 32K context separately, recording whether the model exceeds the requested limit or silently truncates the prompt. Batch processing requires another measurement because throughput can be higher when requests are grouped, but a batch test does not predict the latency of one interactive user. Always state whether results include model loading, prompt evaluation, generation, or only the internal computation loop.
Speed can vary substantially between backends even for the same weights. GPU offload, flash attention, KV-cache precision, and the choice between CUDA, Metal, Vulkan, or CPU execution all influence results. On a unified-memory Mac, the GPU and CPU share bandwidth, so memory capacity and memory bandwidth are often more useful than nominal processor core count. On an RTX 5090, the large memory pool and bandwidth can help larger models, but software support and quantization format still determine actual throughput. A result from a 5090 should therefore be labelled with its model format, context length, and runtime rather than presented simply as “RTX 5090 performance.”
How to Evaluate Quality, Accuracy, and Reliability
Quality evaluation should use a rubric rather than an impression. For factual questions, compare the answer with trusted reference material and mark unsupported claims, omissions, and contradictions. For coding tasks, run generated code against hidden or public tests; for data extraction, compare every field with the source. Translation work needs separate checks for meaning, terminology, grammar, punctuation, and preservation of placeholders. A fluent answer that changes a date, currency, or legal restriction is not accurate. If the application must return JSON, malformed output is a failed result even if the intended content appears elsewhere in the response.
Use multiple scoring methods. Exact-match or regular-expression checks work for names, dates, and labels. An LLM judge can help with style or broad semantic similarity, but it is not a neutral authority and may favor the same kind of wording as the candidate model. Human review remains useful for a small number of high-value cases. A practical local evaluation can combine automated checks with a 1–5 rubric, then report the percentage of prompts that pass all required conditions. Keep the weighting visible: a system that spends 90% of its points on fluency may look strong even when it invents facts. Reliability means not just a high average score but acceptable behavior across repeated runs and different prompt phrasings.
Do not assume that a larger parameter count guarantees better results. Model training, instruction tuning, architecture, quantization, and task fit can outweigh size. Likewise, a local model’s privacy advantage does not remove the need for evaluation. Running locally prevents data from being sent to a hosted API, but it does not prevent sensitive information from appearing in logs, exported chats, backups, or application telemetry. Review the runtime and application settings, especially for tracing and network access. Local deployment is particularly useful for confidential documents, offline operation, predictable costs, and experimenting with translation workflows, but it is not automatically safer if the surrounding software is poorly configured.
Comparing Hardware and Quantization Options
Memory is usually the first practical constraint. A 7B–8B model at 4-bit quantization is a sensible baseline for many 16 GB systems, although the exact footprint depends on the model and context cache. A 12B–14B model often fits comfortably in 24 GB and may need 32 GB for longer contexts or less aggressive quantization. A 70B-class model can run on a high-end workstation, but it will be substantially slower and more expensive to operate than a smaller model. For a new purchase, the relevant question is not how many billions of parameters the marketing material mentions, but whether your chosen model, context length, and application remain responsive under the intended workload.
Quantization trades storage and memory bandwidth against fidelity. Q4 is a common starting point because it reduces weight size substantially. Q5 or Q6 can preserve more capability when memory is available, while Q8 is closer to the original numerical precision and may be appropriate for quality testing. FP16 or BF16 should be treated as a reference when hardware permits. Quantization can change model behavior, not merely file size, so compare the same prompts across at least two levels if quality is important. Do not infer that a Q4 benchmark applies to an FP16 model. Also distinguish quantization of model weights from KV-cache quantization; the latter can affect context performance in a separate way.
| Hardware profile | Models commonly considered | Expected experience | Cost trade-off |
|---|---|---|---|
| 16 GB laptop or Mac mini | 3B–8B, usually quantized | Best for chat, extraction, and light coding; quality depends on model | Lowest hardware cost, but limited upgrade room |
| 24–32 GB unified memory or VRAM | 7B–14B, depending on format | Better reasoning and translation quality with moderate speed | Middle range; often the most practical local setup |
| RTX 5090 with 32 GB VRAM | 7B–30B-class models, format dependent | High throughput and headroom for larger models | Expensive GPU; power and total-system cost rise |
| Multi-GPU workstation | 30B–70B-class models | Higher quality potential, but complex setup and slower generation | High purchase, electricity, and cooling costs |
First, choose one workload and one model family. Download a model from a reputable source, verify its format, and start with a single-user configuration. If you use Ollama, the setup is intentionally simple, but you should still record the exact model tag and quantization. For deeper control, llama.cpp exposes command-line options for context size, GPU layers, threads, and batch settings. Keep those settings constant until the baseline is recorded. A first test should use a short prompt and a small context, such as 1,024 or 2,048 tokens, so that a configuration failure is easy to diagnose before moving to a large document.
Next, create a small evaluation file with prompts, expected facts, and scoring rules. Measure startup time, time to first token, total response time, output length, memory use, and whether the answer satisfies the rubric. Repeat each prompt at least three times, and repeat the most important cases after the machine has been running for several minutes. If the model exposes temperature and top-p, keep them fixed for comparisons. Test a second context size and a second model only after the baseline is stable. This sequence reduces the temptation to collect a large volume of incomparable numbers.
For a practical 2–8 hour evaluation, use a staged plan. Spend the first hour checking installation, model loading, and basic answers. Use the next 1–2 hours for speed and memory tests, followed by 1–2 hours for quality cases and long-context retrieval. A final hour can compare the two best configurations under realistic prompts. The exact duration depends on model size and hardware; a CPU-only 70B model can make the same plan take much longer. Save raw results, not just the final averages, because outliers often reveal thermal throttling, context failures, or unstable quantization.
When to Act and What It May Cost
Act on local benchmarking when privacy, offline access, predictable high-volume use, or model customization matters. A local 7B–8B model on existing hardware can be enough for drafting, summarization, translation assistance, and private document questions. Upgrade toward 24 GB or 32 GB when your evaluation shows that a larger model solves recurring failures. Buy a high-end GPU only if your workload justifies the additional cost or if it is already available. Software such as Ollama and llama.cpp can be free, but the real expense is hardware, electricity, storage, cooling, and the time spent maintaining the system.
Cloud APIs may be more economical for occasional advanced tasks because you pay only for use, while a local GPU represents a fixed investment. Compare that fixed cost against expected volume and latency requirements. Do not assume that a local model is cheaper if you need a new machine solely to avoid a few dollars in occasional API calls. Conversely, a local setup can be economically attractive when it replaces repeated high-volume requests, keeps sensitive material on-site, or supports a workflow that would exceed a hosted service’s limits. In September 2026, pricing and model availability change quickly, so verify current provider and hardware prices rather than relying on an old comparison.
The best time to change systems is when the current one fails a defined threshold: an unacceptable hallucination rate, inability to process a required context length, persistent memory swapping, or an interactive speed below your usable range. If the local model meets those thresholds, a larger GPU may add cost without adding value. If it fails them consistently, compare a better-quantized smaller model, a larger model within available memory, or a hybrid workflow in which local systems handle private or routine work and a hosted model handles difficult exceptions. This is a more defensible decision than choosing equipment based on a leaderboard position.
Common Mistakes and the Best Reporting Format
The most common mistake is benchmarking with unrealistic prompts or changing multiple variables at once. Another is reporting a model’s advertised context length as though it were successfully tested at that length. Some runtimes reserve memory, truncate silently, or become unusably slow near the limit. It is also easy to ignore instruction-following quality, malformed output, and repeated-run stability. Do not compare a model with different system prompts, or treat a single spectacular response as proof of general capability. Finally, avoid publishing a conclusion without stating whether the machine was CPU-only, whether the model was quantized, and whether token speed included prompt processing.
A useful report can be compact but precise. State the date, operating system, hardware, runtime version, model identifier, quantization, context size, generation settings, and test-set composition. Report median and minimum output speed, time to first token, peak memory, and pass rates by task. Include two or three failure examples with enough detail to reproduce them. If results come from a public benchmark, identify the benchmark and version; if they come from a custom set, publish the prompts or describe their categories and scoring rules. This format makes results more useful than a marketing claim such as “runs 100 tokens per second” or “best for AI.”
For AI Translations-related workflows, local evaluation should include language direction, glossary adherence, preservation of HTML or JSON structure, and human review of cultural adaptation. A model that translates prose elegantly may still mishandle a product name, repeated whitespace, or a long table. Test the same document through your actual application, not only the model’s chat interface, because templates, tokenizers, retrieval, and system prompts can change behavior. The decisive result is not the highest score on a general leaderboard; it is the configuration that meets your quality threshold while fitting the hardware, protecting the data, and remaining fast enough for the work.