Running a local large language model requires enough GPU memory to hold its model weights, context cache and runtime overhead. As a practical starting point in 2026, 8GB VRAM handles many 7B–8B models at 4-bit quantisation, 12GB accommodates several 14B models, and 24GB can run selected 27B–32B models with shorter contexts. Larger 70B models generally need around 45–55GB or more.
Local LLM VRAM requirements by GPU memory capacity
The most important hardware specification when running an AI model locally is often memory capacity rather than raw GPU processing power. A graphics card can have excellent computational performance but still fail to load a model if its weights and working memory exceed available VRAM. This is why checking memory requirements before buying a GPU can save considerable money.
The following estimates use Q4_K_M quantisation, a commonly used 4-bit model format, and assume a relatively short context window. Actual requirements vary by model architecture, software framework, context length and runtime settings.
| Available VRAM | Practical model range | Typical use |
|---|---|---|
| 8GB | 7B–8B | Chatbots, coding assistance and basic local AI |
| 12GB | 12B–14B | More capable reasoning and general-purpose models |
| 16GB | 14B–20B | Larger models and longer working contexts |
| 24GB | 27B–32B | Advanced local assistants and coding models |
| 48GB | 70B-class at Q4, depending on context | More capable local inference |
| 80GB+ | Larger models and higher precision | Professional AI development and experimentation |
These are practical buying categories, not absolute compatibility limits. A model may technically load when memory is nearly full, but leave insufficient space for longer prompts or other applications. Keeping some headroom is preferable to choosing a GPU that barely accommodates the model weights.
How much VRAM does a local LLM actually need?
A useful starting estimate for Q4_K_M models is approximately 0.6GB per billion parameters, including a modest allowance for memory overhead. For example, an 8B model needs approximately 6.7GB at an 8K context in the LLM Configurator estimates. A 70B model can require approximately 46GB, making it unsuitable for most single-GPU consumer systems.
There are three major components to consider when calculating memory requirements:
- Model weights: The stored numerical parameters that determine the model’s behaviour. Larger models require more memory.
- KV cache: Memory used to retain information about the current conversation and processed context. Its requirements increase as context length grows.
- Runtime overhead: Additional memory used by the inference framework, GPU operations and supporting processes.
These components explain why downloading a 5GB model file does not mean that a GPU with exactly 5GB of VRAM can run it. The model needs additional working memory once it is loaded and begins processing requests.
What is quantisation, and why does it reduce VRAM requirements?
Quantisation reduces the numerical precision used to store model weights. Instead of representing every parameter with a full 16-bit value, a quantised model uses fewer bits, reducing its memory footprint. This allows users to run larger models on relatively affordable graphics cards.
The most common formats include Q4_K_M, Q5_K_M, Q8_0 and F16. Each represents a different balance between memory consumption and numerical precision.
| Quantisation | Approximate memory per billion parameters | Practical trade-off |
|---|---|---|
| Q4_K_M | 0.5–0.6GB | Lower memory requirements |
| Q5_K_M | 0.625GB | Slightly higher precision |
| Q8_0 | 1GB | Higher precision and memory usage |
| F16 | 2GB | Full 16-bit weight representation |
These figures describe approximate weight storage rather than total GPU requirements. Metadata, tensor formats, architecture and runtime overhead can change the actual allocation. Q4_K_M is a popular starting point because it offers a useful balance between model size and output quality.
For example, an 8B model stored in F16 can require approximately 16GB just for its weights, compared with roughly 4–5GB for a Q4_K_M version. Once the context cache and framework overhead are included, the difference becomes even more important for consumer hardware.
How much VRAM do 8B, 14B, 32B and 70B models need?
Model parameter count provides a useful first estimate, but the actual memory requirement depends on quantisation and context length. The following examples use approximate Q4_K_M requirements and an 8K context, drawing on published model-memory estimates rather than treating every model architecture as identical.
| Model size | Approximate VRAM | Recommended GPU capacity |
|---|---|---|
| 3B–4B | 3–4GB | 6–8GB |
| 8B | 6.7GB | 8–12GB |
| 14B | 10–12GB | 16GB |
| 27B–32B | 20–24GB | 24–32GB |
| 70B | 45–50GB | 48–64GB |
8GB VRAM: Getting started with local AI
An 8GB graphics card is a reasonable entry point for running smaller language models. An 8B model such as Llama 3.1 8B Instruct can fit at Q4_K_M with an 8K context, although available headroom depends on the graphics card and other running applications.
Smaller models are useful for summarisation, basic coding assistance, text generation and everyday question answering. They will not consistently match the reasoning capabilities of much larger models, but they provide a practical way to experiment with local inference without purchasing expensive hardware.
12GB VRAM: A useful middle ground
With 12GB, users can explore several 12B–14B models at 4-bit quantisation. These models offer more capacity for complex instructions and longer responses than many smaller alternatives, although performance still depends on the training quality and architecture.
For example, a 14B Q4_K_M model may need approximately 10–12GB at a relatively short context. A 12GB GPU can therefore be close to its limit, making 16GB a more comfortable choice if the budget allows.
24GB VRAM: A strong consumer GPU configuration
A 24GB graphics card opens the door to considerably larger local models. Selected 27B and 32B models can fit at Q4_K_M, although longer context windows may push memory requirements beyond the available capacity.
The NVIDIA RTX 3090 is an example of a consumer GPU with 24GB of GDDR6X memory and a 350W total graphics power rating. It remains useful for local AI experimentation, especially when paired with efficient quantisation and suitable inference software.
However, a 24GB card should not be treated as a guaranteed 32B-model solution at every context length. A 32B model that fits with a short prompt may exceed available memory when processing longer documents or conversations.
48GB and above: Running 70B-class models
Models with approximately 70 billion parameters require substantially more memory. A 70B model at Q4_K_M can need roughly 46GB before additional context and runtime requirements are fully accounted for. A 48GB GPU may therefore be insufficient for some configurations, particularly when the context window is large.
For more comfortable operation, a 64GB or 80GB memory configuration is preferable. Another option is using multiple GPUs, provided the inference software supports distributing model layers or tensors across them.
Why context length can make a model run out of VRAM
Context length determines how much information a language model can consider during a conversation or task. It includes the instructions, previous messages, supplied documents and the text currently being processed. A longer context is useful for analysing large documents or maintaining extended conversations, but it increases memory consumption.
The KV cache stores intermediate attention information so the model does not need to repeatedly calculate everything from the beginning. Its memory requirements vary according to the model’s architecture, number of layers, attention configuration, context length and cache precision.
Consider a 32B Q4_K_M model. One published measurement using Qwen2.5 32B in Ollama recorded approximately 19.9GB of peak GPU memory at 4K context and 27.2GB at 32K context. That difference is enough to turn a model that fits on a 24GB GPU into one that exceeds its capacity.
There are several ways to manage this problem:
- Reduce the context window if the application does not require extensive conversation history.
- Use a quantised KV cache where the inference framework supports it.
- Choose a model architecture with more efficient attention mechanisms.
- Use GPU and system-memory offloading if the additional performance penalty is acceptable.
One important distinction is that a model’s advertised maximum context is not the same as the context your GPU can practically support. A model may support 128K tokens in its configuration while requiring substantially more memory than your hardware can provide.
Can Apple Silicon Macs run local LLMs without dedicated VRAM?
Yes. Apple Silicon Macs use unified memory, which is shared between the CPU and GPU rather than separated into conventional system RAM and dedicated graphics memory. This arrangement allows compatible applications to use a larger common memory pool for local AI inference.
A Mac with 64GB of unified memory can therefore accommodate models that would not fit inside a conventional 24GB graphics card. However, the entire 64GB is not available exclusively to the model. macOS, applications, model weights, context caches and other processes all compete for the same physical memory.
Memory allocation policies also matter. macOS manages how much memory can be wired for GPU use, and advanced users can adjust certain allocation limits. Changing these settings does not create additional physical memory or remove the operating system’s requirements.
Unified memory capacity should also not be confused with GPU performance. A large model may successfully load on a Mac but generate tokens more slowly than on a high-bandwidth discrete GPU. Memory bandwidth, processor generation, software optimisation and model architecture all influence inference speed.
How Ollama handles GPU memory and context limits
Ollama is one of the popular tools for running language models locally. It manages model loading and GPU allocation, making it easier to experiment without manually configuring every inference parameter. However, its default context settings can affect the experience.
Ollama’s documented default context allocation uses available VRAM to determine the initial context window. Its published defaults include:
| Available VRAM | Default context length |
|---|---|
| Below 24 GiB | 4K tokens |
| 24–48 GiB | 32K tokens |
| 48 GiB or more | 256K tokens |
These are software defaults, not guaranteed maximums. Users can configure context length themselves, but doing so increases memory requirements. A 24GB GPU may run a model comfortably at 4K while struggling with the same model at 32K.
Before assuming that a model is using the entire graphics card, users can check its allocation with the ollama ps command. This helps identify whether the model is running entirely on the GPU or whether some of its layers have been placed in system memory.
Do mixture-of-experts models need less VRAM?
Mixture-of-Experts (MoE) models introduce an important distinction between total parameters and active parameters. Instead of using every part of the model for every token, an MoE architecture activates selected expert networks depending on the input. This can reduce computational requirements during inference.
However, fewer active parameters do not necessarily mean a proportionately smaller memory footprint. The model weights for the available experts generally need to remain accessible, even though only some experts participate in processing an individual token.
For example, a hypothetical 109B-parameter MoE model with 17B active parameters cannot automatically be treated as a 17B model when calculating storage requirements. Its total parameter count remains important because the expert weights still need to be stored somewhere accessible to the inference system.
Frequently asked questions
Can I run a local LLM with 8GB VRAM?
Yes. An 8GB GPU can run several 7B–8B models using Q4_K_M quantisation and a relatively short context window. You may need to close other GPU-intensive applications to maintain sufficient memory headroom.
Is 24GB VRAM enough for a 70B model?
No, not for a typical 70B model running entirely on the GPU at Q4_K_M. Such models generally require approximately 45–50GB or more, depending on context length and architecture. You can use multiple GPUs or system-memory offloading, but performance and configuration become more complicated.
Does increasing context length reduce LLM performance?
It can. Longer contexts increase KV-cache memory requirements and may increase prompt-processing time. If the model exceeds available VRAM, the inference framework may offload some operations to system memory, potentially reducing generation speed.
Is 64GB unified memory on a Mac equivalent to 64GB GPU VRAM?
No. Both the CPU and GPU share the Mac’s unified memory, and the operating system and other applications also need space. Nevertheless, unified memory can accommodate larger local models than a discrete GPU with a smaller dedicated VRAM allocation.
Should I buy a GPU with more VRAM or faster processing power?
For local LLM inference, memory capacity should generally be the first consideration because it determines which models and context lengths can fit. Once you have enough memory for your target models, compare memory bandwidth, GPU architecture and software support to determine generation speed.
For local AI in 2026, 8GB is a practical starting point, 12–16GB offers more flexibility, and 24GB provides access to substantially larger models. Running 70B-class models comfortably generally calls for 48–64GB or more, depending on quantisation and context requirements. If you were building a local AI workstation today, would you choose a 24GB GPU for faster inference or invest in 48–64GB of memory to run larger models?
