Local or cloud: when a model on your own device is enough

Short answer

Decide by what the data is and what the task needs. Run a model on your own device when the data should stay there, when you are offline, or when the tasks are simple and numerous and you already own the hardware. Use a cloud model for the hardest reasoning, for long agent tasks, and when you have no suitable hardware. A model has to fit in your memory first. Our rule of thumb for that is about 0.6 GB per billion parameters at 4-bit, plus room for context.

The decision rules, the fit rule and the five-task test below are our suggestions, not results from a study.

Running example (illustrative). Tomas is a freelance bookkeeper with client files and a laptop with 32 GB of unified memory. He wants to know whether he could run a model there instead of pasting figures into a chat.

How far behind is local

As of August 2025, a consumer GPU under $2,500 (an RTX 5090) could run open models matching the frontier of 6 to 12 months earlier (Epoch, benchmark comparison). In practice a local model can be good enough for many tasks and is still behind the best cloud model on the hardest ones. Tomas can expect a local model to summarise an invoice list, and should not expect it to match the newest cloud model on a hard tax question.

Will it fit

Our rule of thumb: about 0.6 GB of memory per billion parameters for a 4-bit model, about 1 GB at 8-bit and about 2 GB at 16-bit, plus room for context. Ollama's Llama 3.1 downloads are 4.9 GB (8B) and 43 GB (70B) at 4-bit (Ollama library, vendor file sizes), and the rule matches both closely (8 x 0.6 = 4.8, 70 x 0.6 = 42).

Two further assumptions of ours: add 1.5 GB for a short context or 6 GB for a long one, and count only 80% of a graphics card's memory, or 70% of a Mac's unified memory, as usable.

Will it fit? by hand · v1.0

Need = <PARAMS_B> x 0.6 + <CONTEXT_GB> = <NEED_GB>
Usable = <MEMORY_GB> x <USABLE_FRACTION> = <USABLE_GB>
It fits if the need is at or below the usable memory.

Replace the following:

<PARAMS_B>: the model's size in billions of parameters, such as 32.
<CONTEXT_GB>: 1.5 for short chats, 6 for long documents.
<NEED_GB>: your result for the first line.
<MEMORY_GB>: your graphics card's memory, or your Mac's total memory, such as 24.
<USABLE_FRACTION>: 0.8 for a graphics card, 0.7 for unified memory.
<USABLE_GB>: your result for the second line.

For 8-bit or 16-bit, use 1 or 2 in place of 0.6. A worked table at 4-bit with a short context:

Model size	Weights (x 0.6)	Need with short context	Smallest memory, graphics card (x 0.8)	Smallest memory, unified (x 0.7)
8B	4.8 GB	6.3 GB	7.9 GB	9.0 GB
32B	19.2 GB	20.7 GB	25.9 GB	29.6 GB
70B	42 GB	43.5 GB	54.4 GB	62.1 GB

Round up to the sizes on sale. With a long context, add 4.5 GB to the need and recompute.

The calculator does the same sums, starting on Tomas's laptop. Nothing you type is stored or sent.

For Tomas: 32 GB of unified memory gives 22.4 GB usable, so a 32B model at 4-bit with a short context (20.7 GB) fits with little room. An 8B model fits easily. A 70B model does not.

A model that fits may still run too slowly to use, so try it on a real task before you plan around it.

What it costs

Local use has no per-token bill, but the hardware and electricity are costs. An illustration with assumed numbers: a 450 W machine running 2 hours a day at $0.17 per kWh uses 0.45 kW x 2 hours x 30 days = 27 kWh, about $4.59 a month. Put in your own wattage, hours and rate, and compare it with what the same tasks cost you on a cloud plan.

Deciding

Situation	Lean local	Lean cloud
Data you may not share, and your plan's terms do not cover it	Yes	Or a business plan with terms that cover it
The hardest reasoning, or long agent tasks		Yes
Offline or travelling	Yes
High volume of simple tasks, such as tagging or extraction	Yes, if you own suitable hardware	Yes, with a small cheap model
No suitable hardware		Yes

For work material, your employer's AI policy comes first.

The five-task test. Pick five real tasks, run each on the local model and on the cloud model you use, and compare the answers yourself. If the local model fails two or more of the five, stay on cloud for those tasks. Test any new model on your own tasks is the longer routine.

Limits

Try this next

Run the fit check for your computer, with the calculator or by hand. (5 minutes)
Sort the next three things you would paste into the three tiers. (3 minutes)
See Do I need to pay for AI, and for which plan? before you pay for a plan or a bigger machine.

Sources

Epoch AI. Consumer GPUs and the frontier, August 2025. Benchmark comparison. Link
Ollama. Llama 3.1 tags with download sizes. Vendor file sizes. Link

Changes

v1.010 Oct 2026first version