# Local or cloud: when a model on your own device is enough Decide by what the data is and what the task needs, and check by hand whether a model will fit in your computer's memory, using a rule of thumb of about 0.6 GB per billion parameters. Guides · v1.1 · draft · https://hayzlabs.com/guides/local-or-cloud/ In short: - Go local when the data should stay on your device, you are offline, or simple tasks are many and you own the hardware. - Use a cloud model for the hardest reasoning, long agent tasks, or when you have no suitable hardware. - Check fit at about 0.6 GB per billion parameters at 4-bit plus context, then run five real tasks both ways. ## Short answer Decide by what the data is and what the task needs. Run a model on your own device when the data should stay there, when you are offline, or when the tasks are simple and numerous and you already own the hardware. Use a cloud model for the hardest reasoning, for long agent tasks, and when you have no suitable hardware. A model has to fit in your memory first. Our rule of thumb for that is about 0.6 GB per billion parameters at 4-bit, plus room for context. The decision rules, the fit rule and the five-task test below are our suggestions, not results from a study. **Running example (illustrative).** Tomas is a freelance bookkeeper with client files and a laptop with 32 GB of unified memory. He wants to know whether he could run a model there instead of pasting figures into a chat. ## How far behind is local As of August 2025, a consumer GPU under $2,500 (an RTX 5090) could run open models matching the frontier of 6 to 12 months earlier ([Epoch](https://epoch.ai/data-insights/consumer-gpu-model-gap), benchmark comparison). In practice a local model can be good enough for many tasks and is still behind the best cloud model on the hardest ones. Tomas can expect a local model to summarise an invoice list, and should not expect it to match the newest cloud model on a hard tax question. ## Will it fit Our rule of thumb: about 0.6 GB of memory per billion parameters for a 4-bit model, about 1 GB at 8-bit and about 2 GB at 16-bit, plus room for context. Ollama's Llama 3.1 downloads are 4.9 GB (8B) and 43 GB (70B) at 4-bit ([Ollama library](https://ollama.com/library/llama3.1/tags), vendor file sizes), and the rule matches both closely (8 x 0.6 = 4.8, 70 x 0.6 = 42). Two further assumptions of ours: add 1.5 GB for a short context or 6 GB for a long one, and count only 80% of a graphics card's memory, or 70% of a Mac's unified memory, as usable. ```text title="Will it fit? by hand · v1.0" Need = x 0.6 + = Usable = x = It fits if the need is at or below the usable memory. ``` Replace the following: - ``: the model's size in billions of parameters, such as `32`. - ``: `1.5` for short chats, `6` for long documents. - ``: your result for the first line. - ``: your graphics card's memory, or your Mac's total memory, such as `24`. - ``: `0.8` for a graphics card, `0.7` for unified memory. - ``: your result for the second line. For 8-bit or 16-bit, use 1 or 2 in place of 0.6. A worked table at 4-bit with a short context: | Model size | Weights (x 0.6) | Need with short context | Smallest memory, graphics card (x 0.8) | Smallest memory, unified (x 0.7) | |---|---|---|---|---| | 8B | 4.8 GB | 6.3 GB | 7.9 GB | 9.0 GB | | 32B | 19.2 GB | 20.7 GB | 25.9 GB | 29.6 GB | | 70B | 42 GB | 43.5 GB | 54.4 GB | 62.1 GB | Round up to the sizes on sale. With a long context, add 4.5 GB to the need and recompute. The calculator does the same sums, starting on Tomas's laptop. Nothing you type is stored or sent. For Tomas: 32 GB of unified memory gives 22.4 GB usable, so a 32B model at 4-bit with a short context (20.7 GB) fits with little room. An 8B model fits easily. A 70B model does not. A model that fits may still run too slowly to use, so try it on a real task before you plan around it. ## What it costs Local use has no per-token bill, but the hardware and electricity are costs. An illustration with assumed numbers: a 450 W machine running 2 hours a day at $0.17 per kWh uses 0.45 kW x 2 hours x 30 days = 27 kWh, about $4.59 a month. Put in your own wattage, hours and rate, and compare it with what the same tasks cost you on a cloud plan. ## Deciding | Situation | Lean local | Lean cloud | |---|---|---| | Data you may not share, and your plan's terms do not cover it | Yes | Or a business plan with terms that cover it | | The hardest reasoning, or long agent tasks | | Yes | | Offline or travelling | Yes | | | High volume of simple tasks, such as tagging or extraction | Yes, if you own suitable hardware | Yes, with a small cheap model | | No suitable hardware | | Yes | > [!OUR RULE] > Sort what you would paste using the three tiers in [What not to paste](https://hayzlabs.com/start/what-not-to-paste/). The "Never" tier stays out of every chat, local or not. "Redact first" material can go to a local model, or to a business plan your employer approved, when you cannot redact it. "Fine" material can go anywhere. Then run your own tasks both ways before you rely on local. For work material, your employer's AI policy comes first. **The five-task test.** Pick five real tasks, run each on the local model and on the cloud model you use, and compare the answers yourself. If the local model fails two or more of the five, stay on cloud for those tasks. [Test any new model on your own tasks](https://hayzlabs.com/workflows/personal-eval-set/) is the longer routine. > [!CHECK] > Before you rely on a local model, open its settings and check what it sends out. "Local" apps may still connect to the internet, and a local model with tools can still be fed hostile text. ## Limits > [!LIMIT] > The memory rule is a rule of thumb, and real needs vary with the model, the format and the software. The context add-ons and the usable fractions are our assumptions. The lag finding is a snapshot from August 2025. Privacy terms of cloud plans differ and change, so check the live terms of the plan you use. ## Try this next 1. Run the fit check for your computer, with the calculator or by hand. (5 minutes) 2. Sort the next three things you would paste into the three tiers. (3 minutes) 3. See [Do I need to pay for AI, and for which plan?](https://hayzlabs.com/setup/do-i-need-to-pay-for-ai/) before you pay for a plan or a bigger machine. ## Sources 1. Epoch AI. Consumer GPUs and the frontier, August 2025. Benchmark comparison. [Link](https://epoch.ai/data-insights/consumer-gpu-model-gap) 2. Ollama. Llama 3.1 tags with download sizes. Vendor file sizes. [Link](https://ollama.com/library/llama3.1/tags)