Answer

Local LLM or hosted API: how to choose

Data sensitivity, volume, latency, and the capability the job needs decide it — not ideology. What each option costs, and why many businesses run both.

Published July 2, 2026

It is an engineering decision, not an ideology. Data sensitivity, volume, latency, and the capability the job actually needs all weigh in. Plenty of work is better served by a frontier model behind an API with cost ceilings and fallbacks; some of it can never leave your walls. Many businesses end up running both, routed by task.

Local open-weight models compared with hosted frontier APIs
	Local open-weight	Hosted frontier API
Where your data goes	Nowhere — it stays inside your boundary	To a third party, under their terms
Cost shape	Hardware up front, then near-zero per use	Per token, forever, scaling with success
Capability ceiling	Good and improving; below the frontier	The strongest reasoning available
Latency	Predictable, no network hop	Network-dependent, provider-dependent
Who is on call	You — it's your infrastructure	The provider, within their SLA
Model changes	Only when you choose to upgrade	Under you, on the provider's schedule
Best for	Sensitive data, steady high volume	Hard reasoning, spiky or low volume

Start with the data boundary, not the model

The first question is not which model is smarter — it is whether the data is allowed to leave. That single constraint resolves most of these decisions before capability ever enters the conversation. Contracts, medical records, unreleased financials, and anything under a confidentiality obligation you signed on someone else's behalf all point the same direction.

Local deployment is what makes that architectural rather than contractual. A hosted provider promising not to train on your inputs is a policy, and policies are renegotiated, reinterpreted, and occasionally breached. A model running on hardware you own with no route out is a property of the system. When compliance makes a promise insufficient, that difference is the whole point.

Then look at volume, because the cost curves cross

Hosted APIs are cheap to start and expensive to succeed with. Per-token pricing is close to free while you are piloting one workflow, and it scales linearly with adoption — which means the bill grows fastest exactly when the thing is working. Local inference inverts that: real money up front for hardware, then a marginal cost near zero no matter how much you use it.

So the honest version of the question is where your steady-state volume lands, not what the pilot costs. Low or spiky volume favors hosted. High, predictable, everyday volume — document extraction on every inbound order, classification on every ticket — is where owning the hardware starts paying for itself.

Capability is the argument for hosted

Open-weight models have closed much of the gap and keep closing it, but the frontier is still the frontier. For genuinely hard reasoning, long-context analysis, and tasks where quality differences are obvious to the reader, a hosted model does work that a mid-sized local one does not.

The engineering answer is to treat that model like any other production dependency rather than a magic service: a cost ceiling enforced in code, a fallback for when the provider has a bad morning, logging you can audit, and evaluations run against your real data before customers ever see an answer.

Decision record

Context: A client project needed plain-English search across years of contracts and internal records. The documents were covered by confidentiality terms that made sending them to a third-party API a non-starter, regardless of the provider's training policy.
Options considered: Hosted frontier API with a no-training agreement and redaction before sending
Open-weight model deployed on hardware the client controls
Hybrid — local retrieval over the documents, hosted model for the final answer
Decision: Open-weight model on their own infrastructure, with the retrieval index and the internal tools inside the same boundary. Private by architecture rather than by policy, and no per-token bill on a workload that runs all day.
Trade-off accepted: A lower capability ceiling than a frontier model, and the operational burden moves in-house — hardware, updates, and uptime become theirs. Worth it here because the constraint was absolute; on a workload without that constraint I would have started hosted.

Most businesses end up with both

Routing by task is usually the right final architecture, not a compromise. Sensitive, high-volume, well-defined work runs locally; the occasional hard reasoning problem goes to a frontier model with a cost ceiling on it. The decision stops being a one-time platform bet and becomes a per-workflow question with a defensible answer each time.

What matters more than either choice is that the model gets held to the same bar as everything else you run in production — error handling that assumes it will sometimes be wrong, evaluations on your own data, and a human left in the loop wherever a confident mistake would be expensive. That is how AI integration survives contact with real volume, and it is the same discipline behind deciding whether to build or buy software at all.

Questions

Is our data used to train anyone's models?

With local open-weight models the question doesn't arise, because nothing leaves your infrastructure. With hosted APIs, it depends on the terms you sign: business and enterprise agreements from the major providers exclude training on your inputs, while consumer tiers often don't. Wire the integration to send only what each task needs rather than whole records by default.

What is an air-gapped AI deployment?

A setup where the model runs entirely on infrastructure you control, with no route to the outside world. Open-weight models make it practical: contracts, records, and customer data never touch a third-party API, and there are no per-token bills. It's the right architecture when compliance or confidentiality makes "we promise not to look" insufficient.

What hardware does a local model need?

It scales with the model you pick and how many people use it at once. A mid-sized open-weight model serving a department runs comfortably on a single workstation-class GPU; larger models and heavier concurrency want dedicated server GPUs. Size the hardware after the pilot has shown which model actually clears your quality bar — not before.

Related notes

More on ai integration

Related service: AI Integration

Want this kind of engineering on your project?

Tall Karol takes on fractional and project-based engagements for startups and agencies.

Book a working session