Own GPUs vs token pricing: what AI on your own hardware really costs
Whether running AI models on your own GPUs pays off depends mainly on volume. Based on our own calculation with list prices from October 2026, a rented GPU in a German data centre pays off against closed top tier models from about 145 to 204 million tokens per month, and against mid range models from about 360 to 550 million. Against low cost models, own operation only pays off at very high volume and utilization. Utilization, data protection and operational responsibility then decide.
Two cost models with different logic
Whether a company uses AI through a cloud API or runs models on its own GPUs is often decided by gut feeling. Yet both options follow a clear logic that can be calculated.
With pricing per token, you pay for every request. Inputs and outputs are measured in tokens, meaning fragments of words, and billed at a fixed price per million tokens. For getting started, this model is ideal: no investment, no operations, no waiting for hardware. You can start the same day. With every new use case, however, the bill grows linearly. An assistant in sales, a document review in purchasing and an agent in customer service add up month after month.
When you run AI on your own GPUs, the logic is reversed. Hardware, power, cooling, maintenance and staff cause fixed costs, regardless of how heavily the system is used. Each additional request, on the other hand, costs almost nothing as long as the existing capacity is sufficient. Running models yourself rewards utilization; pricing per token rewards restraint.
This leads to the central question of every cost comparison: how many tokens does your company consume today, and how many will it be in one to two years?
Where the cost lines cross
If you plot both models against monthly volume, you get a rising straight line for pricing per token and an almost flat line for running models on your own GPUs. The intersection is the break even point. Our analysis in the article "Das lokale KI Betriebssystem" (The local AI operating system) arrives at the following thresholds:
-
Compared with closed top tier models, the threshold is about 145 to 204 million tokens per month.
-
Compared with mid range models with much lower prices per token, the threshold is about 360 to 550 million tokens per month, and above one billion against low cost models.
Looking at the cost structure is revealing. A rented GPU costs the same every month, whether it is busy or idle. Its cost per million tokens therefore falls with every additional request, while per token pricing stays constant. That is why the list price per token alone rarely settles a decision. What tips the balance is volume, utilization, data protection and the question of who is responsible for operations.
For your own situation, it is worth running the numbers with real consumption data. Our cost comparison calculator shows you where your company sits on the curve.
Comparison table: own GPUs vs pricing per token
| Criterion | Running AI on your own GPUs | Pricing per token via cloud API |
|---|---|---|
| Cost structure | Mostly fixed: hardware, power, maintenance, staff | Variable, grows linearly with every request |
| Startup effort | Procurement, setup, security hardening | Low, usable immediately |
| Cost per additional request | Almost zero up to the capacity limit | Constant per million tokens |
| Economical from | About 145 to 204 million tokens per month compared with top tier models, about 360 to 550 million compared with mid range models | Below these thresholds |
| Data control | Data never leaves your own network | Depends on provider, contract and jurisdiction |
| Operational responsibility | Internally or with an operations partner | With the API provider |
| Adaptation to domain language | Fine tuning and custom models possible | Limited, depending on the provider |
| Predictability | High, costs known over the full term | Fluctuates with usage and price changes |
| Dependency | On hardware and operational expertise | On the provider's prices, model versions and availability |
The items missing from the calculation
Many cost comparisons set the price of a GPU against the price per token and stop there. That leaves out exactly the items that make the difference in daily use. Who monitors the system and notices when answers get worse? Who installs updates for drivers, runtime environment and models, and tests them beforehand? Who responds at night or on the weekend when the service is down? Who maintains permissions, logs and documentation for the next audit?
On premise operation moves operational risk into your own company. That is no argument against running AI on your own GPUs, though it is an argument for a complete calculation. Besides hardware, the fixed costs include staff time for operations and on call duty, redundancy or spare parts, energy and cooling, security updates and the maintenance of the applications built on top of the model. Anyone who leaves these items out is comparing an incomplete bill with a complete one.
On the other side, many comparisons also leave out items: data protection reviews for every new use case, contracts with subcontractors and the risk that prices or model versions change. Then there is the legal situation. The US CLOUD Act applies regardless of server location. A European data center operated by a US provider therefore does not automatically protect against access by US authorities.
For companies without their own operations team, there is a middle path. The hardware sits in your own data center or with a German hosting provider, and operations are contractually assigned to a partner. Your data stays within the company without your IT having to be on call at night. We describe how we take over this part under AI operations.
Utilization and software count too
Whether running AI on your own GPUs is economical depends not only on the hardware but on how many requests it processes in a given time. The software used to serve the models plays a bigger role than many expect. A recent scientific measurement shows around 800 tokens per second with vLLM compared with around 150 tokens per second with Ollama at ten concurrent users.
For the cost calculation, this means the same GPU can carry several times the load depending on the software. A tool that is convenient for testing on a single workstation is not automatically suitable for production use with many simultaneous users. Anyone calculating the cost of running models on their own GPUs should therefore not just count GPUs but plan the entire serving setup: runtime environment, queues, load balancing and a gateway that routes requests to suitable models.
A gateway like this also enables hybrid routing. Simple requests run on a small local model, confidential data always stays within your own network, and only selected tasks involving public data go to an external API when needed. This way you use the strengths of both cost models without committing permanently to either one.
Our recommendation in three tiers
From the thresholds and our project experience, we derive a simple tiered approach:
-
Below 5 million tokens per month: Start in the cloud, provided your data allows it. Use the time to measure consumption, use cases and quality.
-
From 10 million tokens per month or with regulated data: Switching to your own GPUs pays off. With personal, confidential or regulated data, it can be the right choice even below this threshold.
-
From 50 million tokens per month: Plan for multiple GPUs, load balancing and hybrid routing so that capacity and costs grow with your use cases.
Around these thresholds lies a gray zone in which the nature of your data, expected growth and internal expertise decide. What matters is not to make the decision once but to monitor consumption continuously. Many companies underestimate how quickly demand rises as soon as the first use case is in production and other departments follow.
The cost question is rarely decided by the price per token. It is decided by volume and by who carries the responsibility for operations.
In practice we do not start with cost, we start with the use case. The order is: first, the planned use case and how it is expected to grow; second, the expected volume in tokens per month; third, the requirements for data residency and independence from a single vendor. Only then do we calculate cost, with the cost calculator or in the assessment.
Frequently Asked Questions
When do your own GPUs for AI pay off?
Compared with closed top tier models from about 145 to 204 million tokens per month, compared with mid range models from about 360 to 550 million. With regulated data, own operation can make sense earlier.
Which costs are often forgotten when running AI on your own GPUs?
Staff time for monitoring and on call duty, tested updates, redundancy, energy and cooling, as well as the maintenance of permissions, logs and documentation.
Do I have to choose one cost model?
No. With a gateway and hybrid routing, local models and external APIs can be combined depending on data class and task. Would you like to know where your company sits on the cost curve? In an initial consultation, we work through your volume, your data situation and your operational needs together with you and show you which option works for you. Book your initial consultation now
Is own operation only worth it above a certain company size?
No. Volume decides, not headcount. A mid-sized company with one use case that processes many documents every day can be above the threshold, a large corporation with a single internal assistant below it. With confidential or regulated data, own operation can be the right choice even below the cost threshold.
Can we switch from per-token billing to own operation later?
Yes. Many companies start with a cloud API and switch once volume grows or sensitive data comes in. The switch needs lead time for hardware or a dedicated environment, model selection and testing; we plan 4 to 10 weeks for it. It is easier if a gateway sits between application and model from day one, because then only the target of the requests changes, not the application.
Sources and References
- NexPatch AI: "Das lokale KI Betriebssystem" (The local AI operating system), https://nexpatch.ai/de/wissen/beitrag/lokales-ki-betriebssystem-eigene-server-infrastruktur (break even compared with premium and budget APIs, 36 month TCO)
- arXiv 2511.17593, throughput measurement of vLLM and Ollama at ten concurrent users, https://arxiv.org/abs/2511.17593
- US CLOUD Act (Clarifying Lawful Overseas Use of Data Act)
Methodology: Basis: our own calculation with list prices from German data centres and model providers, as of October 2026. Blended price of three parts input to one part output, rented GPU with 96 GB memory including operations.
