Why Deploying Open-Source AI Is Often More Expensive Than Renting It

Deploying an open-source large language model is often more expensive than renting a closed one because the cost of intelligence is no longer in the software, but in the silicon and the talent required to keep it running. OpenAI Chief Financial Officer Sarah Friar made this explicit on September 8, 2026, telling the Goldman Sachs Communacopia + Technology Conference that the company is actively undercutting open-source rivals on cost.[1][2]

Friar noted that OpenAI is focusing on specialized sectors like chip design and life sciences, where enterprises demand measurable returns on their artificial intelligence spending. "If you're deploying Luna and compare that to GLM 5.3, for example, on a cloud layer, we are cheaper," Friar said, referring to Z.ai's popular open-weight model. She added that an 80 percent price cut on OpenAI's lower-cost Luna model recently drove a 10-fold increase in usage.[1][2]

The claim strikes at the heart of a long-held technology assumption: that open-source software, being free to download, is inherently cheaper to run. In traditional software, that is largely true. In generative AI, model weights are free, but the infrastructure required to serve them is not.[6]

The math of large language model inference explains the discrepancy. Inference cost is essentially GPU-time per token. According to a September 2026 analysis by CloudZero, output tokens cost roughly five times as much as input tokens across major providers because generating tokens sequentially is harder work than reading them in parallel.[5]

When an enterprise downloads an open-source model, they shift costs from transparent API fees to hidden operational expenses. A March 2026 report by AI Superior found that deploying an open-source LLM costs between $125,000 and $820,000 annually for most organizations.[3]

The talent required to maintain these systems forms a massive portion of that baseline. Minimal internal deployments require at least three to four engineers, while enterprise-scale systems demand 15 or more. With a machine learning engineer costing roughly $200,000 annually, AI Superior notes that a single engineer "needs to save you 6.6 billion tokens worth of API calls just to break even on their salary alone."[3]

Hardware utilization is the other major trap for self-hosted deployments. A GPU-hour costs the same whether it emits 2,000 or 27,000 tokens per second. Cost per token is the hourly rate divided by the realized throughput, making utilization the dominant variable in AI economics.[6]

Cerebrium, an AI infrastructure provider, published internal benchmarks in July 2026 demonstrating this scale. Serving a 120-billion parameter open-source model at 12 million tokens per minute requires approximately 21 instances of dual NVIDIA H100 GPUs. At $0.11604 per minute per instance, that setup costs roughly $105,000 a month at continuous full load.[4]

The problem for most enterprises is that their workloads are not continuous. If those 21 instances sit idle overnight or during low-traffic periods, the cost per generated token skyrockets. Commercial API providers like OpenAI and Anthropic avoid this by pooling traffic across millions of users, keeping their GPUs saturated and their unit costs low.[4][6]

API providers also leverage massive economies of scale in prompt caching. Anthropic, for instance, bills cache reads at a 90 percent discount, dropping the input cost of its Claude Sonnet 4.6 model from $3.00 to $0.30 per million tokens. Open-source deployments can implement caching, but achieving high hit rates requires sophisticated engineering.[5]

CloudZero highlighted a case study from ProjectDiscovery, the company behind the open-source Nuclei security scanner. By moving dynamic working memory out of the system prompt and setting explicit cache breakpoints, the engineering team increased their cache hit rate from 7 percent to 84 percent, cutting total LLM spend by up to 70 percent across 9.8 billion tokens.[5]

Despite the cost disadvantage, open-source AI remains critical for many organizations. The primary drivers are data sovereignty, privacy control, and the ability to fine-tune models for highly specific domain tasks without sending proprietary data to a third-party server.[6]

As Friar noted at the Goldman Sachs conference, open weights have a place in the ecosystem, but customers pay for the holistic stack, compute, data integration, and enterprise reliability. Until the price of specialized silicon collapses, the free open-source model will remain a premium deployment choice for those who can afford the infrastructure to run it.[1][6]