Per-Token Inference Compute Turns Marginal Costs Linear, Capping AI Gross Margins at 50 Percent

For every $100 a traditional software company collects from a subscriber, it spends roughly $15 to keep the servers running. The remaining $85 is gross margin, the structural advantage that made cloud software the most lucrative business model of the last decade. Generative artificial intelligence breaks that math entirely.[1]

When a user asks a conventional application to retrieve a customer record, the software queries a database. The marginal cost of that retrieval is effectively zero. When a user asks an artificial intelligence agent to summarize that same record, the system must actively compute every single word it generates.[1]

That active computation is called inference, and it carries a hard, physical cost. Because every token requires billions of mathematical operations, the cost of delivering the software scales linearly with how much the customer uses it. The more value the application provides, the more expensive it is to run.[1][4]

This linear cost structure is forcing a reckoning across the technology sector. Financial benchmarks from 2026 show that artificial intelligence applications are structurally capped at roughly a 50 percent gross margin. The software industry is attempting to build the next generation of computing on unit economics that resemble a low-margin service business.[1]

The physics of active computation

To understand why artificial intelligence margins are compressed, you have to look at the physical infrastructure required to generate a response. Traditional cloud hosting relies on central processing units that excel at moving data around. Generative models require specialized tensor cores that perform massive parallel matrix multiplications.[1]

"As AI factories grow from megawatts to gigawatts, building them takes more than advanced computing," Nvidia noted during its October 12, 2026, Open Compute Project Global Summit presentation. The company introduced its Vera Rubin architecture specifically to help scale these massive facilities. The physical footprint is staggering.[2]

Building an artificial intelligence factory costs roughly $60 million per megawatt of power capacity. Those capital expenditures flow directly into the price of rented compute. When an application developer pays a cloud provider for application programming interface access, they are paying to amortize those gigawatt-scale data centers.[2]

Every time a language model generates a token—roughly three-quarters of an English word—it must run its entire neural network forward. A one-trillion-parameter model performs a trillion operations just to output a single syllable. If a user requests a thousand-word essay, the model repeats that massive computational effort over a thousand times.[1]

This is why the marginal cost never approaches zero. In classic software, the development is expensive but the distribution is free. In generative artificial intelligence, the distribution requires active, continuous manufacturing. The software is essentially rebuilding itself for every single query.[1][4]

The structural margin ceiling

The financial impact of this compute requirement is now clearly visible in private market benchmarks. The 2026 Aleph and Benchmarkit metrics report, which analyzed 342 private business-to-business companies, found that traditional software gross margins sit firmly at 80 percent. Top-quartile performers achieve 86 percent or better.

Artificial intelligence companies operate in a different reality. According to ICONIQ Growth's 2026 State of AI data, the average gross margin for an artificial intelligence product is currently 52 percent. That figure is up from 41 percent in 2024, but it remains trapped far below the historical software ceiling.

Venture capital firm Andreessen Horowitz previously flagged this dynamic, noting that artificial intelligence businesses typically run at a 50 to 60 percent gross margin. The firm estimated that application companies spend between 20 and 40 percent of their total revenue simply paying for inference and per-customer fine-tuning.[4]

This margin compression is not a temporary inefficiency that will vanish as chips get faster. When Sequoia Capital modeled the economics of the artificial intelligence ecosystem in its updated "AI's $600B Question" analysis in June 2024, the firm explicitly assumed a 50 percent gross margin for the end-user of the graphics processing unit.[3]

The assumption reflects a structural reality. If an application developer buys compute from Microsoft Azure or Amazon Web Services, they must mark up the cost of that compute to make a profit. Because the underlying inference cost is so high, pushing the gross margin above 50 percent requires charging prices that most software buyers will not accept.[1][3]

Why cheaper compute fails

A common counter-argument is that hardware improvements will eventually solve the margin problem. As Nvidia releases faster architectures and cloud providers optimize their data centers, the cost per token drops. However, falling compute costs have historically failed to improve application-level gross margins.[1][2]

This dynamic is a classic example of the Jevons paradox, which states that increasing the efficiency of a resource increases its total consumption. When inference becomes cheaper, developers do not simply pocket the savings. Instead, they deploy larger, more capable models that consume the newly available compute headroom.[1]

"The gap is narrowing as inference gets cheaper, but it is real and structural, so plan for it and never assume software-grade margins," noted Kanika Mathur in a February 2026 Resourcifi analysis of artificial intelligence economics. The drive for better reasoning constantly pushes developers toward more computationally expensive techniques.[4]

Techniques like chain-of-thought reasoning and speculative decoding require the model to generate dozens of hidden tokens for every word it actually shows the user. The cost per token might fall by half, but if the application uses three times as many tokens to deliver a smarter answer, the total inference cost still rises.[1]

As long as intelligence remains the primary competitive axis, companies will spend their margin improvements on larger models. The 50 percent gross margin ceiling is not a product of inefficient hardware; it is the natural equilibrium point where companies balance profitability against the need to provide the smartest possible product.[1][4]

Abandoning the flat rate

The linear cost of inference makes traditional software pricing highly dangerous. For a decade, the industry relied on flat-rate monthly subscriptions. If a user logged into a project management tool once a month or a hundred times a day, the cost to the provider was virtually identical.[1]

In an artificial intelligence application, a heavy user can actively destroy the company's profitability. If a subscriber pays $20 a month but generates enough text to consume $30 in cloud inference costs, the company loses money on every interaction. Flat-rate pricing effectively subsidizes the most demanding customers.[1]

To protect their unit economics, companies are rapidly abandoning the pure subscription model. According to a 2026 monetization survey cited by The SaaS CFO, hybrid pricing models rose from 25 percent to 37 percent adoption in a single year. It is now the most common pricing structure among software startups.

Hybrid models typically combine a stable base subscription with a usage-based credit system. A user might pay $15 a month for access to the platform and a baseline number of queries, but they must purchase additional compute credits for heavy generation tasks. This aligns the company's revenue directly with its variable costs.[1]

By passing the marginal cost of inference directly to the consumer, companies can defend their 50 percent gross margin. However, this shift fundamentally changes the customer relationship. Buyers are no longer purchasing a software license; they are effectively buying metered access to a digital utility.[1]

Compressing valuation multiples

The permanent shift to lower gross margins has profound implications for how these companies are valued. During the peak of the cloud software boom in 2021, the Bessemer Cloud Index showed public companies trading at 18 to 20 times their next-twelve-months revenue. That premium was justified by 80 percent margins.

By late August 2026, the Bessemer Cloud Index had stabilized near 8 times forward revenue. While growth rates and net revenue retention drive much of the variance between individual companies, the underlying gross margin sets the absolute ceiling on how efficiently a business can convert its revenue into free cash flow.

"If you are co-mingling these AI compute expenses with your standard hosting or burying them inside your R&D lines as 'development tools,' you are committing two critical financial sins," The SaaS CFO warned in September 2026. Distorting the core gross margin hides the true cost of delivering the service.

A company with a 50 percent gross margin must grow significantly faster than a company with an 80 percent margin to justify the same valuation multiple. Every dollar of new revenue simply yields less gross profit to fund sales, marketing, and future development. The math is unforgiving.[1]

Investors are increasingly separating companies that merely augment their software with artificial intelligence from those that are entirely native to it. Augmented businesses can often maintain 80 percent margins because inference is a tiny fraction of their value. Native applications, where the model performs the core work, are valued more like tech-enabled services.[4]

Selling outcomes over software

If the foundation model is rented from a cloud provider and the gross margins are structurally lower, the traditional software moat disappears. A company cannot defend its business simply by having the best code, because the intelligence engine is a commodity available to any competitor with an application programming interface key.[1][4]

Defensibility now comes from the data flywheel and the depth of the workflow. As Andreessen Horowitz has noted, the most successful artificial intelligence companies turn the intelligence problem into a data engineering problem. They capture proprietary user data to fine-tune their systems, creating a localized advantage that generic models cannot match.[4]

Furthermore, the most resilient companies are changing what they sell. Instead of selling a tool that helps a human do a job, they are selling the completed job itself. If an artificial intelligence agent can fully automate a legal review or a customer support interaction, it is no longer competing against a $50 software license.[1][4]

It is competing against a $50,000 human salary. When priced against human labor, a 50 percent gross margin is exceptional. Traditional professional services firms operate on margins of 30 to 40 percent. By replacing labor rather than software, artificial intelligence companies can access vastly larger budgets.[1]

The era of near-zero marginal cost software is ending, replaced by a model where computation is an active, ongoing expense. The 50 percent gross margin cap is not a failure of the technology. It is simply the new economic reality of selling manufactured intelligence.[1]