How much do tokens really cost?

I hear a lot of suspicion that AI tokens are unsustainably priced and that frontier labs are heavily subsidising costs with their VC money. They want to get us hooked now, so they can pull the rug out from under us whenever they want to make a bunch of money. Is that true? My sense is that it’s mostly not. But can I back that up with some data? That’s my goal with this post.

There are many open models available, but here I’m going to focus on Kimi K3. I’ve been using it heavily for the past couple of months, and interestingly, I’ve continued to use it even after Opus 5.5 was released. Opus 5.5 definitely feels a bit smarter than Kimi K3 to me, but I still find myself drawn to Kimi K3 for its cost effectiveness. I want to talk more about this in a future post, because it connects to one of my favourite Posit Assistant features: the cost display. This is a powerful tool for building up your sense of where models spend money and how much a given task should cost.

The Posit Assistant chat box, showing the running cost of the current conversation (34K tokens, $0.94) alongside the model selector set to Kimi K3.

So Kimi K3 is a reasonable proxy for a solid frontier model: maybe it’s not quite as good, but it’s pretty darn close. How much does it cost?

OpenRouter pricing

Because it’s an open model, you can buy Kimi K3 from a range of providers. Here’s the current pricing from OpenRouter. I’ve included a blended column which uses typical ratios for agentic coding: 20:1 input to output and 95% of input from cache. We’ll need that later when we compare costs to self-hosting.

#>  Provider       Input /M Output /M Cache /M Blended /M
#>  Wafer          0.28     9.60      0.13     0.59      
#>  Sail Research  0.22     10.72     0.11     0.62      
#>  Relace         0.21     9.33      0.21     0.64      
#>  Makora         0.36     12.00     0.12     0.70      
#>  Chutes         1.85     9.50      0.18     0.71      
#>  Decart         1.95     9.75      0.20     0.73      
#>  ParetoAlpha    1.00     11.25     0.23     0.79      
#>  Fireworks      2.10     10.50     0.21     0.79      
#>  Phala          2.55     12.75     0.26     0.96      
#>  DigitalOcean   2.55     12.95     0.26     0.97      
#>  InferenceNet   0.23     16.50     0.20     0.97      
#>  Morph          1.42     12.00     0.40     1.00      
#>  DeepInfra      2.85     14.25     0.28     1.07      
#>  Amazon Bedrock 3.00     15.00     0.30     1.13      
#>  Parasail       3.00     15.00     0.30     1.13      
#>  Modal          3.00     15.00     0.30     1.13      
#>  Together       3.00     15.00     0.30     1.13      
#>  BaseTen        3.00     15.00     0.30     1.13      
#>  Moonshot AI    3.00     15.00     0.30     1.13      
#>  Alibaba        3.45     17.25     0.34     1.30      
#>  AkashML        1.20     14.00     1.20     1.81

(Sorry for the crappy formattaing but Substack doesn’t seem to support tables 😡)

You’ll notice that there are a lot of options! And I bet most purchasing decisions are driven by cost, which means there’s a lot of pressure on these providers to compete on price.

What stops some sort of subsidy pushing prices unsustainably low? Well, I bet some of the cheapest providers are spending their VC cash to try and get customers. But there are also six providers all selling around the same price at the high end. One of those providers is BaseTen who we work with to provide models for Posit AI Pass. I can personally vouch for the quality of this service; it’s not the cheapest but our experience with them has been great.

That’s buying Kimi K3 “off the rack”. What if we tried to host it ourselves?

Self-hosting

I’m no expert on local models, so I used Claude to help figure out the costs. I’ve carefully checked individual facts as I wrote this article, but it’s possible Claude steered me in completely the wrong direction and there’s a deeper flaw that I missed. But let’s have a go.

We need to start by buying a computer that can host Kimi K3. How much will that cost? Moonshot AI (the authors of Kimi K3) provide a vLLM recipe that recommends “at least 8x GB300”. One possible option for that is a Supermicro 8U GPU SuperServer, which costs a whopping $550,000! I don’t have half a million dollars lying around, so we’ll need to borrow the money for it. If we assume an 8% interest rate (which is maybe optimistic) for 3 years, that’s going to cost us around $17,000 a month.

You could instead rent the 8 GPUs for around $4/hour each, which adds up to around the same amount when you include the cost of hosting (discussed next). The advantage of borrowing the money and buying it is that you could sell it at the end of the three years, netting you maybe $150,000. The advantage of renting is that you can stop at any time.

Now we need to put that server somewhere. That’s called a server colocation facility, and while pricing information isn’t generally available on the internet, Claude claims it will cost around $3,750 / month, with another $600 for bandwidth, and ~$700 for power. You might consider keeping the server at your house, but it produces a tremendous amount of heat, and you’ll need to buy a dedicated A/C unit for it.

There are a few other smaller bits and pieces that will bring the total monthly cost to around $23,000.

How many tokens can we serve in a month? SemiAnalysis suggests ~6000 tokens/s per GPU, so 48,000 tokens/second, which is 124 billion tokens/month. If you assume we get 75% utilisation (reasonable for an API provider), that’s $0.25 / million tokens. If you assume typical ratios for agentic coding (20:1 input to output, 95% of input from cache), the equivalent API cost (for BaseTen and friends) would be $1.13 per million tokens. Assuming we have high capacity, it seems like the cost of running the server is significantly cheaper than what providers are charging.

Obviously this is a crude analysis, but it’s a reasonable top-level bound on the cost of serving tokens. A real business would benefit from economies of scale, and the ability to negotiate with vendors, but would also have to pay staff.

Subsidies

So I think I’m convinced that token prices are reasonably accurate measurements of the underlying cost. There’s a lot of competition pushing the cost down, but there still seems to be room for a healthy profit margin based on the underlying costs.

That’s not to say subsidies aren’t still happening! I just don’t think they’re on per-token API costs. Instead, they’re happening on subscription plans. A recent SemiAnalysis post suggests that a $200 / month plan gets you somewhere between $2,000 and $10,000 worth of tokens. These plans are only available to individuals and small businesses, not enterprises, which have to pay per-token prices.

It’s also worth noting that my analysis ignores the cost of model training which might have cost several hundred million dollars. Currently that cost is paid by the investors in Moonshot AI (including the Chinese government). You could argue that this is also a subsidy, which is fair. My hope is that more governments decide that training open models is a societal good (compared to having frontier models owned exclusively by private industry) and decide to fund it directly.

What do you think? Did you find this analysis convincing? Do you have other evidence that might suggest how much tokens really cost?