AI-assisted coding feels like magic. You type what you want, and out comes working code. (well, maybe after a few hours of setup − but still)
Like all magic, though, it has a cost. And right now, that cost is mostly hidden − even as the invoices show up every month. What makes it work is a stack of expensive infrastructure: thousands of GPUs, power-hungry data centers, and cloud contracts worth billions.
Worth asking: what are we really paying for, and is it sustainable? Spoiler alert: probably not. But let’s take a closer look.
AI hardware
Starting with GPUs: these are specialized, expensive, and in short supply. Companies like OpenAI are reportedly spending $700,000 a day just to keep ChatGPT running (see Links/resources/news section below for more on this). That’s not for training – just for serving requests. Imagine a SaaS product with that kind of daily burn rate 😳
But it’s not just the chips. You need space to run them, electricity to cool them, and contracts to guarantee access. That’s part of why OpenAI signed multi-billion dollar deals with Microsoft and CoreWeave. Google, which owns its stack end-to-end, has a natural edge. Everyone else rents.
Does pricing reflect demands?
The high cost of running large models shows up in the price users pay. GPT-4, for example, is priced up to $0.09 per 1,000 tokens (input + output) − a price that reflects not just software value, but the GPU time and energy required to generate results.
Because compute is such a dominant cost, many providers operate on thin margins − or even subsidize usage for heavy users. That’s why we’re starting to see more flexible pricing models emerge: subscription plus usage (as with Cursor), or tiered enterprise plans.
Big brains, bigger bills
Training a frontier model like GPT-4 costs more than $100 million. Then you have to align it. That means humans in the loop: labeling, ranking, refining. It’s expensive and slow. But important: that’s how you get models that don’t just parrot StackOverflow.
All of this means the cost structure is front-loaded and back-loaded at the same time. You spend a fortune to build the model, then you spend more to keep it useful.
Losing money at scale
OpenAI made $3.7B in 2024. They also spent $9B. Most of that went to compute and infrastructure. It’s like selling dollars for ninety cents. That strategy might make sense if you’re playing to win the whole market, but it also suggests today’s pricing isn’t built to last.
We’ll likely see more usage-based billing soon − more like a gas station, where you pay from the first drop. Enterprise plans are already heading that way. Flat rates are nice for adoption but unsustainable for profit.
Will it get worse?
Yes and no. Demand is rising. Models are getting bigger: more tokens, longer context windows, more compute. That all pushes cost up. But there’s pressure in the other direction too: better chips, smarter architectures, sparse models. Caching. Routing. If you can serve a query with a smaller model, why fire up the big one?
In the long run, efficiency wins. But in the short term, expect prices to fluctuate. Especially if you’re buying access to models you don’t control.
What it means for you
If you’re a CTO or engineering lead, this matters. Not just because of cost, but because it shapes your organization’s dependencies. You’re betting on vendors who are themselves betting the farm. That might be fine, but understand what you’re signing up for.
AI coding agents aren’t just clever tools. They’re backed by billion-dollar infrastructure bets. Used well, they offer real leverage. Not magic, but a nice productivity shift − less time on boilerplate, more time on real problems. The benefits build over time, and teams that learn to work with them don’t just move faster, they scale better.
Links/resources/news
- ChatGPT costs OpenAI $700,000 a day to keep it running
https://www.reddit.com/r/artificial/comments/12whu0c/chatgpt_costs_openai_700000_a_day_to_keep_it - OpenAI is taking a page out of Big Tech’s playbook by reportedly building its own chips
https://www.businessinsider.com/openai-chip-design-tsmc-broadcom-big-tech-nvidia-2024-10 - OpenAI set to finalize first custom chip design this year
https://www.reuters.com/technology/openai-set-finalize-first-custom-chip-design-this-year-2025-02-10/ - OpenAI is reportedly getting closer to launching its in-house chip
https://www.theverge.com/news/609421/openai-in-house-chip-development - OpenAI has burned through $8.5 billion on AI training and staffing, and could be on track to make a $5 billion loss
https://www.pcgamer.com/software/ai/report-claims-that-openai-has-burned-through-dollar85-billion-on-ai-training-and-staffing-and-could-be-on-track-to-make-a-dollar5-billion-loss/ - AI Leadership Makes for a Difficult Balance Sheet
https://www.deeplearning.ai/the-batch/openai-faces-financial-growing-pains-spending-double-its-revenue/ - OpenAI forges $12bn contract with CoreWeave
https://www.ft.com/content/4b52fdbb-ca8e-4208-bb99-f1e7f9313863 - Amazon to Invest Another $4 Billion in Anthropic, OpenAI’s Biggest Rival
https://www.barrons.com/articles/amazon-stock-ai-anthropic-chatgpt-openai-544ab0e5 - GPT-4 Training Costs
https://news.ycombinator.com/item?id=42476926 - Powering the next generation of AI development with AWS
https://www.anthropic.com/news/anthropic-amazon-trainium
