How Much Does It Cost to Run AI Features in Production?
What drives the ongoing cost of AI features, how to estimate it with a simple formula and a worked example, and practical techniques to keep AI costs under control.
Building an AI feature is a one-time investment. Running it is not. Every question answered, document processed or summary generated has a cost, and unlike most traditional software, that cost grows with every user and every request. Teams that do not model it upfront are sometimes surprised by the first monthly bill, or discover that their most active customers are their least profitable.
This guide explains what it costs to run AI features in production, how to estimate those costs before launch and the most effective ways to reduce them without hurting quality.
Key takeaways
- Most AI running costs come from model usage, billed per token (roughly, per piece of text processed and generated).
- Cost per request is driven by model choice, context size and output length, often more than by request volume.
- Retrieval, infrastructure, monitoring and human review add to the total.
- Choosing the right model per task, trimming context and caching typically cut costs significantly.
- Track cost per feature and per customer from day one.
What you pay for
1. Model usage (tokens)
Most AI providers charge per token. A token is a fragment of text; in English, 1,000 tokens correspond to roughly 750 words. Pricing is usually quoted per million tokens, with separate rates for:
- Input tokens: everything sent to the model, including instructions, retrieved documents, conversation history and the user's question.
- Output tokens: the text the model generates. These are typically several times more expensive than input tokens.
Prices vary widely between models: small, fast models can cost a small fraction of the most capable ones. Prices also change frequently, generally downward for a given level of capability, so always check current pricing for the models you consider.
2. Embeddings and retrieval
Assistants that answer from your documents use embeddings to search content. Creating embeddings is relatively cheap; the vector index itself requires storage and compute, either in your existing database or a dedicated service. See how RAG works.
3. Other AI services
OCR and document AI, speech-to-text, text-to-speech and image analysis are usually billed per page, per minute or per image.
4. Infrastructure
Application servers, queues for background processing, storage for documents and logs, and monitoring. Modest for most features, but not zero.
5. Human review and quality assurance
Reviewing low-confidence results, sampling outputs for quality and maintaining evaluation sets takes people's time, and is part of the real cost of reliable AI.
6. Ongoing improvement
Prompt and retrieval updates, model upgrades and evaluation runs as providers release new models and your content changes.
A simple formula
Monthly model cost ≈ requests per month × (input tokens × input price + output tokens × output price)
The token counts per request matter as much as the number of requests. A request that sends 20 pages of context costs far more than one that sends a short question.
Worked example: an internal knowledge assistant
Assumptions, for illustration only (check current prices for your chosen model):
- 200 employees ask 50 questions each per month: 10,000 questions
- Each request sends instructions, retrieved passages and the question: about 4,000 input tokens
- Each answer is about 400 output tokens
- Assumed price: $3 per million input tokens and $15 per million output tokens
| Item | Calculation | Monthly cost |
|---|---|---|
| Input tokens | 10,000 × 4,000 = 40M tokens × $3/M | $120 |
| Output tokens | 10,000 × 400 = 4M tokens × $15/M | $60 |
| Model usage total | ≈ $180 |
If each employee saves just ten minutes a month, the time saved is worth far more than the model cost. But change the assumptions, such as a model ten times more expensive, 20,000 tokens of context per request or 500,000 requests from a customer-facing feature, and costs grow quickly. That is why modeling matters.
Cost drivers to watch
| Driver | Effect |
|---|---|
| Model choice | Large models can cost many times more per token than small ones |
| Context size | Long documents, full conversation histories and many retrieved passages multiply input tokens |
| Output length | Verbose answers cost more; output tokens are the expensive ones |
| Multi-step workflows | Agents that call the model several times per task multiply cost per task |
| Retries and failures | Every retry is billed |
| Heavy users | A small share of users can generate most of the usage |
How to reduce AI costs
Use the smallest model that meets the quality bar
Classification, extraction and routing often work well with small, inexpensive models. Reserve the most capable models for tasks that truly need them. Your evaluation set tells you where the quality threshold lies.
Route between models
Send simple requests to a cheap model and escalate complex ones to a stronger model. This alone can substantially reduce average cost per request.
Trim the context
Retrieve fewer, more relevant passages; summarize long conversation histories; remove unnecessary instructions. Better retrieval often improves both quality and cost.
Use prompt caching
Many providers discount repeated input, such as long instructions or documents sent with every request. Structuring prompts so the stable part comes first lets you benefit from these discounts.
Cache answers
Frequently asked questions with stable answers can be served from a cache instead of calling the model again.
Limit output length
Ask for concise answers or structured output. Shorter responses are cheaper and often more useful.
Batch non-urgent work
Background tasks such as nightly document processing or bulk classification can use batch processing, which providers often offer at a discount.
Set limits and monitor
Per-user and per-customer quotas, rate limits and alerts on unusual spending protect you from runaway costs, whether from bugs, abuse or unexpected success.
Pricing AI features for your customers
If you sell software with AI features, your pricing must account for usage costs. Common approaches include usage allowances per plan, AI add-ons and outcome-based pricing. See our guide to SaaS pricing models.
Should you run your own models?
Self-hosting open-weight models can make sense at very high, steady volumes, for strict data requirements or for specialized models. But GPUs, operations, scaling and model updates carry significant fixed costs and require expertise. For most businesses, API-based models are cheaper and simpler until volume is substantial.
Build cost visibility in from day one
- Log tokens, model and cost for every request
- Attribute costs to features, customers and users
- Show cost trends in dashboards next to usage and quality metrics
- Review monthly and adjust models, prompts and limits
Frequently asked questions
Are AI costs going down?
For a given level of capability, prices have generally fallen over time. At the same time, new, more capable models and more ambitious features tend to increase usage, so total spend depends on how you use them.
How do I estimate costs before building?
Prototype the feature with real examples, measure average input and output tokens per request, multiply by expected volume and current prices, and add a buffer for growth and retries.
What about the cost of building the feature?
Development is a separate, one-time investment. Our development cost guide covers typical budgets for AI features.
Make AI economics part of the design
AI features can be remarkably cost-effective when designed with the right model, focused context and good monitoring, and surprisingly expensive when they are not. Model the costs early, measure them continuously and optimize where it matters.
At Tyrcom we design AI features with quality and running costs in mind from the start. Talk to us about your AI use case, or read our roadmap for adding AI to business software.
