Not Every Feature Needs an LLM. The Ones That Do Need a Cost Model First.

The ticket categorization feature shipped on a Thursday.

By Monday, it had processed six thousand support tickets, sorted them into the right queues, and nobody had touched it once. The team moved on to the next sprint.

On the fourteenth day, someone in finance forwarded the Azure invoice with one line highlighted and no other comment attached.

The feature was simple by design. A ticket comes in, gets sent to Azure OpenAI with a short prompt, and comes back tagged as billing, technical, or account access. Accurate, fast, and clean in every code review it passed through.

Nobody had asked what it would cost once it stopped being a pilot running against forty test tickets and started running against everything hitting the support queue in production.

The answer was three times the original estimate.

Not because the model was misbehaving.

Because the estimate had never accounted for retries on timeout, for tickets long enough to double the token count, or for the fact that ten thousand tickets a day was the real number, not the four hundred the pilot had run against.

The feature worked exactly as designed.

Nobody had designed for its economics.

This is the same shape of problem this series keeps returning to. Something works, passes review, looks fine in the dashboard, and the real consequence is sitting quietly underneath, waiting for production volume to arrive.

Your Production Health Checks Are Lying to You was about a green signal that stopped meaning anything once real traffic hit the system.

Here, the hidden signal is a cost model nobody built, and the traffic that exposes it doesn't show up on a graph.

It shows up on an invoice.

Cost Is an Architectural Property, Not a Line Item You Check at the End

Most engineering teams treat cost the way they treat documentation, something to get to once the feature works.

That habit is expensive with ordinary infrastructure.

It is dangerous with AI.

Traditional infrastructure often hides marginal cost behind provisioned capacity. You provision a database tier or a compute plan and pay for that capacity whether the application uses every unit of it or not, so the marginal cost of one more request stays close to zero until you cross a scaling boundary you can usually see coming.

AI services make the marginal cost far more visible, and far less forgiving.

Every model request can carry a direct, variable cost, and that cost moves with things that are easy to underestimate at design time.

Input tokens grow with everything stuffed into the prompt.

Output tokens grow with how verbose the model decides to be.

Conversation history, if the feature maintains any session state, can make every subsequent request more expensive than the one before it.

A retry on timeout can turn one failed operation into two or three billable requests.

None of this is complicated.

The problem is that it rarely gets modeled.

A feature that costs eight cents a call looks harmless sitting in a pull request.

At ten thousand calls a day, that is eight hundred dollars a day, close to twenty-four thousand a month before anything else gets added to the picture.

And that number does not sit still.

It moves every time someone adds more context to the prompt, changes the model, introduces a retry policy, or the feature quietly expands into a second workflow nobody costed separately.

The ticket categorization team was not careless. They checked the cost once, against pilot volume, and the assumption was never revisited once the feature moved from forty test tickets to the entire support queue.

That is the same gap described in Scaling the Application: Not an Answer to Every Performance Issue, where the real bottleneck sits one layer away from wherever everyone happens to be looking.

Nobody was looking at cost because cost was never part of the feature's production model in the first place.

Five Questions Before the Architecture Diagram

Before a line of code gets written for an AI feature, there is a costing exercise worth running, and it does not need a spreadsheet with fifty tabs.

It needs five questions, answered honestly and written down somewhere the whole team can see, not held in the head of whoever happened to design the feature.

1. What Does One Operation Actually Cost?

Not the headline number on the pricing page.

The real cost of the complete request the production system will send, including conversation history, retrieved context, system instructions, and metadata, not just the user's single line of input.

The unit of cost is the whole request, not "one prompt."

2. What Is the Realistic Volume?

Pilot volume tells you almost nothing about production volume.

The numbers that matter are expected daily traffic, peak traffic, and the volume a successful marketing campaign or sudden internal automation could generate.

The system does not get to choose when that traffic actually arrives.

3. What Is the Unhappy-Path Multiplier?

A successful request might cost one model call.

A retry policy that allows two retries can turn one logical operation into three physical, billable calls.

And a timeout followed by a retry is worth sitting with, because the first request may already have reached the provider even though your application never received the response and has no way of knowing it was charged for it.

This is the same retry storm problem production systems have always had.

The difference with an LLM dependency is that every additional retry now has a direct price attached to it, not just added load.

4. How Does Cost Grow?

A conversational feature's first turn might send a short system prompt and one question.

Its twentieth turn might send the full prior conversation, retrieved context, and the new question.

If the application keeps resending that history in full, the cost of a single conversation grows with every exchange.

The real question is not what one request costs.

It is what a request costs once a real user has been in the feature for twenty minutes.

5. What Does It Cost to Be Wrong?

If a model miscategorizes a ticket, a human corrects it.

If a generated response contains an error, someone has to catch it before it reaches a customer.

A feature that costs two cents a request but creates five dollars of correction work every time it is wrong has very different economics from one that costs five cents and rarely needs a human to step in.

The provider's price is only one part of the bill.

Five questions, answered before the SDK gets touched.

A team that cannot answer the first one is building blind, the same way a team that ships a health check without asking what it actually verifies is shipping a false signal dressed up as confidence.

Sometimes the Honest Answer Is That the Feature Never Needed an LLM

There is a question that belongs before all five, and it is easy to skip once a prompt is already half written.

Does this problem need an LLM at all?

Go back to the ticket categorization example.

If a ticket needs to land in one of five known categories based on structured fields the user already filled in, an LLM may be solving a far harder problem than the application actually has.

A rules engine is often enough.

A lightweight classifier is often enough.

In plenty of real cases, a straightforward conditional chain against known fields and keywords is enough, and it solves the problem for close to zero marginal cost per request.

That is not an argument against AI.

It is good architecture.

The same pattern repeats everywhere once it becomes visible.

Extracting a field from a document that always follows the same template needs a parser, not a model call.

Validating that an input matches an expected structure needs a schema check, not probabilistic reasoning.

Routing a request across five known conditions needs a switch statement, not an LLM.

Generating output from a fixed template needs a template engine once, at design time, not a model call on every single use.

A deterministic problem deserves a deterministic solution.

An ambiguous, language-heavy, or genuinely open-ended problem is where a model earns its cost.

The mistake is not using AI.

It is reaching for probabilistic machinery on a problem that deterministic logic already solves, and paying for the non-determinism, the added latency, and the new failure mode that comes with a model call nobody actually needed.

Before asking which model to use, ask what the problem actually is, and whether a model is the simplest reliable way to solve it.

Most of the time, the honest answer settles the question before the architecture diagram even gets drawn.

The Same Cost Model, With No AI in It

A different team, the same quarter, was designing a notification system for order status updates.

Every time an order changed state, a notification went out.

Order placed.

Payment confirmed.

Shipped.

Out for delivery.

Delivered.

Five notifications per order, each one a separate call to a third-party push and email provider that billed per message sent.

Nobody ran a costing exercise, because nobody thought a notification system needed one.

It was not AI.

It felt too simple to worry about.

The cost per message was tiny, the expected order volume was a few thousand orders a month, and the feature looked cheap enough that it never came up in the architecture review.

Then a promotional campaign tripled order volume for two weeks.

Notification volume did not just triple.

It went up five times, because peak order days also drove more support-driven status checks, which triggered re-sends of the same notifications through a retry path that had been added for reliability and never load tested against a real spike.

The unhappy path cost more than the happy path, and nobody had modeled the unhappy path at all.

A batched approach, sending one consolidated update per order per day instead of five separate sends per state change, would have cut the message count substantially with almost no change to what the customer actually needed to know.

That decision was available at design time.

It only became obvious once the bill arrived.

This is why the lesson was never really about AI.

AI just makes the economics impossible to ignore, because the invoice shows up fast and the number is hard to miss.

A notification provider billed per message has variable cost.

A third-party API billed per request has variable cost.

A background job that quietly duplicates work has variable cost.

The engineering question is identical in every case:

What happens to the economics once production reality, not pilot volume, actually arrives?

Cost Belongs in Your Telemetry, Not Just Your Design Doc

Once a feature ships, the cost model should not disappear into a document nobody reopens.

The assumptions behind it need to become something you can observe, the same way you would observe latency or error rate.

For an AI-backed feature, this means tracking request volume, input and output tokens, which model handled each request, retry counts, failures, and an estimated cost per request, per user, and ideally per business transaction rather than per raw API call.

If you are already running on Azure, this telemetry belongs alongside everything Application Insights already gives you about how often an operation runs and how it fails.

Azure's cost tooling tells you what the infrastructure and AI service are costing in aggregate.

Together, those two signals answer a more useful question than whether a feature is healthy.

They answer whether it is still economically healthy at the volume it is actually running at.

A feature can be fast, reliable, highly available, and accurate, and still be quietly uneconomical.

That is a production health signal in its own right.

It belongs next to latency, availability, and error rate in the same conversation, not filed separately under finance.

It is also worth remembering that cost does not always grow in a straight line with traffic.

A retry policy can make cost grow faster than traffic does.

A conversation that keeps resending its own history can make individual requests more expensive over time even at flat volume.

A workflow can fan one user action into several downstream calls: document extraction, embedding, vector search, a model call, validation, and perhaps a retry.

What looks like one operation to the user can therefore become several billable operations underneath.

The unit worth tracking is not the cost of one model call.

It is the cost of the entire business operation the user actually asked for.

That is the number that means something in an architecture review.

Treat a Cost Spike Like an Incident, Not a Surprise

Here is the part most teams skip even after they get the costing exercise right at design time.

A cost model built once, before launch, is a snapshot.

Production does not hold still.

Traffic shifts, a prompt gets a few lines longer during a routine change, a retry policy gets tuned for reliability without anyone re-running the economics, and six weeks later the number that was correct at launch is no longer the number that is true.

The ticket categorization team did not fail because their initial estimate was wrong.

Their estimate was reasonable for the volume they tested.

They failed because nothing was watching the gap between that estimate and what production was actually doing, the same way a service can run for months with a slow memory leak that nobody notices until the day it finally falls over.

A cost model without an alert attached to it is not a safeguard.

It is a document.

The fix is to treat a cost anomaly the same way you would treat an error-rate spike or a latency regression: as a production signal with a threshold and an owner, not a line item someone in finance eventually finds.

Azure Cost Management can be used for budgets and cost monitoring, while application-level telemetry can track token usage, request volume, retries, and estimated cost per operation.

Pair those signals together and a cost regression stops being something you discover in an invoice.

It becomes something you discover in a dashboard, at the same time you would discover a spike in five hundred errors, and it gets a ticket, an owner, and a root cause the same way an outage would.

This is also where the unhappy-path multiplier from earlier stops being a design-time estimate and becomes a live number worth watching.

If retries were budgeted at one and a half calls per operation and production telemetry shows they have crept to two and a half, that is not a rounding error.

That is the same kind of silent drift this series has described before, the gap between what a system was designed to do and what it is actually doing under real conditions.

Except here, the consequence is not a timeout.

It is a bill that keeps compounding until someone finally looks.

A cost model that lives only in a design document before launch answers one question well:

Should we build this?

A cost model wired into production telemetry answers a second, harder question that matters just as much:

Is this still true?

The first prevents an obviously bad decision at the start.

The second catches the slow, ordinary drift that turns a reasonable decision into an expensive one, three months in, without anyone changing anything on purpose.

What to Actually Do With This Before the Next Feature Ships

Before the next feature gets built, AI or otherwise, run the five questions against it and write the answers down somewhere the team can see.

What does one complete operation cost?

What is the realistic volume, including the worst case?

What do retries, fallbacks, and failures add to that cost?

Does cost grow in a straight line with volume, or does something in the design make it grow faster?

And what does it cost, in human or customer terms, when the system gets it wrong?

If those answers do not exist yet for something already running in production, that is worth an afternoon this week, not next quarter.

The ticket categorization feature had an architecture review.

The notification system had an architecture review.

Neither had a meaningful costing review, and neither had an alert that would have caught the drift before finance did.

That is the gap worth closing while it is still cheap to close.

And it does not end once the feature ships.

It ends when the cost model has a threshold, an owner, and a place on the same dashboard as everything else that pages someone at two in the morning.

Most of the time, running this exercise costs an afternoon and prevents the conversation that follows a tripled invoice.

Sometimes it does something better than that.

It changes the architecture before the architecture becomes expensive to change.

A rules engine replaces a model call that was never needed.

A batched notification replaces thousands of individual sends.

A tighter retry policy stops duplicate work from quietly doubling the bill.

Occasionally the answer is simply not to build the AI feature at all.

And that is not a failure of AI engineering.

It is a success of engineering.

A production feature is not finished because it works.

It is finished once you understand what happens when it scales, fails, retries, gets expensive, and runs into the conditions the happy-path demo never showed anyone.

Cost is one of those conditions.

AI just makes it impossible to pretend it isn't there.

This is part of The True Code of Production Systems, a series about the decisions that only become visible when something breaks in production.

Sometimes that something is a server.

Sometimes it is a dependency.

Sometimes it is a retry storm.

And sometimes, it is a bill.