Enterprise AI that holds up

A pilot shows that a model can answer. Production requires that it answers correctly, for the right people, at a cost someone has agreed to pay.

Most stalled AI programmes did not fail on model quality. They stalled because the pilot was run under conditions that production will not offer. The demonstration was genuine, and it answered a different question from the one the business later needed answered.

Compare the two settings.

Each row is a place where the pilot hid a problem. The curated extract concealed the join errors between systems. The prepared questions concealed how the system behaves when it does not know. The single administrator concealed the entitlement problem entirely, and entitlement is often the issue that stops a security review and ends the programme.

There is a sequence behind this. Making data answerable comes first. Making AI usable on that data comes second. Letting agents do the routine work comes third. Programmes that attempt the third stage before the first is complete tend to stall, because an agent acting on data nobody trusts only produces wrong actions faster.

Production also introduces work that a pilot never meets. Someone must monitor answer quality over time as data and models change. Someone must handle the questions that fail. Someone must pay for the compute. These are operating tasks and they continue indefinitely, so a pilot that has no plan for them has only postponed the cost.

You can design a pilot to learn about production instead of to impress. Use real data from live systems, even if the scope is narrow. Include users with restricted access. Write the success criterion down before the work starts, in terms of the business question and not of the technology. Agree in advance that a verdict that the work should not proceed is an acceptable result. A pilot that cannot return that verdict is a demonstration.

Finally, ask who will own the result. A pilot with a sponsor but no owner usually ends when the sponsor's attention moves on.

In practice: Before approving the next pilot, go through the table above and mark each row as it will apply to the pilot. Every row that reads as a pilot condition is a risk you are choosing to carry into the production decision. Fix the cheap ones now, starting with live data and restricted users.

Next: B2, A citation shows where an answer came from, and evidence shows that it is supported.

Vector retrieval is the better choice for text and for speed to a first version, and an ontology is the better choice for records that must be joined and counted.

Two retrieval designs dominate enterprise AI. They solve different problems, and arguments that treat one as superior in general usually come from selling it.

Vector retrieval splits content into chunks, converts each chunk to a numerical embedding and, at question time, returns the chunks nearest to the question in meaning. An ontology approach models entities and their relationships explicitly, such as customers, orders and shipments, resolves a question against that model and generates a query over the live records.

Vector retrieval is the better choice in these circumstances.

• The corpus is mostly documents: policies, contracts, manuals, emails, reports.
• The question is about what a text says, or about finding material that resembles another.
• The schema is unknown, unstable or absent, so there is nothing to model.
• You need a working first version quickly, with little preparatory effort.
• The vocabulary is messy, and meaning-based matching tolerates synonyms and spelling that an exact model would not.
• The stakes are low enough that a reader will check the output.

An ontology is the better choice when the answer depends on structure. Relating a customer to their orders and then to their shipments is a join, and a similarity search over chunked tables cannot reliably reconstruct it. Counting, summing and filtering with exact conditions are also queries, not matters of resemblance. A generated query is either correct or fails visibly, and every figure traces to records. Policy can be applied when the query runs.

An ontology has its own weaknesses, and they are significant.

• Building and maintaining the model takes effort and needs an owner. If nobody maintains it, it decays.
• Concepts that were not modelled cannot be asked about. The system fails, which is visible but still a failure.
• It does little for free text. A contract clause is not a row.
• The first useful answer takes longer than it does with a vector index.

The last row is the common case. Real questions often mix a count over records with an explanation drawn from text, and hybrid designs use each approach for what it does well. When you evaluate a product, ask which parts of your question set are records and which are text, and test each part separately.

In practice: Sort your fifty most likely questions into three piles: about text, about records, and both. The proportions between the piles tell you where the weight of the system should lie, and which vendor claims apply to you.

Next: B4, A system that never refuses tells you nothing when it answers.

A public benchmark tells you how a model performs on general tasks, and your data, vocabulary and permissions are not general.

Leaderboards are useful for a first shortlist of models. They cannot tell you whether a system will answer correctly about your customers, in your terminology, against your tables, for a user with restricted access. Only a test built from your own questions can do that.

A workable evaluation set has a few properties. It is drawn from real questions, taken from analysts' request backlogs, support tickets, meeting minutes and the queries people already run. It is large enough that one failure does not decide the verdict, which in practice means a few dozen to begin with, and it should grow. It includes the awkward cases deliberately.

• Questions that require joining several systems.
• Ambiguous questions, where the right behaviour is to ask for clarification.
• Questions the system should refuse because the data does not exist.
• Questions where two users should receive different answers because of entitlement.
• Questions whose answer changed recently, to test freshness.

Each question needs an agreed correct answer, written by the person who owns the figure. This step is often skipped, and it is the most valuable one. It forces the organisation to decide what correct means, which is the same discipline as module A3.

Scoring should cover more than correctness. Record whether the answer was right, whether the evidence supports it, whether a refusal was appropriate, whether entitlement was respected, how long the response took and what it cost. A system that is slightly less accurate but never leaks restricted data may be the better choice. A single accuracy figure hides that trade.

Three practices protect the evaluation itself.

1. Keep a held-out portion of the questions that nobody uses for tuning. Otherwise the system is optimised for the test.
2. Have answers scored without knowing which system produced them, when comparing products.
3. Rerun the whole set whenever the model, the prompt, the data or a definition changes. A system that passed in March has not necessarily passed in June.

The same set lets you compare vendors on equal terms, including products you have already bought. It also gives you a record of behaviour over time, which you will want when an auditor or a board member asks how you know the system still works.

Be cautious with automated scoring by another model. It is cheap and fast, and it has its own errors. Check a sample of its judgements by hand until you know how far to trust it.

In practice: Ask five people who use your data daily for the three questions they ask most often and the three they have given up asking. Write the correct answers with them. Those thirty questions are a usable first evaluation set, and they can be run against any product tomorrow.

Next: B6, Cost per answered question is the unit that makes systems comparable.

Most AI spend is self-inflicted: systems paying to re-read their own context, reasoning at length about lookups, and answering simple questions at frontier prices. The fixes are architectural, and they are known.

When the first production bill arrives, the instinct is to negotiate the price list. The price list is the least interesting part of it. Ungoverned agent workloads spend most of their money on things nobody chose: conversation history re-read on every turn, retrieved documents that are never used, tool schemas carried into every call, long reasoning about questions that deserved a lookup, and output tokens — priced at several times input — with no cap on length.

Five levers move most of this spend.

• Route. Send each class of question to the smallest model that passes your own evaluation for that class. The price spread between tiers is large, and most enterprise traffic belongs in the cheap ones.
• Cache. Stable prompt prefixes are charged at a fraction of the price by most providers; repeated questions can be answered from a semantic cache with no model call at all. Most enterprise workloads are repetitive, which is what makes this lever large.
• Compress. Most tokens in a typical agent's input are low-signal. Compression before dispatch removes the majority of them without measurably hurting answers.
• Cap. Output is the expensive direction. Cap answer length and reasoning depth per question class — short answers for short questions.
• Budget. Every agent carries a spend counter enforced at the gateway, with anomaly alarms. This turns the other four from intentions into facts, and it is what catches a runaway loop at 3am.

Two details decide whether the levers actually work. Prefix caching pays only when the start of the prompt is byte-identical between calls, so prompt construction must be deterministic — stable parts first, variable parts last. And routing pays only when the evaluation behind it is honest: the cheapest model that passes a public benchmark is not the cheapest model that passes your questions on your data.

There is also a counter-movement worth knowing. As per-token prices fall, deliberately spending more tokens on the questions that deserve it — deeper reasoning, verification passes against the evidence — becomes affordable, and for high-error-cost questions it is the correct choice. The skill is not spending less. It is spending the right amount per question class, measured.

In practice: Take last month's bill and decompose it into the categories above: re-read context, reasoning, output, retries, and the questions themselves. Then apply one lever — routing is usually the largest — and measure cost per answered question before and after. If you cannot produce the denominator, that is the first gap to close.

Next: This is the last module in this track. Return to A1 and apply the questions in this series to your own estate.