Custom AI vs off-the-shelf: when building for one workflow beats buying a platform

Every buyer in a regulated industry eventually hits the same fork. There is a SaaS tool that does 70% of what you need, sold by a vendor with a demo and a security questionnaire already filled out. And there is the option to build something around a frontier model that does 100% of what you need, but that you now own. Most of the advice on this question is written by people selling one side of it. Here is the version we use internally when a hospital revenue cycle director or a general counsel asks us which way to go — including the cases where we tell them to buy the SaaS and save their money.

What custom AI actually means in 2026

The phrase carries baggage from 2019, when "custom AI" meant collecting a labeled dataset, fine-tuning or training a model, and standing up GPU infrastructure to serve it. That is not what anyone sensible is doing now for document-heavy work. Frontier models from Anthropic, OpenAI, and Google already read a denial letter or a payer policy better than a model you could train on your own corpus.

Custom AI in 2026 means the system built around the model. Concretely, five things:

• Orchestration. The decomposition of one messy job into steps a model can do reliably. "Handle this prior auth" is not a prompt. "Extract the CPT code, look up the payer's policy for that code, check the chart for each documentation element the policy requires, flag missing elements" is four steps with checkable output at each one.
• Retrieval. Getting the right context in front of the model. Your payer policy library, your firm's prior deal precedent, your institution's transfer credit rules. This is where domain specificity actually lives, and it is the part a horizontal tool cannot ship you.
• Evaluation. A test set of real cases with known correct outcomes, and a harness that scores the agent against it on every change. We write this before the agent code. If you cannot describe what a correct output looks like on 200 historical cases, you do not yet have a project.
• Integration. Reading from and writing to the systems of record — Epic, iManage, Banner, whatever. Usually the least glamorous and most expensive line item.
• Controls. Audit logs, human approval gates, PHI handling, the ability to answer "why did it do that" nine months later during an audit.

Notice that the model is one line in five. That is the actual insight: model capability is a commodity you rent, and it improves under you for free. The differentiated asset is the eval set, the retrieval corpus, and the integration surface. Those are yours whether you build or buy — except that when you buy, they sit inside someone else's product. We wrote more about this distinction in what vertical AI agents actually are.

The three tests for when custom is right

We run every prospective engagement through these three. Custom is justified when at least two are clearly true. If only one is true, buy the tool and revisit in a year.

Test 1: Is the workflow a source of margin or just a source of cost? There is a real difference between a workflow that costs you money and one that determines whether you get paid. Expense reports cost money. Denial appeals determine revenue. Discovery review determines case outcomes. Transcript evaluation determines enrollment yield. If the workflow sits on the revenue line, the difference between 82% and 94% accuracy is worth paying to own. If it sits on the cost line, the SaaS tool at 82% is fine, because the failure mode is mild annoyance rather than a lost claim.

Test 2: Do you have rules that no vendor could know? Not "our industry is complicated" — every industry says that. The test is whether you can name specific decision rules that live in your organization's head rather than in any public document. A payer mix where three plans account for 60% of your denials and each has quirks your billing team learned the hard way. A firm-specific position on indemnity caps that partners will not deviate from. A residency determination policy your registrar interprets in a way the state statute does not spell out. If your team's answer to "how do you decide?" runs to more than a paragraph and cannot be found in a manual, that is the raw material for a custom agent and the exact thing horizontal tools flatten.

Test 3: Is there enough volume to amortize an eval harness? This is the test people skip. Custom AI is not expensive to build anymore. It is expensive to keep correct. Every model upgrade, every payer policy change, every new document format is a regression risk, and the only defense is a maintained eval set. That machinery pays for itself at maybe 500 cases a month and is very hard to justify below 100. Below that threshold, a human plus a good general-purpose assistant beats a custom agent on total cost, and we will tell you so.

The real cost comparison, done honestly

Vendors compare their subscription against a build estimate. That comparison is rigged in both directions, so do the arithmetic yourself with your own numbers.

The buy side is not just the license. It is the license plus implementation plus the integration work you will do anyway plus the internal time spent reshaping your process to fit the tool's assumptions plus the ongoing cost of the cases it cannot handle, which still route to a human. That last item is the one that gets left out. A tool that automates 70% of volume at 90% accuracy still leaves your team touching about 37% of cases — the 30% it never took, plus rework on the ones it got wrong. Run that number for your own workflow before you sign anything.

The build side is not just development. It is the eval harness, the shadow-mode period where you are paying for an agent that produces output nobody acts on, model inference costs, and a named owner for the next three years. Anyone quoting you a build price without a line for ongoing operations is quoting you a proof of concept, which is why 88% of AI proofs-of-concept never reach production (IDC, 2025). A POC has no operations budget by construction.

Two things genuinely favor custom that rarely make it into the spreadsheet. First, inference costs have fallen steadily and the trend is your friend; a custom agent gets cheaper to run over time while a per-seat license does not. Second, the eval set and retrieval corpus are durable assets — they survive a vendor switch, a model switch, and a re-platforming. A vendor relationship does not.

One thing genuinely favors buying that rarely makes it into ours: the vendor has already been through SOC 2, HIPAA attestation, and a hundred security reviews. If your procurement cycle is the bottleneck, that matters more than any accuracy delta. Our comparison of build, buy, and operate paths lays the tradeoffs out side by side.

When you should absolutely just buy the SaaS

We turn down work on these grounds regularly, and the pattern is consistent.

• The workflow is genuinely standard. Eligibility verification, claim status checks, e-signature routing, calendar coordination. If the rules come from a national standard rather than from your organization, someone has already built it better than you will and sells it for less than your build.
• You cannot produce 200 labeled historical cases. No eval set, no custom agent. If nobody can tell you what the right answer was on last quarter's cases, you will have no way to know whether the agent is working, and you will end up trusting a demo.
• Nobody owns the workflow. If four departments each own a piece and none owns the outcome, a custom agent will surface every disagreement between them as a bug report. Fix the ownership first. This is an organizational problem wearing an AI costume.
• You are automating a process you are about to change. Building an agent for a workflow that a system migration will replace in eight months is a way to spend money twice. Wait.
• The volume is low and the variance is high. Forty complex, unlike cases a month is a job for a skilled human with good tools, not an agent. The eval set will never be representative enough.

There is also a middle path more people should take: buy the platform for the standard 70% and build custom only for the specific slice where your rules live. A commercial clearinghouse for eligibility, a custom agent for the three payers that generate most of your denials. That is often the correct answer, and it is the one nobody selling you either product will suggest.

How to decide this month

Pick one workflow. Pull 200 completed cases from the last quarter with their actual outcomes. Have someone who knows the work write down, for twenty of them, what decision they made and why — in enough detail that a competent new hire could reproduce it. That document is your answer. If it reads like a public policy manual, buy. If it reads like institutional knowledge that would leave with the person who wrote it, build.

Then, whichever way you go, insist on shadow mode before cutover. The agent runs on live cases beside your team, produces output nobody acts on, and you compare accuracy against what the humans actually did. This works for purchased tools too — most vendors will resist it, which is itself informative. That comparison is the only evidence worth having, and it is cheap relative to the cost of finding out in production.

Our two-week AI Readiness Map exists to run exactly this diagnostic, and it ends in a written recommendation that sometimes says "buy the tool, here is which one." If you want to see how the shadow-mode period and the eval harness fit together on a real build, the methodology page walks through the sequence.