AI & GenAI Enablement
Evals before endpoints. Cost envelopes before pilots. Architecture that survives contact with a regulator.
§ 01 — Position
The demo works. It nearly always works — retrieval over internal documents is a solved problem at demo scale, and the models are good enough that a promising result is easy to produce. What follows the demo is where projects stall.
Two questions arrive within a week of anyone senior seeing it: how accurate is this, and what does it cost to run. Both are answerable on day one and are usually asked in month six, by which point the architecture has hardened around assumptions nobody tested.
This practice is about doing that work in the right order. Not because measurement is more interesting than capability, but because a system nobody can evaluate is a system nobody can defend.
§ 02 — Method
- 01
Define correct before building
A fixed set of questions with known-good answers, drawn from the real corpus and reviewed by someone who understands the domain. A hundred is usually enough. Without it there is no way to tell whether a change to chunking helped or hurt.
- 02
Model the cost envelope early
Tokens per query multiplied by expected volume is arithmetic that can be done before a line of code. Doing it early occasionally kills a project, which is considerably cheaper than discovering the same numbers in month nine.
- 03
Design the control plane alongside the capability
Agents that call tools need identities, scopes, audit trails, and a kill switch that does not require a deploy. Scoped after the fact, this becomes hardening, and hardening is what gets cut when the date holds firm.
- 04
Build the approval path
Model governance is mostly organizational: who signs off, against what criteria, with what evidence. Architecture that produces that evidence as a by-product is far easier to approve than architecture that has it retrofitted.
§ 03 — Deliverables
- Evaluation harness with a domain-reviewed question set
- Reference architecture for retrieval or agentic workloads
- Cost model with per-query and volume projections
- Identity, scope, and audit design for tool-using agents
- Model governance framework and approval criteria
- Guardrail and failure-mode design
- Architecture decision records for every significant choice
§ 04 — Fit
Best whenA pilot has proven the capability and now has to become something defensible.
Not forChoosing a model vendor. That is a procurement question, and the answer changes every few months.
§ 05 — Field notes
§ 06 — Other practices
A pilot has proven the capability and now has to become something defensible.
Start a conversation