Build notes / Working approach
How will we know an AI solution is useful?
Our measurement approach for the first pilots: a baseline, acceptance criteria, human effort and total cost.
Seeing a first working solution is exciting. Our next question is what it improves for the person using it. As we prepare for the first pilots, we are writing down how that question should be answered.
Question: What would count as useful?
We first need to define the work, its user and the starting point. Which step repeats, where is time lost and which error is unacceptable? We aim to agree acceptance criteria together before building the solution.
Experiment: Make the evaluation questions visible
Our FDE guide brings problem definition, measurement, delivery and operational ownership together. The homepage measurement framework and downloadable discussion briefs prepare the first conversation around these questions.
We include human effort, model usage, compute and maintenance when comparing the baseline with the outcome. This is an approach for preparing a pilot, not a report of completed customer work.
Observation: Time saved and cash impact are separate questions
A faster work step does not mean the same amount of cash has been saved. Review and correction effort also matter. We therefore want to record time, quality and realized cash impact separately.
Next: Test the framework in a permitted pilot
Our next step is to agree a baseline and acceptance conditions with a customer in a scoped pilot. The measurement period, data permissions and permission to share results will be agreed for that work. We have not published verified customer-return results.