How do you measure whether AI training worked?

How do you measure AI training, in short?

Measure at three levels, in order: can people do the task to a standard, do they do it on real work, and did something the business cares about change. Completion rates and satisfaction scores are worth recording, but they show that people attended and liked it, not that they learned.

This mirrors the widely used Kirkpatrick Model, whose four levels are reaction, learning, behaviour and results. Its own definitions describe level 3 as the degree to which people perform the critical behaviours in their environment, and level 4 as the degree to which targeted organisational outcomes occur. For AI training, levels 2 and 3 are where the useful evidence usually is.

What does a good standard for a work sample look like?

A good standard is a short rubric someone else could apply and reach the same view. For an AI-assisted customer reply it might cover: uses only the facts supplied, answers the customer's actual question, matches the house tone, flags anything uncertain and contains nothing that would need correcting before sending.

Score each point simply, such as met, partly met or not met, and keep the reviewer's note. The note is often more useful than the score, because it shows what to practise next.

What does a before and after comparison look like?

Fernhill Outdoors, a fictional 15-person online retailer, trained its four support agents. Before training, each drafted a reply to a sample complaint using ChatGPT. Typical drafts were polite but invented a refund policy detail and missed the customer's question about delivery dates.

After four weeks, each agent drafted a reply to a different complaint against the same rubric. The support lead recorded which points were met and noted that three of the four drafts now flagged the uncertain delivery date rather than inventing one. That is evidence of learning on one task, for four people. It is not proof of faster support or happier customers, and the page should not claim it is.

How do you tell whether people use the skill on real work?

Look for the behaviour in normal work rather than asking people whether they use it. Useful signs include reviewed AI-assisted drafts in the shared folder, a manager seeing fewer corrections of the kind the training targeted, and people raising information boundary questions before acting.

Tool usage reports help with the access side. Microsoft's Copilot usage report, for example, shows active users and last activity dates. Activity shows that the tool was used, not that it was used well, so pair it with a review of actual work.

Can you calculate ROI from AI training?

You can estimate it, but be careful what you claim. A time saving reported by employees is a perception, not a measurement. If a result matters, time a comparable task before and after, or compare reviewed output quality, and record how many people and tasks the estimate covers.

Also count the costs honestly, including learner and review time. Many small teams will find that the clearest evidence is a set of reviewed work samples and a routine people keep using, rather than a single ROI figure. That is a sound basis for deciding whether to continue.

Where does Tosheo.ai fit?

Tosheo.ai is being built so that every lesson ends with a task graded against a visible rubric, with specific feedback, and learners can flag a score they disagree with for review. Managers are planned to see skill progress by role, which supplies the learning evidence. It cannot measure your business results for you.

When is Tosheo.ai not the answer?

If you need a formal evaluation for a funder or board, with control groups or audited figures, you need an evaluation specialist. If you mainly need to track tool licences and activity, your vendor's admin reports do that. Tosheo.ai does not integrate with an LMS at launch.

Sources and scope

Kirkpatrick Partners, The Kirkpatrick Model, checked 16 October 2026 ↗

Microsoft Learn, Microsoft Copilot usage report, 14 July 2026 ↗