What do you mean by an AI skill?
An AI skill, for assessment purposes, is a behaviour you can inspect on a particular task. Supplying an appropriate source, identifying an unsupported claim or explaining a final edit is more specific than saying someone is good at AI. Name the role, the tool and the conditions so a reviewer knows what the evidence actually concerns.
A broad label can hide different abilities. Someone may write a clear prompt but fail to check a citation, or check arithmetic while accepting an unsupported explanation of the cause. Separate those observations rather than reduce them immediately to one overall score. The AI skills guide describes seven practical skills that most knowledge work draws on; an assessment should name which of them a task is meant to show.
What is the assessment for?
Decide the purpose before you design anything, because it changes the stakes. A learning assessment helps someone find what to practise next and should feel safe to fail. A readiness check helps a manager decide where to start a programme. A certification assessment supports a claim to others that someone met a defined standard on defined tasks. Hiring is a separate case, covered below.
Mixing purposes causes most of the trouble. If people believe a diagnostic will feed into performance reviews, they will play safe, avoid the tool or ask a colleague for help, and the result tells you little. Tell people what the result will be used for, who will see it and what will not happen because of it.
Which signals should you keep separate?
Confidence, completion and demonstrated work answer different questions. A self-rating tells you how someone feels. A completion record tells you they reached a programme milestone. A practical task gives them an opportunity to show particular decisions under particular conditions. None should silently stand in for the others.
A useful assessment report states the method and its limits. If a learner completed one fictional drafting task, say that. Do not imply that they can safely handle every confidential or consequential workflow. NIST's AI Risk Management Framework makes a similar point about AI systems themselves, asking that limits on generalising beyond the tested conditions be documented. The same discipline applies when people are being assessed.
| Signal | What it can show | What it does not establish |
|---|---|---|
| Self-rating | Reported confidence or uncertainty. | Demonstrated performance. |
| Completion | A programme milestone was reached. | Competence across every work task. |
| Tool usage | That the tool is being opened and used. | That the output is checked or suitable. |
| Practical sample | Decisions on a specified task. | A universal prediction of job performance. |
How do you choose a practical task?
Choose work that is relevant to the role and supply enough source material to judge the answer. A marketer can revise a fictional campaign email. An HR colleague can improve interview questions against a role brief. A finance colleague can explain a small table while leaving unsupported causes unresolved. A support agent can reply to a customer question using a supplied policy extract.
Make the task safe to practise. Use fictional or approved information and avoid decisions about real people, money or customer communication in an initial exercise. The goal is to inspect the work, not to create a consequential automated decision while trying to measure a skill.
Good tasks have a trap that matters. Plant one realistic problem the learner should catch, such as a policy detail the tool is likely to invent or a figure that needs recomputing. A task with no possible error only measures whether someone can operate the tool.
What does a complete assessment task look like?
Here is a full example for a support role at a fictional furniture retailer, Larchwood Home. The learner receives a customer email asking whether a sofa delivered with a torn cushion can be replaced, and a policy extract that says damaged items reported within 14 days are replaced free, and the customer must send a photo. The customer reported the damage on day 9.
The instruction reads: "Use the approved AI assistant to draft a reply. Use only the supplied policy. Do not include the customer's address or order number in your request to the tool. Submit your request, the tool's draft, your final reply and two or three sentences explaining your edits."
The trap is that generic tools often add a refund option, a delivery date or a goodwill voucher that the policy does not mention. A strong submission keeps to replacement, asks for the photo, gives no date the policy does not support and explains the removal of any invented offer. A weak one sends the tool's draft with light edits. Both may read well; only one is safe to send.
What belongs in a visible rubric?
A visible rubric states the criteria before submission and describes what each level looks like. Useful categories include context, evidence, information handling and review decisions. Explain what a reviewer is looking for in the work, rather than attaching a number without a description. A learner should be able to understand what to improve after reading the feedback.
The rubric below fits the Larchwood task. It is an editorial assessment structure, not a claim of psychometric validation, and it has no validated pass mark.
| Criterion | Meets the standard | Not yet |
|---|---|---|
| Context | The request supplies the policy, audience and constraints. | The request leaves the tool to guess the policy. |
| Evidence | Every promise in the reply traces to the policy extract. | An invented refund, date or voucher remains. |
| Information handling | No personal details were sent to the tool. | The address or order number appears in the request. |
| Review decisions | The explanation names what changed and why. | No explanation, or only "tidied it up". |
Who should score the work, AI or a person?
Either can, provided the criteria are visible and the scoring can be challenged. A person who knows the work is the most trustworthy reviewer but is slow and can be inconsistent across many submissions. AI grading against a rubric is fast and can give specific feedback, but a confident score is not proof of accuracy, and AI graders can miss context a domain expert would see.
A practical compromise is AI feedback for practice tasks, a route to flag disputed scores for a second check, and human review where the result carries consequences. Tosheo.ai's planned design follows that pattern: tasks graded by AI against a visible rubric, with learner flags sent to a review queue. That flow is planned scope, not evidence that an implemented grader has been validated.
Want to see how this connects to a role?
Explore the planned team trial ↗How do you keep reviewers consistent?
Calibrate before you score. Ask two reviewers to mark the same three or four sample submissions independently, compare where they differed and agree what the rubric wording means. Write down the decisions as short notes beside each criterion, such as "a goodwill gesture counts as an invented offer unless the policy allows it".
Re-check occasionally by having a second person review a small sample. NIST's measurement guidance notes that independent review can reduce internal bias. In a small team that may simply mean the founder and a senior colleague each review a handful of the same submissions once a quarter.
How should you interpret repeat attempts?
A second attempt can show that someone understood feedback on that example. It can also benefit from remembering the task. Use a new but comparable example before making a broader inference, and record changes in the tool, source material or assistance available to the learner.
Where a score matters, make the review process inspectable and provide a way to challenge it. AI-generated feedback can be useful to discuss, but it should not be the last word on a consequential outcome.
How should you report the results?
Report what was observed, under what conditions, and what remains unknown. A short, consistent record is more useful than a single score. The template below can be used per person or per role.
Task and conditions
The task, the tool and version or plan used, the source material supplied, and any help available.
Criteria
The rubric used, linked or attached, so a reader can see what was judged.
Observations
What the person did well and what was not yet at standard, criterion by criterion, in plain words.
Limits
What this task did not test, such as confidential data, live customer contact or unfamiliar tools.
Next step
One practice focus and the date of the next comparable task.
How does skill assessment differ from adoption tracking?
Tool usage and engagement can help a manager understand whether a programme is being used. They do not by themselves show that a person is making suitable decisions. A frequently used tool can still produce work that nobody checks, while a careful learner may use it only for a narrow approved task.
Review adoption alongside work samples, policy questions and the support available. State which signals are observed and which are missing. Do not turn a change in login frequency into a productivity claim without an appropriate method and contextual evidence.
What should managers be able to see?
Managers need enough to support people: which skills someone has shown, where they are stuck and whether they are progressing. They rarely need the raw text of every conversation someone had with an AI tool. Reading those logs feels like surveillance, discourages honest practice and can expose information the learner should not have shared.
Decide what is visible and say so before anyone starts. Tosheo.ai's planned manager dashboard is designed to show skill progress and completion, not raw chat logs, for exactly this reason.
How is this different from assessing a candidate?
Learning and hiring serve different purposes. A training task can give an existing employee space to make a mistake and improve. A candidate assessment belongs in a fair, defined hiring process with appropriate human review and clear information for candidates. The same example should not be reused as if the contexts and consequences were identical, and a training result should not be used to make an employment decision it was not designed for.
When should you not run an AI skills assessment?
Do not run one before people have an approved tool and basic information rules; you would be measuring access and confusion rather than skill. Do not run one as a disguised performance review. And do not use a self-reported questionnaire, including the readiness self-check on this site, to rank people, award credentials or make employment decisions.
If you need a validated psychometric instrument for high-stakes decisions, the approach described here is not that. It is a practical method for learning conversations and programme planning, informed by NIST guidance on context and limitations, which does not endorse or validate this employee-assessment approach.
What should a manager do next?
Choose one role-relevant task, publish a small rubric and review the reasoning with the learner. Keep a record of the source, tool and conditions. Decide what the evidence supports and what needs another example. That is a practical starting point for a useful learning conversation.
If the bigger question is whether the team is ready to practise at all, start with the readiness checklist, which covers tool access, information boundaries, review standards and support. The self-check can help open that conversation. Both are discussion aids, not substitutes for demonstrated task performance.