The correct answer in an AI workflow evaluation should be established by people authorised to interpret the relevant business requirement. Developers can help structure labels and check consistency, but should not invent domain decisions. Where the evidence is incomplete or disputed, the expected answer may be review required rather than a definite value.
Ground truth is a practical reference for the evaluation, not a claim that every record contains an unquestionable fact. It needs an owner, documented rules and a way to correct mistakes. Otherwise, a score can reward the tool for reproducing an incorrect assumption.
Separate observation from decision
Consider a fictional job pack for JB-214. The service sheet shows a date of 6 February and a handwritten name. The purchase order refers to site ST-09, while the service sheet lists ST-90. A model extracts all three values clearly.
The observation labels can record those visible values. The workflow label should identify a site conflict, with the pack held for staff review. A developer should not decide that ST-90 is a typo and correct it to ST-09. The responsible operations reviewer must determine what evidence would justify that change.
| Layer | Label | Who establishes it |
|---|---|---|
| Visible content | Service date: 6 February | Trained document reviewer |
| Record comparison | PO site ST-09; sheet site ST-90 | Reviewer using agreed matching rules |
| Business route | Hold for site clarification | Authorised process owner |
| Final release | Not decided by this test | Person with release authority |
Write a short labelling guide
Define each field, acceptable formats and how to mark missing, conflicting or unreadable values. Explain whether a label records the original text or a normalised form. For dates, preserve the original where the format is ambiguous; do not make a regional interpretation silently.
For decision labels, show the rule and required evidence. Include examples of a straightforward case and a case that should be escalated. A guide should let two reviewers understand the task similarly without requiring them to agree on matters outside their authority.
Review disagreements explicitly
Have a second qualified reviewer examine cases where interpretation has material consequences or where the first label is uncertain. Record both views and the reason for the disagreement. An adjudicator with the right domain authority can decide the reference label or retain an unresolved state.
Do not resolve disputes by majority vote when the disagreement is about a business rule that has never been approved. The rule owner needs to settle the requirement first. Similarly, an existing spreadsheet value is useful evidence, but not automatically the correct label if the source documents contradict it.
Keep labels independent of the model
Reviewers should establish the expected answer before being shown the model's proposed answer wherever practical. Seeing a plausible suggestion first can steer interpretation. If labels are revised after an error investigation, record why and re-evaluate affected results consistently.
Keep the evaluation version linked to the rule version and source documents. When the business changes a policy, old labels may no longer represent the current workflow. Decide whether to retain them for historical comparison or create a revised evaluation set.
Test the labelling process itself
- Can a reviewer trace every consequential label to evidence?
- Are uncertain cases allowed to remain uncertain?
- Does the decision stay within the reviewer's authority?
- Are rule changes distinguished from corrected labelling mistakes?
- Can another reviewer reproduce the expected route?
These checks are often more useful than adding another decimal place to an accuracy score. A small, well-understood reference set can expose requirement gaps before the tool is connected to business systems.
Use the labels to support a bounded pilot
For a workflow evaluation, agree the reference answers, disputed cases and acceptance process with the business owner. Report errors against that agreed scope. Keep the final operating decision with people, especially where the test establishes only that information was found and compared, not that a commercial or professional approval is warranted.
Sources and further reading
AI-assisted research and writing, checked against the linked sources. Worked examples are fictional. Product features can change; confirm the details in your own account.