Choose document-pilot examples by covering the variations and failures relevant to the business decision. There is no universal sample size that proves a workflow is ready. Start with a coverage matrix, include ordinary and difficult cases, and decide what evidence is needed before expanding the pilot's scope.
A folder of many nearly identical PDFs may test less than a smaller, carefully varied set. Quantity still matters for estimating how often errors occur, but coverage determines whether important conditions have been examined at all. Keep those two purposes separate when interpreting the results.
Build the matrix around the decision
Imagine a fictional service-pack workflow that checks whether job JB-601 has a matching service record and purchase-order reference before staff review. The test set needs more than clean service sheets. It should include a wrong job reference, a revised PO, a missing page, an unreadable scan and a pack containing evidence from two jobs.
For document DOC-14, the expected extraction might be job JB-601 and PO-778. For DOC-15, the page reads JB-610. Even if the text is extracted perfectly, the second document should not support the first job. The test must capture the workflow decision as well as the extracted fields.
| Variation | Example | Expected route |
|---|---|---|
| Clean matching record | DOC-14, JB-601, PO-778 | Ready for staff review |
| Wrong job | DOC-15, JB-610 | Conflict requiring review |
| Missing page | DOC-16, page 2 absent | Incomplete evidence |
| Revised reference | DOC-17, superseded PO | Version check |
| Poor scan | DOC-18, reference unreadable | Unable to determine |
| Mixed pack | DOC-19, two job identifiers | Separate or investigate |
Use real distributions without ignoring rare failures
Estimate which document families and case types occur in the business's work. Include enough ordinary cases to understand normal operation, while deliberately adding consequential exceptions even if they are uncommon. Label deliberately enriched exception sets so that their results are not mistaken for the natural workload distribution.
Keep counts by customer format, input channel, page count and other relevant characteristics. These fields help explain an error pattern. A single overall score can conceal that every multi-page pack or one customer's revised form is handled badly.
Keep expected answers independent
Have appropriate staff establish the expected fields and route before they see the tool's answer. Where the evidence is ambiguous, record that ambiguity as the expected state rather than forcing a value. The correct behaviour may be to ask for review.
Google Document AI's evaluation guidance compares predictions with labelled test documents and discusses precision and recall. That supports careful labelling, but a processor score does not measure the whole business workflow. Your own test also needs approvals, source links, corrections and unsupported inputs.
Separate development from evaluation
Use one set to develop extraction and rules, then reserve a different set for evaluation. Repeatedly adjusting a prompt against the same examples can make the tool look better without demonstrating how it handles unfamiliar records. Keep a record of which examples informed each change.
Do not split pages from the same document between development and evaluation. Closely related duplicates or revisions can also make the evaluation less independent. Preserve their relationship so the team can make a deliberate split rather than accidentally testing near-copies.
Expand when the evidence demands it
Begin with a manageable pilot and examine every error. Add examples where coverage is missing or where failures reveal a new category. If a critical type has not been tested, state that limitation rather than treating absence of observed errors as proof.
A document workflow pilot should deliver the coverage matrix, expected answers and results by case type. Expansion should follow the business's risk and evidence requirements. Keep untested formats outside the approved scope until they have their own review, regardless of how well the original examples performed.
Sources and further reading
AI-assisted research and writing, checked against the linked sources. Worked examples are fictional. Product features can change; confirm the details in your own account.