Pilots & evaluation

How do you measure a manual workflow before deciding whether to automate it?

A useful baseline describes the whole workflow, including follow-up and correction. This guide explains how to observe work without confusing elapsed time with staff effort, choose consistent counting units and record exceptions. A fictional enquiry illustrates the measurements needed before comparing an automated alternative.

Published 5 October 20263 min readBy Clear Anvil

Measure a manual workflow from the point an input becomes available to the point the intended output is ready. Record active staff effort, waiting time, handoffs, corrections and unresolved cases separately. Use a consistent unit, such as one enquiry or one completed-job pack, before deciding whether automation addresses a meaningful problem.

Do not time only the most visible data-entry step. A process may involve several short visits over days while staff wait for missing information. Automating the first extraction may leave most of that work unchanged, so the baseline should reveal where effort and delay actually occur.

Follow a complete case

Consider a fictional bespoke-product enquiry ENQ-118. A coordinator spends four minutes reading the email, six minutes organising attachments and three minutes drafting missing-information questions. The customer replies two days later. Staff spend five minutes updating the brief and two minutes handing it to an estimator.

Active effort is 20 minutes across those recorded steps. Elapsed time is more than two days. Those are different observations. A tool that prepares the initial brief might affect some active work, but cannot be credited with removing customer waiting time unless the pilot demonstrates that change and its cause.

Fictional baseline record for ENQ-118
StageActive effortWaiting or handoff
Read enquiry4 minutesNone recorded
Organise attachments6 minutesNone recorded
Request missing information3 minutesCustomer reply after two days
Update brief5 minutesNone recorded
Handoff to estimator2 minutesEstimator queue measured separately

Choose the denominator deliberately

Count enquiries, line items, documents and staff touches separately where they explain the workload. One enquiry with 40 equipment lines is not equivalent to one with two lines. Record complexity indicators so that later comparisons do not accidentally match simple pilot cases with difficult baseline cases.

Define when a case starts and ends. For quote preparation, the endpoint might be ready for estimator review rather than quote sent. Keep that boundary identical in the pilot. Otherwise, an apparently faster result can merely reflect moving unfinished work outside the measurement.

Observe ordinary work and exceptions

Include representative periods and the case types the team actually handles. Avoid asking staff to select only clean examples. Record missing references, revised files, duplicate enquiries and cases that never reach completion. Unfinished work belongs in the baseline because it consumes attention even when there is no final output.

Keep the recording method proportionate. A short task log can capture stage, minutes, reason for revisit and current status. Explain that the purpose is understanding the process, not ranking individual employees. Differences may reflect case mix or access to information rather than personal performance.

Record quality alongside time

  • Was the output complete enough for the next person?
  • Did someone return it for correction or clarification?
  • Were source references and document versions preserved?
  • Which important details were missed or misinterpreted?
  • How much later work came from the initial error?

A faster brief that creates more estimator questions is not an obvious improvement. Where quality is disputed, have the appropriate process owner define the expected result. Keep uncertainty visible rather than converting every judgement into an apparently objective score.

Use the baseline to narrow the pilot

For an AI-assisted pilot, NIST AI RMF 1.0 provides broader evaluation guidance: document test sets and metrics, compare performance with benchmarks, and evaluate conditions similar to the intended deployment. The timing example in this guide is our illustrative workflow method.

Review the observations with the people doing the work. Select a bottleneck the proposed tool can plausibly influence, such as locating attachments or identifying missing fields. State what will remain with staff and what evidence would show improvement.

A workflow pilot should compare the same endpoint, case mix and quality requirements. Include review, correction, exception handling and monitoring in the automated route's effort. The baseline then becomes a useful decision tool, rather than a collection of optimistic before-and-after numbers.

Sources and further reading

AI-assisted research and writing, checked against the linked sources. Worked examples are fictional. Product features can change; confirm the details in your own account.

START WITH ONE USEFUL CHANGE

What could your team
spend less time doing?

Tell us about the work, the tools, and the bottleneck. We can explore a focused pilot by email.

jamil@clearanvil.com