Shadow mode: testing automation without touching live work
How an automated system is proved against the people currently doing the work, before it is allowed to do any of it.
Short answer. Shadow mode is running the automated system on real, live work in parallel with the team, while acting on none of its output. Nothing it decides is used, nothing it produces reaches a customer, and nothing is written into live systems. You compare its decisions against the team's and only advance when the agreement rate is good enough.
The problem it solves
Automation is usually tested on examples chosen by whoever built it. Those examples are clean, representative and few. Real work is none of those things.
Going straight to production means the first genuine test of the system is a live customer, a real invoice or an actual claim. If it is wrong, you find out from the consequences.
How it runs
The system receives the same inputs as the team, at the same time, and produces a decision for every item. That decision is recorded and nothing else happens to it.
- No output reaches a customer or supplier
- Nothing is written into the systems of record
- No payment, approval or commitment is issued
- The team is not asked to check its work, and carries on exactly as before
The last point matters commercially. Shadow mode costs the client no additional time. If it required the team to review every machine decision, it would be adding work rather than testing a replacement for it.
What gets measured
The useful number is the agreement rate: on what proportion of items did the system reach the same decision as the person who actually handled it.
The disagreements are more informative than the agreements. Each one resolves into one of three findings, and all three are worth having:
- The system is wrong, and a rule needs correcting
- The system is right, and the manual decision was an error
- Both are defensible, and the written policy is genuinely ambiguous
That third case is common, and it is usually the most valuable output of the exercise. A process that cannot be automated consistently often turns out to be one that is not being applied consistently by people either.
When to advance
The threshold should be agreed in writing before shadow mode starts, so that nobody is negotiating the bar after seeing the results. A typical shape is a stated agreement rate sustained over a set period with no serious errors.
Two to four weeks of live running is usually enough to see the pattern, but the client sets the duration. There is no reason to rush it, and no charge for the period.
What it does not do
Shadow mode does not make automation safe in general. It gives you evidence about one system on one process against one team's current standard. It says nothing about what happens when the policy changes six months later, which is why anything running in production needs maintaining.
Related questions.
Running an automated system on real live work in parallel with the people currently doing that work, while acting on none of its output. Its decisions are recorded and compared against the team's, but nothing it produces is used.
Typically two to four weeks of live running, which is usually enough to see the pattern including a normal spread of exceptions. The client sets the duration and there is no reason to shorten it.
That should be agreed in writing before shadow mode starts, so the standard is not renegotiated after the results are seen. It is normally a stated agreement rate sustained over a set period with no serious errors.
No. The team is not asked to review the system's decisions and carries on exactly as before. If it required review it would be adding work rather than testing a replacement for it.
Describe the process you want to automate.
You will receive a fixed price and a build plan.
There is no charge for the initial call and no obligation. You deal directly with the engineer who builds the system.