Quality
Testing and independent V&V: trust, but verify
Why independent validation catches what the build team can't see, and how to test like production instead of like a demo.
Test execution summary
Run by Build team
Data set Demo What Monday looks like
- 30 employees41,000 employees
- 1 legal entity9 legal entities
- 1 pay group, salaried6 pay groups, 3 unionized
- No retro, no leaveRetro back two periods
- Round numbersMid-period transfers
- Complete historyHistory missing before 2019
- 1 state, 1 currency14 states, 2 currencies
- Nobody terminatedRehires with prior IDs
| Scenario | Result |
|---|---|
| New hire, salaried employee | Pass |
| Standard pay run, one pay group | Pass |
| Approval routing, happy path | Pass |
| Period close, sample volume | Pass |
| Not on the log | |
|---|---|
| Retroactive change | Not runOwner |
| Rehire with a prior ID | Not runOwner |
| Multi-state tax | Not runOwner |
| Full production volume | Not runOwner |
| Mid-period movement | Owner |
| Leave and absence overlap | Owner |
| Deliberately bad input | Owner |
Independent verification
- Reports to
- The program, not the integrator
- Coverage
- Money, employees, regulatory weight
- Authority
- Can say not ready, and be believed
Parallel run Tolerance agreed in writing before the run
- Cycle 1 clean: a data point
- Cycle 2 clean, consecutive: evidence
Go-live decided at the gate
- Zero open critical defects
- Two consecutive clean parallels
- Independent verdict on record
Every program reaches a moment when the build team stands up and says the system is ready. They mean it. They've configured it, walked the demos and watched the happy path run clean, and as far as that goes they're right: the system does what they built it to do.
What they built and what the business needs are two different things, though, and only one of them shows up in a demo.
A test that looks like a demo proves the configuration is internally consistent. It says nothing about whether the business can run on it. Production-like data at production-like volume is the only condition under which the result means anything.
- Employees
- 41,000against 30 in the demo
- Legal entities
- 9against 1
- Pay groups
- 63 unionized, against 1 salaried
- States
- 14against 1
Each row is drawn to Monday’s scale.
Why the build team can't grade its own homework
Dishonesty has nothing to do with it. The people who configured the system are good at their jobs and want it to work. The problem is blind spots, and blind spots come from where you sit.
When you configure something you build a mental model of how it should behave, then you test the paths in that model. You confirm the cases you had in mind and miss the ones you never imagined, because you can't test for a scenario you don't know exists. That gap between what was built and what the business needs is where go-live failures come from.
Independence only holds if three things are set on purpose
A second QA team running the same scripts more carefully adds very little. What independent verification adds is a different vantage point, and without these three it collapses back into the integrator's view of the world.
- A separate line of sight
- The function reports to the program, not to the integrator. Its job is to tell you the truth about quality, whatever that does to a delivery date or a statement of work.
- Who writes their performance review?
- Risk-based coverage
- You can't test everything, so test what matters: the processes that move money, touch employees, or carry regulatory or contractual weight.
- Can they name the top ten by exposure?
- Authority to say not ready
- An assessment nobody is obligated to act on is theater. The function earns its keep when it can hold up a hand before go-live and be believed.
- Has it ever changed a date?
The third one is what gets negotiated away when schedules tighten. Resist that. An independent voice is worth paying for because it can deliver news the program would rather not hear, and one that can be quietly overruled isn't independent.
Test the ugly scenarios on purpose
For testing to mean anything it has to look like the real world on its worst day. These are the cases configuration teams would rather not think about, which is exactly why they belong in the plan with a name against each one.
| Scenario class | What it exposes | Owner |
|---|---|---|
| Retroactive change | A correction dated back two periods. It touches pay, accruals, ledger and reporting at once, and almost never behaves the way the demo implied. | |
| Mid-period movement | Transfers, promotions and cost center changes landing between cycles, where proration and approval routing both have to be right. | |
| Termination and rehire | A rehire carrying a prior identifier, prior balances and prior tax history. The classic source of duplicate records. | |
| Leave and absence overlap | Concurrent leave types, statutory interaction, and the accrual that shouldn't have accrued. | |
| Multi-jurisdiction | Employees working across states or entities, where tax, statutory reporting and entity allocation all disagree if they were configured in isolation. | |
| Volume at the boundary | Period close and payroll running against full production volume, not a sample. Throughput problems only show up at scale. | |
| Deliberately bad input | Missing fields, malformed dates, records that should be rejected. Silently accepting bad data is worse than a hard failure. |
Parallel is the only test that argues back
Everything else compares the system against a specification. Parallel compares it against reality: the same period and the same population, run twice, and the two answers have to agree.
Agree the tolerance in writing before you run, down to the cent. Once it's written down, a variance either falls inside it or it doesn't, and you're making a decision instead of listening to an explanation after the fact.
Then classify every difference by root cause, not by size. Ten small variances from one bad rule matter more than one large keying error.
Programs that run one clean parallel and declare success are reading noise.
1clean cycle is a data point.
2consecutive clean cycles are evidence.
Define severity before testing starts, or it gets argued down
Severity has to be defined by business consequence and agreed before testing starts. Otherwise it becomes negotiable exactly when the pressure is highest, and if severity can be argued down in the week before go-live, your gate has no floor.
-
Define before you test
- What each severity level means in business terms
- Who assigns severity, and who can change it
- The exit threshold for each level
- Which defects block go-live outright
-
Watch during
- Reopen rate, the strongest signal of a superficial fix
- Age of the oldest open critical defect
- Whether severity is drifting downward under pressure
- Defects clustered in one process area
-
Gate on
- Zero open critical defects at the gate, no exceptions
- High-severity items closed, or carrying an accepted and documented workaround
- Two consecutive clean parallel cycles
- The independent function on record with its own verdict
None of this assumes the build team is careless or that anyone is at fault. No team can see its own assumptions from the inside, and that's the whole case for trust, but verify.
Put independent eyes on your program.
If you're betting tens of millions on an ERP program, a candid second opinion is the cheapest insurance you'll buy.