Research reliability audit
An illustrative sample of how a strategy is challenged before anyone increases capital.
ILLUSTRATIVE SAMPLE - NO RETURN CLAIM
Audit decision
The workflow is reproducible, but the current evidence does not support scaling.
NOT READY TO SCALE
Data provenance
PASS
Inputs and dates can be reproduced.
Leakage controls
ATTENTION
Universe formation uses later knowledge.
Out-of-sample
FAIL
Holdout result is unstable.
Trading costs
FAIL
Turnover cost is understated.
Decision lineage
PARTIAL
Orders cannot all be linked to decisions.
Critical findings
| Severity | Finding | Why it matters | Evidence needed |
|---|---|---|---|
| HIGH | Candidate universe is not frozen as of the research date. | Later winners can leak into earlier tests. | Point-in-time universe snapshot. |
| HIGH | Cost model excludes observed turnover and execution drag. | A positive backtest can become negative after implementation. | Broker-aware fee, spread, slippage, and fill model. |
| MEDIUM | Promotion compares models built on different data snapshots. | The apparent improvement is not a controlled comparison. | One immutable snapshot and shared execution contract. |
Recommended research actions
Freeze evidence
Create one point-in-time data snapshot and keep future labels outside the discovery window.
Replay with real costs
Run the same rules through a broker-aware execution model before comparing candidates.
Block release
Keep the strategy in research until the holdout and lineage gates are independently satisfied.