AI Quant Builder

Research reliability audit

An illustrative sample of how a strategy is challenged before anyone increases capital.

ILLUSTRATIVE SAMPLE - NO RETURN CLAIM

Audit decision

The workflow is reproducible, but the current evidence does not support scaling.

NOT READY TO SCALE
Data provenance
PASS
Inputs and dates can be reproduced.
Leakage controls
ATTENTION
Universe formation uses later knowledge.
Out-of-sample
FAIL
Holdout result is unstable.
Trading costs
FAIL
Turnover cost is understated.
Decision lineage
PARTIAL
Orders cannot all be linked to decisions.

Critical findings

SeverityFindingWhy it mattersEvidence needed
HIGH Candidate universe is not frozen as of the research date. Later winners can leak into earlier tests. Point-in-time universe snapshot.
HIGH Cost model excludes observed turnover and execution drag. A positive backtest can become negative after implementation. Broker-aware fee, spread, slippage, and fill model.
MEDIUM Promotion compares models built on different data snapshots. The apparent improvement is not a controlled comparison. One immutable snapshot and shared execution contract.

Recommended research actions

Freeze evidence

Create one point-in-time data snapshot and keep future labels outside the discovery window.

Replay with real costs

Run the same rules through a broker-aware execution model before comparing candidates.

Block release

Keep the strategy in research until the holdout and lineage gates are independently satisfied.