Your backtest needs 13 years to prove what it claims
We found this by discovering our own tests were underpowered by one to two orders of magnitude. The best result in one experiment sat below the threshold that experiment could detect.
A directional strategy claiming a 52% hit rate sounds modest. Detecting one, at 80% power, needs 4,900 independent observations. On a single instrument at a daily horizon that is 13.4 years. A 51% edge needs 53.7.
| Independent observations | Smallest detectable hit rate |
|---|---|
| 270 | 0.5852 |
| 900 | 0.5467 |
| 2,695 | 0.5270 |
| 5,000 | 0.5198 |
| 20,000 | 0.5099 |
How we found it
By auditing our own work. One experiment ran 2,695 rows per cell and could resolve nothing below 54.67%. Its best cell measured 52.10% — the best result in the experiment sat below the threshold the experiment could detect. A real 52% edge would have produced exactly the data we saw, and we would have missed it every time.
The trap is the independence count
Those 2,695 rows were really 898 independent observations. Decisions were taken three times a day against a one-day horizon, so each measurement overlapped the next twofold. Overlapping windows inflate the apparent sample threefold and the apparent t-statistic by the square root of three.
A second experiment looked like 234 observations and was 80: 90% of its events fell within a minute of another, making them episodes rather than independent draws.
What it means for anyone selling a signal
Most crypto strategies are validated on a few hundred trades. At that sample the smallest detectable edge is around 58% — far above anything realistic. The honest statement after such a test is not "this works" or "this fails" but "this test cannot tell."
We now require every experiment to state, before it runs, the effect size it can detect, the independent observations that requires, and how long collecting them takes. Several of our own would not have been run under that rule.