How to test a trading idea without fooling yourself: pre-registration and sealed hold-outs
Anyone can find a profitable back-test: try enough settings and one of them will look good by luck. The hard part is knowing whether it will survive. Here is the process we use for every strategy on our review site, why each step exists, and what happened when it caught us out.
The problem: the garden of forking paths
Suppose you test an indicator with 40 different periods, 5 stop levels and 3 exit rules. That is 600 combinations. Even on pure noise, the best of 600 will often show a pleasing profit and a respectable t-statistic. The number you see is the maximum of many tries, and the maximum is biased upward. Published example from our own work: on synthetic data with no edge the best in-sample configuration reached a t-statistic of 2.07. That is what "selection bias" looks like in a single sentence.
The seven steps
1. Pre-register the rules
Before looking at any result we write down: the universe, the signal, entry and exit, position sizing, capital, costs, the pass criteria, and what would make us reject the idea. Interpretations of ambiguous wording are stated up front. Any change afterwards is a new version, labelled as such, not a silent edit.
2. Use three windows
- Development (roughly the first half to 55% of the data): where rules and parameters may be examined.
- Validation (the next 25%): run once, with the chosen rules, to see whether the result survives out of sample.
- Sealed hold-out (the last 20%): never touched until the owner explicitly asks, and only once per strategy. If a diagnostic accidentally peeks, we disclose it.
3. Count costs honestly
Brokerage, STT, exchange fees, GST, stamp duty and slippage, with whole lots. Then stress them at 1x, 2x and 3x: an edge that vanishes at 2x costs is smaller than the error in the cost estimate. See what the 2026 STT rise did to costs.
4. Compare with random behaviour
For every signal-based strategy we run the same exits with random entry times (30 repetitions). If random entries earn as much as the signal, the signal adds nothing: that was the finding for an EMA option seller where random-time selling lost nearly as much as the signal.
5. Correct for the number of tries
We apply a reality check: a bootstrap of the best result across all variants examined. In a 27-variant grid on an RSI strategy, the best variant had a reality-check p-value of 0.89: statistically indistinguishable from the noise. See the RSI study.
6. Look for plateaus, not peaks
If a strategy only works at a 14-day period and fails at 13 and 15, it is a coincidence. Neighbouring parameters must also work.
7. Publish the failures
Failed ideas get the same page as promising ones: every trade, the audit checks, the costs, the controls. Most strategies on our review site are labelled "no edge", which is the honest base rate.
Examples where the process changed the result
- The rules had to be amended, openly. In an RSI study our risk brake deadlocked (it could never re-arm after a pause), which distorted the first run. We documented the bug, amended the rule with a dated note, and re-ran; the first run's result is kept in our research files. A reader can see the path, not just the final number.
- The brief's own constraints contradicted each other. A gamma-based spread strategy required a 0.20-delta short leg and a 25% credit: 291 of 292 signals failed the credit rule. We recorded an amendment before seeing profit or loss: see why a 0.20-delta credit spread pays about 14% of its width.
- A good-looking candidate stayed a candidate. A re-costed overnight strangle passed its pre-registered checks, but every period had been seen during development, so we call it a candidate and run a forward paper test instead of promoting it: overnight strangles.
Why a sealed hold-out is not enough alone
A hold-out only works if it is genuinely untouched. Every look at it, even to "just check", turns it into a second development set. We keep a rule: open it once, on explicit instruction, after the rules are frozen. If we open it and the strategy fails, the idea is closed; we do not tweak and reopen.
A checklist you can apply to any back-test you are shown
- Were the rules fixed before the results? Is there a dated version?
- How many variants were tried in total? What is the corrected significance?
- Which data were never used to choose anything?
- What is the gross result per trade, and the charges per trade?
- Does it survive 2x costs? What does the random-entry control earn?
- How much of the result depends on modelled prices or assumed fills?
- Do neighbouring parameters also work?
Limits
None of this guarantees future profit; it only filters out many false positives. A strategy that passes can still fail when markets change, and a strategy that fails our tests may have worked for reasons our data could not capture. We say what we could and could not test on each page. Examples above are from our public Strategy Review; nothing here is a recommendation to trade.