The +25% a day back-test: how a rolling-strike bug nearly fooled us
Our first run of a machine-learning option-buying strategy turned ₹1 lakh into more than ₹90 lakh in 18 months, with a 62% win rate. It was false. Here is how we caught the error, what the corrected test showed, and how to check your own results for the same kind of mistake.
The result that looked too good
The study trained a LightGBM model, retrained each quarter on past data only, to predict whether a long call or put would make a net profit under fixed exit rules. The first development run gave these figures for its best settings.
| Measure | First (wrong) run, best setting |
|---|---|
| Trades | 2,021 |
| Win rate | 61.8% |
| Profit factor | 1.63 |
| Net profit on ₹1 lakh | ₹91.5 lakh |
| Average per day | 24.7% of starting capital |
The target for that study was 1% a day, which is already very ambitious. A result 25 times the target should not be celebrated. It should be treated as a bug report until proven otherwise.
Finding the cause
We did not look at the code first. We asked what the model was using. A model's ability to rank winners above losers is measured by AUC, where 0.50 is chance. Intraday market prediction rarely gets far above that, and this model scored about 0.68 to 0.71.
So we retrained it with groups of features removed, and that pointed straight at the problem.
| Features used | AUC |
|---|---|
| All features | 0.708 |
| Index price features only | 0.523 |
| Option-derived features only | 0.695 |
Index price features alone were close to chance, as you would expect. Almost all of the apparent skill came from features describing the option itself, chiefly its delta. That suggested something was wrong with how the option's path was connected to its features, rather than any real market insight.
The bug: a label that moves
Our data provider supplies expired options as rolling series: "at the money", "one strike above", and so on. The strike behind each label changes as the index moves. Our trade simulator picked a label at entry and followed that label through the trade.
That is not the contract anyone would hold. Someone who buys the 24,000 call keeps the 24,000 call. In the simulation, if the index moved 50 points, the "same" position silently became a different strike, at a different price. Those jumps created price moves that never happened to any real position, and the size of the jump depended on where the option sat relative to the strike. The model found that pattern and exploited it, as it was built to do.
The fix was short: record the strike at entry, and at each minute find whichever label carries that strike. Our earlier studies already tracked by strike, so they were not affected. The bug was new, and it appeared in code written for this study.
After the fix
The same quick check showed the change at once: AUC fell to about 0.54, and the win rate of the top-scored trades fell to about 44%, against 38% for all candidates. We discarded the first results, disclosed the error in the study's written plan, and re-ran everything.
| Measure | Corrected result |
|---|---|
| Settings tested in development | 2,880 |
| Share with profit factor above 1 | Under 1% |
| Validation return per day, by capital | -0.04% to +0.05% |
| Validation profit factor, by capital | 0.94 to 1.05 |
| Validation drawdown | 27% to 39% |
| Overfitting check across all settings | p = 0.99 |
The honest answer was no edge, roughly 500 times smaller than the false result at best. The model did carry some information, since it beat random entries, but that information was far too small to pay for spreads, tax and brokerage. The full tables are on the research page.
A checklist for results that look too good
- Compare with what is plausible. Very high win rates with high trade counts are rare in liquid markets. Treat them as a reason to audit.
- Remove feature groups. If nearly all of the skill sits in one group of features, ask how that group could know the future.
- Check identity through time. Is the thing you bought the same thing you sold? This applies to rolling option labels, continuous futures and index membership.
- Replay one trade by hand. Take a single trade and read the actual contract's prices minute by minute.
- Check timestamps. Find out whether a bar is stamped at its start or its end. A one-bar error can make a useless signal look strong. We also found that one data request format silently dropped the first bar of the day.
- Use a null test. Random entries with the same exits should not do well. If they do, the exits or the data may be the cause.
Why we publish this
Most back-tests shown in public report only the final figure. Errors like this one tend to survive when the result is pleasant, because nobody has a reason to look for them. Our pre-registration approach asks for the rules in writing before results are seen, and for any change to be disclosed. It does not stop bugs, but it makes them harder to hide.
This is research on historical data, not a recommendation. See our disclaimer and editorial policy.