A backtest reports a 24% return after a researcher tries 200 variations. The winning version looks precise, but its position at the top of the table is partly the reason it was selected. William Eckhardt’s discussions of systematic research direct attention to that selection process before the result becomes evidence of skill.
The practical questions concern overfitting, sensitivity to parameters, transaction costs and dependence on a few unusual periods. A smooth chart can conceal weaknesses in all four. The independent hypothetical experiments below show what those weaknesses look like in numbers.
William Eckhardt examines the research behind the result
Eckhardt Trading’s company biography describes trading beginning in 1974 after four years of doctoral research in mathematics, without claiming a completed doctorate. It describes systematic futures research across trend and nontrend approaches and multiple timeframes.
In a 2018 interview hosted by the firm, Eckhardt discusses scrutiny of overfit systems and profits concentrated in a few periods, alongside the importance of distribution tails. Those observations motivate the tests here. None of the examples reproduces the firm’s models, current settings or returns.
For the opening 200-variation experiment, the first missing fact is how all 200 were evaluated. Reporting only the selected 24% result leaves the reader unable to see the search that produced it.
The best row carries the history of the search
Suppose a researcher tests ten lookbacks, five exit settings and four volatility filters. That creates 200 combinations. Even when the underlying ideas have no durable advantage, some combinations can fit the particular historical sample better than others.
It’s therefore necessary to assess the selected winner as the outcome of a search. A performance statistic calculated as if this single rule had been specified in advance would leave out part of the selection process.
I read the 24% result as one selected observation within the full 200-row table. The neighbouring rows and the number of attempts help explain why that number deserves more scrutiny than a genuinely predeclared test of one rule.
The historical chart study guide offers a related lesson for discretionary work. Selecting only the clearest winning charts can create a visual version of the same problem. The unsuccessful candidates and ambiguous examples remain part of the research history.
A stable region is different from an isolated peak
Imagine three adjacent lookbacks produce net annualised results of 3%, 24% and 2% in the chosen sample. Another family produces 9%, 10% and 9%. The 24% result sits between 3% and 2% neighbours, which I read as an isolated peak. It’s a reason to examine why a one-step parameter change alters the outcome so sharply.
The smoother family isn’t automatically better. Its returns could still reflect a common error, an unsuitable sample or the same hidden dependence. It’s a comparison that reveals sensitivity concealed by a single best result.
A researcher could inspect which trades enter or leave when the lookback changes. If one exceptionally profitable event qualifies only under the middle setting, that event may explain most of the peak. That’s a more specific investigation than a general claim that the parameter is optimal.
The Donchian channels guide provides an example of a rule whose lookback can be varied. The channel’s simplicity doesn’t exempt a search across many lengths from selection effects.
A later sample loses independence when it becomes a tuning set
Suppose five years of fictional data are used to choose a rule and the next two years are reserved for evaluation. It’s a chance to examine whether the rule travels beyond the sample that selected it.
If the researcher inspects the two-year result, changes the filter and checks it again, that period has now influenced development. Repeating the process until performance looks good doesn’t preserve an untouched test merely because the file is named out-of-sample.
I read the first revision after the two-year result as a change in the experiment: that evaluation set has now influenced the design. That’s the timestamp that matters more than the label on the folder.
A chronological evaluation also has limits. Two later years may contain few relevant events or a very different market environment. A disappointing result can be informative without identifying a single cause, and a favourable result still leaves uncertainty about the next period.
Costs can reverse a small apparent advantage
Consider a hypothetical strategy averaging 0.12R gross across 100 trades, with R defined as initial planned risk. If average round-trip costs are 0.05R, the net mean is 0.07R. At 0.15R of costs, it becomes minus 0.03R.
It’s the same signal sequence in both calculations. Spread, commissions and slippage assumptions determine whether the small gross result survives. A test that uses idealised fills at the exact trigger can therefore misrepresent the economics.
Costs may also rise precisely when the strategy trades most actively. A volatility surge can widen spreads and reduce available liquidity. Using one constant average cost is an assumption worth examining rather than a complete description of execution.
Position size adds another dimension. A fill assumption reasonable for one contract may be unrealistic for a much larger order. A result that scales linearly on a spreadsheet may not scale linearly in the market.
One large winner can be both essential and uncertain
Suppose 100 trades produce a total of 20R, with one trade contributing 25R. The other 99 sum to minus 5R. Removing the large winner makes the strategy look poor, but that removal may also eliminate the type of event the strategy was designed to capture.
It’s the dependence itself that provides the useful finding. The next questions concern how the event entered the sample, whether its fill was plausible and whether the method could remain active through the losses preceding it.
A record containing only one such event leaves its frequency especially uncertain. It doesn’t establish that the event will recur at the same interval or magnitude. Equally, treating it as an outlier to be discarded automatically could misdescribe a strategy whose intended return distribution is uneven.
The Nassim Taleb profile provides related reading on unusual outcomes. The comparison is useful for thinking about tails, while the two bodies of work shouldn’t be collapsed into one trading prescription.
Dependence reduces what the trade count tells us
A study may report 300 trades while 100 of them occur in closely related markets during the same three-week move. Counting every trade as an entirely independent observation would overstate how many separate opportunities the record contains.
There’s a common event capable of producing a cluster of profits or losses. Examining results by period, market group and shared exposure helps reveal that concentration. The choice of grouping needs explanation because it can also influence the conclusion.
Suppose five simultaneous positions each have a planned $500 loss and all gap adversely to twice that amount. The combined loss is $5,000. Reviewing each position in isolation would hide the portfolio-level event.
The historical volatility guide describes one input to risk analysis. Recent variation can be measured accurately while remaining an incomplete model of joint stress.
A research log preserves the decisions that disappeared
Imagine the final rule uses a 40-session lookback, but earlier versions tried 15, 20, 30, 50 and 60. A report describing only the final 40-session version makes the research appear more predetermined than it was.
A useful log records the reason for each change, which data were visible and which alternatives were rejected. It can distinguish a correction to a data error from a parameter change chosen because it improved the equity curve.
The log also gives an independent reviewer a way to ask whether the results survive reasonable alternatives. If the rule depends on one particular price adjustment, one cost estimate and one narrow sample, those dependencies deserve attention before the headline return.
This is a demanding form of transparency because failed experiments take space. They’re still part of the evidence needed to interpret the successful one. Deleting them can make an ordinary search look like a remarkable first insight.
Books and the editions behind the titles
Michael Covel’s The Complete TurtleTrader offers an account of the experiment, while Curtis Faith’s Way of the Turtle provides a participant’s perspective. Jack D. Schwager’s The New Market Wizards includes an interview with Eckhardt, identified in its contents as The Mathematician. These are separate books by their named authors, with the print formats identified below.
Disclosure: As an Amazon Associate, I earn from qualifying purchases.
The Complete TurtleTrader by Michael Covel. Paperback edition.
(US) The Complete TurtleTrader
(UK) The Complete TurtleTrader
(DE) The Complete TurtleTrader
Way of the Turtle by Curtis Faith. Hardcover edition.
(US) Way of the Turtle
(UK) Way of the Turtle
(DE) Way of the Turtle
The New Market Wizards by Jack D. Schwager. Paperback edition.
(US) The New Market Wizards
(UK) The New Market Wizards
(DE) The New Market Wizards
Keep the other 199 rows
The selected 24% result becomes more meaningful when the reader can see the 199 alternatives, the later evaluation period and the cost assumptions. Those details may support the method, weaken it or leave the conclusion unresolved. Recording which assumption changes the answer most gives you a specific place to seek new evidence. That gives the next research step a purpose beyond admiring the headline return.
Eckhardt’s research perspective rewards that scrutiny. The useful output is a claim whose dependencies can be examined, with enough of the unsuccessful work preserved to show how the claim was selected.
Learn the pattern. Ride the trend. Keep the gains.
Educational content only. Not investment advice. Trading involves risk. You are responsible for your decisions.
Get the free Market Wisdom e-book
Join Trends and Breakouts — historical winners, breakout studies, and risk lessons. No spam, unsubscribe anytime.
