Parameter Performance Summaries: backtest multiple parameter values in one go

Jul 21, 2021 in Backtest, Parameter Summary

We have just added new functionality to the site that makes it possible to backtest numerous different parameter combinations in one go. We have started with two Parameter Performance Summaries, one focused on relative strength and the other on mean reversion:

• RS Parameter Performance Summary
• TRD Parameter Performance Summary

Parameter Performance Summaries are available to annual subscribers, both regular and pro, and can be accessed from their respective backtests.

Set the weight, min, max and step / increment for each required parameter, then click 'Run Backtests' and the tabulated results will be displayed:

Backtesting is the only way to know if a strategy works, but it obviously does not guarantee good future performance. Following a solid testing procedure, however, puts the odds in your favor. To that end, we recommend keeping the following in mind when using the Parameter Performance Summaries:

• Backtest results shouldn't be used to justify a model; it needs to have a sound underlying rationale to begin with.
• Choose a range of parameter values that make sense for the strategy. i.e., very short lookbacks aren't suited to a strategy targeting longer-term trends and vice versa. Sticking to appropriate parameter values lessens the possibility of being misled by an isolated / lucky result.
• Larger step / increment values can be used initially to identify the approximate range of parameter values that have produced the best returns. More detailed testing, with smaller step values, can then be done on that range of parameter values..
  i.e., begin with a wider spread between min and max and a larger step value. After identifying a narrower min / max spread, the step value can be reduced.
  Alternatively, when employing 2 or more factors, choose 'Serial' rather than 'All Backtests'. Rather than backtesting every parameter permutation, Serial employs a multi-stage process that reduces the total number of backtests performed, thereby allowing longer periods of time and/or a wider range of parameter values to be tested.
• When examining the results:
  • A robust model will be moderately sensitive to small differences in parameter value. i.e. performance will vary, but slightly different parameter values should not produce wildly different returns. Larger differences in parameter value, by contrast, should be expected to have a bigger impact on performance. (note: adjacent monthly parameters will exhibit larger performance differences than adjacent daily lookbacks.)
  • If the top performing parameter's returns are far above the rest, then it indicates that its results likely benefited from good luck.
  • The best parameter / lookback values are generally those that show consistency over time. i.e. parameter values that were hugely successful in favorable environments but performed poorly in other conditions, are less desirable than parameter values that produced solid returns across different market environments.
  • The overall backtest should be of sufficient length to include a mix of environments; up, down and sideways markets. Examining shorter periods within that backtest is also worthwhile, as it’s unlikely that the best overall performers were strongest in each and every sub-period. Recognizing that even the best strategies have endured periods of under-performance can help set realistic expectations.
  • The Parameter Summary provides an overview, but it’s important to go beyond the headline statistics once a set of parameter values has been identified. Run a backtest and examine the return and drawdown for each year vs your benchmark. Was it a wild ride? Could you have actually stuck with it (be realistic)? Does huge out-performance in only one or two years mask under-performance the rest of the time? If so, was that because the model does well only in a particular environment, (if so, can you live with that?) or, was it just lucky at certain times?
• The future is all that matters and it won't be exactly like the past, so there's little point in trying to precisely optimize historic performance.

You may need to refresh the site to force your browser to download all the necessary files from the server afresh in order for the pages to render correctly. To do this, hold down the Shift key on your keyboard and click on the circular arrow icon to the left of address field at the top of the browser window.

Follow ETFreplay on