Notice: New Feature Added To Backtesting

Feb 25, 2011 in Backtest | Relative Strength

Last summer, we added a feature to the portfolio relative strength backtest application. We added a line that showed what the 'provisional picks' would be if the update period had just ended. The provisional picks listed are the exact same as if you ran the ETF screener using the same list of ETFs and same parameters. We added this feature just as a convenience so it wouldn't be necessary to open another tab and check what was strongest now -- rather than at the last update period.

Depending on the list you use, it is of course possible that the 'provisional pick' is not the final pick (hence the term 'provisional'). But what if we just looked up the provisional pick on the next-to-last day of a given month and bought that ETF the next day on the close and held it for the subsequent period? What would the performance look like assuming we did that? Would it be similar?

Let's look at a simple example just to discuss the mechanics of what we mean. We will compare using the 'next-to-last day picks' with the 'last day picks.' The holding periods will be the exact same, we are just reading the picks of the ETF screener with a one day offset. Of course, for many periods the picks (and therefore performance) will be the exact same.

In this example, we will use one of the Ivy Portfolio lists of 5 basic ETFs and a semi-monthly update period. We are choosing the top 1 of 5 and holding it on 2-week intervals. The settings are the exact same except we are checking the box "Invest in next to last day pick(s) near the top of images below.

Using Regular 'Last Day' Picks:

Using Next To Last Day Picks:

We can see that the returns follow similar paths but that there was a difference -- even for this list of just 5 ETFs. In this case, the next-to-last day picks actually performed better -- which means very little in and of itself. We simply are pointing out the mechanics of how it works. Users should test this using their lists across various other settings and draw their own conclusions. Our view is that this doesn't change anything -- if your backtest is well thought-out, then this extra analysis will very likely show a result that is in the same ballpark as the original method.

Final note is that today is Friday, February 25th -- so the next-to-last days picks will be locked in and known after todays close. The final picks will update using Monday's (Feb 28) closing price. In both cases, the backtests will assume the cost-basis for March performance is the closing price as of Feb 28.


Follow us on Twitter:


Comments (24) -

Feb 25, 2011 08:48 #

Could you give us some insights about why you thought that adding this feature was important enough for you to develop it?

Paul Belgium

Feb 25, 2011 09:31 #

It is just a small feature we thought was interesting enough to add.   If the 'picks' are actually specified a full 24 hours in advance of the actual update, how often does that extra day create a change in the final pick -- and to what extent does it actually make that much of a difference when it is off?   This feature doesn't totally answer that -- but its a step towards it.

There have also been some interesting anecdotal things that occured in 2010.   For example, lets say you bought the 'next-to-last-day' pick but then there was a very late change.   Let's say you missed the close for whatever reason and/or couldn't determine the final pick accurately.    Then on the first of the next month, the market gaps and trends up.   In this situation, thinking about this actually must be a multi-stage process.  

First, are you in the wrong ETF(s)?   If you are, who is to say that the wrong ETF doesn't outperform the 'final pick'  ETF on the first day of the new month?   We noticed this actually happens a lot.    That is, the next-to-last-day pick actually outperforms the last day pick on the first day of the new period.   When this happens, you could actually swap to the final pick on the close of the first day and lock in that performance and actually be better off doing this.

We don't mean to say this is a robust strategic thing to do --- we just want to highlight a few of the factors that go into some of these nuanced short-term decisions.

Chris United States

Feb 25, 2011 16:07 #


I think this is an interesting feature.  Your last comment touches on one of the difficulties of making purchases on the first day on the month.  The first on the month tends to be an outperformance day; hence, it is almost impossible to buy at the previous end-of-month closing price.  The first day of the month more often that not is a huge gap-up day.

Ernest H Harris United States

Feb 26, 2011 06:52 #

I wonder how different it would be if the cost-basis is also offset by one day???.

Ernest H Harris United States

Feb 26, 2011 08:02 #

In actual trading practice, why should it make a difference what day you buy as long as you're following a disciplined approach and have demonstrated (backtested) that the day you've chosen will produce the level of outperformance you're looking for?   Are there events (dividends / distributions) or herd trading patterns that favor purchase at month close?

It would be a great tool enhancement to be able to specific which trade day to purchase by using a +/- value.   Example:  Buy at month close +5 trade days or buy at month close -4 trade days.  Using this example, you've implemented a capability to model a purchase at month close -1 trade day.

Mike United States

Feb 26, 2011 10:36 #

A very useful new feature to implement in backtesting would be to add "trading costs".

Implemented either as:
- a fixed percentage of the traded amount (percentage to be entered as a variable by user)
- a fixed dollar amount per trade (amount to be entered as a variable by user)
- a combination of both of the above

This would allow us to better estimate what the "real-world returns" would be of our ETF relative strength portfolios, since a cost would be incurred when we buy and when we sell. Higher-frequency trading (e.g. weekly versus monthly), would result in significantly higher trading costs, which may lead us to prefer less frequent trades.

Can we expect this to be added in the near future?

Paul Belgium

Feb 26, 2011 11:02 #


I know you probably understand this but just for clarification for all, the timing of the purchase (and sale) in the example above is no different no matter what the pick: all trades are assumed to be executed at the closing price of the last day of the relevant period.    The difference is that sometimes the ETF chosen will be different because it makes the decision on what to buy/sell 1-day earlier.  

The benefit to testing this is that you do get a full days notice of what the model is going to buy and sell on the next close.   As it stands now, we have weekly, semi-monthly, monthly and quarterly options --- and each of these 4 choices can now also be tested with the next-to-last day pick(s).   This is a pretty strong set of choices.    

Chris United States

Feb 26, 2011 16:42 #

I agree with Mike’s suggestion and am also for the ability to change the day (of e.g. the month) you do your rebalancing.
--- that might give the backtesting of a favorite portfolio more credibility (in case the variations of the results of all days are small over the whole period between “standard” updates)  
--- or it might also show the days most favorable for rebalancing (that then could be further tested/confirmed with different portfolios).

hugos Germany

Feb 27, 2011 07:02 #

Thanks for the clarification Chris!  What I understand from your description is that provisional buy signals can be provided the day before the period closes and the actual buy transaction still occurs at the period close.   The enhancement I’m suggesting would model the buy signal & buy transaction occurring on some day other than the period close.  Specifically, the 1-n trade date of the month (1st day, 2nd day, 3rd day … nth/closing day).

Intuition tells me that there is an outperformance advantage to trading at certain times relative to month close.   Just wish I could quantify that!   Mike

Mike United States

Feb 28, 2011 18:36 #

I compared the next-to-last day with last-day on five of my portfolios, and saw large drops in gains with much larger draw-downs and much lower Sharpe ratios on all five portfolios. This causes me to question the reliability of my back tests. Either there is a last-day-of-the-month bump in the markets, or I have inadvertently introduced a data snooping bias into my test results. I really have no easy way of knowing which it is. Since one has to wait until after the market closes to get the ranks, in reality one is forced to trade the next day in any case. A more realistic back testing feature would be to compute back test gains from market OPEN on the day of ranking based on market CLOSE. Otherwise it is impossible to trade the same as the back test strategy.

Jim United States

Mar 01, 2011 02:59 #


I agree with your thoughts completely!

If I could vote on enhancements, these would be my requests:

1.  Export to Excel (Spreadsheet) feature.
2.  (See Jim's comment above)
3.  (See Mike's comments above)

Ernest H Harris United States

Mar 01, 2011 07:41 #

We have recorded the requests.  

Today is a perfect example of what the original point of this blog served to highlight.  

The portfolio pick for the list of 5 ETFs highlighted above surely looked like it was DBC.   DBC had been the top pick for multiple days inlcuding Feb 25 (the provisional pick day).    The next-to-last day model (the pick known since Friday and easily observable) bought DBC on the Feb 28 close.  

In a almost strange surge on the last day,  VNQ replaced DBC as the top pick.   But here we are an hour into trading on the first day and DBC is up and VNQ is down.   The spread is  +1.90% so far.  

FWIW, REITs (including VNQ, ICF, IYR etc) represent a market segment that does quite poorly when the S&P 500 drops.   DBC has far lower correlation.  

Jim, the other point of this blog post was to show that you don't have to buy the open.   I think you may have missed this --- if you like DBC and you want to capture the forward return of DBC in March, the best (and perhaps only) way to ensure this is to buy the close of the last day of the period, which was yesterday.     That said, you could buy right now and its only a 0.50% difference.  

I may follow up with a new short blog after todays close to summarize this.  

Chris United States

Mar 01, 2011 19:58 #

     Chris, if I understand you correctly, to be able to buy the close on the last day of the month using rankings for the last day of the month, ETF Replay would have to use real time data, so that if I ran rankings a half hour before close, I would have a good chance of catching late changing rank switches. I thought ETF Replay used end of day data.
      Or are you saying to just use the next-to-last day rankings to buy the last day close? If the latter, I did catch that point in your article, but am alarmed at how much using next-to-last day rankings smacked down my returns.
     Here is an interesting stat: I compared next-to-last day trades with last day trades over 62 monthly trades on one of my portfolios. Of the 15 times when the next-to-last day ranks were different than the last day ranks, those 15 trades on the next-to-last portfolio were winners only 50% of the time, versus a last-day portfolio win-rate average of 70%. The 15 trades on the next-to-last portfolio lost about 30% total. The 15 trades on the last-date portfolio had gains of about 10% - a spread of about 40% total over 62 months.
     This kind of pattern held even for portfolios I created with simplistic rules, such as buying the top 25 market cap ETFs. So my hypothesis is that last-day switches are important as they catch the next wave of momentum. At least this seems to be the case with my monthly trading strategies.
     Thanks - Jim

Jim United States

Mar 01, 2011 22:24 #

I have found that more often than not -- the last day picks have in recent years better performance than the next-to-last-day picks with the lists I have tested as well.    But there is a second point mentioned in my first comment above that was written on Feb 25th.

I mentioned that analyzing this doesn't end so quickly.   Note that in the original blog example above,  DBC was the next-to-last-day pick.   DBC ranked highest on Feb 25th and therefore would be purchased on Feb 28th per the rules of the next-to-last day method (1 day advance notice of the next-day trade).    Recall that we do this so that we can ensure we capture the following period as closely as possible.  

Now in actuality, VNQ shot up on Feb 28th and became the final pick to be recorded in the model.    So here you have the exact example I mentioned --- DBC was the buy but then VNQ shows up as the official last day pick.   Now you have a decision, which is it you want to own?    Regardless of the answer,  today DBC went up and VNQ went down sharply.    If you wanted, on todays close you could have sold DBC and bought VNQ.   You would thereby lock-in better performance than the official last-day model.    I have seen this happen many, many times.  

So the real point with understanding these backtests is that it is just not as simple as 'the final picks did much better than the next-to-last therefore the next-to-last picks are no good.'   The next-to-last day picks may or may not do worse than last day picks in the future -- but either way, there is no rule that says you now MUST hold the next-to-last-day pick(s) for the month of March.  

In this case, I actually think the VNQ late spike on Feb 28 was a bit of a fluke -- this list I used was simplified to just 5 ETFs for this example.   However, I am happy I used it as it emphasizes the point perfectly.  

FWIW, DBC makes more sense to me as it has been consistently strong -- and on my lists with more ETFs, DBC has ranked higher all along.   This is just a case of judgment and logic over a statistical fluke of a shortened list.          

Chris United States

Mar 26, 2011 15:34 #

When I first ran backtests with the Next to Last Day option set, most of my portfolios had much wose results than what I was getting with Last Day.   I adjusted my parameters (Time Periods, Weights) and now I get results equal to and in some case much, much better than with Last Day.  So, I am maintaining two sets of parameters for my portfolios (one for Last Day & one for Next to Last Day).  With the ability to pick on Next to Last Day and buy near the end of the Last Day will allow me to better track my forward testing.  Thank you very much for this additional feature.

Ernest United States

Mar 27, 2011 03:56 #

I wonder whether tuning the parameter set to get better results is merely an example of "curve fitting".

The selected parameters may indeed yield optimal results with the historical data set used, but may yield optimal (or even "good") results in the future.

The fact that there seems to be so much difference in returns between "last day" and "next to last day" returns, makes me more skeptical about the reliability of the method used. At first site, a one-day difference should not produce such widely differing results?

Paul Belgium

Mar 27, 2011 04:15 #


A very valid point, indeed!

Ernest United States

Mar 27, 2011 07:58 #

If you use extremely volatile ETFs, the changes in rank from day to day will be higher.   As volatility increases, the importance of your timing increases.    While we have made volatility a factor in the rankings, if you create a long list of highly volatile ETFs, be sure that the results are not going to be immune to these inherently risky securities.  Since the screener runs off <b>ranks</b>, it is possible for anything to rank well while still being highly volatile if you build your list in such a way.

Remember that the goal is not to maximize the backtest returns -- it is quite easy to create monstrous results given some testing.   The goal in the end is to create a portfolio that provides good return at reasonable risk and to have a process for executing your plan.

The advanced relative strength backtest allows you to model some aggressive (high-volatilty) strategies and yet still maintain a conservative core portfolio.   This 2-portfolio framework will stabilize your results and allow an overall portfolio to be enhanced with the satellite strategies --- yet if/when these strategies face inevitible volatility, you have a core portfolio so that you manage/lower your drawdowns.

Chris United States

Mar 27, 2011 08:36 #

Chris, it would be very informative for your paying subscribers to see a few well-documented examples of what your describe above.

It would be nice if you could create some sample portfolios, only visible to subscribers, which show (1) an example of the highly-volatile portfolios you discribe above, and (2) what you would consider to be a well-balanced core satelite portfolio.

Then apply the 'last-day' and 'next-to-last' day effect is, etc.

This would make it a lot easier to DEEPLY understand what you explain above.
Such educational portfolios would add a lot of value to your site, help new subscribers get a better/faster start in really using the tools.

Thanks for considering this.

Paul Belgium

Mar 28, 2011 16:01 #

We have some good ideas for doing this Paul.   We will push forward on this once you see what we have been working on recently.    

Chris United States

Apr 12, 2011 16:34 #

What time (Eastern Standard) does the system get updated? Do you recommend extended hours trading to ensure the closing day price? I often miss the closing price at the next days open and end up paying more since the ETF's have high momentum. Over time the difference is eroding my return.
Thank you.

Al Canada

Apr 13, 2011 09:46 #


Remember that if there is a gap up on the open of the first day of the month, then the holdings in your portfolio will have gapped up as well.    Likewise, if there is a gap down -- it is likely that the holdings in your portfolio will have gapped down.     The difference of the 'day 1 move' will always be there if you are comparing day 1 entries vs end of month closing prices.    This is not an erosion of your returns, this is just how you make proper comparisons.    FWIW, if you buy on the first day of a new month, then the relative performance should be vs the first day of the month closing price for the benchmark you are using.     The top application on the backtest page has a custom date to perform this comparison.

Technically, if you want to compare to the end of the month -- then you should include what you owned on the last day of last month and its return for the first day of the new month and link that return with the return since that time to form a view of the full month.   However, this is complete overkill and not worth the time as this is just not going to be that important over the long-run.   Some months go down and you have a drawdown, some months you are in the right ETFs and you catch some large trends.    The markets are noisy -- but in the end it pays off to update your portfolio and be rewarded for the bigger trends.


Chris United States

Apr 25, 2011 20:03 #


I appreciate your answers to the concerns above. However, in all my portfolios I find that using "next to last days picks" increases volatility and decreases returns. Using yesterdays picks and buying at the next open, with a difference of even 0.20% - 0.5% adds up big  time over 26 semi-monthly periods per year (or 52 periods for those following a weekly strategy). After all we are buying the ETF with the highest momentum. The odds are this will increase more the next day than the ETF with lower momentum which we are selling.

Chris, is there any way you can update the system by 3.00 pm everyday? This will give us the opportunity to buy at the close on the same day.

Thank you Chris.


Al Canada

Apr 25, 2011 21:54 #


In addition to my last comment, with Leveraged ETF's the account gets hurt big time suffering large draw downs using "next to last days picks". Updating by 3.00 pm every day would make it much safer to use the volatile 2x and 3x funds, even as a satellite strategy.

Thank you Chris.


Al Canada

Comments are closed

Follow ETFreplay