Survivor Bias

Aug 19, 2012 in Backtest | Stocks

Survivor bias is a real problem in individual stock backtesting.   A quick statistic:   ~30% (90 / 308) of the stocks in the DJ Financial services index on Dec 31, 2007 have since been removed from the index.     

Said another way, if you were running a stock backtest using all of the current individual stock components of this index as a starting point to backtest today, you would have a massive positive bias in your study.   Your list would not include names like:   

Wachovia (failed bank), Merrill Lynch (likely insolvent in 2008),  Fannie Mae (insolvent), Lehman Brothers (bankrupt), Washington Mutual (failed), Bear Stearns (insolvent), Countrywide (BofA acquisition disaster), MF Global (bankrupt) etc...

Nearly 1/3 of the list is gone. That is a TON of survivor bias.

Back up a few years and it was names like Worldcom (bankrupt), Enron (bankrupt), Global Crossing (bankrupt) along with all the other Tech, Media & Telecom (TMT) companies that lost -98% of their value and were removed from their respective indices.

Of course, if you operate at a level of long-established indices (as with ETF backtesting), you can largely avoid the survivorship bias problem as the historical index data reflects all of the securities that left the index due to bankruptcy and/or similar resons.   That is, while the DJ Finanicals Index no longer has those 90 companies in the index -- the index RETURN of course does indeed reflect the effect of all those failed/insolvent firms.   Survivorship bias is no longer relevant if using established index ETFs. This is one key benefit you get with using securities that actually traded at the time --- and not just theoretical indices (an index is NOT an investment, an index FUND is what you invest in).

The issue in ETFs is that of newly-created indices that were created only to sell a financial product.   Many of these are going away and the ETF analyst should differentiate between what is a 'real' index and what is a financial gimmick. Does anyone think that the S&P Financial Sector Index isn't going to still exist in 10-20 years? It will. MSCI Emerging Markets -- yes it will too.

New important indexes do come along -- but you should be discerning in this and really focus most of your efforts on long-established indices. Learn about new ETFs as part of your efforts -- but spend 95% of your time on what is already out there.

RIP these Direxion ETFs: Direxion Decides To Close 9 3x ETFs

Note also that having valid indices is no guarantee that YOU will be the provider of choice --- Russell Investments is giving up on its ETF effort after starting far too late and hence gaining virtually zero traction in the marketplace: Russell Pulls Plug On ETFs


See Also: New ETF Tracking Error Nuances

Follow us on Follow etfreplay on Twitter


Learning Through Portfolio Backtesting - The Short-Term Isn't That Meaningful

Jun 10, 2012 in Backtest

In this post, we will check back in on the 'Sample Portfolio' -- which was a portfolio we set up over 2 years ago and included as the starter example portfolio for new members.   The idea behind it was to show one type of simple mix of ETFs that can serve as good complements to each other in terms of a rotation strategy.

As seen below,  this rotation portfolio has performed pretty well over the past 2 years.   

But the more specific point we wanted to make here was in the statistical profile that accompanies the report (marked with an arrow at the bottom).  

In the last 'out of sample' 18 monthly results (2011 & 2012), while this strategy is well ahead of the major indexes -- it has actually underperformed the S&P 500 in 10 out of 18 of those months.   Yes, that is right --  it has handily beat the S&P 500 on a cumulative basis while still underperforming more times than not in terms of months.  That seems counter-intuitive -- but this actually isn't that unusual in investing.  

Many times you run a backtest and it shows good results --- make sure to scan down to that number in the table below and take a look at how often is outperforms your inputted benchmark.  If you do this a lot, you will start to understand some of the common pitfalls of bahavioral finance issues.

This is the important part --- if you didn't know this, imagine how your psychology might affect your investing process.  The start to 2012 is a good example.  Our allocations board portfolio fell behind the S&P 500 return on a Year-To-Date basis in March (relative basis only).   Was this underperformance meaningful?  No.  Not if we believe that our strategy is solid and that we don't care about the short-run --- we only care about the equity curve over time.  Through backtesting, we are armed with the knowledge that good strategies might indeed underperform in 40% or more of the months and this does nothing to change the validity of the strategy.

Summary:   We all want to outperform the indexes in every single period --- but if you obsess about this kind of thing --- chasing a benchmark around in every short-term timeframe -- it will ultimately be your un-doing.   The behavioral/mental side of investing is hugely underrated.   The specific numbers in this one example are not the point.   The point here is to use the portfolio backtesting process as a way to battle against the behavioral biases that keep you from doing well in investing.   In our view, backtesting ideas can greatly help because if you are like us, you have to SEE it and imprint it on your brain to actually believe it.  Only through a good and never-ending research process will we ever truly believe in these kinds of things.


Follow us on Follow etfreplay on Twitter


Drawdown Nuances In Backtesting Results

Oct 21, 2011 in Backtest | Drawdown | Video

We have improved the quality of the backtesting applications again.   All backtests will now display the path of returns on a daily basis, regardless of which update schedule you are on.   This allows for more precise chart visual --- and a few supporting statistics have been improved.   The video below explains this -- there is a nuance that we go over that may not be so obvious here:



Also, just a snapshot of a few indices for YTD drawdowns -- the higher the volatility, the higher the drawdown.


See Also: Drawdown Blog Posts

Follow Us On Twitter:


Comparing Allocations Introduction

Jun 29, 2011 in Backtest | Video


A brief introduction to the versatile yet easy-to-use 'Compare Allocations' application located on the Backtest Page



Follow Us On Twitter:


Backtests Become Forward Tests

Mar 18, 2011 in Backtest

It is well accepted in professional money management that having a quantitative aspect to your investment process is additive. That is, quantitative methods can greatly help in screening and monitoring lists of securities into a manageable ranking for further analysis. The vast majority of institutional-oriented firms do this kind of thing.

The classic, basic steps of an investment process involve:

1. Install a (Quantitative) Method To Rank A Relevant Universe of Securities
2. Take The Top-Ranked Securities And Do Further Research
3. Construct a Portfolio Using Securities That Pass The First 2 Steps
4. Monitor And Update The Portfolio
5. Repeat

We view backtesting as a very practical and useful part of the research process. The way you rank securities should be based on something consistent with your beliefs on what actually works.

Once you have done some research and found a method to rank securities, run some backtests. You will learn about your method greatly and can understand more aspects and characteristics of such strategies. You will speed up your understanding and you will be forced to think through all the details of how you are going to execute your process.

Once you have created a portfolio (per Step 3 above), that portfolio effectively now becomes a ‘forward-test.’ Monitoring these portfolios and seeing these various rotation strategies behave is a crucial part of the process. Track your various rotation strategies. You can learn a lot by running many simultaneous ‘forward tests’ at once. Importantly, you will get a feeling for the ‘short-term noise’ that occurs around your strategy. Even for professionals, the psychological aspects of short-term volatility will cause them to doubt themselves. Back/Forward tests will make you much more aware of the kind of thing you will face in the future. You must get used to this as psychological aspects to investing are absolutely critical.

Back/forward-testing accelerates the learning process and you can then feed the incremental improvements you discover back into your actual portfolio. The point is that just doing a backtest and then ‘stopping your research’ is very limiting. Don’t stop learning, don’t stop improving. Small incremental improvements add up over time.

We are entering the golden age of active indexing. The specialized, targeted index fund is really a somewhat new phenomenon. Index funds in the 1980s were all very broad vehicles. Many specialized index products (Vanguard REIT index mutual fund, country fund ETFs) actually only have histories back to 1996. You can simulate prior performance -- but they weren't so inexpensively accessible. TIPS securities weren’t even issued by the Treasury until 1997. The World Equity Index products (mutual fund AND etf) didn’t launch until 2008.

Important new areas of future investment may come from newly investable products. For example, the emerging markets small cap index might become as mainstream a product as some other well-accepted benchmarks are today. Understanding the important indices of tomorrow might be as good an idea as understanding REIT and emerging markets indices when they were new PRODUCTS (ie, investable and tradeable at reasonable cost).

Understanding and processing relative performance, relative volatilities and observing relative drawdowns in present ‘forward-test’ environment strikes us as a pretty good idea.

iShares is bringing to market an Internationally focused preferred stock index. Wisdomtree just launched an Asia-Pacific regional intermediate bond ETF. Remember that at one point, the emerging markets index mutual fund was brand new (1994).  Today it is a primary index everyone follows.   Growth vs value wasn’t mainstream until the 1990s -- and indexing these products came later.  The world evolves.  Embrace the change and learn from it – let many simultaneous forward tests accelerate the learning.

Follow us on Twitter:




Notice: New Feature Added To Backtesting

Feb 25, 2011 in Backtest | Relative Strength

Last summer, we added a feature to the portfolio relative strength backtest application. We added a line that showed what the 'provisional picks' would be if the update period had just ended. The provisional picks listed are the exact same as if you ran the ETF screener using the same list of ETFs and same parameters. We added this feature just as a convenience so it wouldn't be necessary to open another tab and check what was strongest now -- rather than at the last update period.

Depending on the list you use, it is of course possible that the 'provisional pick' is not the final pick (hence the term 'provisional'). But what if we just looked up the provisional pick on the next-to-last day of a given month and bought that ETF the next day on the close and held it for the subsequent period? What would the performance look like assuming we did that? Would it be similar?

Let's look at a simple example just to discuss the mechanics of what we mean. We will compare using the 'next-to-last day picks' with the 'last day picks.' The holding periods will be the exact same, we are just reading the picks of the ETF screener with a one day offset. Of course, for many periods the picks (and therefore performance) will be the exact same.

In this example, we will use one of the Ivy Portfolio lists of 5 basic ETFs and a semi-monthly update period. We are choosing the top 1 of 5 and holding it on 2-week intervals. The settings are the exact same except we are checking the box "Invest in next to last day pick(s) near the top of images below.

Using Regular 'Last Day' Picks:

Using Next To Last Day Picks:

We can see that the returns follow similar paths but that there was a difference -- even for this list of just 5 ETFs. In this case, the next-to-last day picks actually performed better -- which means very little in and of itself. We simply are pointing out the mechanics of how it works. Users should test this using their lists across various other settings and draw their own conclusions. Our view is that this doesn't change anything -- if your backtest is well thought-out, then this extra analysis will very likely show a result that is in the same ballpark as the original method.

Final note is that today is Friday, February 25th -- so the next-to-last days picks will be locked in and known after todays close. The final picks will update using Monday's (Feb 28) closing price. In both cases, the backtests will assume the cost-basis for March performance is the closing price as of Feb 28.


Follow us on Twitter:


Backtesting: Combining Relative Strength With A Moving Average Filter

Jan 11, 2011 in Backtest | moving average | Relative Strength

We have added an optional moving average (MA) filter feature to the RS backtest app.  With the recently expanded date start and stop functionality, the applications continue to get more versatile.  

Combining a long-term moving average within the construct of relative strength has been highly requested and we wanted to discuss one idea when considering whether to use it (note that you can just leave it set to ‘off’ as well).  

If you build a relative strength list of say 10 ETFs and you are choosing the top 2,  you could protect your portfolio by including 2 bond funds.   You don’t need a moving average filter because the bond funds will naturally be the ones with the relative strength when equity markets are dropping.   This method actually can get more interesting because you can make better use of more type of ETFs.   Rather than just use cash-like bond funds, you might want to extend the potential holdings to an intermediate bond fund like IEF (7-8 year duration) or others. You don't HAVE to restrict yourself to just stocks and cash.   

Another way to test is by using a moving average.   If you do it this way, then you will inherently be out of ETFs as they go into extended downtrends.  You don’t have to proportionally keep X number of bond funds in your list if you do it this way.    

But a lot of indexes can go above or below a long-term moving average and still not really be a source of market leadership and enhance your return.   Moreover, you may save a lot of money between the time ETFs lose relative strength and the time they actually cross below the moving average.   For these reasons, we believe adding relative strength to a MA strategy will generally be more robust.   Adding MA to a RS strategy is optional -- and you may find works better or worse than your existing method.   Continuous testing leads to better decisions.




Follow us on Twitter:


Tactical Asset-Allocation (TAA) And Core-Satellite ETF Techniques Intro

Dec 02, 2010 in Advanced Relative Strength | Backtest

Charles Schwab recently published some summary statistics on their accounts.   They said that while 90% of accounts own individual stocks – just 15% of accounts hold ETFs.    So while many people may have been using ETFs for many years now – using Schwab as a proxy, about 85% of people still need some introduction to ETFs.

There are 3 basics schools of thought on active management in general:

1) Those that think it’s impossible

2) Those that think it’s easy

3) Those that think it’s a challenge -- but rewarding

For those that think passive management is the way to go, your job is still not over.  You will still need to figure out an asset allocation.  “Just buy index funds” actually doesn’t get you far --- there are over 1000 ETFs that follow various indexes and there is no single answer to what exactly constitutes a passive strategy.  I can’t imagine there exists a professionally trained investment advisor who advises putting 100% of money into a S&P 500 index -- but I guess never say never.  So absent that, there are still allocation decisions to be made about stocks vs bonds -- as well as the mix of international stocks and how much to include in alternatives like REITs & Precious Metals.  These are all classic passive indexing discussion points.

For those that think like #2 above – that active management is easy, this crowd is likely just plain dangerous.  They simply ‘don’t know what they don’t know.’      

The third group acknowledges the challenge and comes at the issue with some humility.  We sit firmly in the 3rd camp.   It takes some analysis and testing and hard work to do well in investing.  You don’t need to  just come up with ideas – you need an investment PROCESS.  That is, some type of structured approach.  

So let’s introduce the core-satellite approach.   It’s excellent for its flexibility in balancing return & risk.  Here is a visual that Blackrock has used for this:

Source:  Blackrock

What makes this framework really powerful is that you can pursue more aggressive strategies --- but still rein in the overall risk of a portfolio.   You can stay within your specific risk tolerance but still buy/overweight some attractive (but perhaps more-volatile) segments.    Or perhaps you are an advisor and your clients have different risk appetites.    You can use your same ideas across these different accounts but just change the allocations between the conservative core and the more aggressive satellite strategies to more precisely target the appropriate risk-budget.  

A typical investor thought might be --- I like the idea of investing in Emerging Asia – but how do I do it?  How do I think about the risk involved in the short-run if and when some problem emanates out of South Korea or India or China?      

So for a very basic start, let’s mix the core idea of completely passive-indexing with just 1 or 2 actual ideas.  In this example, rather than invest in just the benchmark S&P 500, we will invest 80% in the S&P 500 and 10% in each of two other market segments --- Real Estate Investment Trusts and Gold.  Below is a look from our free Backtest Portfolio Allocations App to observe how this portfolio looks Year-To-Date (total return):

Note the 3 percentage point pick-up in performance this year for some pretty simple ‘tilts’.   Importantly, note also that there has been some diversification benefit (reduction in volatility).  Other than returns being better, the core-satellite portfolio went down less over the summer.   So it went down less and yet has added 3%.  Your tilts added significant value here.   Not bad for a start.   But we can do better.

If you are a registered user on the site – whether you registered 8 months ago or yesterday – you may have noticed you had 1 starter portfolio we created called ‘Sample Portfolio.’   Rather than use REIT’s and GLD as our satellite strategies, what if we instead used this sample portfolio top relative strength pick as 20% of the portfolio.  Here is that example using our new TAA Application (for members)

Notice that the result here is +14.8% return with even more diversification benefit (it went down even less during the summer correction).  Remember, it is 80% S&P 500 in the first place so this is actually an excellent result.   Standalone, this ‘sample portfolio’ that we created in the Spring time has had a banner year --- see for yourself within the portfolio backtest app.  

As you will see when using the TAA application – and as we will discuss in upcoming blogs, you can change the percentage allocations from the core to the satellite and back to the core and see how it affects the overall results -- not just in returns but in ‘smoothing out the ride’ with lower volatility.   This is the very essence of TAA.

All of this is just a start for those who are new to ETFs.   We have many more options available in this core-satellite framework.   A more conservative investor – perhaps older and less willing to take even average risk --- could start with a bond-only core portfolio and then add equities as satellite strategies.   Or we could start with a more diversified core (say mixing REITS & Gold & Junk Bonds in the core) and then add country or sector funds in the satellite(s).   These and other ideas will be future blog topics.



Follow us on Twitter:



A Fair Example of Global ETF Relative Strength

Oct 20, 2010 in Backtest | Relative Strength

We showed a few examples of some very basic relative strength techniques at a recent investor event.

This example was meant to show a case of a relative strength strategy that easily could have been thought of as unbiased at any point in past 10 years -- or right now for that matter.

For simplicity, we have used the global regional ETFs and have simply covered the major parts of the 'developed world'.   We know that emerging markets have been excellent performers over the past 7-8 years -- as have selected other assets like Gold -- but we have intentionally left these out.   We wanted to show something that had some poor performing picks, a few good ones and therefore representative of someone who didn't pick ETFs particularly well but implemented a sound technique to stay with leaders and avoid large underperformers.

Importantly, we believe investors SHOULD decide first which ETFs they would like to be involved with and exclude those they have no interest in on a fundamental basis -- this is value-add to a basic, mechanical technique such as this.   

So for this example we use the World Index as a starting point and seek to just 'cover' the developed regions:

Current Approximate Weightings in Global Indexes are:

1) United States                 42.0%

2) Developed Europe         25.0%

3) Japan                              7.0%

4) Developed Pacific [1]       7.0%

5) Canada                          4.5%

***Emerging Markets       Excluded

[1] Developed Pacific is ~97% Australia/Hong Kong/Singapore and 3% 'other' (New Zealand etc..)

From an indexing perspective, Canada is always kind of the lonely child.   Canada is generally not included within a broader regional ETF -- and is not generally lumped with the United States for a North American ETF.   For conservatism here, I will exclude Canada and use the first 4 only.  These are logical regions in our view.   We get coverage of many different countries within this regional framework so this seems quite fair and hardly something that could not have been thought of many years ago -- or even today.

We will include the first 4 from above and SHY, a <2-year duration U.S. Treasury ETF which will act as a benchmark for positive returns.   If no region of the world is beating 2-year maturity short-term fixed-income -- then SHY will by default be thought to be the highest Relative Strength ETF.  

Here is the result using a 6-mo/3-mo 2-factor relative strength model with monthly re-balancing.

Note all historical results are purely hypothetical and meant to show the mechanics of the backtesting application.  This does not represent investment advice.

Symbol list here is:  SPY,EWJ,IEV,EPP,SHY


Follow us on Twitter:



Moving Average ETF Backtest For Portfolios: New Functionality

Sep 20, 2010 in Backtest | moving average

If you have created a portfolio list on ETFreplay, we are building new applications to leverage your ETF lists.  (If you are unsure on how to create a personalized ETF list, click here for a quick video: Setting Up Portfolio Lists Short Tutorial Video )

We have two new modules out that we have been working on for the past few months.  These applications offer simplified views to help us try to understand larger forces at work in the global marketplace.  

Building upon academic research regarding the use of moving averages, these apps save investors time by allowing many calculations and quantitative analyses to be simplified into a few clicks.  We think that creating specific entry/exit rules and creating a detailed strategy report adds value to better understanding a concept. That is, we create apps that convert concepts into tangible, specific techniques.  The accountability of these techniques is built into the very architecture of the website. On any day, you are just a click away from an updated view of the profit and loss history of a particular strategy.

Importantly, this type of research should be used as a complement to other forms of research. We suggest you think about which types of ETFs you want to be involved with over the long-run and then use techniques such as relative strength and moving average backtesting to help you research methods that reduce risk of a large drawdown, while potentially offering to enhance your return as well.



The example below uses 4 key smaller developed markets outside Europe & the U.S.

1.  EWA   iShares MSCI Australia Index
2.  EWC   iShares MSCI Canada Index
3.  EWH   iShares MSCI Hong Kong Index
4.  EWS   iShares MSCI Singapore Index





Follow us on Twitter:




Follow ETFreplay