Survivor bias is a real problem in individual stock backtesting. A quick statistic: ~30% (90 / 308) of the stocks in the DJ Financial services index on Dec 31, 2007 have since been removed from the index.
Said another way, if you were running a stock backtest using all of the current individual stock components of this index as a starting point to backtest today, you would have a massive positive bias in your study. Your list would not include names like:
Wachovia (failed bank), Merrill Lynch (likely insolvent in 2008), Fannie Mae (insolvent), Lehman Brothers (bankrupt), Washington Mutual (failed), Bear Stearns (insolvent), Countrywide (BofA acquisition disaster), MF Global (bankrupt) etc...
Nearly 1/3 of the list is gone. That is a TON of survivor bias.
Back up a few years and it was names like Worldcom (bankrupt), Enron (bankrupt), Global Crossing (bankrupt) along with all the other Tech, Media & Telecom (TMT) companies that lost -98% of their value and were removed from their respective indices.
Of course, if you operate at a level of long-established indices (as with ETF backtesting), you can largely avoid the survivorship bias problem as the historical index data reflects all of the securities that left the index due to bankruptcy and/or similar resons. That is, while the DJ Finanicals Index no longer has those 90 companies in the index -- the index RETURN of course does indeed reflect the effect of all those failed/insolvent firms. Survivorship bias is no longer relevant if using established index ETFs. This is one key benefit you get with using securities that actually traded at the time --- and not just theoretical indices (an index is NOT an investment, an index FUND is what you invest in).
The issue in ETFs is that of newly-created indices that were created only to sell a financial product. Many of these are going away and the ETF analyst should differentiate between what is a 'real' index and what is a financial gimmick. Does anyone think that the S&P Financial Sector Index isn't going to still exist in 10-20 years? It will. MSCI Emerging Markets -- yes it will too.
New important indexes do come along -- but you should be discerning in this and really focus most of your efforts on long-established indices. Learn about new ETFs as part of your efforts -- but spend 95% of your time on what is already out there.
RIP these Direxion ETFs: Direxion Decides To Close 9 3x ETFs
Note also that having valid indices is no guarantee that YOU will be the provider of choice --- Russell Investments is giving up on its ETF effort after starting far too late and hence gaining virtually zero traction in the marketplace: Russell Pulls Plug On ETFs
See Also: New ETF Tracking Error Nuances