Beware of Improper Backtesting in ETFs

Recently there has been a good deal of discussion based upon how just simply using a moving average has outperformed buy and hold in the S&P. This was published recently by a newsletter writer in a book and then Fidelity did their own twist to the test adding a second moving average and finding similar type results.

I'd caution everyone to be careful with these test results. I'm sure the people who publish these tests mean well but their testing methods are very incomplete. And here are the things you should look for before trading any moving average system, especially one with ETFs.

1. Was the test run on one exchange traded fund (like the SPY) or on a basket of ETFs with a minimum of 20 broad-based ETFs like we did in our book High Probability ETF Trading (and we now run our tests on a universe of 100 of the largest non-leveraged, non-inverse ETFs). Most of these moving average tests were run on the S&P. It's like running a test on one stock and saying its applicable to all stocks. It's only applicable if there is consistent behavior over many ETFs, not one.

2. What was the performance of the moving average system up until 2008? This is a key question. Most moving average crossover tests fared poorly versus buy and hold until 2008. Then because the systems went into cash in early 2008 and missed one sell-off, their numbers look better versus buy and hold. But what about all those years it unperformed or lost money?

3. The only moving average crossover test which have shown any type of long term edges have been in commodities (commodities tend to trend). But the draw downs are gut wrenching. Stomaching 30%-40% draw downs for many months and sometimes years is not for everyone. But this is the one place where there is any evidence that a moving average system is robust and holds over many types of commodities over many years. In no other place have we seen any long term evidence that moving average crossovers make money over a longer term period of time.

Back testing is becoming more mainstream and you'll see more and more people publishing historical test results. Some will be right on the surface but make sure you dig deeper before committing your capital to their ideas. You'll see what I mean if you run the moving average tests using the simple parameters I mention above.

Next week we'll go further into how to protect yourself from invalid testing methods and how to properly run your own back tests. Have a great weekend!

This is from Larry Connors Daily Battle Plan which he publishes each morning. If you'd like to take a free trial click here, or call 1-888-484-8220 ext 1 to start your free trial today.