Traders' Roundtable

Backtesting – Tools for Fools?

At the end of the day, my belief, and certainly my experience is that however carefully I think things out, however much I back test, I am still constantly surprised at how in practice things turn out completely different. Well, perhaps "used to be surprised" is more accurate.

Everything about back testing is a bit of a con really but as I have so often pointed out, it is all we have got to go on. A nice smooth equity curve results from many tests over many parameters of the same system on the same portfolio over the same time period. Well we all know what to think about that. Different time periods, different start dates, different portfolios must be looked at. Need I go on.

But say you do not go on - you spend a long time on one system and convince yourself the level of drawdown, volatility and return all look realistic no matter how many time frames and parameters you test it over. Monte carlo analysis with re-sampling may help to bring you back down to earth. It tells you there is an alternate universe where those nice month long periods of increasing equity could easily be replaced by nice long periods of decreasing equity. It shows just how your equity curve MIGHT look or have looked if things had turned out differently. Random entry tests can also help you to see reality.

And, funny old thing, time and time again I have noticed how the figures produced overall from countless monte carlo runs seem to bear a much closer relationship to the real life long term track records of the long surviving CTAs than any of my carefully planned theoretical equity curves do.

Tools for fools - all of this stuff. At the end of the day we just have to stick our fingers in the air, see the direction of the wind and dip our trading toes firmly in the water, knowing that things are highly unlikely to turn out as we had expected.

Just to re-iterate: I am not for a minute suggesting trading blind. It would be foolish indeed not to back test very thoroughly. But don’t imagine for an instant that back testing has any predictive power. You will always face surprises and disappointments. As in every area of life you just have to do your best and accept that life will probably not turn out as you had planned.

Anthony - you might enjoy reading this...... more against the 'positive thinking movement' as being all you need. Not so much about trading, but i think upon reading it, a lot can be applicable to this horrible profession we have joined.


http://www.amazon.co.uk/Antidote-Happin ... 735&sr=1-1

Well I am never going back to stone age again

Excellent book, thanks!

Odd how some people on Forums such as those over on Linked In just don't seem to get the nuances or perhaps even understand the language one uses.

My heading for this post was designed to be deliberately provocative to attract attention. I do, very firmly believe in back testing and I am devoted to systematic trading. My real purpose was to point out how, if you are not very careful, you can so easily fool yourself.

In the trend following world many point to the long track records of Dunn etc but fail to mention the countless thousands of CTAs which have bitten the dust for one reason or another. Let alone the psychologically un-livable with drawdowns - one of Dunn's programs is still in a 71% DD which started in the mid 2000s. I do believe in general though that long term trend following has a basis in reality, whether it is following broad long term economic growth through the stock markets or the advance / decline of a particular country/currency through the forex markets. Or whatever.

I believe that simple, "cockroach" like systems can follow trends and that back testing does help to show that this has been the case in the past. Although as to the future....who knows.

Curve fitting and fooling ones self are the real danger of back testing as well as unrealistic assumptions on a myriad of matters including the frictional costs of trading.

The greatest unknown and danger in trend following is whether your strategy can survive the noise and the chop so that it can eventually benefit from the re-emergence of "clean" long term trends. Systems have to constantly adapt - hence the old, original Turtle methodology has bitten the dust - as least on the parameters and way they operated it. Tough game.

It is always amusing to have the young Turks brashly beating their drums and throwing out insults. Rather therapeutic really; like listening to the waves roll in off the ocean. A kind of ever present background of white noise.