Back Testing - Tools for Fools?
Posted: Wed Feb 20, 2013 6:22 am
Backtesting – Tools for Fools?
At the end of the day, my belief, and certainly my experience is that however carefully I think things out, however much I back test, I am still constantly surprised at how in practice things turn out completely different. Well, perhaps "used to be surprised" is more accurate.
Everything about back testing is a bit of a con really but as I have so often pointed out, it is all we have got to go on. A nice smooth equity curve results from many tests over many parameters of the same system on the same portfolio over the same time period. Well we all know what to think about that. Different time periods, different start dates, different portfolios must be looked at. Need I go on.
But say you do not go on - you spend a long time on one system and convince yourself the level of drawdown, volatility and return all look realistic no matter how many time frames and parameters you test it over. Monte carlo analysis with re-sampling may help to bring you back down to earth. It tells you there is an alternate universe where those nice month long periods of increasing equity could easily be replaced by nice long periods of decreasing equity. It shows just how your equity curve MIGHT look or have looked if things had turned out differently. Random entry tests can also help you to see reality.
And, funny old thing, time and time again I have noticed how the figures produced overall from countless monte carlo runs seem to bear a much closer relationship to the real life long term track records of the long surviving CTAs than any of my carefully planned theoretical equity curves do.
Tools for fools - all of this stuff. At the end of the day we just have to stick our fingers in the air, see the direction of the wind and dip our trading toes firmly in the water, knowing that things are highly unlikely to turn out as we had expected.
Just to re-iterate: I am not for a minute suggesting trading blind. It would be foolish indeed not to back test very thoroughly. But don’t imagine for an instant that back testing has any predictive power. You will always face surprises and disappointments. As in every area of life you just have to do your best and accept that life will probably not turn out as you had planned.
At the end of the day, my belief, and certainly my experience is that however carefully I think things out, however much I back test, I am still constantly surprised at how in practice things turn out completely different. Well, perhaps "used to be surprised" is more accurate.
Everything about back testing is a bit of a con really but as I have so often pointed out, it is all we have got to go on. A nice smooth equity curve results from many tests over many parameters of the same system on the same portfolio over the same time period. Well we all know what to think about that. Different time periods, different start dates, different portfolios must be looked at. Need I go on.
But say you do not go on - you spend a long time on one system and convince yourself the level of drawdown, volatility and return all look realistic no matter how many time frames and parameters you test it over. Monte carlo analysis with re-sampling may help to bring you back down to earth. It tells you there is an alternate universe where those nice month long periods of increasing equity could easily be replaced by nice long periods of decreasing equity. It shows just how your equity curve MIGHT look or have looked if things had turned out differently. Random entry tests can also help you to see reality.
And, funny old thing, time and time again I have noticed how the figures produced overall from countless monte carlo runs seem to bear a much closer relationship to the real life long term track records of the long surviving CTAs than any of my carefully planned theoretical equity curves do.
Tools for fools - all of this stuff. At the end of the day we just have to stick our fingers in the air, see the direction of the wind and dip our trading toes firmly in the water, knowing that things are highly unlikely to turn out as we had expected.
Just to re-iterate: I am not for a minute suggesting trading blind. It would be foolish indeed not to back test very thoroughly. But don’t imagine for an instant that back testing has any predictive power. You will always face surprises and disappointments. As in every area of life you just have to do your best and accept that life will probably not turn out as you had planned.