Further thoughts on Backtesting
Posted: Wed Jun 30, 2004 9:32 am
I put a piece on here a few weeks ago about walk forward testing .. I would love to hear more feedback and myself have some further thoughts ...
I still dont see how walk forward testing can be too heavily criticised. If you take a 10 year stretch of data to be tested , for example , 1990 - 2000 .. you then design a system that seems to work well .. perhaps an MAR of 2. You then roll it forward and test it on 2000-2004 and you get an mar of say 0.6 and the performance seems to have fallen dramatically away.. is this still a good system and if you hadnt of tested it on an out of sequence length of data surely you'd never really know if you optimised and if the sysytem works going forward.
What I am trying to do is to design something on 1990 - 2000 and then try it on an out of sample 4-5 year data period and see if i get a semblence of robustness .. i.e perhpas mar declined to 1.2 .. it my book this could still be a good system.
One other final thought .. sure this wll stirr a few people up .. I seriously think its a bad idea to backtest any form of trendfollowing system on the years 1980 - 1989 ... i think a one armed , blind lobotomised monkey couild have made money trendfollowing with any kind of system in this period since the whole nature of markets has now changed with the onset of computers and trendfollowing and communications .. Agreed people human nature nver changes and thats what drives markets but the characteristics of markets when you have several hundread thousand trendfollowers in the market do change I believe every 10 years or so .. this would back up Richard Dennis in his assesment that its only worth backtesting over a 10 year period and not 20 which is percieved wisdom ..
Any thoughts warmly appreciated.
Chris
I still dont see how walk forward testing can be too heavily criticised. If you take a 10 year stretch of data to be tested , for example , 1990 - 2000 .. you then design a system that seems to work well .. perhaps an MAR of 2. You then roll it forward and test it on 2000-2004 and you get an mar of say 0.6 and the performance seems to have fallen dramatically away.. is this still a good system and if you hadnt of tested it on an out of sequence length of data surely you'd never really know if you optimised and if the sysytem works going forward.
What I am trying to do is to design something on 1990 - 2000 and then try it on an out of sample 4-5 year data period and see if i get a semblence of robustness .. i.e perhpas mar declined to 1.2 .. it my book this could still be a good system.
One other final thought .. sure this wll stirr a few people up .. I seriously think its a bad idea to backtest any form of trendfollowing system on the years 1980 - 1989 ... i think a one armed , blind lobotomised monkey couild have made money trendfollowing with any kind of system in this period since the whole nature of markets has now changed with the onset of computers and trendfollowing and communications .. Agreed people human nature nver changes and thats what drives markets but the characteristics of markets when you have several hundread thousand trendfollowers in the market do change I believe every 10 years or so .. this would back up Richard Dennis in his assesment that its only worth backtesting over a 10 year period and not 20 which is percieved wisdom ..
Any thoughts warmly appreciated.
Chris