Interesting test results (for a budding tester at least)
Posted: Fri Nov 07, 2003 1:34 pm
In testing a set of parameters recently for a medium-term trend following system, I was interested to see the robustness of the system tested over various time periods. Little did I know that I would end up learning an interesting lesson from this little exercise. Please read on to see if you have seen the same in your testing or if this is an aberration and I am just plain loony.
OK here are the steps I took. First, I optimized the parameter set over a 20 year time period with a basket of 22 commodities. Actually, it was 20.75 years (1/83 through 9/03). I took caution to not pick the absolute best parameters, but chose the ones that were in the middle of a smooth range. The results from the test were encouraging with annual returns just above 80%, MAR slightly above 2.00 and max DD around 38%. The actual test statistics aren’t as important as the comparison of the numbers through out this test.
Now to test for robustness, I re-ran the tests with the same system parameters in increments of 1 year. For example, my next test period was 1/84-9/03, then 1/85-9/03, and so on with my last test period being 1/03-9/03. The results seemed pretty good and some-what robust with the MAR ranging from 1.01-3.04 with an average of 2.00. The distribution was skewed to the right, which I consider to be a good thing. I did notice lower returns and DD’s if I had started trading this system over the last 5 years.
This prompted me to take it one step further and re-optimize the parameters for the last 5 years (10/98-9/03) to compare the results from the 20 year optimization. After getting my new “optimalâ€
OK here are the steps I took. First, I optimized the parameter set over a 20 year time period with a basket of 22 commodities. Actually, it was 20.75 years (1/83 through 9/03). I took caution to not pick the absolute best parameters, but chose the ones that were in the middle of a smooth range. The results from the test were encouraging with annual returns just above 80%, MAR slightly above 2.00 and max DD around 38%. The actual test statistics aren’t as important as the comparison of the numbers through out this test.
Now to test for robustness, I re-ran the tests with the same system parameters in increments of 1 year. For example, my next test period was 1/84-9/03, then 1/85-9/03, and so on with my last test period being 1/03-9/03. The results seemed pretty good and some-what robust with the MAR ranging from 1.01-3.04 with an average of 2.00. The distribution was skewed to the right, which I consider to be a good thing. I did notice lower returns and DD’s if I had started trading this system over the last 5 years.
This prompted me to take it one step further and re-optimize the parameters for the last 5 years (10/98-9/03) to compare the results from the 20 year optimization. After getting my new “optimalâ€