TK,in another topic TK wrote:After running a couple of tests myself, I came to a sad conclusion that no matter how prudent you are about your entry/exit rules, position sizing models, optimization process and Monte Carlo simulation, in the end it is all about... market selection. If you haven't done it yourself yet, I suggest you do the following:
1. Open the file Demo.set in the Futures Sets folder.
2. Replace the symbol PA (palladium) with SI (silver).
3. Save the file and run the test again on the Demo portfolio.
What you have just done is replace only one (out of 15) market in the portfolio with another market from the same sector (Metals), so you wouldn't expect a major difference in the results, would you? Well, run the test and see the CAGR drop from 169.2% to 52.7%, the MaxDD rise from 39.1% to 51.0%, and MAR fall from 4.44 to 1.03...
All said and done, it seems market selection is the key.
I think you've hit upon the limitation of testing with too little data, as much or more than the effect of changing one market.
With five years data, you simply don't have enough information to draw valid trading conclusions. Even 10 years of data is too little in my book (notwithstanding Richard Dennis' comments otherwise).
People often talk about there being too few trades, I look at it more from the perspective of too few trends. It's really only the bad periods and the good periods that count. A single good trend at the right time can make a huge difference in a five year test. Likewise a single series of losses at the wrong time, can make an equally bad difference.
On average, my guess is that in any given market you might get one good trend or less in five years, and perhaps a couple smaller winners.
So even across a basket of 30 markets, the data that matters most is in very short supply. Perhaps 20 to 30 good trades in a sample of five years. This is simply too little data to make statistically significant judgements on.
All those short periods of little losses or smaller wins don't really matter that much, because even if you have hundreds of data points from the perspective of trades, there are very few of those that make much difference.
The other problem is the effect of a single year on the results. In five years on a compounded basis a single year affects the entire results considerably. CAGR% can move around quite a bit based on the outcomes of only a few trades. The solution again is getting more data.
Your example is illustrative because Palladium happened to have six good trades over the test period:
Feb 2000 - 54.7%
Oct 2000 - 29.6%
Jul 2001 - 25.7%
Dec 2001 - 16.9%
Nov 2002 - 22.9%
Mar 2003 - 35.5%
where as Silver probably lost money during the same period.
Over a 20 year period, the differential in results would not have been so dramatically different.
MAR is particularly vulnerable to short test periods because a drawdown impacts MAR calculations in both the numerator and denominator, since a large drawdown reduces CAGR% as well. This is especially true in a short test where you might only have one period (or none) of large drawdowns. The presence or absence of the large drawdown will change the MAR more than any other result.
Now, that having been said, I do believe that market selection is very important but not as easy as it might appear, but that is a subject for another topic.
- Forum Mgmnt