Robust Optimization Question

jameskuah · Post by **jameskuah** » Wed Jun 01, 2011 1:44 am

Hi all,

I realise that the issue of robust optimization and the problems of overcurvefitting have been mentioned in this forum. However, I would appreciate everyone's comments on the proper sequence for optimization. Assuming I have accurate end of day futures data for the last 25years years and I'd like to optimize a Donchian System, there are a few ways in which one can optimize on the data:

A) Optimize the system on the entire set of 25 years of data and selecting the best parameter that lies in the centre of the broad plateau of good performing parameters.

B) Optimize the system on the recent 10 years of data from 2000-2010 and look at system performance on out of sample data from 1985-2000.

C) Optimize the system from 1985-2000 and test it on out of sample data from 2000-2010.

D) Optimize the system from 1985-1995 and test it on out of sample data from 1995-2010.

With regards to my optimization , what I did was optimized from 1998-2008 and test the system on two different sets of out of sample data: firstly, 2008-2010 and secondly 1985-1998. Is this the correct form of system optimization and testing?

Testing was done on a $200,000 account, using 2% core equity fixed fractional MM and performed on 28 diversified worldwide markets. i have attached screenshots for your reference.

I would really appreciate the advice of the professionals on this forum on the correct sequence of optimization and it would be great if everyone could share their experience and tips on the proper procedure of optimization. Thanks to all.

Chris67 · Post by **Chris67** » Thu Jun 02, 2011 11:51 am

James

FWIW AND IMHO - I think you have done a very good job with your approach. Not only that but your results - rather trhan screaming out with an unrealistic MAR of 3.7 and a max d/d length of 2 days (I jest) - 5 months , for example , look somewhat realistic.
I think your approach is pretty much perfect.
Having said that I also do not believe there is a guranteed "righ / wrong" way to do it - because whatever you do the future maybe different - its an imperfect science after all ?
There seems to me - and this is just my opinion - so I'd love to see wiser heads comment - that there is a certain amount of common sense in testing / optimising a system over , approximately, teh last 10 years - AND THEN seeing if it worked well when markets - APPARENTLY - were easier to trend follow i.e. 1980-1995 - that way if it worked in the hard bit , performed really well in the easier bit - then common sense dictates that unless the markets stop trending - that it will probably work going forward to some degree ?
C

jameskuah · Post by **jameskuah** » Fri Jun 03, 2011 7:50 am

Hi Chris,

Thanks so much for your reply and compliment. I agree with you that there is no right or wrong solution and it all boils down to individual preference. As what Shane Wisdom from Wisdom Financial has said, "There are more than a few ways to skin the cat." Have a great weekend ahead!

Eventhorizon · Post by **Eventhorizon** » Fri Jun 03, 2011 9:40 am

jameskuah,

I am not a professional, but I did sleep at a Holiday Inn Express last night,so here's my 2c.

I agree with Chris that you seem to be doing all the right things. I would raise the following issues for consideration:

1) In / Out Of Sample Testing.

I have wrestled with this issue at some length and have yet to arrive at a good solution. The 2 problems you (and I) face is having a small set of contracts to test (making it hard to have a sufficiently large sample sizes after you partition the data) and concerns about the changing nature of markets over time (making it hard to partition the data over the time dimension - the approach you have adopted).

I think the best solution is to use multiple-fold validation techniques. I also think the best way to create your in and out of sample sets is to partition the data randomly by contract and by time i.e. if you have 10 years of data and 10 contracts randomly remove 20 contract/year combinations, re-splice the continuous contracts across any gaps and optimize on the rest. I haven't done this myself - on the to-do list!

2) Start Conditions.

You will discover there is a butterfly effect in trading system back-testing. The results of a back-test display characteristics of chaotic systems in that starting conditions can have a scary effect on the final equity curve. Two starting conditions in particular come to mind: start date and initial equity. Consider 2 back-tests that differ early on by only one contract in one trade, this initially small difference can grow and grow over time. It is worth experimenting with a range of start dates and starting capital to get a better estimate of the system's expected goodness measure rather than rely on one back-test only.

LeviF · Post by **LeviF** » Sat Jun 04, 2011 12:59 am

I dont like to partition the data much because there are never enough years to go around. And if it performs good or bad on a couple of years of out of sample data, what does that really tell you?

I think a good alternative is changing up the portfolio/instruments. Optimize on half, then add or swap the other half, etc. There is a "random portfolio generation" block around here that will help you run some tests like this.

Coincidentally, I just learned today that my broker added a handful of new instruments available for trading last month. So I went and downloaded virgin data for the new instruments, ran them thru my systems, and viola! better performance metrics all around [pat self on back...]

rhc · Post by **rhc** » Sat Jun 04, 2011 10:19 am

Further to the above excellent suggestions, I also like to perform rolling test windows using the "start date stepping" feature in TB
You have defined your prefered parameters over a nice long slice of time but how do those perform in (say) 5 year rolling blocs?
i.e. from 1981-1986, then from 1982-1987, then 1983-1988 etc etc all the way up to 2011
This can help to show how your system performed with start date dependancy minimised.
You would think that a good 'robust' system might perform well in all 5 year windows from 1985 to present
I find long 15-20+ year tests can misleading sometimes and I like to run the above tests as well.

Also I note that you are using a 200K Account @2% bet size.
This amount of money in 1985 was more than adequate to trade all futures contracts using a LT system and because of this the magic of diversity worked in your favour and results in later years were enhanced.

Using this amount of cash in the last couple of years may not have been sufficient to trade things like the Energies & Metals & Stock indices & hence these trades may have been filtered out. (assuming you are using a LT system and not some ST swing system)
$200K in 1985 is equivalent in buying power to maybe $600K today (assuming an Inflation rate of 4%)

You can see the effect that a 200K account has on your results by doing the afore-mentioned 5 year rolling test windows & seeing how many trades are NOT taken vs how many trades are taken in recent times.
Maybe it makes no difference. Maybe it does

Haraldino · Post by **Haraldino** » Sun Jun 05, 2011 7:28 am

four additional suggestions:

CAGR is influenced mainly from the start date; the better alternative is RAR (a kind of regression)...

MAR is influenced mainly from the only one max dd; the better alternative is to use i.e. an average of the last five greatest dd...

rMAR = RAR/avgdd

maximize the numbers of instruments

maximize the starting capital (in order to have no problems with contract specifikations)

jameskuah · Post by **jameskuah** » Mon Jun 06, 2011 11:04 pm

Hi to all,

Thanks so much to all for your great advice. I look forward to implementing all the suggestions and will post the updated optimization results. Thanks and good luck to all in your trading.