I just posted an article in the Trading Library that covers a phenomenon which I call the Optimization Paradox which often leads to unrealistically good results coming from the process of optimizing parameters.
www.tradingblox.com/articles/optimization_paradox.htm
Any questions, comments or dissent should be made here in this topic.
- Forum Mgmnt
The Optimization Paradox
-
Forum Mgmnt
- Roundtable Knight
- Posts: 1842
- Joined: Tue Apr 15, 2003 11:02 am
- Contact:
-
blueberrycake
- Roundtable Knight
- Posts: 125
- Joined: Mon Apr 21, 2003 11:04 pm
- Location: California
Re: The Optimization Paradox
That's a great article c.f.. Too many people seem to have an irrational fear of sytem optimization. I think a key question that your article brings up is, what is the value to be optimized in order to maximize future profit. Is it simply maximum historical profit? Is it the "smoothness" of the profit chart against pertrubation in parameter value?
In my experience optimizing any single value, has always led to a poorly predictive optimization, leading me to look at all relevant measures and make a qualitative decision on the correct optimization values. What I mean is that I run a few optimizations based on things like net profit, profit factor, average trade size, etc., but after that I have to manually examine all the top contenders and play around with each one to see if the rules still make logical sense, look at the stability of results over time, differences between long/short performance, parameter stability, statistical summary (dispersion, t-statistic, etc), etc. It always feels like a cop out, but I could never create a single measure that when optimized created the highest future profit.
This becomes even more of a challenge when your optimization includes not only changing several parameter values, but also adding/removing rules as part of the optimization.
-bbc
In my experience optimizing any single value, has always led to a poorly predictive optimization, leading me to look at all relevant measures and make a qualitative decision on the correct optimization values. What I mean is that I run a few optimizations based on things like net profit, profit factor, average trade size, etc., but after that I have to manually examine all the top contenders and play around with each one to see if the rules still make logical sense, look at the stability of results over time, differences between long/short performance, parameter stability, statistical summary (dispersion, t-statistic, etc), etc. It always feels like a cop out, but I could never create a single measure that when optimized created the highest future profit.
This becomes even more of a challenge when your optimization includes not only changing several parameter values, but also adding/removing rules as part of the optimization.
-bbc
c.f. ,
It's a very good article. I totally agree with what you've said.
I have been confused in my discussions with other system developers when they continually say that optimization is not good and it is no predictor of future results. We all know the basis of their argument but I always argue that if something has worked well in the past then why should't it work well in the future ?? and the best systm from the past is prob. better to use than the worst right ??
Another question... if you are working with a general theory/ methodology of trading such as trendfollowing such that it is robust and nearly all parameters generate a positive result ( some with bigger drawdowns than others) then surely you have that as an extra safety net when optimising ... i.e if you know that anything from 30 to 70 days as a breakout , all else considered equal , will be profitable then surely picking the best historical performing parameter is not that risky since you know all parameters around the optimised number are profitable ??
just a thought.
regards
c
It's a very good article. I totally agree with what you've said.
I have been confused in my discussions with other system developers when they continually say that optimization is not good and it is no predictor of future results. We all know the basis of their argument but I always argue that if something has worked well in the past then why should't it work well in the future ?? and the best systm from the past is prob. better to use than the worst right ??
Another question... if you are working with a general theory/ methodology of trading such as trendfollowing such that it is robust and nearly all parameters generate a positive result ( some with bigger drawdowns than others) then surely you have that as an extra safety net when optimising ... i.e if you know that anything from 30 to 70 days as a breakout , all else considered equal , will be profitable then surely picking the best historical performing parameter is not that risky since you know all parameters around the optimised number are profitable ??
just a thought.
regards
c