A systematic way to portfolio optimzation
Posted: Thu Jul 12, 2007 9:32 am
*** Warning: Written by a trading newbie, terrible english inside and just a leisure time researcher of trading systems. That said I hope you enjoy reading it ***
One day I was struck by the post of Ted (see viewtopic.php?t=3820) presenting an equity curve I've never ever seen before in my tests. It made me really thought of how to achieve equity curves (EQ) looking like straight lines. After several attempts fiddling with the systems itself I came up with the following approach of fiddling with the portfolio. You'll see after reading through that IMHO I found a systematic approach. I would be glad to dicuss it thouroughly here. Is this approach over-optimization? Is it simply wrong? Or maybe it's common knowledge and there are several sources to read much more about it ? Maybe you will just feel a moment as I did where some things like diversification, correlation and portfolio optimization did much more sense.
The idea: Each instrument contribute to the EQ (equity curve). Unfortunately (see the several diversification discussions) the instrument contributions are usually dependent of each other. But if we would know the EQ's of all combinations of instruments with a given system we would certainly choose the one looking the most as a straight line (at least I would do
).
Mathematical solution: Given a set of N instruments and wanting to choose the best K instruments out of it there are (N K) possible combinations of instruments, where (N K) is the binonimal coefficient.
Let's say our future dictionary has 80 markets and I want the best EQ with 10 markets this gives (80 10) = 80! / (10! * (80-10)!) = 1646492110120 possibilites...wow, quite a few test runs. Now it's clear
to me that no two EQ look the same and portfolio construction is considered an art of its own. Given todays limited computing power we should better find another way solving it. One possible answer is certainly Local optimization. If I can not have the smoothest EQ I still want a very good one (think of chess programms).
Practical solution: Two ways appeared to me. Beginning with 0 markets I steadily add the one market to the set of markets which makes the EQ look the best (-> I've chosen R-Squared as my optimzation criteria).
Having added 10 markets I am done with the above example. This also takes some computing time but it's feasible (80 runs + 79 runs + 78 runs..755 test runs in total). Because I bothered that I add 10 super smooth but bad markets I took the other way. Beginning with the whole set of markets I steadily exclude one market after another so that R-Squared is maximum. Hope you get it...for safety a short example: having 80 markets, I make 80 test runs with 79 markets each, calculate R-Squared and take the one combination of 79 markets with the highest R-Squared value. Taken that combination I run 79 test with 78 each and so on. I do this as long as I have the wanted number of markets. As we surely all agree that it can be programmed I unfortunately own only the professional version of Trading Blox...so (urgs!) let's do it ... manually.
Just doing it: I took a simple standard Donchian 5-20 system with a long term filter (see attached Images). Nothing unusual, just default values. A set of 111 future markets overall. I want to choose 20 markets that give a pleasant equity curve according to the above rules. To limit the task I simplified it further by searching each step only a market which considerably enhanced R-Squared but I did not look for the one market with maximum R-Squared enhancement. Note that it was just a study so that it didn't matter that the resulting markets are tradable at all. I also set commission and slippage to 0. This doesn't limit the validity of the approach IMHO which can be used with any system under any settings. Important to say I intentionally didn't care about the resulting changes in MAR, profit factors and such. On the contrary I excluded some well known profitable markets as the Eurodollar giving a smoother curve in that step. To avoid over-optimization I steadily changed optimzation window. That means I excluded e.g. 5 markets based on the R-Squared of a 5-year period, then examined the whole EQ, choosed another 5-year period where the EQ was most jagged and excluded the next markets. That said the task of excluding market became more difficult at the end because each market got more and more influence on the total R-Squared. An interesting observation was that the whole EQ seems getting better and better this way. Btw. there was no deep thought for using 5-year periods. I just have CSI data for 10 years overall. Therefore the attached EQ to become a bit more jagged at the left end. There were simply missing the data for MACD priming etc. Those of you who own 20 years and more data will get better results I think.
Correlation: This brings me to another good old topic - market correlation. Personally I found out that the correlation coefficient of two markets not truely show whether they loose win or loose money at the same periods. Using human sense you are sure that they are strongly correlated but mathematically the correlation coefficient must not always show it. Using R-Squared as the optimzation criteria - giving the requested straight EQ line - the choice of markets reflects their perfect neutralization of each other over the investigated period and for the chosen trading system. What one market looses at a time will another market win (and some more...) resulting in a steady increase of equity. Regarding correlation there are in my opinion two possibilities. Either the markets are totally uncorrelated (I'm quite sceptical of this) or they are optimally correlated for the applied trading system. Either way I would say one is best diversified with that choice of markets.
Results: Now that the portfolio optimization was done I examined the results (yet not completely) and was (and still I am) reasonably enthusiastic of the results. A nice MAR of 2.48 with an R-Squared of 99.69
and this with a simple standard Donchian system. There also would have been better MAR's in the optimization process. Those people with mathematics background will know that you also can apply any other target function to the optimization process instead of f(x)=R-Squared. This way you also could optimize according to some other measures. Naturally having nearly a straight line one can alter start date and duration of the test and always get a steady gain in equity. A superb property came immediately to my mind. While varying the risk with the fixed fractional money manager R-Squared will be constant. See the attached test for confirmation. This last property is very powerful on its own to adjust the system to one's own needs. I'm curios which other properties such a approach to portfolio optimization reveals.
The answer: Last things to say...Back again to my initial motivation. Im totally sure that given e.g. 123 markets I could come up with a system using 85 markets and a resulting equity curve with R-Squared at 99.88%. My gut feeling says that I would take a LTTF system if I want that EQ hold for quite some time because of the diversification matter (see above).
Ted somewhere wrote in this forum that a CTA should give investor what they want. I think that it could be.
Please give me your opinion. Is it a valid approach? Does this approach exist already? For me portfolio optimization was a large black hole yet ...
Please give me your opinion.
One day I was struck by the post of Ted (see viewtopic.php?t=3820) presenting an equity curve I've never ever seen before in my tests. It made me really thought of how to achieve equity curves (EQ) looking like straight lines. After several attempts fiddling with the systems itself I came up with the following approach of fiddling with the portfolio. You'll see after reading through that IMHO I found a systematic approach. I would be glad to dicuss it thouroughly here. Is this approach over-optimization? Is it simply wrong? Or maybe it's common knowledge and there are several sources to read much more about it ? Maybe you will just feel a moment as I did where some things like diversification, correlation and portfolio optimization did much more sense.
The idea: Each instrument contribute to the EQ (equity curve). Unfortunately (see the several diversification discussions) the instrument contributions are usually dependent of each other. But if we would know the EQ's of all combinations of instruments with a given system we would certainly choose the one looking the most as a straight line (at least I would do
).Mathematical solution: Given a set of N instruments and wanting to choose the best K instruments out of it there are (N K) possible combinations of instruments, where (N K) is the binonimal coefficient.
Let's say our future dictionary has 80 markets and I want the best EQ with 10 markets this gives (80 10) = 80! / (10! * (80-10)!) = 1646492110120 possibilites...wow, quite a few test runs. Now it's clear
to me that no two EQ look the same and portfolio construction is considered an art of its own. Given todays limited computing power we should better find another way solving it. One possible answer is certainly Local optimization. If I can not have the smoothest EQ I still want a very good one (think of chess programms).
Practical solution: Two ways appeared to me. Beginning with 0 markets I steadily add the one market to the set of markets which makes the EQ look the best (-> I've chosen R-Squared as my optimzation criteria).
Having added 10 markets I am done with the above example. This also takes some computing time but it's feasible (80 runs + 79 runs + 78 runs..755 test runs in total). Because I bothered that I add 10 super smooth but bad markets I took the other way. Beginning with the whole set of markets I steadily exclude one market after another so that R-Squared is maximum. Hope you get it...for safety a short example: having 80 markets, I make 80 test runs with 79 markets each, calculate R-Squared and take the one combination of 79 markets with the highest R-Squared value. Taken that combination I run 79 test with 78 each and so on. I do this as long as I have the wanted number of markets. As we surely all agree that it can be programmed I unfortunately own only the professional version of Trading Blox...so (urgs!) let's do it ... manually.
Just doing it: I took a simple standard Donchian 5-20 system with a long term filter (see attached Images). Nothing unusual, just default values. A set of 111 future markets overall. I want to choose 20 markets that give a pleasant equity curve according to the above rules. To limit the task I simplified it further by searching each step only a market which considerably enhanced R-Squared but I did not look for the one market with maximum R-Squared enhancement. Note that it was just a study so that it didn't matter that the resulting markets are tradable at all. I also set commission and slippage to 0. This doesn't limit the validity of the approach IMHO which can be used with any system under any settings. Important to say I intentionally didn't care about the resulting changes in MAR, profit factors and such. On the contrary I excluded some well known profitable markets as the Eurodollar giving a smoother curve in that step. To avoid over-optimization I steadily changed optimzation window. That means I excluded e.g. 5 markets based on the R-Squared of a 5-year period, then examined the whole EQ, choosed another 5-year period where the EQ was most jagged and excluded the next markets. That said the task of excluding market became more difficult at the end because each market got more and more influence on the total R-Squared. An interesting observation was that the whole EQ seems getting better and better this way. Btw. there was no deep thought for using 5-year periods. I just have CSI data for 10 years overall. Therefore the attached EQ to become a bit more jagged at the left end. There were simply missing the data for MACD priming etc. Those of you who own 20 years and more data will get better results I think.
Correlation: This brings me to another good old topic - market correlation. Personally I found out that the correlation coefficient of two markets not truely show whether they loose win or loose money at the same periods. Using human sense you are sure that they are strongly correlated but mathematically the correlation coefficient must not always show it. Using R-Squared as the optimzation criteria - giving the requested straight EQ line - the choice of markets reflects their perfect neutralization of each other over the investigated period and for the chosen trading system. What one market looses at a time will another market win (and some more...) resulting in a steady increase of equity. Regarding correlation there are in my opinion two possibilities. Either the markets are totally uncorrelated (I'm quite sceptical of this) or they are optimally correlated for the applied trading system. Either way I would say one is best diversified with that choice of markets.
Results: Now that the portfolio optimization was done I examined the results (yet not completely) and was (and still I am) reasonably enthusiastic of the results. A nice MAR of 2.48 with an R-Squared of 99.69
and this with a simple standard Donchian system. There also would have been better MAR's in the optimization process. Those people with mathematics background will know that you also can apply any other target function to the optimization process instead of f(x)=R-Squared. This way you also could optimize according to some other measures. Naturally having nearly a straight line one can alter start date and duration of the test and always get a steady gain in equity. A superb property came immediately to my mind. While varying the risk with the fixed fractional money manager R-Squared will be constant. See the attached test for confirmation. This last property is very powerful on its own to adjust the system to one's own needs. I'm curios which other properties such a approach to portfolio optimization reveals.
The answer: Last things to say...Back again to my initial motivation. Im totally sure that given e.g. 123 markets I could come up with a system using 85 markets and a resulting equity curve with R-Squared at 99.88%. My gut feeling says that I would take a LTTF system if I want that EQ hold for quite some time because of the diversification matter (see above).
Ted somewhere wrote in this forum that a CTA should give investor what they want. I think that it could be.
Please give me your opinion. Is it a valid approach? Does this approach exist already? For me portfolio optimization was a large black hole yet ...
Please give me your opinion.