A systematic way to portfolio optimzation

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Jens Albrecht
Senior Member
Senior Member
Posts: 40
Joined: Thu Feb 08, 2007 3:19 pm
Location: Dortmund, Germany

A systematic way to portfolio optimzation

Post by Jens Albrecht » Thu Jul 12, 2007 9:32 am

*** Warning: Written by a trading newbie, terrible english inside and just a leisure time researcher of trading systems. That said I hope you enjoy reading it ***

One day I was struck by the post of Ted (see viewtopic.php?t=3820) presenting an equity curve I've never ever seen before in my tests. It made me really thought of how to achieve equity curves (EQ) looking like straight lines. After several attempts fiddling with the systems itself I came up with the following approach of fiddling with the portfolio. You'll see after reading through that IMHO I found a systematic approach. I would be glad to dicuss it thouroughly here. Is this approach over-optimization? Is it simply wrong? Or maybe it's common knowledge and there are several sources to read much more about it ? Maybe you will just feel a moment as I did where some things like diversification, correlation and portfolio optimization did much more sense.

The idea: Each instrument contribute to the EQ (equity curve). Unfortunately (see the several diversification discussions) the instrument contributions are usually dependent of each other. But if we would know the EQ's of all combinations of instruments with a given system we would certainly choose the one looking the most as a straight line (at least I would do :D).

Mathematical solution: Given a set of N instruments and wanting to choose the best K instruments out of it there are (N K) possible combinations of instruments, where (N K) is the binonimal coefficient.
Let's say our future dictionary has 80 markets and I want the best EQ with 10 markets this gives (80 10) = 80! / (10! * (80-10)!) = 1646492110120 possibilites...wow, quite a few test runs. Now it's clear
to me that no two EQ look the same and portfolio construction is considered an art of its own. Given todays limited computing power we should better find another way solving it. One possible answer is certainly Local optimization. If I can not have the smoothest EQ I still want a very good one (think of chess programms).

Practical solution: Two ways appeared to me. Beginning with 0 markets I steadily add the one market to the set of markets which makes the EQ look the best (-> I've chosen R-Squared as my optimzation criteria).
Having added 10 markets I am done with the above example. This also takes some computing time but it's feasible (80 runs + 79 runs + 78 runs..755 test runs in total). Because I bothered that I add 10 super smooth but bad markets I took the other way. Beginning with the whole set of markets I steadily exclude one market after another so that R-Squared is maximum. Hope you get it...for safety a short example: having 80 markets, I make 80 test runs with 79 markets each, calculate R-Squared and take the one combination of 79 markets with the highest R-Squared value. Taken that combination I run 79 test with 78 each and so on. I do this as long as I have the wanted number of markets. As we surely all agree that it can be programmed I unfortunately own only the professional version of Trading Blox...so (urgs!) let's do it ... manually.

Just doing it: I took a simple standard Donchian 5-20 system with a long term filter (see attached Images). Nothing unusual, just default values. A set of 111 future markets overall. I want to choose 20 markets that give a pleasant equity curve according to the above rules. To limit the task I simplified it further by searching each step only a market which considerably enhanced R-Squared but I did not look for the one market with maximum R-Squared enhancement. Note that it was just a study so that it didn't matter that the resulting markets are tradable at all. I also set commission and slippage to 0. This doesn't limit the validity of the approach IMHO which can be used with any system under any settings. Important to say I intentionally didn't care about the resulting changes in MAR, profit factors and such. On the contrary I excluded some well known profitable markets as the Eurodollar giving a smoother curve in that step. To avoid over-optimization I steadily changed optimzation window. That means I excluded e.g. 5 markets based on the R-Squared of a 5-year period, then examined the whole EQ, choosed another 5-year period where the EQ was most jagged and excluded the next markets. That said the task of excluding market became more difficult at the end because each market got more and more influence on the total R-Squared. An interesting observation was that the whole EQ seems getting better and better this way. Btw. there was no deep thought for using 5-year periods. I just have CSI data for 10 years overall. Therefore the attached EQ to become a bit more jagged at the left end. There were simply missing the data for MACD priming etc. Those of you who own 20 years and more data will get better results I think.

Correlation: This brings me to another good old topic - market correlation. Personally I found out that the correlation coefficient of two markets not truely show whether they loose win or loose money at the same periods. Using human sense you are sure that they are strongly correlated but mathematically the correlation coefficient must not always show it. Using R-Squared as the optimzation criteria - giving the requested straight EQ line - the choice of markets reflects their perfect neutralization of each other over the investigated period and for the chosen trading system. What one market looses at a time will another market win (and some more...) resulting in a steady increase of equity. Regarding correlation there are in my opinion two possibilities. Either the markets are totally uncorrelated (I'm quite sceptical of this) or they are optimally correlated for the applied trading system. Either way I would say one is best diversified with that choice of markets.

Results: Now that the portfolio optimization was done I examined the results (yet not completely) and was (and still I am) reasonably enthusiastic of the results. A nice MAR of 2.48 with an R-Squared of 99.69
and this with a simple standard Donchian system. There also would have been better MAR's in the optimization process. Those people with mathematics background will know that you also can apply any other target function to the optimization process instead of f(x)=R-Squared. This way you also could optimize according to some other measures. Naturally having nearly a straight line one can alter start date and duration of the test and always get a steady gain in equity. A superb property came immediately to my mind. While varying the risk with the fixed fractional money manager R-Squared will be constant. See the attached test for confirmation. This last property is very powerful on its own to adjust the system to one's own needs. I'm curios which other properties such a approach to portfolio optimization reveals.

The answer: Last things to say...Back again to my initial motivation. Im totally sure that given e.g. 123 markets I could come up with a system using 85 markets and a resulting equity curve with R-Squared at 99.88%. My gut feeling says that I would take a LTTF system if I want that EQ hold for quite some time because of the diversification matter (see above).
Ted somewhere wrote in this forum that a CTA should give investor what they want. I think that it could be.

Please give me your opinion. Is it a valid approach? Does this approach exist already? For me portfolio optimization was a large black hole yet ...

Please give me your opinion.
Attachments
Donchian520Settings.png
Donchian520Settings.png (20.68 KiB) Viewed 6243 times
FromAllMarketsOptimzedTop20.set
(88 Bytes) Downloaded 586 times
LogarithmicEquityGraph_P1.png
LogarithmicEquityGraph_P1.png (30.29 KiB) Viewed 6245 times

Jens Albrecht
Senior Member
Senior Member
Posts: 40
Joined: Thu Feb 08, 2007 3:19 pm
Location: Dortmund, Germany

Post by Jens Albrecht » Thu Jul 12, 2007 9:36 am

just more test results to add...
Attachments
SteppingRisk.png
SteppingRisk.png (13.64 KiB) Viewed 6241 times
SteppingStartDateFor1500Bars.png
SteppingStartDateFor1500Bars.png (14.52 KiB) Viewed 6241 times
ResultForDonchian520with20markets.png
ResultForDonchian520with20markets.png (10.45 KiB) Viewed 6241 times

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Thu Jul 12, 2007 3:56 pm

1. What is the advantage of setting commissions and slippage to zero? It doesn't make the simulations run any faster. But it does distort the results.

2. Maybe you could optimize the portfolio over a subset of the historical data, like for example 1990 thru 2002. Then you could take the resulting "optimum portfolio" and simulate its performance on previously unseen data from 2003 thru 2007. It may help you decide whether or not the optimization procedure is Curve Fitting At Its Worst.

Angelo
Roundtable Fellow
Roundtable Fellow
Posts: 90
Joined: Fri Apr 29, 2005 4:31 am
Location: Italy

Re: A systematic way to portfolio optimzation

Post by Angelo » Fri Jul 13, 2007 10:43 am

Jens Albrecht wrote:
Im totally sure that given e.g. 123 markets I could come up with a system using 85 markets and a resulting equity curve with R-Squared at 99.88%.

Yes, you can be totally sure to get all results you like but just FOR THE PAST......
What is the solid motivaton for assuming R squared (as any other performance measure) will be constant in the future?

After all, we are all forced to trade in the future, regardless of the correlations one has been able to find in the past.

Jens, I do not want to be rude and - most of all - I'm far away from having all the answers I'd need, but I strongly advise you to read c.f. Book at p 50-51, when he shows 17 losing trades in cocoa...... that have no predicting power of the large winning trades that could have occurred in the future.
Or p 214 (and the whole chapter 13) on a missed trade on coffee, market in which the Turtles had been steadly losing money before.

Simply put, IMHO, there's no stationarity in the trading result of a traded instrument, in any measure you'd name.
The other way around, your technique should conquer the world very soon.
Before putting real money to risk- however - try to apply Ted's second advice not only on historical data, but on some real time paper trading too.
If I'm wrong, and you have something valuable, there's no need to hurry: there's always another day to trade the markets.


PS Do not be too eager in front of a straght equity curve.
Speaking in general terms, the trading arena is full of theorethical straight equity curve that happen to change slope when real time trading begins (I'm not talking of Ted's one, of which I don't know any details, but in very broad terms) .

Jens Albrecht
Senior Member
Senior Member
Posts: 40
Joined: Thu Feb 08, 2007 3:19 pm
Location: Dortmund, Germany

Post by Jens Albrecht » Sat Jul 14, 2007 3:50 am

Ted,

wouldn't everything be a distortion? If I set it 120$ I would have had a hard time just to care for the profitability of that Donchian system and hadn't focus on the task itself. Setting comssion to 0 seemed just logical to me to avoid that factor completely.

I fully agree on the testing procedure. I better should have done that before. I'll do it asap and add the results here.

Angelo,

I want to emphasize that it's just a study with the purpose to think about, not to trade it neccessarily. But as long as I see results somewhere I can not explain to myself I will search for an answer. Surely I've a hard time to accept that portfolio optimization (which we all do by some means or other!) could not be solved systematically.

I think we'll agree that we always only adjust for the past. But to give you just an example. The given default value of 1.8 for the ATR stop (I'm sure you know) is still the optimum although the value was chosen some time ago. I can not see why that shouldn't hold for the chosen instruments. While I don't want to predict the future, the approach just (at least could IMHO) tend to choose instruments that are correlated in a good way...even for the future. That is why I've chosen R-Squared. As the system uses a MACD long term filter the implied long term correlation of the markets is what makes me hope of holding up for the future.

Eventually only thorough testing will show I am about to implement the approach to test it large scale. A few hand-run tests are quite unsatisfying.

Chelonia
Roundtable Knight
Roundtable Knight
Posts: 497
Joined: Mon Apr 30, 2007 3:37 pm

Post by Chelonia » Mon Aug 06, 2007 5:48 pm

Hi Jens,

Do me a favor and test your ATR stop between 0.1 and 2 and then show me the results. You be suprised!

Chelonia
Roundtable Knight
Roundtable Knight
Posts: 497
Joined: Mon Apr 30, 2007 3:37 pm

Post by Chelonia » Mon Aug 06, 2007 5:50 pm

Code: Select all

       Ending                     Modified  Annual  Max Total  Longest
Test   Balance    CAGR%   MAR     Sharpe    Sharpe  Equity DD  DD (mo)    # Trades 
 .................................................................................
  1      $0.00    191.2%  4.27     2.32      1.89     44.8%      7.3       11,771

Post Reply