Correlation

Discussions about Money Management and Risk Control.
Post Reply
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Correlation

Post by Chris67 » Tue Dec 16, 2003 3:52 pm

This is my second question and has caused me huge headaches for years
Correlation has a tendency to shift dramatically over short time period and long time periods .. 2 markets that are correlated at 0.9 over 50 days may suddenly be correlated at -0,4
for example witness this year - U.S stocks up - U.s dollar down ... last year the opposite applied
Since correlation is an important part of the system how do you monitor correlation and its change .. what time periods ... what do you do if you are long 2 products and they have no correlation that is significant but after 40 days , say , you are still in both positions and suddenly the correlation is very large ??

I use excel to work out correlation and have tried comparing different time frames but again you see that 2 markets have no correlation over 30 days , are correlated at 0.99 over , say , 60 days , and have no corrleation over 150 days ?? it confuses me a great deal ...
Help !! I need somebody !! hahah
thanks in advance

George
Full Member
Full Member
Posts: 19
Joined: Thu Jul 31, 2003 11:42 pm
Location: Atlanta

Correlation 12 year study

Post by George » Tue Dec 16, 2003 4:00 pm


DrHendricks
Roundtable Fellow
Roundtable Fellow
Posts: 81
Joined: Tue May 13, 2003 12:21 am

Post by DrHendricks » Tue Dec 16, 2003 5:09 pm

I believe a couple of observations are worthwhile:

1) The pertinent time window of correlation is the same as the expected trade duration. For short term strategies, correlation drift over 6 months probably doesn't matter. For longer time frames, a generalized algorithm such as close and loose correlation that c.f. uses with Veritrader is a more robust solution.

2) In a panic, all correlations approach 1.0. That is why I believe limiting total portfolio risk is a higher priority than individual position correlation.

David

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Post by kianti » Wed Dec 17, 2003 3:12 am

You can find a relationship between any two items if you look hard enough. It may be entirely spurious and have no predictive power, but you will find one. To give you an idea, you'll always find what we call data miners who will show you that there is a 100 percent correlation between his great aunt's blood pressure and the back-month Nikkei volatility. When you're a trader you get a lot of calls from people who found relationships that can produce a 10 Sharpe ratio. That means it's almost impossible to lose money on the trade. Sure enough, when you start trading you realize that the relationship was not there. Trading has fewer biases than statistics.

Nassim Taleb
I share the same feelings about diversification and correlation.

I would like to have a correlation algorithm to minimze the risk of a diversified portfolio; the best things I’ve found so far are c.f.’s strong and lose correlation, Ed Seykota’s total heat and trade window duration.

I also found this interview with Nassim Taleb particularly interesting

http://www.derivativesstrategy.com/maga ... 1296qa.asp

Javelin
Senior Member
Senior Member
Posts: 41
Joined: Fri Jul 11, 2003 10:22 pm
Location: Georgia

Post by Javelin » Wed Dec 17, 2003 6:59 pm

Kianti,
Could you point to a source for more info on Seykota's trade window duration? Thanks.
As I learn more about correlation/diversification, I find that the meaning of a smooth equity curve takes on more depth. Very interesting subject.
J

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Correlation

Post by kianti » Thu Dec 18, 2003 12:47 am

Javelin,

I borrowed 'trade window duration' from DrEndricks, I don't know if Seykota used that specific term. I guess it's preatty simple, the less you trade the less you spend in commissions and the slippage on a 120 days breakout is lower than on a 20 days breakout.

Thinking about correlation I have the feeling that there is some kind of relationship with the historical low level of interest rates in US and Europe.
I tested a few trend following systems on the last 12 years, averages, bollingers, channels and other stuff and most of the systems made very good money until the end of the 90'.

I would like to share some ideas with the other friends in the forum. Maybe there is a relationship of the trend following lengths with the level of Fed Funds rates.
Is it possible that with a 1% risk-free interest rates the arbs are trying to squeeze every single tick from every single financial instrument?
Trend following could be considered trading against greed and fear, is there another arbs bubble behind the corner?
Maybe the last few years are just a trading range in the general trend following strategy.

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Thu Dec 18, 2003 8:28 am

Both of these statements are true:
  • Several of the most widely known long term trendfollowing systems have suffered poor performance in the most recent 2-5 years, compared to their performance in the 1990's
  • Many trendfollowing CTA's have enjoyed excellent performance during the same period of time, better than their own performance in the 1990's
If these statements are correct (they are), what conclusions can you draw? How does it affect you if at all?

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Thu Dec 18, 2003 10:14 am

Ted ,
Interesting:

The ONLY conclusion that can be drawn from that is that the trendfollowing CTA's do not use the most popular or renowned trendfollowing systems but have eveolved to totally new methods ?? am I right ?

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Thu Dec 18, 2003 5:37 pm

This question about the valid time period of correlation leaves me a bit cold.

There are two answers that spring to mind:

- use back testing. You'll see whether your time frame for correlation has any impact on your results.

- as far as I'm concerned, I compute the correlation for as long as I have data for, and use the current value at the time I initiate the trade, with not so poor results. Granted, past data of 6 years ago somehow influence today's value. And so what? I use bands: if the abolute value is > 0.6666 then strong otherwise, if the absolute value is > 0.3333 then weak, otherwise then no correlation. I'm not sure what a finer grained system, or more "timely accrurate" system would bring to the party ...

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Fri Dec 19, 2003 1:54 am

Verec ,
Thanks for your reply.
I guess my point is this..
If you look at correlation for the last 10 years ( this is just one example .. there are many many more) .. and you had a signal to buy US dollars and Us stocks at the same time then you would nt take signal and load in both since on most long term historic correlation calculations they are highly correlated.
However for the whole of this year the correlation has gone from pretty much 0.9 to -0.9 .. i.e u.s stocks up now seems to bring USD down.

My point is that if correlation can change dramatically and the persist for say 6 months then you can miss many good signals and also find yourself fully loaded in 12 directional units that are uncorrelated at initiation but after a few weeks of holding they all move exactly together .. such that you may have 3 times the risk on you assumed .
Also I notice that when the proverbial ''poo'' hits the fan all correlation data goes out the window.
Take sept 11 th for example. You could have a nicely balanced portfolio on of stock indeces , currencies and commodities but find that every position you have stops you out at the same time and if you have 20 % of open portfolio heat that can be an ugly situation.
Another point : if you take the original turtle observations on what is strongly and loosely correlated u may find many of these relationships dont hold true anymore. Some may hold true over a look at 10 years but be completely wrong for a 6 month duration .. therefore by not calculating correlation change you may be putting on 3 times the size position you think you are .
I guess my final though is that financial markets have a huge tendancy to shift theyre focus for large periods of time and that dramatically changes what are percieved to be strongly and loosely correlated units
I would warmly appreciate any other views on this

Regards
Chris

-w.
Roundtable Fellow
Roundtable Fellow
Posts: 95
Joined: Wed Nov 19, 2003 11:02 am

Post by -w. » Fri Dec 19, 2003 6:53 am

Very interesting topic, Chris.
Chris67 wrote:[...] if you take the original turtle observations on what is strongly and loosely correlated u may find many of these relationships dont hold true anymore. Some may hold true over a look at 10 years but be completely wrong for a 6 month duration .. therefore by not calculating correlation change you may be putting on 3 times the size position you think you are.
While it certainly may be tempting to account for correlation change because of the reasons you cited, doing so presents it's own dangers I think.

If you look at correlations over a longer time span (5 or 10 years), two markets might show a zero correlation which is the wished-for correlation from a diversification point of view, as I understand it. Now, does that mean those markets never correlate for shorter periods within those longer time frames? Probably not.

The more one would shorten the time frames or even account for correlation changes, the less the chance to come up with uncorrelating markets: correlation would fluctuate from positive to negative and back, exactly as you describe it in the stocks/US$ example. But if all markets you trade are either positively or negatively correlated at all times, the concept of risk reduction through diversification doesn't work anymore, IMO.

To me this looks very much like over-optimization, just on the portfolio level. Tempting, as I said above. But I don't think it would hold.

-wojo

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Fri Dec 19, 2003 7:31 am

To me this looks very much like over-optimization, just on the portfolio level. Tempting, as I said above. But I don't think it would hold.
That's excatly what I think. BTW, the example given when the correlation went to +0.9 to -0.9 is probably a text book example, in that whether the two markets are correlated or ANTI correlated means the same thing from a "diversification" point of view. That's why I use "bands" on the ABSOLUTE value of the correlation. What this correlation thing really says is the degree of "dependence" or "common root cause"; whether they are behaving the same way, or behaving in opposite way still means that there are some factors that is the source of both observed behaviors. Which, from my point of view, means that I pay no attention, at all, to whether we're talking about a correlation or an ANTI correlation.

The only question is then "do they react at more or less the same time to some unknown common cause, even though the way they react does differ, or not?". And that's what the ABSOLUTE value of the correlation tells me. Again, only by testing will you see whether you need 3 bands (as I do) or more bands, or less bands, but I doubt very much that shortening the time frime will provide you with better results.

A classical example in statistics is the minimal amount of samples before you can start drawing some conclusion. And I would tend to say that anything less than 3 month (60 days) history is biased beyond repair.

But that's just me :)

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Fri Dec 19, 2003 7:43 am

Thanks for the postings Chaps.

I still maintain that if you dont take account of say 50 day correlation that if you are operating on a 55 day breakout then you may load up in correlated positions ... it may well be that over the long term the correlations hold pretty constant but my capital will run out long before correlations come back into line..
And remenber in the long term .. '' we're all dead''

If i go long usd today and long corn ( randon exampple ) and they are non correlated over 10 years but for the last 6 months they have moved in tandom for whatever reason .. then I would suggest you have a 95 % chance of having a position on with double the risk you think you have.

From my researh Im beginning to believe its a art and not a science .. I think common sense prevails .. If I got a signal today to buy dollars and stocks I would do both even though 10 year correlation is very high because , for example it may be over the last 4 months the corrleation is -0.4 which is acceptable ..
I think this area is important because it s telling us that there will always be areas of trading that human judgement is more important than anything else .. you have to know what the mkt is focusing on at the moment else youre in trouble

More responses plse .. this is an importanat and interesting subject

Kind regards

-w.
Roundtable Fellow
Roundtable Fellow
Posts: 95
Joined: Wed Nov 19, 2003 11:02 am

Post by -w. » Fri Dec 19, 2003 8:07 am

Chris67 wrote:[...] and they are non correlated over 10 years but for the last 6 months they have moved in tandom for whatever reason .. then I would suggest you have a 95 % chance of having a position on with double the risk you think you have.
I beg to differ strongly on this one :)

I have exactly the risk I choose to take if the system has been designed with long term backtesting. In order to account for this risk (uncertainty), I have the money management algorithms such as the rules for "how many (directional) units".

The problem just cannot be solved on the portfolio level, IMO. But thank you for being so insisting. (The correlation topic is a big issue for me as I'm testing stock portfolios. This discussion helped me to get some more insight.) 8)

-wojo

jimsta
Full Member
Full Member
Posts: 13
Joined: Sun Jun 15, 2003 2:45 am

Post by jimsta » Sun Dec 21, 2003 8:55 pm

Interesting topic. I have found correlation to be like picking direction ( long or short ). Because the instruments go up and down in an apparent random, so must the interaction between them. Fund managers are always rebalancing their portfolios in an attempt to avoid one area becoming the dominant. There has been a move toward global as well national diversity to avoid this. Themes here are diversification and total risk (as opposed to individual trade risk). Perhaps the problem that is being insuated is the killer drawdown (apart from maximising profits). Maybe what should be questioned is a "gunslinging" risk in search of higher returns. Having been hit hard in the past my risk levels are much tighter than ever before the hits. Its survival first!, anything else is a bonus.
The suggestion about art is an interesting one, like faith when the losing streak arrives. A psycological response to the random part of random, yet the bell curve is the ordered part of random, science got us here from the jungle.

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Sun Dec 21, 2003 11:17 pm

Heres another thought.

To me correlation is likely to be on again off again and sometimes ridiculous. IMO the best I can do is pick a portfolio that are unlikely to be correlated (and historically dont seem to have been too correlated too often) and use that.

An example would be:
2 currencies, JY and EU
2 energy, CL and NG
A metal, GC
Some grain C and W
Bonds/Int TY and ED
More? CT and RR

and maybe if I wanted to be exotic Palladium and something overseas. Perhaps also another currency crossrate like GB/AD or something. In the end this will be as good as a more scientific choice because in the end too much data mining becomes mouse milking.

John

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Fuzzy Correlation .... or just fuzzy

Post by kianti » Tue Dec 23, 2003 4:28 pm

Here's my latest fuzzy thoughts:

Rules
IF the world is ever-changing and non-stationary THEN I follow trends
IF world is uncertain and dynamic THEN divergences and changes may be dramatic an unanticipated
IF trend-following THEN no prediction of fair values


Uncertainty principle
Trade-off between two bell curves. As one bell curve grows narrower and thus more certain, the other grows wider and thus less certain (Werner Heisemberg, published the popular uncertainty principle in the late 1920s).

Centroidal Defuzzier

1) Kiwi's diversification:Portfolio= financials+
currencies+
energy+
metals+
softs+
meats+
indices

2) Turtles Max & Closely fuzzy correlated market
3) Last but not the least: max long,max short units; to reduce single sectors correlation.


.... and few fuzzy thougths

Into every tidy scheme for arranging the pattern of human life, it is necessary to inject a certain dose of anarchism

Bertrand Russel
or
So far as the law of mathematics refer to reality, they are not certain. And so far as they are certain, they do not refer to reality

Albert Einstein

Fuzzy thoughts produced under the influence at home of one extreme event, fat tail or black swan (Parmalat).

Merry Christmas and Happy New Year
[/b]

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Risk/Reward and Positive Profit Expectation

Post by kianti » Fri Jan 02, 2004 2:29 am

Abraham Trading Company’s trading methodology is a systematic, long-term, trend-following approach implementing filtering techniques that avoid trends with adverse risk/reward characteristics. While the goal is to capture long-term trends, the system only enters the market during periods when the risk/reward of a trade is heavily in the trade’s favor. If unacceptable risk characteristics exist, the system will even avoid trends that have a positive profit expectation.
http://www.abrahamtrading.com/method2.htm#Portfolio

I think that concept of positive profit expectation is clear to all the trend-following strudents and traders. I will study more deeply the risk/reward concept, any input for research would be very welcome.

In the above link I found also an interesting example of diversification.
Abraham Trading Company strives to trade the most diversified futures portfolio possible for our investors. Our portfolio contains 51 markets covering the currency, interest rate, grain, soft, meat, metal, and energy sectors. The markets are individually weighted for diversification purposes, resulting in an overall 38% financial-based and 62% commodity-based exposure.

Currencies: 19%
CME British Pound
CME Canadian Dollar
CME Euro
CME Swiss Franc
CME Japanese Yen
CME Australian Dollar
CME Mexican Peso
FINEX Euro-Pound Cross
FINEX Euro-Yen Cross
Interest Rates: 19%
CME Eurodollar
LIFFE Euribor
TFF EuroYen
CBOT US 10-Year Note
CBOT US 30-Year Bond
ME Canadian Gov't. Bond
LIFFE Long Gilt
EUREX Euro-German Bund
TSE JGB
SFE Aussie 10-Year Bond
Grains: 11%
CBOT Wheat
KCBT KC Wheat
CBOT Corn
CBOT Soybeans
CBOT Soy Meal
CBOT Bean Oil
WCE Canola
Softs: 16%
NYBOT Cotton
NYBOT Sugar #11
LIF London Sugar
NYBOT Coffee
LIF Coffee Robusta
NYBOT Cocoa
LIF London Cocoa
NYBOT Orange Juice
Meats: 8%
CME Lean Hogs
CME Pork Bellies
CME Live Cattle
CME Feeder Cattle
Metals: 15%
NYMEX Platinum
COMEX Silver
COMEX Gold
COMEX HG Copper
LME Aluminum
LME Nickel
LME Zinc
Energy: 12%
NYMEX Light Crude Oil
IPE Brent Crude Oil
NYMEX Heating Oil
IPE Gas Oil
NYMEX Unleaded Gas
NYMEX Natural Gas

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Fri Jan 02, 2004 8:40 am

The thing that matters most is: what do you see in your own testing? Until you run your own tests and obtain your own results, you've got nothing concrete.

If and when you do run tests and you do get results, you will be able to struggle with more sophisticated problems like
  • My test system makes big profits in Yen and big losses in Lean Hogs. What should I do now?
  • My test system makes a lot more money when I eliminate all the metals from my portfolio, than when I leave them in. What should I do now?
  • I ran my test system on a big portfolio that included both Brent-Crude and Crude-Oil, then I ran it again with just Crude-Oil at double position size. The results were basically identical. Should I leave Brent-Crude in the portfolio, should I trade Crude-Oil at double size, or what?
  • I did a test where I allowed the number of correlated positions (and/or the Group Total Risk) for the Currencies to be 50% bigger than I was testing before. The test results were noticeably better. Is it okay if I tip the balance of risk in favor of Currencies a little? After all, I'm not curve-fitting to a single market, I'm just favoring a proven trendy sector? What should I do now?
  • I tested the Original Turtle System and the ATR Band Breakout System together. Should I use completely different markets for the OTS than for the ATRBBS? Or should they be identical portfolios? My results show that OTS does great in the Bonds-InterestRates while ATRBBS does great in Currencies. What should I do now?

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Post by kianti » Fri Jan 02, 2004 12:04 pm

Ted,

Thanks for the reply, those are the actual problems I’m facing. These are the answers I’m giving to these questions.

Yen/Lean Hogs - My Portfolio should include also contracts losing money in the past, Live Cattle this year has been one of the best trending contracts; past performance is no guarantee for the future.

Sectors – My Portfolio should include all the main sectors, including stock-index futures; the new contracts introduced like e-mini can help to limit sector exposure.

Correlation – Exclude highly correlated positions like IPE/NIMEX Crude Oil or Soy/Soybeans. Define rules for controlling loosely correlated positions, like max loosely and/or long/short positions.

Sectors Weightings: I don’t know yet; maybe use the volumes traded. Maybe use the so called ‘risk/reward’. I don’t know yet how to define risk/reward; you mentioned above the interest-rates sector, 1% in the US, it could be a good example of risk-reward opportunity.

Post Reply