Blending noncorrelated (or anti-correlated) equity curves

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1428
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF » Mon Feb 21, 2011 11:03 am

LeviF wrote:The concept is simple. Finding un/anti correlated equity curves is not.
I probably should have been clearer. The concept is simple. Finding un/anti correlated equity curves that improve portfolio gain/pain metrics is not.

Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon » Mon Feb 21, 2011 1:09 pm

LeviF wrote:
There must be some other factors at play here aside from just equity curve correlation. I can come up an uncorrelated system with negative expectancy that hurts overall portfolio performance. Or what about two systems with -1 correlation (system 2 takes the opposite trade of system 1).
Allocation is what's missing. The excellent expositions above are using equal allocations to make the math easier and to make a particular point very clear.

Consider Sluggo's proposition:
if ECY is a lovely nice equity curve and ECZ is a complete and total stinker, with gut-wrenching drawdowns, psychotic swings, and enormous hand over fist losses, then adding ECZ to ECY will not increase your happiness! It will reduce both pain AND gain, and you won't like the result.
Let's build an example where one equity curve has standard deviation of 0.01 and another has 0.1 (10 times larger). Let's say their correlation is -0.7. A portfolio allocation of 93.5% to the first system and 6.5% to the second results in a minimum variance portfolio with a standard deviation of 0.0007!

Jez,

Here's another equation builder you might like (but you need LaTeX installed in your blog code): LaTeX Editor

Great thread!

Edited for typos / clarity.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Mon Feb 21, 2011 5:26 pm

Eventhorizon wrote:Let's build an example where one equity curve has standard deviation of 0.01 and another has 0.1 (10 times larger). Let's say their correlation is -0.7. A portfolio allocation of 93.5% to the first system and 6.5% to the second results in a minimum variance portfolio with a standard deviation of 0.0007!
In the special case of blending exactly two equity curves, the optimum allocation has a simple closed-form solution, shown below. For Eventhorizon's example, the optimum allocation to ECA, the low volatility equity curve, is 93.043% (107/115) of the account. ECB, the high volatility equity curve, gets 6.957% (8/115).

Image

Edit - added later - Let's do a couple of sanity checks
  • When ECB's volatility is zero, the formula says the optimum allocation equals 1 for B and zero for A. Sounds right.
  • When ECB's volatility is infinity, the formula says the optimum allocation equals zero for B and 1 for A. Sounds right.
  • When ECB and ECA have the same volatility, the formula says the optimum allocation equals 1/2 for B and 1/2 for A, no matter whether correlation (rho) is +1, 0, -1, or anything in between. Sounds right, yes?
Last edited by sluggo on Mon Feb 21, 2011 7:37 pm, edited 1 time in total.

Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon » Mon Feb 21, 2011 7:05 pm

By way of explanation ... the differences in solution arise because I simulated 2 series of length 1000 with the characteristics mentioned above and jammed them through some code I use to calculate the eigen-decomposition of a covariance matrix of real multiple data series!

A sledgehammer compared to Sluggo's elegent closed-form solution!

Reminds me of a joke:

Several scientists were all posed the following question: "What is 2 × 2 ?"
The engineer whips out his slide rule (so it's old) and shuffles it back and forth, and finally announces "3.99".
The physicist consults his technical references, sets up the problem on his computer, and announces "it lies between 3.98 and 4.02".
The mathematician cogitates for a while, then announces: "I don't know what the answer is, but I can tell you, an answer exists!".
Philosopher smiles: "But what do you mean by 2 × 2?"
Logician replies: "Please define 2 × 2 more precisely."
Medical Student : "4." All others looking astonished : "How did you know?" Medical Student : "I memorized it."

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Tue Feb 22, 2011 4:54 am

i was wondering:

does the use of root mean square measures(i.e. StDev) induce any parculiar effects when doing all these variance analysis for weightings?
i.e. if we used a Rescaled MAD or AAD (median and average absolute deviations, respectively), would that be more sensible?

I understand that Stdev responds better to the wilder stuff which is most likely the bits that matter to us in terms of "pain", but AAD is more arithmetically consistent over a large dataset with frankly completely unknown underlying population distributions(returns of asset timeseries transformed by application of tradelogic).

personally it is nice to understand these principles, but i wouldn't try and engineer this stuff too much, it is rather too "top down" to my taste.
- i prefer to plant a nice mix of good seeds and accept the wood at the end, than try and engineer a perfectly even forest canopy with heavy manipulation.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Tue Feb 22, 2011 8:20 am

rabidric wrote:does the use of root mean square measures(i.e. StDev) induce any parculiar effects when doing all these variance analysis for weightings? i.e. if we used a Rescaled MAD or AAD (median and average absolute deviations, respectively), would that be more sensible?

Certainly, feel free to do it that way. In fact if you operated a Fund of Funds, you could emphasize this detail in your marketing presentations and literature, thereby showing how you stand apart from the herd of other FoF sellers. You'd probably want to exhibit a couple of real-world examples where Markowitz's Nobel Prize winning variance-based approach yielded noticeably inferior results compared to your Rescaled MAD / AAD based approach. Just to drive home the point that you're not merely arguing a theoretical distinction, but rather putting more risk-adjusted returns in your investor's pockets, in the real world, here and now.

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Tue Feb 22, 2011 10:02 am

lol, thanks for the sarcy reply. i should have expected that i guess. I wonder if that is your equivalent to "you could take that to your trading tribe". :oops:
problem is, this is my goddamn trading tribe!!!! :?

ah well. 8)
sluggo wrote:You'd probably want to exhibit a couple of real-world examples where Markowitz's Nobel Prize winning variance-based approach yielded noticeably inferior results compared to your Rescaled MAD / AAD based approach.
heh, i've read those papers too. I also ran simulations on my own "deliberately nasty" synthetic distributions. in the end it didn't make any real difference, sometimes it was better, sometimes not. couldn't be arsed to test for confidence, as that seemed to me to be squeezing bollox out of a bag of nuts at that point.

I still think the whole approach outlined in the OP , whilst theoretically valid, is rather too contrived, though i don't discount it's benefits when they are gained indirectly.

My woodland/forest canopy based analogy to this can be further expanded that i will type up in full if people would like to hear it. I think it can intuitively expose the danger of relying on this approach.
Since you are all quite enlightened anyway it seems , i'll leave it for now tho-
maybe if the thread gets to another page.....

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Tue Feb 22, 2011 10:40 am

actually screw my bullshit analogy, Jez Liberty's EXCELLENT post in this thread says it all just fine really.

We know that over the long haul correlations of even combined equity curves tend to be about as stationary as speedy gonzalez on speed, bouncing around inside the large hadron collider.
As such, if we drink too much kool-aid from this particular trough, and get cocky enough to base our leverage on derived pain statistics of these carefully selected equity curves, then we are likely to choke when the percieved benefit of such alchemy displays large sensitivity to changes in "rho", and our predicted pain statistic goes out the window.

Having said that, I don't think there are many on this forum who disagree.
Enjoy the indirect benefits of diversification when they occur(less pain on a day to day basis, better compounding etc), but don't chase them too much(base leverage choices on them too heavily). seems like a good rule to live by.

Good discussion though, credit to the way the OP and others have laid out all the underlying reasoning in such clear homemade formats. Above all I think it has shown me that I need to up my game in the data manipulation and presentation stakes!

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Feb 23, 2011 7:26 pm

I hope folks don't misinterpret Jez's chart. He started with the assumption that all N of the equity curves had the same correlation to each other -- which simplifies the math tremendously. But please don't interpret his bottom most red line as some kind of a "Sound Barrier" which is impossible to breach. It's merely the limitation of this plot plus this assumption, nothing more.

To illustrate that it is indeed possible to do better, I've attached a set of four equity curves. They all have the same volatility but they don't all have the same correlation to one another; some correlations are positive (but small), while other correlations are negative (and large). When you blend these four equity curves together (with 25% allocation to each), the blend has a volatility of 0.13. This is marked with a black dot on the chart. It's way beyond the bottom most red line (the red line crosses N=4 at volatility=0.50). There is no sound barrier.

In fact if you think about it for a few minutes, you'll be able to invent a procedure for generating N different equity curves that blend together to give volatility (reduction) V, for any N and V you care to specify. This demonstrates that there is no "Sound Barrier" - you've got an algorithm to generate N equity curves that blend together giving any V you choose. Even zero. (Eventhorizon's post may get you started).

In the spreadsheet, the four equity curves are called "Lucy", "Ricky", "Ethel", and "Fred". :P
Attachments
Ecurves_Four.xls
Four equity curves that blend nicely
(58.5 KiB) Downloaded 256 times
Mod_Four.png
Black dot at N=4, V=0.13 shows the volatility of this blend of 4 equity curves
Mod_Four.png (136.42 KiB) Viewed 7922 times

Jez Liberty
Roundtable Knight
Roundtable Knight
Posts: 123
Joined: Tue Nov 03, 2009 8:49 am
Location: London
Contact:

Post by Jez Liberty » Wed Feb 23, 2011 9:36 pm

sluggo wrote:I hope folks don't misinterpret Jez's chart. He started with the assumption that all N of the equity curves had the same correlation to each other -- which simplifies the math tremendously.
Very true. I think I would still be working out the maths if I had not made this simplifying assumption!
sluggo wrote:But please don't interpret his bottom most red line as some kind of a "Sound Barrier" which is impossible to breach. It's merely the limitation of this plot plus this assumption, nothing more.
The chart/assumption definitely has some limitations, an important one of them being that only positive correlations were plotted on the chart (the reason I did not plot negative correlations is because the formula errors out after N = a small number, depending on the value of rho).

As I mentioned at the end of the post, by adding negatively correlated equity curves, you could do much better (than the red line). Heck, even theoretically obtain a zero volatility!
Jez Liberty wrote: We could even actually calculate, for a given number of equity curves N, the required correlation between all equity curves (assuming equal volatility and correlation) to reach a blended volatility of 0: rho = -1 / (N-1)

A few examples:
N=2 -> rho = -1
N=3 -> rho = -0.5
N=5 -> rho = -0.25
N=11 -> rho = -0.1
N=26 -> rho = -0.04
N=51 -> rho = -0.02
N=101 -> rho = -0.01
...
The chart expands on bobsyd and sluggo's "Vol vs N" charts (which assume rho = 0 - in effect plotting the same red line in the chart). I wanted to check: "What if can not produce multiple uncorrelated equity curves? Would I benefit in a similar, albeit less favorable way, by throwing up hundreds of strongly correlated equity curves?"
The chart seems to point to the answer being a "No: focus on finding low/negatively correlated equity curves, rather than throwing hundreds of near-identical ones", which basically points back to the point in your initial post, sluggo.

As the negative rho example and sluggo's update clearly show, picking only a few equity curves can result in a more dramatic vol reduction.

"It's quality, not quantity!"

I have actually updated the chart to plot the negative rhos as well. Some curves go slightly "off-tangent" towards the bottom, as the errors start creeping in just before 0, but it definitely shows that there is no "sound barrier".
I have also plotted sluggo's "special blend" of Lucy, Ricky, Ethel and Fred (black dot), which is close to rho=-0.3, if all curves had the same correlation. They do not, but their average correlation is not far off at -0.24, if my calcs are correct. Maybe the chart/formula is not a bad estimator even in cases where curves do not display all equal correlations (the estimation error would depend on the standard deviation of the correlations I presume)
Attachments
Chart-Vol-vs-N-rho-2.png
Chart-Vol-vs-N-rho-2.png (176.18 KiB) Viewed 7895 times

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Thu Feb 24, 2011 3:57 am

rabidric wrote: As such, if we drink too much kool-aid from this particular trough, and get cocky enough to base our leverage on derived pain statistics of these carefully selected equity curves....................
Quite. Much as I applaud and appreciate the research in the above posts ....................I suspect that real life long term success (in investment or otherwise) probably also requires a healthy dose of scepticism and involves a large measure of chance and an acceptance that the most carefully laid plans are at best only a rough sketch drawn with a blunt stick in ever moving sands.

Or am I just feeling more than usually puzzled by a world largely governed by tribal thugs where "truth" and "reason" seem so often curiously lacking and which is emphatically not inherited by the meek.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Thu Feb 24, 2011 5:17 am

"Determinism" is the name of a broader philosophical view that conjectures that every type of event, including human cognition (behaviour, decision, and action) is causally determined by previous events. In philosophical arguments, the concept of determinism in the domain of human action is often contrasted with free will. The argument called indeterminism(otherwise "nondeterminism") negates causality as a factor and opposes the deterministic argument. Determinists believe the universe is fully governed by causal laws resulting in only one possible state at any point in time.
As a 32-year-old assistant professor of mathematics at Helicon University, Hari Seldon visits Trantor from his native Helicon to attend the Decennial Mathematics Convention. He presents a paper which indicates that one could theoretically predict the Galactic Empire's future. At first, Seldon has no idea how this could be done in practice, and he is fairly confident that no one could actually fulfill the possibility. Shortly after his presentation, he becomes a lightning rod for political forces who want to use psychohistory for their own purposes. During his flight to escape the various political factions, he discovers how psychohistory can be made a practical science.
Once I have found the Philosopher’s Stone and achieved Gnosis I will be able to guide the assembled group further towards enlightenment and truth.

I may be some time, so please don't hold your collective breath.

:lol:

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Thu Feb 24, 2011 9:00 am

loving that hari seldon story AFJG. lmao!
also loving jez's second post as much as his first- they highlight the relationships brilliantly and provide good meat for further discourse:

if anything the new chart shows better the perils I am arguing- the potential adverse dV/dRho gets nutty for small portfolios with nice anticorrelations that then mutate in the future.
The "quality" , if it changes for the worse, can drop the "benefits" dramatically.

"quantity", while being a quality all of it's own(stalin?), is perhaps less susceptible , but even so, still suffers the dV/dRho problem.

Which brings us to the nub of it all: the greater the benefits of "s[m]oothing the pain" that derive from this methodology, the greater the undoing when it goes wrong. The going wrong part will depend on the stationarity of the correlations etc YMMV, But even if it only goes wrong 0.1% of the time, that could still be once every 3 years.

In the end, it is just another optimization problem to be abused:
the smaller the number of series being optimized, the lower the stability, the larger the number being optimized, the more mediocre the performance*, and no matter what you do, shit will always happen.
Leverage will decide how much you get hurt.

P.S. I don't mean to be a Jezebel proclaiming it is all useless. In reality you might be surprised at the choices i have made-I actually do prefer quality over quantity personally, and enjoy taking a swing for the fences. I just think provided it doesn't badly cripple your ability to play the game, one can never be too long on scepticism!

* see new "Subset Trading" thread.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Thu Feb 24, 2011 9:53 am

rabidric wrote:I don't mean to be a Jezebel proclaiming it is all useless.
Agreed. The work is illuminating and instructive. Like all properly conducted and intelligent back-testing.

You do the work, and then hope that what your have observed as having been effective in the past, is a workable method for at least some period of time going forward.

It is all you can do.

Until Hari perfects psychohistory and/or the determinists are proved right and come up with something similar.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Re: Blending noncorrelated (or anti-correlated) equity curve

Post by sluggo » Thu Feb 24, 2011 10:40 am

sluggo wrote:Since volatility is the denominator of the Sharpe Ratio, as rho(AB) approaches -1.0, the denominator sigma(A+B) approaches zero and so the Sharpe Ratio approaches +infinity. However, after the manner of the Options Greeks, I suggest that you calculate the sensitivity d(Sharpe) / d(Correlation). You will find that the sensitivity goes to infinity as well. You might want to ponder that for a moment, in the context of In-Sample vs. Out-of-Sample correlations.

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Thu Feb 24, 2011 10:51 am

yes. you wrote that.
yes. i agreed after reading it...
yes. i had to fill out many superfluous lines of text to express that! :lol:

it was worth highlighting 8) for one small paragraph in an entire thread of well written content, it was by far the most important thing of all. It should probably be bolded in red and supersized and capitalized. but I know how you love to just leave nuggets of wisdom in plain sight but easily passed by!

LeviF
Roundtable Knight
Roundtable Knight
Posts: 1428
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF » Fri Feb 25, 2011 12:58 am

I just came across this blog post that made me think back to this discussion.

http://www.michaelcovel.com/2011/01/09/ ... thout-you/

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Fri Feb 25, 2011 6:11 am

I did find this comment by Eckhardt gelled with my own thoughts concerning trading and investment:
Aristotle’s Ship of Theseus gradually had every one of its parts replaced. Was it still the Ship of Theseus? Over the decades every part of our systems has been modified. It’s our scientific approach that persists and supports the whole structure.
Thanks to TK for pointing out the interview. http://www.futuresmag.com/Issues/2011/M ... spx?page=3

This thread emphasizes with great clarity the benefits of diversification amongst lowly correlated instruments in a portfolio. My rather badly expressed comments are merely designed to say: "agreed but the relationships demonstrated between various instruments in the past may not hold in the future". Obvious, trite perhaps.

Comrade Lenin's comments on trading systems are just as applicable to a fund of funds. ( He does look just like my pal Vlad Ilyich, does he not?).

A colleague and I advise (inter alia) a fund of CTA funds. Our suggested constituents were naturally chosen partly on past performance figures and back testing (as well as on qualitative and more subjective considerations). Will the suggested mix still be the same in 10 years time? That seems unlikely. People will die, funds will go out of business, styles will drift.

None of this invalidates the research on this thread. It merely means that it will be an on-going and never ending process. There is no finality, only change and adaptation.

In back testing one can devise smooth, low volatility equity curves pointing upwards and to the right. It will be harder to match those results going forward.
“The past is a foreign country, they do things differently thereâ€

MarkS
Roundtable Fellow
Roundtable Fellow
Posts: 70
Joined: Wed Dec 15, 2010 12:10 pm
Location: Chicago, IL

Post by MarkS » Fri Feb 25, 2011 7:44 pm

I am a fan of research like this, and having run smaller variations of this for a grad school project, I can appreciate the computing power necessary to do it! Thank you sluggo.

As is mentioned by drm7, rabidric, and AFJ Garner, it is also good to take such research and keep an eye towards the market reality, which as one of my mentors has told me is that "the market will always move in the direction that will hurt the most amount of people" -- another way of stating the contrarian approach to trading.

Yes, the biggest flaw is the changing correlations, which can switch on a dime for no apparent reason. Take oil and the S&P. At one point, the two were very positively correlated: economy is strong and growing economies drive more usage of oil: oil prices rise hand in hand with the stock market. Then like a light switch, oil surpasses some point where market sentiment suddenly sees that it is a hazard to future economic growth if it goes any higher. Now oil's correlation swings to the negative side: if oil is up, the stock market has pressure to the downside. Move forward past the recession, and once again, oil and equities are positively correlated: if stocks are picking up, that means the recession must be drawing to a close, so oil demand will pick up. Enter the Mideast turmoil, and oil passes that magic level that once again flips correlation the other way.

Similar to what can happen to spread traders. Ask calendar-spread traders in the wheat pit from 2-3 years ago... You won't find them in the pits, because after 20+ years of trading the spread the same way, things changed. From the story, as I was told from a floor trader that stood in the pit, three of these gentlemen lost over a cumulative $50M+ and all "blew out" as they kept putting on larger and larger positions while they waited for a mean reversion that did not come. A small Long-Term Capital Management situation, if you will.

Kenyes: "The markets can remain irrational longer than you can remain solvent".

I also think a lot of quant firms found out in 2008 that even long-term correlations are not a guarantee of continued correlation. (CSI has a great correlation database with access for only $99, allowing you to test one asset versus another, or one against all, from 2 to 40 years. I find it very interesting and use it quite a bit.)

BUT.... until someone comes up with a better approach, then Markowitz optimization is still the way to approach the market and the way most, if not all, professionals still invest. I think the adjustment has been that more people now understand that correlations are not stationary, and that in times of crisis, they tend to all move towards 1.0; therefore as stated above by several members, do not base your maximum risk / leverage figures solely on the backtest and be cognizant of the fat tail risk. For example, if by investing in real estate and equities and corporate bonds together lowers your volatility by 50% over a single market, you probably may want to increase your exposure by somewhat less than the 100% that is implied...

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Sat Feb 26, 2011 3:03 pm

I hope you realize that we are measuring the correlation of Equity Curves

Equity Curves are the outputs of multimarket trading systems

Price data (such as oil prices and S&P prices) are the inputs to multimarket trading systems

Because modern futures and forex trading systems do not buy-and-hold, and because they sometimes take short positions, and because they use leverage, these systems produce outputs which are not equal to their inputs. Therefore the correlation of their outputs (equity curves) is very different than the correlation of their inputs (price data). This is a second, underappreciated, free lunch on Wall Street.

System 1 trading portfolio P produces equity curve 1. System 2 trading the same portfolio P produces equity curve 2. The correlation between (equity curve 1 and equity curve 2) is VERY different than the correlation between (prices of the various instruments in portfolio P).

Repeat after me: A trading system is a function that transforms prices into an Equity Curve

(In my experience, the really good systems are nonlinear and non-differentiable functions (i.e. their derivatives are discontinuous)).
Last edited by sluggo on Sat Feb 26, 2011 3:59 pm, edited 1 time in total.

Post Reply