Musings on (worthless) Past Performance

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Musings on (worthless) Past Performance

Post by DPH » Mon Jan 30, 2012 12:30 am

I was working on a blog post today and whipped together a few tables and charts I thought might be of interest to some.

The general theme was to prove the worthlessness of most past performance statistics for CTAs. Primary target was the Sharpe Ratio, and frankly this tends to cover them all anyway.. Sharpe Ratio Covers Them All

I used a VERY simple strategy of buying the top 5 CTAs in 2007 as measured by their Sharpe Ratio.

I then compared that to my results five years later at the end of 2011 to see how I did. (below)..

[I am not able to embed the interactive tables and graphs here (I tried), you can see them, and download the data on my blog at:]

http://managedfutures.ws/2012/01/29/is_ ... worthless/

In a nutshell, it failed miserably. Performance fell apart, the out of sample Sharpe Ratios fell an average of 70%. Returns fell by roughly 60% and maximum drawdowns increased by roughly 30%.

I had hoped that maybe at least their peer group rankings remained somewhat constant (I.E. everybody fell apart equally) but nope, their peer group rankings went out the window too!

Conclusion........ Sharpe Ratio (and related measures) are reporters of yesterdays news only. Their value as tools to help predict or forecast future results is near zero! :shock:

stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Re: Musings on (worthless) Past Performance

Post by stopsareforwimps » Mon Jan 30, 2012 2:02 am

DPH wrote:The general theme was to prove the worthlessness of most past performance statistics for CTAs ... the out of sample Sharpe Ratios fell an average of 70%. Returns fell by roughly 60% and maximum drawdowns increased by roughly 30%.
Would it be possible to compute a survivorship-bias-corrected prediction of future performance eg using Bayes's theorem, reverse engineering the survivorship bias from the results you report?

I have a second question: if the past performance of your back tested methodology for picking CTAs is good, does this mean it will continue to be good?

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Mon Jan 30, 2012 2:41 am

There is no question that survivorship bias is a MAJOR issue. As far as I am concerned it is the industry’s “dirty little secretâ€

rajivm
Roundtable Knight
Roundtable Knight
Posts: 129
Joined: Fri Jul 25, 2003 12:50 am
Location: New Delhi, India

Post by rajivm » Mon Jan 30, 2012 3:47 am

I don't know about any ratios..But Past performace normally is an indication of future good performance..Look at Soros, Jim Roger or other long time good performers..They STILL are doing well..
If Sluggo has written insightful articles in past .. I would bet he would write more in future so I better off reading his posts :) ..
I think in life in general Past performance does predict to a large extent what can happen in future..

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Mon Jan 30, 2012 4:29 am

past performance tells you how well they played the most recent iteration(s) of The Game.

The Game can always change it's rules faster than you can adjust them with a backward looking method.

But a Game can maintain it's current rules for longer than you can bear if you try to prempt a rule change too often.

as ever, there are games within games. within games. wrapped up in a game.

At the end of the day you either gotta be very selective about where and how you get involved if you want to do better than the average. Or, you just keep moderate stake in the game through thick and thin, and weather the long droughts, so you are still in play when the good times roll and everybody else in washed out on the sidelines. Option 2 is intellectually/mechanically easier, but psychologically harder.

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Mon Jan 30, 2012 5:04 am

Raj,

I am afraid you’re “in the trapâ€

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Mon Jan 30, 2012 6:27 am

i think , fwiw, its best just to understand that there will probably never be a CTA or fund that has an MAR of > 1 if it lasts 20 years and therefore allocating assets on the basis of teh last few years performance where a new CTA has great stats or an MAR of 3 is a ridiculo9us exercise
I wrote before about how 4 months after I launched my MAR was about 20
Well now its about 0.01 after 15 months
Thers nothing wrong with that but investors should work out what teh long term average is for a strategy and probably sell the MARS of 2 and 3 and buy the MARs of much much lower
Some things never change and I believe investors however will keep making teh same mistakes
Then you get Rentech of course - but that needs some serious thinking about

As for the sharpe ratio - its just nionsense and always has been
for a start nobody can agree on or even define the RFROR

Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Post by Moto moto » Mon Jan 30, 2012 8:18 am

DPH - you reminded me of a time that GS showed us a system in the late 1990s that was going to help the risk management team adequately minimise/"eliminate" risk to our long volatility equity option trading books from 8 different traders covering approximately 50 stocks
(paraphasing)
"we took your books, combined them, did in depth risk analysis and through our great system worked out that the best hedge for your combined portfolio is to be short 1.2 million AAA stock"

needless to say the risk guys were impressed and bought/leased the system at a small fortune because the big guys used it

.....even after we told them that AAA stock only traded less than 500,000 shares a day, was a single mine gold stock (that had rumours of predators and ended up getting taken over) and that as long volatility option players we actually wanted dislocated markets and high volatility to help hedge their other internal firm portfolios.....

in the end they scrapped it and went for that other great backward looking - but in its defense - best guess - measure of risk Var.

I guess in the end all you can do is understand that these are not perfect measures, know their weaknesses and work out when to and when not to apply them.....or just talk them up and sell, sell, sell. (note; not short, short, short)

Chuck B
Roundtable Knight
Roundtable Knight
Posts: 469
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B » Mon Jan 30, 2012 8:25 am

Past performance can be an outstanding predictor of future success! That success is however in the marketing of said fund/adviser to the latest performance chasing idiots. Suck in the new money in the largest possible quantities in the shortest amount of time possible.

Of course we are just entering a period of mutual fund advertising of 3yr returns given the 1st quarter of 2009 market action. Marketing managers always look forward to these periods in time as your marketing window opens wide to suck in the unwary.

rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric » Mon Jan 30, 2012 9:52 am

chuck B is near the mark imho.

regularly re-calibrate yourself to "the idiots"(a changing population, as they die out to be replaced by new idiots who think they have a better take on things) , and then fade their game.... ***AFTER THEY HAVE TIPPED THEIR HAND***

I don't know if you can see which bit i have tried to highlight there :)

It is amazing, eerie and fun to watch the orderflow then turn your way and shovel money into your pockets.

Playing the Game means , playing the players.

whether the game is trading price, investing in funds, or a game of "spoof" down the pub.

LeviF
Roundtable Knight
Roundtable Knight
Posts: 1428
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF » Mon Jan 30, 2012 10:20 am

DPH wrote:Here is an exercise I have for you. I will send you all the historical data of all the CTAs including those who blew out and are no longer in today’s survivorship bias results. I will remove their names so you have no way to peak into the future to know how they would have performed. THEN tell me who the 5 or so programs you would invest in are. Then we will hold those managers for the next 5, 10 or 20 years and see how you did.
I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.

babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader » Mon Jan 30, 2012 10:32 am

I recently read this blog post which has a nice graphical summary of this problem. Given the possibility of this sort of situation actually occurring, why would anyone trust any historical performance record as a good indication of future returns, whatever metric one chooses? Not to mention that you might be entrusting your money to the next MF Global, LTCM... etc.

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Mon Jan 30, 2012 11:49 am

chuck b / Rabidric - yes I agree entirely and its name is
WINTON

marriot
Roundtable Knight
Roundtable Knight
Posts: 350
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot » Mon Jan 30, 2012 12:46 pm

I have yet to find a Cta that started with low gains and ended last years in a great way.
Is depressing the evidence that even after ten years of trading you can find sharply drops.
The only consistent point for survivors seem to be that first years are the best.
It will be nice a test like:" if the first year Ror is at least 100 %, jump in and exit after one of to years"
But this has to be done on all the universe, archived program too.

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Mon Jan 30, 2012 1:27 pm

LeviF wrote:
I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.
My question is why only LTTF and not the others? Is it possible you are using hindsight bias because you know today, with the benefit of hindsight that LTTF did better than those other approaches?

Did you know as much about LTTF 10 or 20 years ago as you do today? If not, then what makes you think you would have made this choice back then?

trackstar
Roundtable Fellow
Roundtable Fellow
Posts: 76
Joined: Sun Mar 09, 2008 11:03 pm
Location: Houston Texas

Post by trackstar » Mon Jan 30, 2012 1:40 pm

LeviF wrote:
DPH wrote:Here is an exercise I have for you. I will send you all the historical data of all the CTAs including those who blew out and are no longer in today’s survivorship bias results. I will remove their names so you have no way to peak into the future to know how they would have performed. THEN tell me who the 5 or so programs you would invest in are. Then we will hold those managers for the next 5, 10 or 20 years and see how you did.
I will play this game if you include the description of the program so I can throw out the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, arb, basically anything but LTTF.
my feelings exactly 8)

LeviF
Roundtable Knight
Roundtable Knight
Posts: 1428
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF » Mon Jan 30, 2012 1:41 pm

I have no idea know how the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, and arb guys did over that time period. LTTF is the only method I believe has a chance over the long run. Influenced by hindsight? Probably, but what is the alternative?

rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Post by rhc » Mon Jan 30, 2012 8:22 pm

Dean thanks for sharing your thoughts in your blog (as linked above)

I’m wondering if I am reading the tables and graphs correctly since the 2007-2011 values differ from table to graph

For Emory Partners;

The table ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = -0.02
The graph ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = 0.15

For Shinnecock;

The table ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.33
The graph ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.47

For Blenheim;

The table ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.32
The graph ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.69

rajivm
Roundtable Knight
Roundtable Knight
Posts: 129
Joined: Fri Jul 25, 2003 12:50 am
Location: New Delhi, India

Post by rajivm » Mon Jan 30, 2012 9:39 pm

LeviF wrote:I have no idea know how the option traders, cattle traders, S&P traders, stock pickers, short term, counter trend, and arb guys did over that time period. LTTF is the only method I believe has a chance over the long run. Influenced by hindsight? Probably, but what is the alternative?
Frankly LeviF,
I feel LTTF wasted a large part of my life :D in my early search for systems to trade...
My thoughts are unless someone has 20 million plus...How can one possibly even think of making a living out of Trading as LTTF...

Returns are low ....and drawdowns are Long streching more than 1 year often..

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Tue Jan 31, 2012 1:22 am

rhc wrote:Dean thanks for sharing your thoughts in your blog (as linked above)

I’m wondering if I am reading the tables and graphs correctly since the 2007-2011 values differ from table to graph

For Emory Partners;

The table ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = -0.02
The graph ‘says’
2002-2006 sharpe ratio = 1.48 and the 2007-2011 sharpe ratio = 0.15

For Shinnecock;

The table ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.33
The graph ‘says’
2002-2006 sharpe ratio = 1.11 and the 2007-2011 sharpe ratio = 0.47

For Blenheim;

The table ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.32
The graph ‘says’
2002-2006 sharpe ratio = 1.38 and the 2007-2011 sharpe ratio = 0.69
Whoops! Thank you for pointing that out! I had about 20 spreadsheets open on my desk and I accidentally used the one thru 2010 instead of 2011. Fortunately BarclayHedge clearly notates your data on every export. I have included some screen shots because it also shows the basic filters I employed in the data. See below:

2002-2006 data:

Image


2007-2010 data:

Image

2007-2011 data:

Image

The correlation results using the data through 2011 (instead of 2010) was even more confirming. The correlation coefficient on the Sharpe Ratios went from roughly (-.07) to (-.14).
Last edited by DPH on Tue Jan 31, 2012 4:56 pm, edited 1 time in total.

Post Reply