Musings on (worthless) Past Performance

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Aaron01
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Thu Jun 02, 2011 1:21 pm

Post by Aaron01 » Tue Feb 14, 2012 4:09 pm

marriot wrote:Well, Ray is trading stocks with some kind of fundamental filter.
This sound very different from what we usually discuss here.
Then he said: "How can you work 3000 trades every year?
I try to keep down numbers of trades at a max of 500 per year."
So, he his using Stop and Limist orders.
The simulation he has show us is not different from the one i have put on here.
This go back some years, but this does'nt not mean that i have trade it from the starting point.
Overall i have to say that his simulation is stimulant.
His simulation shows even less than 500 trades per year.

I have to wonder if he's even using any constraints or filters for liquidity. Researching his older posts shows he states this is a stock market system, but the stock market has the same problems for liquidity that we futures traders have. Though they have their own unique problems of a higher preponderance of HFTs. His older posts also seem to imply he's not using any sort of size constraints
rajivm wrote:But I feel the problem will arise when postions sizes to be taken becomes large.
Unrealistic sized positions in backtesting can lead one to unrealistic expectations of gains.

I question whether or not the signals are simply the product of the systems trader's boogeyman, curve fitting. Especially when he states the following
rajivm wrote:And never had a negative year..
Though, that is simply a statement regarding his backtesting, as he stated in March

[quote="rajivm]I trade it...But I have traded it only for 2-3 months so far. [/quote]

So he can't have had enough real world trading time to make such a statement. Even at this date, such a statement would be invalid.

You are correct in that his simulation is stimulating and it could probably lead to fruitful discussions, though without more information (some of the other charts generated by TB), there's not much to discuss besides an equity chart and some vagaries.

CyTrend
Roundtable Fellow
Roundtable Fellow
Posts: 57
Joined: Mon Oct 25, 2010 5:28 pm

Post by CyTrend » Wed Feb 15, 2012 5:48 am

I might be misunderstanding the original post but in my mind it doesn't really say much more than "Beware of Regression to the Mean".
I think it is quite obvious that if you peak your managers solely as the top sharpe ratios it would be surprising if you actually did not see those sharpe ratios decline. It as is you believe in the Sports Illustrated Cover jinx.

Let's put things in perspective here. TMS Capital Mgmt returned 19% Cagr in the next five years. If i had a measure that told me to invest in them during that period i would consider it hubris to say that my measure failed me miserably.

Don't get me wrong i am not a big fun of Sharpe Ratios but i don;t think this is the way to discredit it.

stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Post by stopsareforwimps » Wed Feb 15, 2012 4:24 pm

AFJ Garner wrote:
rajivm wrote: Anyway, This is a system I use...And never had a negative year..so I am happy...

Let us be entirely clear: you posted above a simulation created using Trading Blox for a system back test which commenced in 2003 and since that date has recorded a theoretical CAGR of almost 300% with a theoretical maximum drawdown of under just 25%. What you did not do was to post your actual results in trading this actual system from that actual date.

So, since 2003 have you continuously traded this identical system and have you achieved an CAGR of 300% with a maximum peak to valley drawdown of just under 25%? If you have not , then with respect your reference to the Country of the Blind is as irrelevant as your back test is misleading.

If by contrast you have, in real trading, achieved this outstanding performance using a significant sum of real money then I congratulate you in all sincerity and wish with equal sincerity that I had your very considerable trading skills. You are my dear sir, a true Wizard.
I would be very impressed if rajivm would post brokerage statements validating the claimed performance. Until I see that, Bayes's theorem would suggest it is more like this (http://en.wikipedia.org/wiki/Energy_Catalyzer) than this (http://www.atrader.com/manager-profile/ ... ket-wizard).

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Thu Feb 16, 2012 1:52 pm

CyTrend wrote: Don't get me wrong i am not a big fun of Sharpe Ratios but i don;t think this is the way to discredit it.
Hey Cytrend,

I agree that what I presented is not exhaustive and conclusive "proof" discrediting the Sharpe Ratio. It was not meant to be (notice I titled it "Musings" and not "The Definitive Report On" ). The idea was a more generalized observation that much of the retail Managed Futures / Hedge Fund industry is foolishly over reliant on these past performance metrics when giving recommendations or allocations…There seems to be an underlying assumption that past performance (from which Sharpe is computed) is indicative, or highly correlated to future performance. My testing has shown me that this isn’t the case. Jack Schwager drew similar, (but not identical) conclusions years earlier in a similar study. (He found more "relative" correlation than I did). Conventional wisdom says that a higher Sharpe Ratio is better (and from a historical perspective I agree), but my studies show me that investing in funds with unusually high Sharpe Ratios is not the best idea. I see little correlation that a fund with a high Sharpe Ratio today will maintain that for any considerable period. So, if you are investing this way you had better have some outstanding timing skills.

For a quick confirmation just look at the option writers or Long Term Capital Management who had stratospheric Sharpe Ratios (or whatever ratios) just before their implosions. Or, if you want more detailed confirmation of this then I am suggesting you look at the kinds of data I am presenting.

I do think there are potential solutions to this……..I’m not implying a crystal ball, but how about some unconventional and robust predictive analysis techniques that proved successful across a broad range of funds with out of sample data? (where care was taken not to have inadvertently “peekedâ€

mojojojo
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Mon Oct 31, 2005 2:07 pm

Post by mojojojo » Thu Feb 16, 2012 3:08 pm

DPH wrote:
CyTrend wrote: Don't get me wrong i am not a big fun of Sharpe Ratios but i don;t think this is the way to discredit it.
Hey Cytrend,

I agree that what I presented is not exhaustive and conclusive "proof" discrediting the Sharpe Ratio. It was not meant to be (notice I titled it "Musings" and not "The Definitive Report On" ). The idea was a more generalized observation that much of the retail Managed Futures / Hedge Fund industry is foolishly over reliant on these past performance metrics when giving recommendations or allocations…There seems to be an underlying assumption that past performance (from which Sharpe is computed) is indicative, or highly correlated to future performance. My testing has shown me that this isn’t the case. Jack Schwager drew similar, (but not identical) conclusions years earlier in a similar study. (He found more "relative" correlation than I did). Conventional wisdom says that a higher Sharpe Ratio is better (and from a historical perspective I agree), but my studies show me that investing in funds with unusually high Sharpe Ratios is not the best idea. I see little correlation that a fund with a high Sharpe Ratio today will maintain that for any considerable period. So, if you are investing this way you had better have some outstanding timing skills.

For a quick confirmation just look at the option writers or Long Term Capital Management who had stratospheric Sharpe Ratios (or whatever ratios) just before their implosions. Or, if you want more detailed confirmation of this then I am suggesting you look at the kinds of data I am presenting.
That's the issue (in bold). I'm sure you can pull any stat out that is based on historical data and show similar results.

No stat should be taken in isolation. They need to be used as a part of a larger picture. But it's much easier for people to say .. He has a high sharpe ratio, give him money.

Chuck B
Roundtable Knight
Roundtable Knight
Posts: 469
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B » Thu Feb 16, 2012 3:23 pm

mojojojo wrote:
DPH wrote:
CyTrend wrote: Don't get me wrong i am not a big fun of Sharpe Ratios but i don;t think this is the way to discredit it.
Hey Cytrend,

I agree that what I presented is not exhaustive and conclusive "proof" discrediting the Sharpe Ratio. It was not meant to be (notice I titled it "Musings" and not "The Definitive Report On" ). The idea was a more generalized observation that much of the retail Managed Futures / Hedge Fund industry is foolishly over reliant on these past performance metrics when giving recommendations or allocations…There seems to be an underlying assumption that past performance (from which Sharpe is computed) is indicative, or highly correlated to future performance. My testing has shown me that this isn’t the case. Jack Schwager drew similar, (but not identical) conclusions years earlier in a similar study. (He found more "relative" correlation than I did). Conventional wisdom says that a higher Sharpe Ratio is better (and from a historical perspective I agree), but my studies show me that investing in funds with unusually high Sharpe Ratios is not the best idea. I see little correlation that a fund with a high Sharpe Ratio today will maintain that for any considerable period. So, if you are investing this way you had better have some outstanding timing skills.

For a quick confirmation just look at the option writers or Long Term Capital Management who had stratospheric Sharpe Ratios (or whatever ratios) just before their implosions. Or, if you want more detailed confirmation of this then I am suggesting you look at the kinds of data I am presenting.
That's the issue (in bold). I'm sure you can pull any stat out that is based on historical data and show similar results.

No stat should be taken in isolation. They need to be used as a part of a larger picture. But it's much easier for people to say .. He has a high sharpe ratio, give him money.
Amazingly, I can recall having these same discussions/arguments back in ~1994 during a seminar. Eighteen years later, one can only guess at the humongous sums that have been lost chasing stuff like Sharpe ratios or "managers" or "programs."

mojojojo
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Mon Oct 31, 2005 2:07 pm

Post by mojojojo » Fri Feb 17, 2012 2:05 pm

Chuck B wrote:
mojojojo wrote:
DPH wrote: Hey Cytrend,

I agree that what I presented is not exhaustive and conclusive "proof" discrediting the Sharpe Ratio. It was not meant to be (notice I titled it "Musings" and not "The Definitive Report On" ). The idea was a more generalized observation that much of the retail Managed Futures / Hedge Fund industry is foolishly over reliant on these past performance metrics when giving recommendations or allocations…There seems to be an underlying assumption that past performance (from which Sharpe is computed) is indicative, or highly correlated to future performance. My testing has shown me that this isn’t the case. Jack Schwager drew similar, (but not identical) conclusions years earlier in a similar study. (He found more "relative" correlation than I did). Conventional wisdom says that a higher Sharpe Ratio is better (and from a historical perspective I agree), but my studies show me that investing in funds with unusually high Sharpe Ratios is not the best idea. I see little correlation that a fund with a high Sharpe Ratio today will maintain that for any considerable period. So, if you are investing this way you had better have some outstanding timing skills.

For a quick confirmation just look at the option writers or Long Term Capital Management who had stratospheric Sharpe Ratios (or whatever ratios) just before their implosions. Or, if you want more detailed confirmation of this then I am suggesting you look at the kinds of data I am presenting.
That's the issue (in bold). I'm sure you can pull any stat out that is based on historical data and show similar results.

No stat should be taken in isolation. They need to be used as a part of a larger picture. But it's much easier for people to say .. He has a high sharpe ratio, give him money.
Amazingly, I can recall having these same discussions/arguments back in ~1994 during a seminar. Eighteen years later, one can only guess at the humongous sums that have been lost chasing stuff like Sharpe ratios or "managers" or "programs."
I think you will pretty much always have this situation. Sales people are always going to try and use something that is simple to explain to present to potential clients. At least as long as allocation decisions are made by discretion.

IMHO humans will always try to simplify a complex issue and doing so will almost always lead into this trap.

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Fri Feb 17, 2012 11:56 pm

I agree, BUT with a strong caveat….

Sometimes the bullshitters (salesman masquerading as experts) will use complexity as a tool in their deception process....

For example, I have watched one fairly visible Managed Futures advisory company produce a series of long, “floweryâ€

Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Post by Moto moto » Sat Feb 18, 2012 4:32 am

interesting DPH.
I thought it was about the three Ps - people processes performance and the combination of those.....
however - all three can be so so so easily wallpaper over-ed.

I know of two people at present who seem to tick all the boxes, however I would not invest with them with my enemies money as i know their motivation is to rip off clients - they are asset accumulators rather than asset managers. (this is as opposed to at least asset accumulators who have your best interest at heart)
They have fooled a lot of people - mostly those who did no due dilligence, but even those who do can be biased for other reasons.

A simple measure - completely generalist of course - audit (reveals all the charges), performance only in good times - if they dont outperform OR at least do exactly as they say during those times then you may as well not bother. ie; you will never make money when they cant make money when everything lines up for them.....regardless.
Plus you know they will talk up and provide as much information - too much often- as possible to you when they are trying to highlight their best times, and yet the numbers show this to be wrong.

Of course you need to then understand your own biases and not get sucked into the risks when times are bad. :)

Robwynge
Senior Member
Senior Member
Posts: 42
Joined: Wed May 25, 2011 10:33 pm

Post by Robwynge » Sat Feb 18, 2012 5:04 pm

Chuck B wrote: Amazingly, I can recall having these same discussions/arguments back in ~1994 during a seminar. Eighteen years later, one can only guess at the humongous sums that have been lost chasing stuff like Sharpe ratios or "managers" or "programs."
Chuck,

so what would be the right thing to chase in making such selections?

Chuck B
Roundtable Knight
Roundtable Knight
Posts: 469
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B » Sat Feb 18, 2012 7:30 pm

Robwynge wrote:
Chuck B wrote: Amazingly, I can recall having these same discussions/arguments back in ~1994 during a seminar. Eighteen years later, one can only guess at the humongous sums that have been lost chasing stuff like Sharpe ratios or "managers" or "programs."
Chuck,

so what would be the right thing to chase in making such selections?
I would imagine it would be different things depending on one's objectives, but the overall thing I would think would be important is no matter how great some program has performed, etc, assume that over the long term AnnROR/MaxDailyDD would be lucky to not fall below 0.4-0.5.

The flip side are programs/managers who have a long and outstanding track record but are in fact trading a negative expectation system over a long measuring period that may exceed what one considers a "long time". There is an unbelievably strong bias in humans to "believe" a real-time record is profoundly indicative of how great a manager/program "is". A bias that is strongly emotional and difficult for most people to see in themselves. I'm reminded of that story I've mentioned here before the late Ray Kelly told me about an options trader from the early days of CBOE hardly ever had a losing week for 10+ years then lost more than he ever made in the space of two days in Oct 87...all because he took the signals from his system (like he always did for 10+ years highly successfully) on the Friday before the crash.

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Sat Feb 18, 2012 7:49 pm

Robwynge wrote:
Chuck B wrote: Amazingly, I can recall having these same discussions/arguments back in ~1994 during a seminar. Eighteen years later, one can only guess at the humongous sums that have been lost chasing stuff like Sharpe ratios or "managers" or "programs."
Chuck,

so what would be the right thing to chase in making such selections?
Hi Rob,

In my opinion you need to chase things that “make senseâ€
Last edited by DPH on Sun Feb 19, 2012 9:56 am, edited 1 time in total.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Sun Feb 19, 2012 8:41 am

Chuck B wrote:
The flip side are programs/managers who have a long and outstanding track record but are in fact trading a negative expectation system over a long measuring period that may exceed what one considers a "long time". There is an unbelievably strong bias in humans to "believe" a real-time record is profoundly indicative of how great a manager/program "is". A bias that is strongly emotional and difficult for most people to see in themselves. I'm reminded of that story I've mentioned here before the late Ray Kelly told me about an options trader from the early days of CBOE hardly ever had a losing week for 10+ years then lost more than he ever made in the space of two days in Oct 87...all because he took the signals from his system (like he always did for 10+ years highly successfully) on the Friday before the crash.
I have a deeply uncomfortable feeling that Chuck may be right. I also believe that DPH is absolutely correct: whatever the truth of the situation (systems work/do not work over the long term) leverage will eventually and inevitably kill you even if the system itself does not.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Sun Feb 19, 2012 11:29 am

Just to clarify, my "belief" in TF remains intact. Hilariously, the system I jigged around to pep up the Dead Turtles has performed brilliantly over the last couple of years in back testing. Would I had been trading it.

Back testing is as misleading as musing over worthless past performance figures. It has to be done, it has value but it is inevitably curve fitting. If you want a more realistic view on how your system will perform in the future it behoves you to take Monte Carlo analysis very seriously. It is certainly likely to disabuse you of any notion of an MAR of over 0.4 in the long term.

After many years and many very costly mistakes I know that greed is the real killer in this game; it inevitably leads to big losses, crippling fear and nemesis. As for "wet dreams" of annual returns of 300% and max DDs of 25%....well, what can I say....... enjoy the dream, wet or otherwise but wise up and realise that a dream is all it is and if you try and put it into practice it will inevitably turn into a nightmare.

Turn down the leverage, get real, survive.

marriot
Roundtable Knight
Roundtable Knight
Posts: 350
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot » Sun Feb 19, 2012 12:52 pm

Dean started this thread on past Cta performances.
Then we drifted to test and simulations.
Now it is not clear if last numbers are related to results after 2/20 fee.
There will be a huge difference.
Going back to test.
Done have everything one think smart for minimize curvefitting what else we can do?
Choose worst parameters because it cant be real?

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Sun Feb 19, 2012 2:56 pm

marriot wrote:Dean started this thread on past Cta performances.
Then we drifted to test and simulations.
I don't consider it drift. I consider the link directly and vitally relevant. The two are inextricably linked. If I knew what to do about the problems you outline I would have found the Stone (Philosopher's). I have been looking for it all my life but it remains resolutely hidden. Like a magic wand, the Stone is the answer to all human anxiety and suffering; it promises a world of serenity, plenty and peace. Keep looking, that is my recommendation.

Aaron01
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Thu Jun 02, 2011 1:21 pm

Post by Aaron01 » Mon Feb 20, 2012 4:56 pm

DPH wrote:There seems to be an underlying assumption that past performance (from which Sharpe is computed) is indicative, or highly correlated to future performance. My testing has shown me that this isn’t the case. Jack Schwager drew similar, (but not identical) conclusions years earlier in a similar study.
I believe NewEdge did a research paper in which they found historical implied Sharpe was not indicitave of realized Sharpe in the future.

DPH
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Tue Nov 17, 2009 1:07 am
Location: Central Pennsylvania
Contact:

Post by DPH » Tue Feb 21, 2012 2:11 am

marriot wrote: Done have everything one think smart for minimize curvefitting what else we can do?
Choose worst parameters because it cant be real?
It’s ironic, but I now believe that fundamental traders may end up “getting the last laughâ€

ifyousayso
Full Member
Full Member
Posts: 20
Joined: Sun Feb 19, 2012 11:47 am
Location: UK

Post by ifyousayso » Wed Feb 22, 2012 3:46 am

chris67, is your concern with WCM that they've become the Big Ugly Gorilla of the CTA space (in terms of hogging inflows), or that they're no longer really pure TFF? AFAIK (and the FSA Register tends to bear this out) they still have more or less the same execution as they did in the early noughties (i.e. they're systematic, but many of the trades are placed by people) and have recently (under their new CEO) moved heavily into single equities (ostensibly in search of uncorrelated investments for the WFF) and away from pure momentum trading. If so, are they that big a problem for smaller managed futures shops? Personally, I can't see why any HNW/UHNW individual would want to buy a hybdrid strategy like that (as for most the point of TFF/MF is to hedge the inevitable exposure to equities and to real estate), but presumably it appeals to a lot of this new institutional money??
Last edited by ifyousayso on Wed Feb 22, 2012 5:20 am, edited 1 time in total.

Demon

Post by Demon » Wed Feb 22, 2012 4:31 am

Hi ifyousayso, it's early and I'm still getting my brain up to speed - could you define AFAIK?

Post Reply