What to make of this chart?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

What to make of this chart?

Post by LeviF »

Should any conclusion be drawn about the recent performance of this system?
Attachments
monthlyreturnsgraph_p1.jpg
monthlyreturnsgraph_p1.jpg (136.94 KiB) Viewed 13966 times
Mathemagician
Full Member
Full Member
Posts: 20
Joined: Thu Jun 28, 2007 1:46 pm
Contact:

Post by Mathemagician »

Yes, the recent performance has not been good.

jj
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

maybe i should rephrase. should i be concerned with the recent performance of the system?
Mathemagician
Full Member
Full Member
Posts: 20
Joined: Thu Jun 28, 2007 1:46 pm
Contact:

Post by Mathemagician »

One should always be concerned about the performance of their system. That said, there isn't anywhere near enough information available here for us to comment one way or the other intelligently.

jj
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

Yes, the recent performance has not been good.

One should always be concerned about the performance of their system.
Who are you? Ed Seykota?
Mathemagician
Full Member
Full Member
Posts: 20
Joined: Thu Jun 28, 2007 1:46 pm
Contact:

Post by Mathemagician »

Are Ed Seykota's initials 'jj'?

jj
Kobeyashi
Full Member
Full Member
Posts: 23
Joined: Fri Oct 26, 2007 10:36 am
Location: London, UK

Post by Kobeyashi »

Hi Levijean,
I can only concur with Mathemagician that there isn't enough data in this instance to pass a judgement.
However, this did raise a question. How do we differentiate between an elongated losing streak, and system death?

Kobeyashi
Mathemagician
Full Member
Full Member
Posts: 20
Joined: Thu Jun 28, 2007 1:46 pm
Contact:

Post by Mathemagician »

Step 1. Generate an honest and unbiased backtest (most people can't/won't do this)
Step 2. Use appropriate resampling techniques to construct confidence intervals at your chosen level of confidence
Step 3. Wait for the system to operate outside its confidence interval

jj
Dallas
Senior Member
Senior Member
Posts: 36
Joined: Wed Dec 31, 1969 7:00 pm

Look at the whole chart, not just the recent months

Post by Dallas »

It seems to me that this would be a very difficult system to trade from a psychological standpoint, due to what I perceive to be extreme volatility.
In round #'s if you can be up or down more than 10-20% per month your account could be severely damaged pretty quickly. If you can be up 5 months straight you can be down 5 months straight just as easily. Turn the heat down, way down. Just my opinion. I may be an overprudent risk assessor.
LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Re: Look at the whole chart, not just the recent months

Post by LeapFrog »

Dallas wrote:It seems to me that this would be a very difficult system to trade from a psychological standpoint, due to what I perceive to be extreme volatility.
For some, maybe not for others. Here is a pic of the results of a well regarded, long term hedge fund. You can add back the 20% fee taken off the top, but that only adds to the volatility of the chart.

I agree though that the 5 big losing months in a row would be hard to handle.
Attachments
Monthly Volatility of a public hedge fund.
Monthly Volatility of a public hedge fund.
Monthly Vol.PNG (10.41 KiB) Viewed 13630 times
Algonquin
Roundtable Knight
Roundtable Knight
Posts: 158
Joined: Sun Jan 28, 2007 9:38 pm
Location: Greenwich, CT

Post by Algonquin »

Given the mammoth moves we have experienced recently in so many instruments, I would be concerned about the system's seeming inability to turn a profit, especially if it is designed to be trend friendly. Just my $.02.
stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Post by stopsareforwimps »

Mathemagician wrote:Step 1. Generate an honest and unbiased backtest (most people can't/won't do this)
Step 2. Use appropriate resampling techniques to construct confidence intervals at your chosen level of confidence
Step 3. Wait for the system to operate outside its confidence interval

jj
I have spent the last couple of months trying to do this. It is a lot harder than I expected.

The problem I have experienced is that the returns data (eg monthly returns, from which you might sample) are not independent identically distributed (IID) points from a stationary distribution.

When I optimize on say the last 10 years of data and test on old data, I get brilliant returns (Sharpe = 2.5-4). This set off my bogusity alarm.

When I optimize on a random subset of months and test on another subset of months the results are almost as brilliant.

When I optimize on old data and run on recent markets, the results are dismal - worse than random strategies.

Doing roll-forward tests, optimizing every 9 months, I get fairly good results - similar to what you see for funds on IASG. I get Sharpe ratios around 0.7. So there still seems to be some juice there. The problem is that a roll-forward test is just one test and it's hard to know how to get a sense for how random it is.

I tried optimizing at various differing intervals, e.g. every 6 months or 9 months or every month, and limiting trades to every 'n' days as ways to introduce some randomness.

One can try subsets of markets but they are correlated, risk on/off and all that.

I welcome any ideas. Including bad ideas - knowing someone tried something and it didn't work is useful.
akhoury
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Mon Feb 28, 2011 11:07 am
Location: Stuart, Florida

Post by akhoury »

Is there a benchmark comparison and backtest to add to the absolute data?.....sometimes that helps a bit and maybe could have some clues...ack
babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader »

stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.
stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Post by stopsareforwimps »

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.
Thank you.
squaredQ
Full Member
Full Member
Posts: 23
Joined: Wed Feb 01, 2012 5:27 pm

Post by squaredQ »

Stopsareforwimps,
I think you are looking at the right things, from my experience.
IID is a concern, but you can get into non-IID analysis as well. Few do.

OP, I would be concerned and scale down... Without even doing the statistical analysis, it seems highly unlikely you would have 5+ neg consecutive months in a row by chance-- a flag of caution. It also helps to look at what the underlying dynamics of what you are capturing are doing (why are they changing or not?).
mojojojo
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Mon Oct 31, 2005 2:07 pm

Post by mojojojo »

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.
There are so many good packages in R. I'm only starting to scratch the surface now.
stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Post by stopsareforwimps »

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.
This opened up a whole new vista of testing and validation ideas.

The statisticians have come up with ways of measuring data snooping bias and the results are disconcerting. Doing a roll-forward test does not make you immune by any means. Here is an example:

"A reality check on technical trading rule profits in US futures markets".

http://ageconsearch.umn.edu/handle/19039

I am not yet 100% convinced. There seems to be a weakness in these methods - if you dump vast numbers of obviously poor strategies into the mix you can rig a bad result.

In "Data-snooping technical trade performance and the bootstrap" by Ryan Sullivan et al, we read that the rule universe of 26 different rules considered in an earlier paper worked quite well out of sample (semi-significant at a confidence level of 15%, and possibly fully significant if the more accurate SPA test had been used) (see figure 8).

In contrast a larger rule universe generated strongly negative results, suggesting that any positive performance was due to data snooping.

Note that this study has problems, because most of their testing was on DJIA index data which ignored dividends.

They then applied the rules derived from this history into the future, with unsurprisingly poor results. However I would suggest that you cannot ignore 50% of the return, which is about what dividends provided historically, and expect to get good results when applied to futures, which implicitly include dividend returns.

You could spend weeks researching this. I plan to try it out myself. The big question is "What is the universe of rules which I considered, directly or indirectly, when putting together my strategy?"
trackstar
Roundtable Fellow
Roundtable Fellow
Posts: 76
Joined: Sun Mar 09, 2008 11:03 pm
Location: Houston Texas

Post by trackstar »

stopsareforwimps wrote:
babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.
This opened up a whole new vista of testing and validation ideas.

The statisticians have come up with ways of measuring data snooping bias and the results are disconcerting. Doing a roll-forward test does not make you immune by any means. Here is an example:

"A reality check on technical trading rule profits in US futures markets".

http://ageconsearch.umn.edu/handle/19039

I am not yet 100% convinced. There seems to be a weakness in these methods - if you dump vast numbers of obviously poor strategies into the mix you can rig a bad result.

In "Data-snooping technical trade performance and the bootstrap" by Ryan Sullivan et al, we read that the rule universe of 26 different rules considered in an earlier paper worked quite well out of sample (semi-significant at a confidence level of 15%, and possibly fully significant if the more accurate SPA test had been used) (see figure 8).

In contrast a larger rule universe generated strongly negative results, suggesting that any positive performance was due to data snooping.

Note that this study has problems, because most of their testing was on DJIA index data which ignored dividends.

They then applied the rules derived from this history into the future, with unsurprisingly poor results. However I would suggest that you cannot ignore 50% of the return, which is about what dividends provided historically, and expect to get good results when applied to futures, which implicitly include dividend returns.

You could spend weeks researching this. I plan to try it out myself. The big question is "What is the universe of rules which I considered, directly or indirectly, when putting together my strategy?"
One other thing to add is that for the "ALX" rule they used percentage moves from a recent high or low to initiate and exit a position. Using percent move calculations on old(backadjusted, or really any adjusted) futures data is incorrect and inaccurate.
stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Post by stopsareforwimps »

squaredQ wrote:Stopsareforwimps,
I think you are looking at the right things, from my experience.
IID is a concern, but you can get into non-IID analysis as well. Few do.

OP, I would be concerned and scale down... Without even doing the statistical analysis, it seems highly unlikely you would have 5+ neg consecutive months in a row by chance-- a flag of caution. It also helps to look at what the underlying dynamics of what you are capturing are doing (why are they changing or not?).
A link to a zip file of papers on reality check and some enhancements. RC can be too pessimistic - as I suggested earlier but I did not know how.

http://4xtutor.com/autotrade/maths/real ... ck-papers/
Post Reply