Traders' Roundtable

Should any conclusion be drawn about the recent performance of this system?

Yes, the recent performance has not been good.

jj

maybe i should rephrase. should i be concerned with the recent performance of the system?

One should always be concerned about the performance of their system. That said, there isn't anywhere near enough information available here for us to comment one way or the other intelligently.

jj

Yes, the recent performance has not been good.

One should always be concerned about the performance of their system.

Who are you? Ed Seykota?

Are Ed Seykota's initials 'jj'?

jj

Hi Levijean,
I can only concur with Mathemagician that there isn't enough data in this instance to pass a judgement.
However, this did raise a question. How do we differentiate between an elongated losing streak, and system death?

Kobeyashi

Step 1. Generate an honest and unbiased backtest (most people can't/won't do this)
Step 2. Use appropriate resampling techniques to construct confidence intervals at your chosen level of confidence
Step 3. Wait for the system to operate outside its confidence interval

jj

It seems to me that this would be a very difficult system to trade from a psychological standpoint, due to what I perceive to be extreme volatility.
In round #'s if you can be up or down more than 10-20% per month your account could be severely damaged pretty quickly. If you can be up 5 months straight you can be down 5 months straight just as easily. Turn the heat down, way down. Just my opinion. I may be an overprudent risk assessor.

Dallas wrote:It seems to me that this would be a very difficult system to trade from a psychological standpoint, due to what I perceive to be extreme volatility.

For some, maybe not for others. Here is a pic of the results of a well regarded, long term hedge fund. You can add back the 20% fee taken off the top, but that only adds to the volatility of the chart.

I agree though that the 5 big losing months in a row would be hard to handle.

Given the mammoth moves we have experienced recently in so many instruments, I would be concerned about the system's seeming inability to turn a profit, especially if it is designed to be trend friendly. Just my $.02.

Mathemagician wrote:Step 1. Generate an honest and unbiased backtest (most people can't/won't do this)
Step 2. Use appropriate resampling techniques to construct confidence intervals at your chosen level of confidence
Step 3. Wait for the system to operate outside its confidence interval

jj

I have spent the last couple of months trying to do this. It is a lot harder than I expected.

The problem I have experienced is that the returns data (eg monthly returns, from which you might sample) are not independent identically distributed (IID) points from a stationary distribution.

When I optimize on say the last 10 years of data and test on old data, I get brilliant returns (Sharpe = 2.5-4). This set off my bogusity alarm.

When I optimize on a random subset of months and test on another subset of months the results are almost as brilliant.

When I optimize on old data and run on recent markets, the results are dismal - worse than random strategies.

Doing roll-forward tests, optimizing every 9 months, I get fairly good results - similar to what you see for funds on IASG. I get Sharpe ratios around 0.7. So there still seems to be some juice there. The problem is that a roll-forward test is just one test and it's hard to know how to get a sense for how random it is.

I tried optimizing at various differing intervals, e.g. every 6 months or 9 months or every month, and limiting trades to every 'n' days as ways to introduce some randomness.

One can try subsets of markets but they are correlated, risk on/off and all that.

I welcome any ideas. Including bad ideas - knowing someone tried something and it didn't work is useful.

Is there a benchmark comparison and backtest to add to the absolute data?.....sometimes that helps a bit and maybe could have some clues...ack

stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.

Thank you.

Stopsareforwimps,
I think you are looking at the right things, from my experience.
IID is a concern, but you can get into non-IID analysis as well. Few do.

OP, I would be concerned and scale down... Without even doing the statistical analysis, it seems highly unlikely you would have 5+ neg consecutive months in a row by chance-- a flag of caution. It also helps to look at what the underlying dynamics of what you are capturing are doing (why are they changing or not?).

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.

There are so many good packages in R. I'm only starting to scratch the surface now.

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.

This opened up a whole new vista of testing and validation ideas.

The statisticians have come up with ways of measuring data snooping bias and the results are disconcerting. Doing a roll-forward test does not make you immune by any means. Here is an example:

"A reality check on technical trading rule profits in US futures markets".

http://ageconsearch.umn.edu/handle/19039

I am not yet 100% convinced. There seems to be a weakness in these methods - if you dump vast numbers of obviously poor strategies into the mix you can rig a bad result.

In "Data-snooping technical trade performance and the bootstrap" by Ryan Sullivan et al, we read that the rule universe of 26 different rules considered in an earlier paper worked quite well out of sample (semi-significant at a confidence level of 15%, and possibly fully significant if the more accurate SPA test had been used) (see figure 8).

In contrast a larger rule universe generated strongly negative results, suggesting that any positive performance was due to data snooping.

Note that this study has problems, because most of their testing was on DJIA index data which ignored dividends.

They then applied the rules derived from this history into the future, with unsurprisingly poor results. However I would suggest that you cannot ignore 50% of the return, which is about what dividends provided historically, and expect to get good results when applied to futures, which implicitly include dividend returns.

You could spend weeks researching this. I plan to try it out myself. The big question is "What is the universe of rules which I considered, directly or indirectly, when putting together my strategy?"

stopsareforwimps wrote:

babelproofreader wrote:stopsareforwimps,

If you're looking for more statistically sophisticated tests than a simple rolling window back test, try reading the reference manual for this R package and the papers referenced in it.

This opened up a whole new vista of testing and validation ideas.

The statisticians have come up with ways of measuring data snooping bias and the results are disconcerting. Doing a roll-forward test does not make you immune by any means. Here is an example:

"A reality check on technical trading rule profits in US futures markets".

http://ageconsearch.umn.edu/handle/19039

I am not yet 100% convinced. There seems to be a weakness in these methods - if you dump vast numbers of obviously poor strategies into the mix you can rig a bad result.

In "Data-snooping technical trade performance and the bootstrap" by Ryan Sullivan et al, we read that the rule universe of 26 different rules considered in an earlier paper worked quite well out of sample (semi-significant at a confidence level of 15%, and possibly fully significant if the more accurate SPA test had been used) (see figure .

In contrast a larger rule universe generated strongly negative results, suggesting that any positive performance was due to data snooping.

Note that this study has problems, because most of their testing was on DJIA index data which ignored dividends.

They then applied the rules derived from this history into the future, with unsurprisingly poor results. However I would suggest that you cannot ignore 50% of the return, which is about what dividends provided historically, and expect to get good results when applied to futures, which implicitly include dividend returns.

You could spend weeks researching this. I plan to try it out myself. The big question is "What is the universe of rules which I considered, directly or indirectly, when putting together my strategy?"

One other thing to add is that for the "ALX" rule they used percentage moves from a recent high or low to initiate and exit a position. Using percent move calculations on old(backadjusted, or really any adjusted) futures data is incorrect and inaccurate.

squaredQ wrote:Stopsareforwimps,
I think you are looking at the right things, from my experience.
IID is a concern, but you can get into non-IID analysis as well. Few do.

OP, I would be concerned and scale down... Without even doing the statistical analysis, it seems highly unlikely you would have 5+ neg consecutive months in a row by chance-- a flag of caution. It also helps to look at what the underlying dynamics of what you are capturing are doing (why are they changing or not?).

A link to a zip file of papers on reality check and some enhancements. RC can be too pessimistic - as I suggested earlier but I did not know how.

http://4xtutor.com/autotrade/maths/real ... ck-papers/