How to realise the system is obsolete

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
AceofAce
Senior Member
Senior Member
Posts: 49
Joined: Thu Sep 30, 2010 5:06 am

How to realise the system is obsolete

Post by AceofAce »

The turtles system, highly succesful as it was in the 70s and 80s became obsolete with its standard parameters in the 90s. Drawdowns and bad performance persisted since then. The system whilst profitable still had temporary periods of bad performance, sometimes going for years before a turnaround.

When someone trading a successful system starts having periods of bad performance and drawdowns, how would he know if its just a temporary down period (so the system doesn't needs changes) or his system is gradually becoming obsolete.

Any ideas?
mojojojo
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Mon Oct 31, 2005 2:07 pm

Post by mojojojo »

That's a tough question and I don't think there is a correct answer. I also feel it depends on the system.

If you system is designed to exploit a specific event, you will have to decide if the market has some how changed to a point where that event is no longer valid.

In a general sense, the only thing that you really have to go off of is your testing up to that point. Do returns/drawdowns still reasonably match what your testing has shown?

I'm definitely looking forward to hearing what others have to say.
bobsyd
Roundtable Knight
Roundtable Knight
Posts: 405
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd »

(1) Search for Walk AND Forward in this forum; (2) search for Robusti in this forum; and (3) Google Walk Forward.
leslie
Roundtable Knight
Roundtable Knight
Posts: 135
Joined: Tue Apr 12, 2011 11:46 am

Post by leslie »

Robert Pardo's old book on Design & Testing of Trading systems has a LOT of good measures. A LOT. Try some.

Perry Kaufman made two suggestions in this regard--don't know if he published these or not. Backtest the strategy using position sizing method as if every day was the first day of trading. (In other words, you are NOT compounding.) Load quarterly returns into Excel, Numbers, MatLab and plot linear regression. If LinReg slopes down, strategy is dead.

This test ought to take under 10 minutes.

Fancier version does same for Quarterly Max DD. If QMDD Linear Regression Slopes UP, strategy is dead.

You might be able to delay system retirement by Walk Forward optimization, perhaps...



Perry's fancier method: Using Non-Compounded EQ as before, divide in-sample data into 2 parts. Adjust risk.Equity, (by playing with system.allocationPercent) until your MaxDD = 1 million dollars. Dump daily equity and compute mean, kurtosis, skew. Do same for second part of in sample. Now, using same risk.Equity settings, run out-of-sample data. Has your Kurtosis & Skew changed?

This method is more sensitive since change in Skew occurs more rapidly then change in mean.

Standard methods such as Student's t-distribution are described in many text including Adams, Booth, et.al.: Investment Mathematics. (Chi-Square distribution & testing is closely related and often used.)

L
Chuck B
Roundtable Knight
Roundtable Knight
Posts: 481
Joined: Thu Apr 17, 2003 6:34 am

Re: How to realise the system is obsolete

Post by Chuck B »

AceofAce wrote:The turtles system, highly succesful as it was in the 70s and 80s became obsolete with its standard parameters in the 90s. Drawdowns and bad performance persisted since then. The system whilst profitable still had temporary periods of bad performance, sometimes going for years before a turnaround.

When someone trading a successful system starts having periods of bad performance and drawdowns, how would he know if its just a temporary down period (so the system doesn't needs changes) or his system is gradually becoming obsolete.

Any ideas?
There is no pure answer to your question. Since we're not dealing with science but instead a pathological, fractal market(s), no matter how much "testing" you do whether it be "in-sample", "out-of-sample", "walk forward", etc, the future will not look like your test results. If you set parameters like 1.5 times my max test drawdown is my cutoff point, and then your system goes to 1.8 times that value and rebounds sharply to new equity highs, and then goes on to great performance for the next 3 years, was it "broken" just because of your decision? (this is just one example of many similar measures people choose to arbitrarily cutoff a system).

I think it is ignorance or arrogance or both perhaps that traders believe they have captured all they need to know to risk capital based on historical data and even real time data. Perhaps real time data results are the most potentially damaging since people place far more faith in those than is warranted (this fact has been proven time and again over the years...many times).

If there was a pure answer to this question, then trading would be pretty simple long term. I think the "real" answer for professional traders is when the customer fund outflow reaches the point of the business not being able to keep the doors open any longer (said facetiously but actually it's pretty accurate).
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

I totally agree with Chuck

Take it one step further as an extreme example - everybody assumes the original turtle system is dead - what about when yuo run TB through in the year 2125 (someone out there probably will) - they look at the equity curve and listen to that 140 year old story of the turtles - they then say

" well that system blew the doors off for 15 years before going into a 10 year period of absolutely nothing - completely flat - but then look at the next 30 years - an amazing performance !! and then yet another decade of crap - before another 20 years of explosive profits !!!!

This is just an example of how we play with fire by assuming 20-30 years of data means something ? its like looking at 1000 years of weather and making big statements about global warming - it means absolutely zilch !!

Of course 10 years of a system doing nothing will blow most businesses up (unless of course your jwh and Bill Dunn who have already banked a billion) - but these guys are smart have no fear and why do you think it is they havent really changed too much ???

I think a heavy dose of inflation will see a bog standard 20 day break out 10 day stop loss (2 atr stop) probably explode into life ? over the last 10 years its probably been rubbish

Too many people make too many assumptions about dead systems and what is likely to work in the future

The secret of this business is not identifying what is a dead system / what isnt a dead system its understanding and dealing with the fact that any system could go in and out of a 10 year period of nothingness and how does that affect your business - the answer is of course badly - that is the real issue
C
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

I do think it is immensely helpful to keep toying with new ideas. I like (and appreciate) Sluggo's recent posts (footnote) and simple systems. I was recently working on the humble double moving average and realising that even that can be vastly improved with a bit of tinkering.

Simplicity, robustness - all the old coconuts help to give confidence that this is not just a game, a fad. It won't go away. It may need tinkering and adjustment but it is real; it builds assets and wealth. It is a hell of a lot better than the great majority of alternatives on offer out there.

The last few days have been miserable; draw downs always are. But at least one learns to temper the ferocious despondency which can kick in at such times with the reasonable expectation that all will come round again in the (hopefully) not too distant future.

This is yet another of those areas of "mechanical" trading which requires a hefty input of discretion and common sense. No amount of "Skegness" measuring, student's spot-tests. z-tests and all the rest of the gizmos in the statisticians magic box can make up for it.

(footnote - added by Moderator): These are in the Blox Marketplace: (1), (2), (3), etc.
rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric »

Yes. I refrained from saying it earlier, as I wasn't sure if it might come across too strong and personal, but for what it is worth, ALL of the methods outlined in longmemory's post are some of the worst ways to approach this kind of problem that you can find. The more arcane and obtuse the method , the more lag and disconnection from the raw underlying signal(+noise).

Simple, actionable, timely. These are the watchwords to bear in mind when tackling this problem. Just like trading rules. Furthermore, like Chuck B and Chris highlighted, you cannot expect to get this solution tuned correctly. Better to be wrong early(though within reason) and have enough bullets left for another crack, than try to hard to be right.

Context speaks a lot in this kind of situation. If in a backtest you never exceed a 10% DD in 10years, and then you go 10% DD in your first 6months of live trading, this carries far greater alarm bells with it than if you go >10% DD after 6 years of live trading.

Likewise a backtest that has 9 years of 5%dd and one 20% dd is to me in many ways nicer than one that has 10 years of 7-13% DD. In the first case you know the system has either worked well or failed hard. it is harder to say in the second case. Thus decisions and appropriate risk reduction methods are easier to implement in the first case. Then again system 1 could morph into system 2. Who can say?

Which brings me back to: whatever you do, make sure you can keep on playing another day.
Post Reply