Backtested performance help

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Backtested performance help

Post by Chris67 »

Dear All.

I designed a system on jan 1990 to jan 2000 that produced great results .. 50 % cagr ..mar of 2.4 .. I then took that system and applied it to 1980 - 1990 .. again th ersults were great .. approx the same ...
I then wlked it forward from 2000 to 2004 and the results diminished greatly ??
Is there a chance this has something to do with the positions going into the year .. and coming out at finish ..
For example in the 2000 - 2004 test the 2000 performance is 17.7 % ar .. if i run the system from 1990 to 2004 it tells me I made 45 % in 2000 ??

Should I be put off by the 2000- 2004 results ? Is there a logical explanation for this ?

Basically the system from 1984 - 2004 works extremely well ...

Any thoughts would be warmly appreciated

Regards
Chris
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Chris,

Yes, the first year won't do as well if there were good trades at the beginning because VeriTrader only takes new trade signals.

This is what would account for the discrepancy for 2000.

We'll try to add some capability for testing assuming you had all open positions as of the start of the test date (rather than looking at new signals) in 2.0. This would give you exactly the same results for 2000 in the case above, but would not be reflective of what you might want to do in real life.

So you have three possibilities:
  • Enter Midstream - This is what most platforms do. The tests don't reflect what you might have done because they enter you into positions mid-stream rather than at the first signal. This often results in significant drawdowns at the start of the tests because you enter positions at times of higher risk.
  • Enter Only New Trades - This is what VeriTrader does right now.
  • Take an Equity Snapshot - This is what I propose as an alternative for 2.0 (actually we'll let you do all three). VeriTrader would simply start the year with all open positions that you would have had if you had been trading from a point considerably before the test start date.
When testing over shorter test periods you can expect periods where the results are both much better and much worse than over longer periods.

How much did the results diminish? Are they still largely positive?

- Forum Mgmnt
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

c.f. .. Yes the results drop off significantly.. what I am beginning to see is that the results , in particular , for the original turtle system , with modifications , or without , is diminishing rapidly .. average 100 % returns in the 80 ' s 60 % in the 90's and 20-30 % so far this millenium .. I dont think the future bodes well for this kind of thing .. too many people are now on the case .. and I think the majority of these peolple prob do stick to the rules .. its a worry
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Chris,

I'm not sure what you're seeing is a universal.

I just tried comparing the Turtle system with the 1.5 default parameters (slight variations of the Original parameters) and I draw the opposite conclusion.

01/94 to 12/03:
CAGR% - 50.4%
MAR - 1.12


01/99 to 12/03:
CAGR% - 65.3%
MAR - 1.45

The Max drawdown in each case was the same period, last year. This period followed the three best contiguous years of the 10 year test, the same period you identified as being poor or worse.

If you had run the same tests at the end of 2002, you would have thought the system was getting better not worse.

I think there is more variation for this sort of thing than one might expect. Also fairly slight variations in the parameters might result in one period being worse than with a different set of parameters. So the level of uncertainty is higher than we would like. That's also the reason the oppotunity still exists. Most people can't/won't make decisions with this level of uncertainty.

I also believe that markets and systems run in cycles, really good years tend to be followed by worse drawdowns than medium years. If you just finished a bad year, you will tend to believe the recent past is better than it was. One year of more typical returns might change things dramatically.

This is one of the reasons I think it is misleading to look at things from the perspective of one or two numbers. You really need to get a look at the equity curve in detail to understand the performance of the system.
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

c.f. ,

Thanks as always for your reply .. I think we are hitting something here which is more important than anything else..

Try doing this .. put the simple or slightly varied original turtle rules into the system and test 1980 to 2004 ... Look at the equity curve and its clear to the eye that the system is tailing off dramatically ...as I said before returns were almost always 100 % + in the 80's .. and then they diminish .. th ecurve is very interesting

Let me put it to you in none mathematical terms ... With the advent of the computer , the publication of turtle rules and the publiication of the turtle story , with the mass availablity of backtesting facilities , plus globalisation in general ( via the internet ) , and lastly with the huge influx of hedge funds entering the arena trading systemataically .. do you not agree to some dgree that degradation of trendfollowing results is obvious ... also PLEASE understand that in no way does this mean markets are not trending .. they are probably trending more now than ever ..its just you gotta be a little street savvy to stay in the trade and the trends are harder to catch .. all IMVHO

Regards
Chris
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Start Date Issues

Post by William »

I have found that what i randomly choose as a start date, can eventually become the bedrock of my performance results. For example, my first year gets curve fitted to the point where its a great year and from then on my results are unrealisitically scewed upward. I was wondering how other people try to work around the issues of overweighting the first year.
stancramer
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Sat Feb 07, 2004 10:34 am
Location: Washington DC

Post by stancramer »

Assuming you're testing profitable systems :) the first year is the one with the lowest equity and the fewest # contracts and the most sensitive to rounding. One thing you can do to cool down the first year is make sure you are rounding down when calculating the number of contracts to trade. If your calculations say "trade 0.836491 contracts" then make sure your backtest engine trades zero contracts (i.e. skips the signal). The effect of this rounding becomes less and less as your equity climbs and when it hits 3 or 4 million dollars, the rounding is insigificant. You can even put in code that operates more conservatively still, something like

Code: Select all

NEWCONTRACTS = RoundDown((EQUITY*fixedfraction/RISK) - 0.20)
Another thing happens the first couple years: you're only taking signals from the "old men" contracts, the ones that have been traded since the 1970's. The newer contracts don't appear till later in the tests. So your results for the first years are unrealistically weighted towards the "old men". The easy solution is to move your simulation start-date forward until it has all (or 95% of all) contracts trading. But this may be too few years for your taste.

A third thing you can do (and I have done) is to post-dictively "cheat". Run a normal backtest your normal way, with proper rounding-down, and print out the equity curve. Find a spot where the equity curve just begins to go into a rather deep and rather prolonged drawdown; for many LTTF systems, 01/01/1983 is a reasonable choice. Then use that date as the beginning of your backtest. Voila, you immediately go into a drawdown and claw your way out slowly. This makes sure you don't get an artificial boost from a great first-year showing.
Demon

Post by Demon »

Hi Stan, I think you've also highlighted a very valid point with regard to backtesting and optimising a system over as much data as possible:
Another thing happens the first couple years: you're only taking signals from the "old men" contracts, the ones that have been traded since the 1970's. The newer contracts don't appear till later in the tests. So your results for the first years are unrealistically weighted towards the "old men".
Clearly the optimised parameters would also be weighted to the "older men" yet they would carry no more weight in the portfolio.
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Post by William »

Stan,

Thanks for your input and your ideas. One of the ideas i have bounced around is actually varying between two start dates. For my testing, i was orignally testing starting from 1.01.90 and what i noticed was that my results were great for the first year, but when i walked it forward i went from around a CAGR of 113% to about 85% (MAX DD% 40% so not a tradable system) once i walked it up only one year.

With the alternating 90 - 91 start date, i have found that my start date is less magnificent and the overall performs degrades a bit, however the upside is that instead of the first year being a great year and the rest off a decent amount, the first few years are all hovering in the same performance area and it suffers from less of a dramatic drop off. My initial belief is that it enables you to find a happy median between those two periods, i think creating a more stable parameter set.

I erased the original data since i am using a laptop and my sytem slows down quite a bit with the results files, so it screws the whole comparison idea up but here is a look at the alternating start date's initial few years. MAX DD% and Longest Drawdown stayed the same between the two.

1990 - 87.5
1991 - 84%
1992 - 85%
1993 - 77.5%
1994 - 71.%
1995 - 73.5%
1996 - 81.5%


I havent done extensive tests (only 2 days worth) but i thought the results were interesting enough to post here.
Post Reply