Too Good to Be true ???

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Too Good to Be true ???

Post by Chris67 »

Ok heres the thing

What would people do if using veritrader 1.5 with a simple mix of system came up with a robust system with an mar of high 3's over the last 10 years .. great risk adjusted returns and sturdy parametrs working across a bunch of random portfolios ?? It sounds too good to be true so now what do i do?
I am thinking I spend the next few weeks doing everything I can to prove this cannot be right and if i', still left with a good system then trade it.

It seems to me its not too hard to find a good mix of sysytems that produce 50 % average per year off of max drawdowns of 20 % or marginally under ....Why wouldnt everybody in the mechanical world use these rules .. is it simply a case that they always interefere or cannot stand the 20 % drawdown .. AM I MISSING SOMETHING ?
Jake Carriker
Site Admin
Site Admin
Posts: 1493
Joined: Fri Sep 12, 2003 10:32 am
Location: Austin, Texas

Post by Jake Carriker »

You might consider testing over a longer time period (20+ years) as well as testing on out of sample data. You might also look at "stress testing" (as you implied you will be doing over the next couple of weeks).

This could include imposing very negative slippage assumptions like buying at the high and selling at the low, greater rollover costs, etc. It could also include having your test start on different dates, like right before a big drawdown. Monte Carlo testing by scrambling the daily % changes and running thousands of iterations may also give you some good information. You could test on data from a different data provider and see how much variation in results you get. You could change the manner in which you backadjust your data and see what happens.

There are endless variations on the hoops you can make your systems jump through, but those are a few that come to mind. The biggest one to me would be the length of the test. If these are medium to long term systems, I would not be comfortable with less than 20 years of testing. This will probably produce 3,000 trades plus, and if your systems don't have too many parameters (degrees of freedom) that should be a pretty good sample. The more parameters you have, the more trades you will need to sample to have a good chance of avoiding curve fitting.

Jake
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Stress Testing

Post by William »

Stress Testing

I was wondering what the forum members think is an approriate level of stress for a system and how do they gauge if the system adequately passed the test?

So far, the two parameters i have been tweaking is slippage and commissions. With slippage, i'm not sure how to interpret 100% slippage test reults, since things typically go to hell. With a high slippage test, are we looking for minor profitability, 5 - 15% with high Max DD's or for the system to retain a decent MAR rating, say 60% of its regular MAR? OR, is this all moot since a 100% slippage stress test, is not a good test setting?
Jake Carriker
Site Admin
Site Admin
Posts: 1493
Joined: Fri Sep 12, 2003 10:32 am
Location: Austin, Texas

Post by Jake Carriker »

Hi William, I agree that max slippage assumptions will kill the profitability of almost any system, however, IMHO there is still some value in it. I believe Mark Johnson once said on this forum or another, I am not sure, (paraphrasing) that he does not know what a good system is, just whether a system is "good" in comparison to another. In other words, don't consider the results in a vacuum, as they won't be terribly meaningful. However, when compared to other system's results under the same conditions, one might glean valuable information.

That said, I find a more useful test is to increment the slippage and note at what point a system begins to break down and become unusable for one's purposes. Once you find that point, you can determine whether slippage could reasonably be limited to such an amount or less. I usually run the max slippage test simply because, in my opinion, a system that turns any profit under these conditions is probably worth considering further.

To answer your question directly, The "real world" test that I use is what Veritrader users would call 20% slippage. That is, my fill price is 1/5th of the way between my targeted entry price and the worst price of the bar. Others may use greater or lesser percentages, and some may be happy with a fixed dollar amount (i.e. the Futures Truth standard $75 per contract). I also run $75 per contract just for comparison's sake, but the 20% slippage test is what I bank on.

My two cents.

Jake

P.S. - I am talking about relatively liquid markets. If you want to trade propane, throw all these figures out the window.
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Post by William »

Most helpful. Thanks for your insights.
MarkH
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Fri Dec 05, 2003 10:24 am
Location: Corpus Christi, Texas

Post by MarkH »

The amount of slippage also depends on the type of order your system uses to enter and exit trades. Stop orders tend to have the most, market orders next and limit orders have the least. For stop (and market) orders, I use $85.00 per contract per trade for most markets (higher amounts for equity indexes). For limit orders, I use $35.00.

I trade multiple systems on multiple markets. I began using limit orders (for both entry and exit) for my most recently designed system, motivated partly by an interest in controlling slippage better. I was pleasantly surprised to discover that I got good test results, even though using limit orders means that when the entry order is elected, the market is temporarily moving opposite of the direction of the trade.

There is an occasional problem with using limit orders that I have not fully resolved yet. There are rare times when the limit order happens to be the high/low of the day and you will not get a fill in real time trading. This does not happen, of course, in the back tested simulated trading results. I have been handling this issue on a case by case basis; however, it remains on my R&D list to test various ways to systematically deal with this on a consistent basis. I was wondering if anyone else has a method or suggestion for dealing with this problem?

Mark
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

Mark ,, The only problem with your theory is that if you use limit orders you risk not getting filled at all .. if you are trading a trendfollowing system then by design you are chasing the market .. if you dont use a stop or market order to get in you risk not getting in at all ... and by default in the real world we all know that the best trends will be the ones where you miss getting in .. its murphys law and it lives in trading... I find the whole argument of slippage amazing and today is an excellent example .. I have executed 4 trades ..3 stops and one stop entry .. the minmum slippage I recieved was 150 usd on all 4 orders .. god knows what would happen if i were executing 50 lots plus
MarkH
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Fri Dec 05, 2003 10:24 am
Location: Corpus Christi, Texas

Post by MarkH »

Chris -

I, too, was a little surprised that a limit order entry for a trendfollowing strategy would backtest very well -- but, I do get good backtested simulated results and my real time trading is tracking my simulated trading --- with the rare exception of sometimes not getting a fill when the limit order is the high/low of the day.

It turns out that the problem you describe (missing getting on board a strong trend) only happens if the market takes off and never retraces. Because I will keep getting entry signals, one of the retracements will likely eventually get the limit order (which would also move up/down as the market moves up/down). Of course, if this happens after several days of trending movement, the limit entry will be worse than what an earlier stop or market entry would have been. These worse entry trades seem to be more than offset by better entries on other trades when the market doesn't race immediately after the other entry conditions have been met.

Anyway, it is a pleasant experience to get into and out of trades with no slippage --- although I will admit that in trading doing what "feels good" is rarely the right answer.

Mark
Post Reply