Too Good to be True?

[steady_jake]

MODERATOR'S NOTE: This topic was split from the "Trading the Equity Curve" topic in the Money Management forum.

Forum Mgmnt,

I have developed a system, which has confused me a little bit. I used to be day trading S&P's for a couple of years in the 90ties and after a break, I considered going into daytrading the Forex pairs on one of so many poular Internet platforms today.

I decided, that the best way to go about it , since I no longer have the time and faith (!) to stick to the monitor all day long, is to devise a mechanical intraday trading system, into which I could incorporate my years of "tape reading" experience. I have used a short term overbought/oversold indicator for entering the market on a trading range basis with a medium term trend-following filter for confirming the market direction. I used trailing volatility stops and a volatility based profit target. The parameters I chose were derived from what I think about the time frames in day trading and my general assumptions about the nature of the markets, so the rule of "don't trade until X happens" came into play just 3 or 4 times for the whole system, which has 6 degrees of freedom (2 for the entry, 2 for the trailing stop and two for the profit target).

The result on the original market was very nice, until I incorporated the transaction costs, which are pretty high in the retail forex. But this is not the issue.

To my surprise, after I applied the system to daily stocks data, which was completely out-of-sample data (and remains so until now ), I saw the nicest thing ever in my system design experience. It looked great on every single stock out of 30 Dow stocks for the last 10 years of data. After combining that into a portfolio and applying 15%-of-equity-invested-per-stock-at-a-time money management, the equity curve looks so suspectingly brilliant to me..... I tried that also on some other stocks, and most of them were not only profitable, but had nice reward-to-risk profiles... What the heck?

The same goes for the 0.75%-of-capital-risked-on-the-initial-stop-basis money management (which is pretty much the same in results)and, most of all for the raw trades!

The system has expectancy of (65.09% * $1.68 - 34.91% * $1.05)/ 1.05 = $0.69 per dollar risked. The opportunity factor is 0.75 trade per day (the system did about 1900 trades during the 10 year period). The average trade takes about one week. Using various money management models, the average annual ROR is 4,75 times bigger than the maxDD trading 30 stocks! The system's signals are symmetrical and both sides of the market perform equally well. The system does not resemble the index fund in any way, the slope of the equity curve is steady in both the bullish and the bearish market. I guess the last two characteristics are more typical for non long term trend following systems.

I have not much experience with execution in stocks, but I have deducted $0.05 per share per trade for commissions and $0.015 for slippage. Is it realistic? What else could be wrong?

I would appreciate any comments or please tell me what data would be more useful for analysis. I can attach all the trades of the system, if anyone cares to have a look and do some numbers crunching.

Best regards,

Jakub

[damian]

Good stuff Jake.

I finished a system design phase the other day. It had a MAR of 3.6, win % of 48%, profit factor of 2.9. At least these were the results when I tested on a portfolio with no dynamic position sizing, just position sizing wrt each individual markets returns. I do this to get a feel for how each market performs individually under the system. The MAR of 3.6 etc comes from summing the individual equity curves. Position sizes in each maret are volatility matched however the position size for any single market is not a function of current capital derived from trading all markets to date in the portfolio. This is not how we trade in reality, unless you allocate 100k to each market and never substitute capital between markets. After 5 years CL might have turned it's 100k allocation into 400k whilst KC turned it's 100k allocation into 10k. The position size in each is independent of the others equity curve.

Anyhow, I proceeded to do a 'proper' portfolio test using dynamic position sizing and it was good bye to the high MAR. Overall return was obviously a lot higher, but the equity curve was shot in terms of shape. It did raise the question yet again in my mind: why should the KC position size benefit from the stella results returned by CL?

Sorry Jake, this was not really related to your current research, but I thought I would mention my what appeared to be too good to be true until I did some proper testing. Your instance of possible TGTBT looks far more promising.

[drm7]

Jakub,


Your system results sound very encouraging.

One other step that I would take is to look at the system performance in trending markets vs. choppy markets.

A lot of stock based systems look great on 10 years of data, because most of those years have had a steady up or down trend. Since your strategy seems to buy dips in an uptrend or sell rallies in a downtrend, your system benefits from the long trend periods.

The problem is that the next few years may not have long trends, and your system may not perform very well in choppy markets. (It might actually do very well, but you need to see for yourself)

Another way to test this is to try it on a number of commodities or forex pairs with different characteristics (i.e trendy, choppy, mix), to see if the system does equally well.

I hope this helps.


-Doug

[steady_jake]

Damian,

thanks for encouragement. Please explain what do you mean by CL and KC? I did test the system with some mm models I know and all look more less equivalent. I still have to code a volatility-normalized model, though.

Doug,

thanks so much for your ideas.

One other step that I would take is to look at the system performance in trending markets vs. choppy markets.

Actually, the system is sufficiently short-term to be independent of what the markets do medium to long term. Short term (say, 20-30 days) volatility, or range, seems to be a sufficient playground for the system to generate profits. I have attached a graph of the system's equity vs buy & hold for the whole market for you to see. It also depicts short vs long trades.

Another way to test this is to try it on a number of commodities or forex pairs with different characteristics (i.e trendy, choppy, mix), to see if the system does equally well.

This seems the most helpful to me. I have done it on 5 min Forex EUR/USD. Because of the pip value and relatively high spreads for such a short time frame, the system is not tradable, although it has good expectancy if transaction costs are dropped out. I think, it gives the system additional credibility, although, as I said, the system is not tradable because of the market specific arithmetics (what do you think?). I might try that on other markets, but I do not have sufficient data.

I have attached an excel file with trades.


Best regards,

Jakub

[Bondtrader]

Some backtesting packages try to prevent you from "making postdictive errors", and other softwares don't. A classic example of a postdictive error is to use today's closing price (or today's high or today's low) in calculations that ultimately affect the way you trade today. This often occurs when you accidentally use a moving average that include's today's prices, in deciding what to do today.

Many times when I've seen backtest results that are too good to be true, further analysis shows that a postdictive error has accidentally been made. Postdictive errors are possible in custom software, and in Excel, and in Trading Recipes. I don't think they're possible in Tradestation or Metastock, but I may be mistaken.

Another thing that can mislead you is fills on stop orders or limit orders. Just because your stop order (or your limit order) price is above the low and below the high, doesn't mean that you could get filled at that price. Prices gap all over the place, generally to the detriment of stops and limits. This is particularly important for systems that trade using weekly or monthly bars. The solution is to use higher resolution data bars in testing.

And the best-known reason for "too good to be true" results is Omega Research's infamous "bouncing ticks" mess in Tradestation. It attempts to guess whether the high of a bar occurred before, or after, the low. If you've got two orders and one-cancels-the-other, you need to know which one is filled first. Bouncing Ticks makes a guess. Often wrong.

[steady_jake]

Dear Bondtrader,

I think your input is very valuable. I agree that post dictive errors can be an issue if the system is not programmed properly. MetaStock is not free from this at all. Try to devise a system on the basis of the infamous ZicZac indicator, for example. The signals for trades made today disappear when new data comes tomorrow.
My system first checks the setup for a trade and enters at the market on the next day's open. Stops and profit targets are even further away from the last calculation in terms of bar data.

Jakub

[Forum Mgmnt]

Jakub,

Besides looking for post-dictive errors which are very possible in WealthLab, I suggest the following:

• Test using more years of data. Preferably 20 years of data using many different markets.
• Your commissions are actually pretty high. You can pretty easily get $0.01 per share each way for a $0.02 total per trade. Your slippage is a decent estimate but you may find that an occasional overnight gap will blow out your slippage estimates. Try the test using liquid NASDAQ stocks. The slippage on NYSE listed stocks can be higher than your estimates for non-Dow stocks.
• Damian was referring to CL (Crude Oil - Light) and KC (Coffee) futures prices

If you can't find any problems with the system's results, I'd start trading it (small at first). You'll pretty quickly find out if your actual results don't match the testing, especially if you run historical tests over the same period that you trade after a few months.

The results you indicate are not atypical for a good short-term system, so they don't strike me as necessarily "too good to be true". The key then becomes managing the trades and execution. For the thirty stocks in the Dow Jones Industrials, this is not a major problem. If you expand the universe this can be a management nightmare.

[steady_jake]

Forum Mgmnt,

Damian was referring to CL (Crude Oil - Light) and KC (Coffee) futures prices

Silly me.. What a shame... I was fixed on something referring to money management...

You'll pretty quickly find out if your actual results don't match the testing, especially if you run historical tests over the same period that you trade after a few months.

A decent approach, thanks.

The key then becomes managing the trades and execution. For the thirty stocks in the Dow Jones Industrials, this is not a major problem. If you expand the universe this can be a management nightmare.

Would you advocate to manage execution in a way described in your paper about turtle trading? And what amounts of capital can I trade with this system on unchanged portfolio without influencing the results too much?


Thanks, again.

Jakub

[Sabrinian]

steady_jake,

I do a lot of system proto-typing in WealthLab and have coded 4 similarly “Great” systems. Unfortunately, in each and every case I discovered I had created a post-dictive error. Wealthlab is quick, cheap, and, if you understand simple programming, very easy to use. But it is VERY, VERY easy to make these types of errors. As far as I’m concerned, everything I do in WealthLab is suspect.

You may have already checked your code against the possibility of using information you would not normally know before hand. However, most of the errors I have made were the result of ORDER in which my entries and exits were coded.

For example, if you code

Line 1: Entry
Line 2: Profit target
Line 3: Trailing stop
Line 4; Protective stop

You will have a sort of Type 2 post-dictive error. In this example, bars where both your stop and profit target might have been hit, will always result in your profit target being hit BEFORE your stop. You would be assuming the best case scenario. Namely, that your profit target was always hit before you were stopped out.

So the first step is to try reordering your code. Try

Line 1: Entries
Line 2; Protective stop
Line 3: Trailing stop
Line 4: Profit target

If you still get a great curve, you are really looking good because now you have assumed the worst case scenario! The next thing to check is the PriceSeries related code. It’s very easy to make errors here. Personally, I never use them anymore because of this issue. If I need to calculate the ATR I’ll use

Code: Select all

MyATR := EMA(Bar, TrueRangeSeries,15);

inside the bar loop rather than use the PriceSeries functions. So, your next step is to recode any Priceseries code (if you have any) into the actual BAR loop rather than doing it before hand.

Usually, these two steps will change my EQ greatly (Boo, hoo) . If you still have a similar EQ, then you are lookin’ real good!

If you have TradeStation, I would suggest recoding in TS. TS pools orders and then either use intraday data to determine the order in which your various orders were executed, or it generates a semi-random sequence to determine which was hit first.

Sabrinian

[gfullmer]

[quote="Sabrinian"]I do a lot of system proto-typing in WealthLab and have coded 4 similarly “Greatâ€

[LeapFrog]

[quote="Sabrinian"]steady_jake,

I do a lot of system proto-typing in WealthLab and have coded 4 similarly “Greatâ€

[BARLI]

I found that testing with market orders can never have "peeking", unless you put your trade on yesterday's bar knowing what it was Happy Holidays to everyone!

[Tortoise]

I found that testing with market orders can never have "peeking", unless you put your trade on yesterday's bar knowing what it was Happy Holidays to everyone!

Some scripts on the wealthlab site have logic along the lines of "if todays close is greater than today's open, then buy at market" - a classic example of forward 'peeking', the "buy at market" becomes a buy at today's Open dependent on a future (end of day) close.

[BARLI]

thats right! This is if you forgot to put (bar+1,'') to buy at market tomorrow, otherwise that'd peek and would give you gazillions in your backtest

[RedRock]

This post from the past has me thinking. I wonder if Steady Jake traded his system and what the results have been in real time? Also, with many of us going longer and longer in time frames seeking the whale trends... What success have any of you had trading shorter term swing systems with the average trade about a week as described by Steady Jake. My experience has been to find a system which works great for a few years at a time, then begins acting exactly opposite. Goes from cant loose, to cant win. Anybody have success in the short term consistently?

Curious rr

[BARLI]

one of my short term system did very well on Crude Oil and Soybeans, but did bad on Currencies.

[RedRock]

one of my short term system did very well on Crude Oil and Soybeans, but did bad on Currencies.

What caught my attention with Steady-Jakes stated system, It seemed very robust, although not sure if he tested on futures etc or only stocks. Was profitable even to intraday, though couldn't beat its costs intraday he said. c.f. commented that it wasnt unusual for a shorter term system to perform in... Well, he said:

"The results you indicate are not atypical for a good short-term system, so they don't strike me as necessarily "too good to be true". The key then becomes managing the trades and execution. "

[BARLI]

in order to trade a short term system(1-2 days hold) that I've created the backtest shows that the comissions should be as those of the exchange members or its better to be the brokerage house yourslef(when it gets to 1000 contracts a day). Now I understand why such guys as Monroe Trout or Mark Fisher have their own brokerages/clearing companies.With a holding period they trade and size they do they pay less than 1$ per contarct