In Sample vs Out of Sample Run

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
rbuck
Contributor
Contributor
Posts: 1
Joined: Thu Dec 17, 2015 8:26 pm

In Sample vs Out of Sample Run

Post by rbuck » Sat Dec 19, 2015 12:27 am

The August 2015 market pull back provides a challenge once again to try to develop a market timer that would have pulled one out of the market in time to avoid loosing money. The NASDAQ Summation Index developed by subtracting new lows from new highs may provide the needed heads up. However, when committing the approach to a set of rules I have been, so far, not seeing much success with OOS results.
The approach used was to develop two mechanical strategies and try to marry them, hoping to get the best of each, not the worst of each. The watch list consisted of 14 Vanguard ETFs and the strategy held a maximum of one position. It was trained on 18 months of data, then applied to the next 6 months. The time frame was then advanced 6 months and the calculations for parameters was repeated. Optimizing over the whole test period of 9/30/2010 through 12/18/2015 produced a annualized return of some 19% with a maximum equity draw down of some 8%. But submitting the approach to the 18 month train, 6 month apply walk forward only produced a 5% annualized return over the OOS period of 9/30/2010 through 9/30/2015. The OOS maximum equity draw down was some 16%, trading at the next open with a $10 per trade cost. The approach resulted in a 63% exposure during the OOS periods.
The real disappointment was that the last OOS application period of 3/31/2015 thru 9/30/2015 had a 15.36% DD, essentially matching the DD of the OOS period 3/31/2011 through 9/30/2011.
Hopefully, some progress will be made down the road.

Tim Arnold
Site Admin
Site Admin
Posts: 8500
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Re: In Sample vs Out of Sample Run

Post by Tim Arnold » Sat Dec 19, 2015 8:53 am

Thanks for sharing -- keep us posted on your progress.

fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Re: In Sample vs Out of Sample Run

Post by fab1usa1 » Thu Dec 24, 2015 10:44 pm

Hi rbuck, I agree that the market began to take a nosedive in August of this year. I am a swing trader, not a trend-follower, so I don't know how much of what I say can be applied to your systems. One of my system's inputs is the "mid-line factor", a number in the range of 0 to 1 where 0 says "be conservative" and 1 says "be aggressive". In swing trading, "aggressive" can mean "go long on a minor pull back" and "conservative" can mean "wait for a deeper pullback". It is a floating point number so my system can become increasingly more aggressive or less so.

Back-testing with a fixed value of 1.0 produced impressive results over the past 15 years until of course we got to August of this year when it began to fall apart. After performing a simple stepped test on "mid-line factor" I discovered that a value of 0.75 was a good compromise although the overall return was lower. I started to think that I could improve on it by adjusted the value of "mid-line factor" as a function of an index.

I've been trading systems similar to this for a number of years and observed that they are particularly attracted to small-cap stocks so naturally I chose the Russell 2000 index. Next, I calculated the 2-period RSI and then applied a 100-period simple moving average. That gives an RSI value in the range of 0 to 100 but practically speaking the value bounces between the extremes of 35 and 70. Next, I apply a function that scales RSI of 35 to a mid-line factor of 0.30 and an RSI of 70 to a mid-line factor of 0.95. I discovered those function coefficients through the use of a multi-step parameter test. I chose the values for "slope" and "intercept" that sat in the middle of a vast plain of high MAR values, figuring that it is the most robust.

I have attached an Excel graph of mid-line factor as a function of the Russell 2000 and the coefficients of the scaling function. I am happy to say that it significantly boosted the CAGR when compared to a fixed value of 0.75 while maintaining high MAR and low DD.
Mid Line Calculation over 2000-2015.jpg

bluefightingcat
Roundtable Knight
Roundtable Knight
Posts: 339
Joined: Fri Nov 20, 2015 3:09 pm

Re: In Sample vs Out of Sample Run

Post by bluefightingcat » Tue Dec 29, 2015 4:17 am

Hi,

This is a really interesting thread. I have long been looking for a market timer to help me reduce my drawdowns. I currently trade US stocks and fab1usa1 your system seems interesting. I will have to look further into it.

You say you are a swing trader. How long do you hold your stocks for on average? I assume you only enter the market when your market timer allows it but once you are in the market do you keep your position until your exit signal comes in or do you immediately exit your positions if the market timer indicator goes against you?

fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Re: In Sample vs Out of Sample Run

Post by fab1usa1 » Tue Dec 29, 2015 8:16 am

Average Trade Duration is 5.26 days but I've had some trades close the very next day. I'll have to run some stats on it to give you standard deviation. The market timer never completely takes me out of the game but it does affect two things when the indicator is bearish: (1) it limits the number of candidate stocks, (2) it exits a position after just a modest reversal. For example, in today's lousy market the system can find only one or two candidates out of a population of 28,000 and if it should take a position it will close only after a modest rise in the 2-day RSI. As the market becomes more bullish the portfolio manager will find more candidates (up to a maximum of 10) and if it should take a position it will close after an even greater rise in the 2-day RSI.

fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Re: In Sample vs Out of Sample Run

Post by fab1usa1 » Tue Dec 29, 2015 8:44 am

I can't think of a good reason to ever take yourself out of the market no matter how bearish your indicator. It's good to have some skin in the game but keep your exposure low. With regards to making a market timer work with trend following may I suggest looking into setting conservative profit targets for open positions. And for entry, if you are using breakouts, wait for a 2nd or 3rd retest before entering. In a bullish market you can take the initial breakout and then set higher profit targets. These are just my thoughts. Thoroughly backtest it.

bluefightingcat
Roundtable Knight
Roundtable Knight
Posts: 339
Joined: Fri Nov 20, 2015 3:09 pm

Re: In Sample vs Out of Sample Run

Post by bluefightingcat » Tue Dec 29, 2015 8:51 am

Ok. So is your mid-line factor calculated per stock or for the entire market? Does each stock have it's own mid-line factor floating value or does e.g. the entire SP500 have just one mid-line factor floating value?

If I understood correctly you use the mid-line factor defines whether you enter a position and if you enter how long you stay within the trade.

If found an interesting article here that does something similar with the Rate of change the an indicator called TSI. http://systemtradersuccess.com/two-dime ... nt-filter/

You can find the tradestation easylanguage code at the end of the article.

I plan to look into it and see whether I can implement it in my strategies. My strategy is a swing trade strategy which is based on mean reversal.

fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Re: In Sample vs Out of Sample Run

Post by fab1usa1 » Tue Dec 29, 2015 9:18 am

Mid-Line Factor describes the overall mood of the market. In my case I've chosen it to be a function of the Russell 2000. For individual stocks, the key indicator is the Mid-Line which is the SMA(100, RSI(2, instrument.close)). Now, multiply the stock's Mid-Line by the market's Mid-Line Factor. That is a key level for determining when to get out of a trade, and it is just one of several operands used for determining the list of candidates for entry.

Thanks for the link, I'll check it out.

Post Reply