Number of optimized variables and ranking of system goodness

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Number of optimized variables and ranking of system goodness

Post by RedRock » Sun Oct 22, 2006 6:00 pm

I am wondering about an index or indicator, which takes into account the number of variables a system contains. Should a systems measure of performance be adjusted to compensate for the number of optimized variables required to achieve the back tested result? Based on the belief that a higher number of curve fit variables is inferior or less likely to replicate in the uncertainty of the future. Thoughts???

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Re: Number of optimized variables and ranking of system good

Post by sluggo » Sun Oct 22, 2006 6:55 pm

RedRock wrote:Based on the belief that a higher number of curve fit variables is inferior or less likely to replicate in the uncertainty of the future.
How might you go about testing this belief, to determine whether it is true?

If one is contemplating using Blox Builder to test this belief, an obvious question is: does this belief make correct predictions for the ten trading systems that come pre-supplied with the software? These systems have a nice variety of #variables which control their actions:
  • ATR Breakout: 4
  • Bollinger Breakout: 3
  • Bollinger Counter Trend: 5.5 (counting a true/false variable as 1/2)
  • Donchian: 8
  • Dual Moving Average: 4.5
  • MACD system: 4.5
  • RSI Trend Catcher: 5.5
  • Stochastic system: 12.5
  • Triple Moving Average: 6.5
  • Turtle: 10.5

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Sun Oct 22, 2006 7:35 pm

Sluggo,

Thanks for pointing out a superb starting point question.

A quote by c.f. from another current post sparked the thought initally.

"That's perhaps the best rationale for having very simple systems. We can have a much stronger basis for their continuing to work because the amount of truth required for their continued success is very low. In other words, we only have to be right about one or two things to be successful." CF

viewtopic.php?t=3326&highlight=

Statistically defining the validity of the supposition is now on my 'list'...

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Tue Oct 24, 2006 5:50 am

RR, nice thread. I taught from Mark Fisher to test only those situation where tarders are " caught" , so they rush and cover if they were short or vice verse. Sometimes such moves last for 2 weeks, sometimes for even months... as I see it, there are traders who have some biased opinions (ego is usually involved) and they keep on Shorting the market thinking fundamentals are in their favor and so on, at some point of time they might have profits on their P/L statements, but its only temporary, so they have lotsa shorts and even increase them, them some other guys come in with technical bias or some new fundamentals (the first ones dont know about) and start buying like crazy, first group keeps on shorting, their profits get erased in a week or so, they're still biased it'll go lower. then "intelligent" public joins and pushes it higher, first grooup sits in losses and ego doesnt let them cover (or those are hedgers who dont care cos they have their cash crop going higher so that offsets losses). Good example is Soybean market, my Chicago broker kept on saying how bearish fundamentals are and that kinda bull ****, same was with Wheat, he had 4k loss per contract in Wheat trade. Mark said that MACD and RSI are inferior indicators, check out the NYMEX symposium

http://www.clicklive.com/NYMEX/symposium_2003/

there were 3 days intensive classes about it

cramnhoj
Contributing Member
Contributing Member
Posts: 8
Joined: Thu Jul 06, 2006 5:27 am

Post by cramnhoj » Mon Nov 06, 2006 6:06 pm

Simply put, using the number of indicators a system uses as a measure of system robustness. I think it could give a rough indication but that there'd be a point where lines get blurred. The system's concept would best tell whether or not it's robust, it's easy to see if it's obviously not robust but the rest seems to be a grey area that can't be neatly boxed by order of robustness.

Also, sometimes things like portfolio diversification become the factor if a system's backtest result can be achieved in actual trading.

Just my thoughts.... :roll:

painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

robustness - just my idea

Post by painless » Tue Nov 14, 2006 11:26 pm

Just for a giigle I made up Blair's Law. It goes like this:
Since almost everything increase at a square.
The robustness of a tool decreases at the square of the number of its parameters.

So a tool with one parameter(p) = (p)^2 = 1
The holy grail has no parameters and thus = 0.
So a tool with 3 params is more than twice as bad as a tool with two. Of course none of this is proven it is just my way of thinking.
For the record I mostly use 2 but I am not counting stops or such because I see them as part of the money management part as opposed to the tool which just generates a market status like long or short.

jaym

Post by jaym » Thu Nov 16, 2006 10:32 pm

Hi,

Just as something can be overfit it can also be underfit. Both equally as dangerous. "Everything should be as simple as possible but no simpler." The number of tweakable or tweaked parameters are only one small part of the this question.

I think the best advice I saw in this post was sluggo's. You need to test the testing method like you test your "system".

Jay

danZman
Roundtable Knight
Roundtable Knight
Posts: 117
Joined: Thu Mar 10, 2005 10:07 am
Location: Newport Beach,CA

Post by danZman » Fri Nov 17, 2006 4:13 am

I have a stock trading system with a large amount of variables and it works REALLY well. Anyone else have a system that's not so simple, yet works well over time? From the posts I've read, most don't stray too far from the donchian, ATR breakout, moving average x-over type of systems. Anyone else go crazy with Trading Blox?

D

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Fri Nov 17, 2006 7:15 am

hi DanZ, I look more for price action such as OHLC relationship. What kinda indicators you use?

GENX
Full Member
Full Member
Posts: 21
Joined: Thu Oct 12, 2006 7:12 pm

Post by GENX » Fri Nov 17, 2006 10:12 am

BARLI WROTE
taught from Mark Fisher to test only those situation where traders are " caught"
BARLI,

I was wondering how do you identify those situations? I assume that it could be documented by being on the floor and watching the buzz and forlorn faces while monitoring how much ground a market moved in such and such a time frame. But for someone who has not been on the floor and is just looking at data from a screen, how would you identify those periods where traders were "caught"?

Is it safe to assume that a signal would be generated by the ferocity with which a market moved in such a short time, and then key in on looking for an extension of that move as more and more traders are figuring out that they are caught?

In other words:
IF KEY REVERSAL AND/or ATR < OR > STDEV OF LAST 10 BARS BY 3.5% THEN ALERT "UH-OH "

How would you know traders are "caught" and it was not just another valid break out? Are they one and the same?

I appreciate your insights. Forgive me if I ask something proprietary, you just sparked my curiosity when I perused your post. Thanks

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Fri Nov 17, 2006 11:16 am

hi GENX, its a proprietary study of Mark Fisher, I will bring you a simple example of that. Mark explains about it also in details in his book Logical Trader. He has a system called ACD, its based on this system. ACD uses opening range as a corner stone.

Here's the example of being "caught" in the March 2007 Wheat market:
Image

Usually, when the situation like that happens, Open for the day is also the Low of the day. Same applies to getting "caught" when traders were Long and had to exit their longs at the Open next day.

painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless » Fri Nov 17, 2006 11:37 am

Barli,
That is the dream system for me, something just based on a simple pattern. All my efforts at coming up with something have not been too good. But you give me hope. Do you think that is a good book to buy then and can you suggest any others in that specific field?
Thanks
Blair

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Fri Nov 17, 2006 11:52 am

I bought this one cos I read Paul Jones' review:

http://www.mbfcc.com/book/paul_tudor_jones.shtml

Paul and Mark been friends for a very long time, Mark taught traders his systems at Tudor. If you watched that speach of Jones at NYMEX 2003 Symposium, you heard that Paul came to speak only because Mark asked him to, that was kinda funny. Paul didn't reveal anything of his stuff of course. Another book I can suggest in this field is of Tom Demark called: New Market Timing Techniques: Innovative Studies in Market Rhythm & Price Exhaustion

painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless » Fri Nov 17, 2006 12:04 pm

Excellent! Thanks a lot :)
I just took a quick look at the idea. I have a tool I tested recently based on a similar idea but I was just using time and/or range and not factoring in volatility (that was on weekly bars), so I was having trouble pegging the size of the opening range. It won't be too difficult to test I imagine. I shall order the book(s) and try to bang up some tests over the next few weeks.

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Fri Nov 17, 2006 12:07 pm

how did you test time in your system?

painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless » Fri Nov 17, 2006 12:24 pm

Well since I wanted to start with a weekly one I used the last (n) minutes of Friday or the first (n) minutes of Monday.
The results showed a success rate greater than would be acceptable if that period was not significant. then I tried reading back (Friday) or forward (Monday) if the range was too small. Then I tried the same if it was too big. Then I decided that I was missing something because those results were not better. So I shelved it for a while and tidied up my code so I can just feed ticks at a time rather than minutes at a time.
Over the last few weeks I have been using my new version to update the tests using my existing tool, just a moving average based thing. Not much exciting or different but just to reassure myself.
Thanks to you and this website I will have another go at it. Using ATR or some measure of volatility sounds like a real good idea. Anyway better people thanme have it already sorted so if I can get something close then I will be happy with that.
Do you think it goes ok on weekly ?

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Fri Nov 17, 2006 1:54 pm

yes weekly works, monthly works, yearly works. it all depends on your time frame!

danZman
Roundtable Knight
Roundtable Knight
Posts: 117
Joined: Thu Mar 10, 2005 10:07 am
Location: Newport Beach,CA

Post by danZman » Fri Nov 17, 2006 9:51 pm

BARLI wrote:hi DanZ, I look more for price action such as OHLC relationship. What kinda indicators you use?
Price relationships...no canned indicators. I like look for gaps and consolidations within a trend.

D

GENX
Full Member
Full Member
Posts: 21
Joined: Thu Oct 12, 2006 7:12 pm

Post by GENX » Sat Nov 18, 2006 12:16 pm

Barli,
A picture is worth a thousand words as they say.
Thanks for the chart. I will have to check that book out.

Post Reply