Parameter Assignments and Optimization

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Parameter Assignments and Optimization

Post by al2000 »

I am a new member, and this is my first post. My question concerns using the Turtle entry system with the same parameter values for all stocks or optimizing on individual stocks. For example, suppose in backtesting, you find that XYZ gives historically best results using 55 in/20 out. However, YXZ gives best results using 40 in/15 out. ZYX gives best results with 25 in/10 out, etc., etc. By "best results" I mean as measured by net profits, MAR, CAGR, RAR, W/L, etc., over a 10+ year history. Is it considered overoptimizing or over curve fitting to use unique parameter values for each stock you trade? I'm asking because I've noticed in system development research that stocks have their own "personality" and don't all behave the same using the identical parameter values as others. So, why not have unique parameters for each stock? In fact, why use the same parameter values for all stocks if the worse-performing ones bring down the better- performing ones? The underlying assumption here is that the parameters for each individual stock are already fairly robust. In other words, for XYZ, 55/20 works just as well as 50/15 or 60/25 or 48/18 or 63/28, etc., but performance degrades when the parameters approach 40/15. Your trading portfolio is only going to be made up of 5 or 6 stocks anyway.

I've often wondered about this, especially after reading about fractal efficiency ratio (FER), which describes a situation in which some stocks simply don't behave well in any system and others do extremely well (in backtesting). FER, which apparently was coined by Perry Kaufman, states that a stock has to have some non-random movement to be predictable. It's the total change in price over a given period, divided by the sum of the absolute values of all the daily changes in price. If a stock has too small a directional component, then it's a poor candidate for any system, regardless of how many filters or refinements you add. You're better off using all that firepower on a better target. The FER can be likened to a given stock's "personality."

I'd be very interested in anyone's insights in this matter. Thanks a lot.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

al2000, First, welcome fo the Trader's Roundtable.

You bring up a very interesting question that has been discussed here before but not specifically as it relates to stocks.

The reason I think this subject is so interesting and seems to be confusing to many of us is because there are really several interlocking and related issues that come into play. I'll lay out what I consider to be the separate issues that are relevant to you question:
  • Curve Fitting - Curve fitting is about mapping a complex system onto too little data so that the system is fit to the test data specifically and is not likely to perform well using other data, especially that set of unknown data I'll call "the future".

    I often find it useful to think in terms of extreme examples. Consider a test of one market that has one very large trend where the optimal entry is a 39 day breakout and with an exit of 14 days against. Let's say this single trend represents 80% of the profitable trades for the system.

    This single trend dominates the test results so any changes to the system that improve the results for that one trend will improve the overall results significantly. Noticing this, you examine the big trend and you see that just before the market broke down it made an 8 day low and then attempted to make a new high coming within $0.25 of the previous high.

    You decide to add a new rule to the system which says: "If the market makes a new 8 day low and then subsequently comes within $0.50 of the high, exit on the close. This new rule greatly improves the results.

    The question then arises: What confidence can you have that the system will perform as well in the future for that market? Since the results have been dominated by a single trend it comes down to the question of how representative that trend is of the likely trends that may develop in that market in the future. Since we are talking about a single very large trend, the answer is probably very little confidence.

    Now consider the opposite extreme. You have a system tested on the entire NASDAQ 100 over 25 years. No single trade accounts for more than 1% of the profits in any given year. No single market accounts for more than 5% of the profits in any given year.

    Now ask the same question: What confidence can you have that the system will perform as well in the future for that market? The answer in this case is certainly a lot more confidence than in the previous case.
  • Market Differences - There are definitely differences in the behavior of different markets and market sectors in the stock market. So it does seem to make sense that trading markets differently will work better than trading all the markets the same.

    The main problem is that you need to make sure that you are not biasing your results using information that would not have been available at the time of your trades.

    Again, an extreme example might serve to illustrate the danger here. Suppose you run your system over all the stocks in the NASDAQ Composite for the last 20 years and then pick the top 20 performing stocks as your trading portfolio. Let's say for the sake of argument your results end up being 150% CAGR.

    How likely are you to get these results in the future? The answer is not very. How likely is it that Microsoft's stock will perform in the future as it did in the last 20 years? Cisco's stock?

    So if you are going to select markets you need to do that using information that is only available before the trades take place. You can then test this portfolio selection algorithm as part of the testing simulation.
  • Statistical Significance - There is a statistical concept called Statistical Significance that is important for testing. The main idea being that one can be more confident in the conclusions drawn from a larger number of samples than those drawn from a small number of samples. For testing, this means more trades are better than less trades. Optimizing stocks individually suffers from generally smaller sample sizes. A very small sample often has no predictive value.
  • Representative Sampling - There is another statistical concept called Representative Sampling that is also important for testing. The idea here is that one can be more confident in the conclusions drawn from a sample that is representative of the entire population than those drawn from a sample that is not likely to represent the entire popuplation. For testing, this means testing over a few years (say the late 1990s in the U.S. stock market) is not as good as testing across many years and overall stock market conditions. A test run over unrepresentative markets has little predictive value.
I hope this begins to answer the question you asked.

- Forum Mgmnt
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Parameter Assignments and Optimization

Post by al2000 »

Thanks, Vince, for your reply. From a purely technical standpoint, the Turtle system, as I understand it, is simply a breakout system in which a buy or short signal is given after a consolidation and after a theoretical loss. So, if I ask myself why do the rules work, the answer is simply that they supposedly capture the beginning of a trend that the market has signalled through the breakout from a sideways consolidation. I have no idea why a 20-day breakout may work better for one stock and a 55-day for another other than differences in "personalities" of the stocks. Admittedly, a large, fast move up could take out both the 20-day AND the 55-day highs at the same time. But I don't think this will happen that often. So, that leads me back to my original question. A stock that responds better to a 20-day breakout may simply be more (or less) volatile than one that works better on a 55-day. Maybe I'm not following your point.

Regarding your comment about examining a graph of a stock that rockets up vs. one that gives a nice, moderately advancing trend both appearing exactly the same, I presume you mean the equity curves, right? I have not observed what you have. In my system development, using the same system for a number of stocks, I get hugely different equity curves for each stock (not the Turtle system). So, maybe I don't follow your second point, too. Perhaps you could elaborate a bit? Are you saying that, once a system has been developed, all stocks should give extremely similar equity curves in backtesting?

Thanks a lot for taking the time to reply.
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Parameter Optimization and Assignments

Post by al2000 »

c.f.:

Very interesting and appropriate reply. It raises some more issues I'd like to expand upon. First of all, in regards to your point about curve fitting, your extreme example where your system gives you one big R-multiple profit that significantly influences your total profitability is a point well taken. However, one of my assumptions that I failed to mention but should have when I first articulated my question was that the distribution of profitable and losing trades was such that there were a statistically significant no. of each to give you confidence in your optimization. I would never use optimized parameter values that were based on only one or a few trades. That would, indeed, be folly. Your other extreme example was more in tune with what I was getting at.

Second, regarding market differences, you gave an example of a trader running his system over the entire NAZ for 20 years, then sorting on the most profitable stocks, and picking the top 20 to trade. That is not exactly what I meant. What I would do is take your example one step further. After sorting on net profits, I would then pick the top 20 stocks and optimize each separately over a period having both bullish, bearish, and sideways properties, such as the period 1/2/98 to 12/31/01. After getting the optimum parameter values for each of those 20 stocks over that bull/bear/sideways period, I would then further test on the out-of-sample (OOS) period from 1/2/02 to present time. That would tell me if my system were robust in regards to each of those stocks. No doubt, some of the stocks would fall apart, and you might wind up with only 4 or 5 worth trading. This is what I was getting at with my question. I think OOS testing is crucial for determining whether a system should or should not be used in real trading. Perhaps I shouldn't have used the Turtle breakout system as my example (which gives few big winners and lots of small losers) but rather a shorter term system with an average trade duration of 3 or 4 days. That way, you get many more trade opportunities upon which to make appropriate judements on the validity of the system and its expectancy. So, my original question relates to that scenario. Am I making any sense? I'd appreciate your insights in light of this expanded explanation. Thanks again for sharing your experience and expertise.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

al2000 wrote:What I would do is take your example one step further. After sorting on net profits, I would then pick the top 20 stocks and optimize each separately over a period having both bullish, bearish, and sideways properties, such as the period 1/2/98 to 12/31/01. After getting the optimum parameter values for each of those 20 stocks over that bull/bear/sideways period, I would then further test on the out-of-sample (OOS) period from 1/2/02 to present time. That would tell me if my system were robust in regards to each of those stocks.
Hmmm, first, I commend you on the idea of using an out-of-sample test to see if your idea works. I'm an empiricist so I say: why not? Test this idea out. Test any idea out that you think will work. Don't take my word for it or anyone else's. I've often found the conventional wisdom to be off, sometime by a lot, sometimes by just enough to be dangerous.

I suspect you'll find that single set of parameters tuned across all the markets will perform better in the out-of-sample testing than the market-optimized parameters. I've done some of this kind of testing and that's what I've found. I've also heard of some similar testing done with some commercially available systems that had the same findings.

I don't think that trading the top 20 stocks based on their performance over a number of years will work very well as a strategy. My own testing has indicated that while performance in the shorter term (i.e. less than a year or so) is indicative of better trading markets, that doesn't hold up if you look back over many years. However, as with many things, conclusions draw in one circumstance may not apply in others. For instance, my testing was done using long-term and medium-term systems, my assumption might not hold for short-term (less than 5 day average hold) systems.

Let me know what your testing reveals.

Finally, I would be very careful to make sure that you test the system over different market conditions. It's very easy to make a short-term system that works well for a few years and then doesn't work well at all after that. This will make sure your testing doesn't fall prey to unrepresentative sampling. Many short-term system testers make this mistake and then don't understand why their systems stop working.

- Forum Mgmnt
Vince
Roundtable Fellow
Roundtable Fellow
Posts: 61
Joined: Wed Jun 11, 2003 12:40 pm

Post by Vince »

suspect you'll find that single set of parameters tuned across all the markets will perform better in the out-of-sample testing than the market-optimized parameters. I've done some of this kind of testing and that's what I've found. I've also heard of some similar testing done with some commercially available systems that had the same findings.
I'd second that motion!
You've got to think about big things while you're doing small things, so that all the small things go in the right direction. Alvin Toffler
Vince
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Post by al2000 »

My own testing has indicated that while performance in the shorter term (i.e. less than a year or so) is indicative of better trading markets, that doesn't hold up if you look back over many years. However, as with many things, conclusions draw in one circumstance may not apply in others. For instance, my testing was done using long-term and medium-term systems, my assumption might not hold for short-term (less than 5 day average hold) systems.
c.f.:

Your comment above raises another question. If performance in the shorter term is indicative of better trading markets, does that suggest that re-optimization of one's system parameter values should be done on a relatively frequent basis (say, every year, or perhaps even every six months) to stay in tune with changes in market conditions or personality? Again, I'm assuming here a shorter term system rather than a long-term trend-following system.

Thank you, Forum Mgmnt, for your insights. They help a lot.
Vince
Roundtable Fellow
Roundtable Fellow
Posts: 61
Joined: Wed Jun 11, 2003 12:40 pm

Post by Vince »

Al,

Re-optimize on the day that human nature changes immutably and not before.

Vince
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Reoptimising

Post by Kiwi »

Hi Al,

I would suggest reluctant reoptimisation of even short term systems. By all means evaluate them every 6 months or, based on my experience, when the markets seem to have changed.

I think you need to understand how your system extracts money from the market to do this well. This will allow you to see how a new parameter set changes your mining tool from the old one (is the shovel wider, do you take it out of the stream quicker). Then you can ask "has something changed about the distribution of gold in this stream that means that I should now dig this way ... and will it be likely to be true later on this year?"

If you change too quickly and easily you will be like the discretionary trader who keeps changing indicators.

Human nature doesnt have to change immutably but you are looking for a change in human behaviour that is likely to continue for a useful period.

John



PS There can be an exception to all this caution. Your mining may be sensitive to small changes in some parameter (decrease in daily move or some such; it might be different in summer to the rest of the year) which does move around over measurable periods. In that case you may even want to optimise monthly for the next month - but you really need to understand why! Many will tell you that, instead, you should design your system with adaptive parameters.

PPS I would suggest noting c.f.'s comment about making short term systems that only work for a few years. Select test periods with a range of market conditions and make sure you can be profitable or breakeven in all of them or you'll join the ranks of those system vendors who are despised by their customers.
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Post by al2000 »

Point well taken, Kiwi. Thank you.
ssidley
Contributor
Contributor
Posts: 1
Joined: Thu Sep 11, 2003 5:25 am
Location: South Africa

how often to optimize, and a comment on OOS

Post by ssidley »

Kiwi wrote 'Human nature doesnt have to change immutably but you are looking for a change in human behaviour that is likely to continue for a useful period. '

This is the rub. Because you dont know when it is that trader behaviour is going to change (until it is too late, and your system is losing money), does it not make sense to re- optimize every six months or so? If trader behaviour has changed, your coefficients will change. If not, no harm done.

On the subject of OOS, I would suggest that any system which does not do OOS testing is on shaky scientific grounds. In our case, we have rolled coefficients forward to OOS periods a year in front of the in-sample periods, sometimes iterating 10 times over the last 10 years, and have found, to our despair, that most of our systems collapsed in OOS periods, indicating over fitting. Better this than finding out after you start trading.
al2000
Full Member
Full Member
Posts: 12
Joined: Sun Sep 07, 2003 5:25 pm

Post by al2000 »

Good point, SSidley. I suspect that, for long term trend-following systems like the Turtle breakout system, one might not need to re-optimize as often since the time horizon is longer. However, for shorter term, counter-trend type systems or even short term trending systems (3-4 days duration), re-optimization every so often (6 months to a year) might be the prudent thing to do. Regarding OOS testing, certainly, I agree that OOS testing is an extremely important part of anyone's arsenal and should not be overlooked.
trend456
Contributor
Contributor
Posts: 3
Joined: Mon Oct 27, 2003 1:05 am
Location: Chicago

Post by trend456 »

I don't think there's a difference between optimizing long-term and short-term systems except that the difference would be how long it'll keep working.

This is my personal opinion but the hard thing about optimization is knowing when to quit or when to re-optimize. Whatever your choices are, you need to quantify when to reoptimize or quit. Having a systematic plan on when to re-optimize or quit will help a lot.

Just my 2 cents.....

Good Trade!!!

Trend :)
trend456
Contributor
Contributor
Posts: 3
Joined: Mon Oct 27, 2003 1:05 am
Location: Chicago

Post by trend456 »

Forgot to add that the above statement was based on short-term systems.

Good Trade!!

Trend :oops:
Post Reply