When to change system rules

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
enigma
Roundtable Fellow
Roundtable Fellow
Posts: 99
Joined: Tue Apr 29, 2003 6:56 am
Location: UK

When to change system rules

Post by enigma »

Are there general rules that can be used in order to indicate when one should be changing the rules of one's system? I remember either Dennis or Seykota talking about finding better rules to change the rules in Market Wizards. Or is this a highly subjective issue?
edward kim
Roundtable Knight
Roundtable Knight
Posts: 344
Joined: Sun Apr 20, 2003 2:42 pm
Location: Silicon Valley / San Jose, CA USA
Contact:

Post by edward kim »

Ok Enigma, no one answered this one yet, so I'll take a stab at it:

In the purest sense, if you found better rules, trade the better rules.

Having said that, no one can predict how good a system WILL BE, so maybe you can slowly roll into the new system. For example, you can allocate 25% of the money into the new system, and keep the rest in your current one. As you evaluate the performance of your new system, you can increase the allocation over time until the old system is phased out.

For me, I have a very good feel for my own systems, and I've always dumped the old rules for the new ones right away. So in a way, it's part confidence and part knowledge of the inner workings of your system that make it easier or harder to switch.

Edward
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

All of this is said assuming that your rules are valid and riobust in the first place:

My dad (one of the people who's money I trade) is in constant agony trying to get me to change my system rules. I can understand why the system characteristics cause him so much discomfort... but I will not be changing the rules in the way that he wants. In fact I have openly invited any of my family and friends to take their money and run rather than ask me again to change the rules. What they do not realise is that they are talking about a different type of system, not just modified rules. I offered a complimentary system to trade in tandem with the current system, however we need more cash for that.

What is the point of this? Make sure you change the rules for the correct reason, and think 5 more times about it before doing it if you are in a DD. Also differentiate between changing rules that represents curve fitting the last year of trading ("but we should have exited those trades much earlier!") and modifying your rules to reflect a misunderstanding that you originally had about the original risk of the system.

Also differentiate between modifying rules and changing systems. Do you want to now use a 40 day MA rather than the 75 day MA you have been using? Do you want to use an XMA rather than an SMA? If all of this is based on the last year or so of trading then I think you are better of just stopping the system and starting again when you have a system that you have confidence in.

Also for thought:

Is changing your portfolio the same as/has more gravity than changing rules?

Is changing % risk/trade the same as/has more gravity than changing rules?
Last edited by damian on Fri Jul 11, 2003 10:30 am, edited 1 time in total.
enigma
Roundtable Fellow
Roundtable Fellow
Posts: 99
Joined: Tue Apr 29, 2003 6:56 am
Location: UK

Post by enigma »

damian, Edward ... thanks for the replies.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

I find this subject very interesting.

I like to run my tests over at least 20 years of data. I only trade systems that perform well when tested over the entire period.

Therefore I change parameters or systems when I find a set that performs better over the period than those I am currently using. What this means in practice is that for existing systems I tend to change the parameters only very slightly over time.

Consider, one new year of data may change the optimal results over a twenty year test, but the single year is only one year in twenty, so it is unlikely to change it very much.

If a 10 day breakout worked really well for 20 years but a 12 day breakout worked almost as well, one year in which the 12 day breakout worked much better than a 10 day breakout might throw enough weight on the overall results to cause a switchover to a 12 day breakout being better overall.

If the 12 day breakout worked poorly over 20 years, one year in which it worked well won't suddenly make 12 day breakouts the way to go.

As far as a new system, again I use the same long-term perspective. If I can find a system that is better over the same time period, if it correlates highly with a system I am already trading, I'll replace the old one with the new one as soon as I am confident in the results, because I have verified them by looking at the trades and the charts; have analyzed the system and believe it to be suitably robust; and have confidence that I can implement the system given my existing resources.

If the system doesn't correlate with a system I am already trading, I will add it to the mix of systems since its addition will improve the risk adjusted returns.
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

Hi Forum Mgmnt,

Regarding the 12 month re-assessment of rules and parameters:

I do not mean to sound possibly impertinent, but is the above what you would ideally aim to do, or what you have actually done over the years?

To me it seems like a rational approach and am interested if it has also proven to be a practical and productive approach in reality as well.

Also, as an extension, if trader were to speculate using 5 minute bars, they could possibly re-assess the system every month or so, perhaps sooner?

damian
Dutchtrader
Roundtable Fellow
Roundtable Fellow
Posts: 58
Joined: Wed Apr 30, 2003 4:35 pm
Location: Netherlands

changing rules.

Post by Dutchtrader »

One af my mental rehearsals of trading is that I try to imagine what it is follow your system day after day year after year while in the mean time I am learning new trading techniques and mm algoritmes you never tested/ heard before and then find out that you are using old fashioned rules....

I think it is very hard to decide "half way" to change your rules. Even when when it's part of the plan I think that I will have big mental problems with the timing of changing. Do you have a fixed date for changing? I won't probably change when I am in a big DD and nor when my EC is skyrocketing.

Changing the rules during the years maybe a good/technique but I won't feel comfortable with it....

Just my 2 cents,

Marc
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Damian,

First, I never changed systems when trading as a Turtle since we didn't have the ability to test like we do today.

What I proposed is what I actually do, and suggest.

However, I was using a year only to illustrate one possible timeframe which could provide data which would indicate that different parameters worked better. I was not suggesting that this analysis only be done on an annual basis.

I make these decisions when I have enough data. After doing research on a system, you reach the point where you understand the system and have enough data to make an assessment of its effectiveness. At this point, I decide to include/trade the system or to discard it because it does not improve my overall results, or is too similar to a better system.

These decision points might come once a month, once a quarter, or once every two weeks. It all depends on how active I am in trading research.

I don't think there is any problem with running tests on every system you are trading once a month to see if the dynamics of the system have changed. If you have a robust system, they will only change very slightly, but if the results indicated that the small change improved the system, I'd change the system at that time.

Remember, I'm talking about testing using a lot of historical data and running a system where the parameters are constant across the entire portfolio for over 20 years, so one month is a very small fraction of the test period, therefore it is unlikely that a single month will cause large changes in the optimal parameters. But if it does, why not trade the new numbers?

Also, optimal parameters are not necessarily those that produce the best results. One should also consider the shape of the curve and where the optimal point appears in relation to that curve.

So if you have results of say:

Code: Select all

Parameter               MAR
---------               ---
10                      1.2
11                      1.6
12                      2.4
13                      2.3
14                      2.3
15                      2.2
16                      2.1
I'd probably trade the 14 value because the parameter value which produces the best results, 12, is very close to the edge where the values fall off dramatically. If I trade with a 12 and the market shifts the curve, my risk-adjusted returns could drop off dramatically. However, if I trade with a value of 14 and the markets change to shift the curve slightly, I will be more likely to achieve results that are consistent with the testing.

This is a bit oversimplified since I don't consider only one test result value (i.e. MAR) when determining the "best" parameters.
Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann »

A pretty smooth example, which reminded me of this 2.5 month old prediction.
In post #882 made Wed Apr 30, Mark Johnson wrote:However, there are some wonderful advantages once you go to an all-dataset-encompassing measurement like Sharpe Ratio. One of them is, when you plot (Sharpe Ratio) versus (System Parameter X), you get smooth curves!! Anyone who has tried to plot (MAR Ratio) versus (System Parameter X) can testify of the frustratingly choppy, spiky, discontinuous curves that MAR plots often produce.

Here's a prediction: I predict that if someone uses c.f.'s VeriTrader to make a plot of (MAR Ratio) versus (Number of days in System Two breakout), that plot will NOT be smooth. But a plot of (Sharpe Ratio) versus (Number of days in System Two breakout) will be smooth. How's that for a testable, falsifiable hypothesis?
So, was he right :?: :roll:
Sir G
Moderator
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Post by Sir G »

Are there general rules that can be used in order to indicate when one should be changing the rules of one's system? I remember either Dennis or Seykota talking about finding better rules to change the rules in Market Wizards. Or is this a highly subjective issue?
This is an area that I have targeted to find the answers for myself. When I was active on Chucks forum about two years back, I recall someone with the name of yingyang was quite adamant about the virtues of re-optimizing. As usual, most everyone jump on him for that belief.

My first thoughts also were that he is mistaken, fortunately a second thought came rapidly to me that was… “how do I know if it works or not? I have never tested it.â€
Post Reply