Confirming Robustness

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Sir G
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Confirming Robustness

Robustness. This just might be the mother of all topics.

How does one "confirm" robustness? Is there a way to quantify it?

Here is my take on it. Among other things, I run my logic through as much data as I have available. I use the same parameters on all the markets, I then expect all sectors, if not all markets to show consistency.

That is probably one of the reasons why I have put much focus on Short Term trading. Because I can get a ton of trades generated. I'm more prone to believe the results of 30,000 trades that span 30 commodities then 1,000 trades from the same data based on a long term system.

This way of doing it is more an art form. How do others do it?

Inquiring minds want to know.

Sir G

Forum Mgmnt
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:
Hmmm, by robustness I assume you mean probability of continued effectiveness and consistency of results under the most varied markets and circumstances.

We never optimize individual markets. We only optimize over the entire period of our complete data set using a single set of parameters. We might do initial testing on a 6 to 8 year sample for more rapid turnaround but only for rough tuning.

From a purely statistical perspective, I don't think you get significant confidence increase going from a sample of 1,000 to 30,000. The really large gains come from going from 20 to a few hundred. While trades admittedly do not exhibit the behavior of normal distributions, the statistics for them are still reasonably good yardsticks.

Confidence intervals use factors of Standard Deviation / Square Root of (sample size).

This means 1/171.2 versus 1/30.6 for your example. Even at a sample size of 1,000 this is a pretty low factor 3% of the standard deviation for two intervals gives a 95% confidence level.

To sum up, I think a sample size in the low thousands still gives pretty good confidence. I'm usually more concerned with the other sampling problem, the diversity of the sampling areas and their being characteristic of the entire universe.

Consider a poll taken with 20,000 people all of whom located in a poorer neighborhood in Washington D.C. The results of this poll on almost any subject won't apply to the U.S. in general and certainly not to the world.

To me this problem shows up most often when people focus only on sample sizes but neglect to test over all the years of data they have. For example, someone testing a system against stocks using the years 1997 through 1999 with a short-term system might have 10,000 or more trades and hence a very large sample size but they would be missing the fact that all the samples were taken in the same type of market conditions. They'd also be in for a rude awakening a few months later as they started trading their "very profitable" system. (a bit like moving from D.C. to the bible belt)

I think that testing over at least 20 years, with single parameters across all markets is about as good as one can do....unless you can find 30 years of data

- Forum Mgmnt

P.S. There does seem to be a tension between "robustness" and profitability in the shorter-term. When compared to "robust" system, systems that are optimized to slightly favor the recent past will perform better over the next few months most of the time. However, they exhibit a greater tendency to work significantly worse some of the time.

Maxwell Cintes
Full Member
Posts: 14
Joined: Wed Apr 16, 2003 10:08 pm
Location: Hartford, CT
In reading Bill Eckhardt's interviews, I thought this was one of the more important aspects of trading. Why haven't any of the more experienced traders answered?

NickR
Full Member
Posts: 12
Joined: Mon Apr 21, 2003 9:09 pm
Sir G, I'll put a differnt spin on "robustness" which may be shot down in flames. To me robustness is understanding exactly why the system makes money. This, in my view, is easy with trend following. All markets, at some stage, will trend. You simply need a mechanism to capture these trends and a basic breakout will do exactly that. The premise of trendfollowing is straightforward math; how much you win when you win and how much you lose when you lose. A trend follower will have a high win/loss ratio, say > 2. Most type of basic trend systems have win percentages of around about 40%, hence the positive expectancy.

In essence its the underlying philosophy that markets will always trend and a trending market will always give you a higher win/loss ratio. This concept is why its robust.

We can then break that philosphy down and compare longer term trends to shorter tremd trends etc etc but the underlying robustness will always be there. I recently started tinkering with my stop levels on some short term systems. By tightening the stops right up you start to get the same type of stats as you do in a trend following system; that is, a high win/loss ratio ( > 3), a decrease in win rate (30% - 40%), and in a single market system, a decrease in drawdown.

In laymans terms, my definition of robustness is understanding that profitability will come from increasing the win/loss ratio almost regradless of the win percentage (and certainly regardless of how the numbers are generated).

N

bloom
Senior Member
Posts: 35
Joined: Thu Apr 17, 2003 12:45 am
Location: SARS infested HONG KONG..ahh

assumptions and robustness

I will drop in my two cent..

I think robustness of a trading system is closely related with the assumptions under which the trading system is design.

The turtle /breakout system so robust because it 's so simple. Basically, the only assumption is making is that " if the market is going to trend up/down , It MUST make a new high/low" which is perfectly logical and this assumption is valid under any market and market environment.

Compare a breakout to say a system which use days of the week/month and limit trading only to these days. and the day of week assumption that works well on one market may fails miserably on other markets. This assumptions is not valid under most condition and not of a universal nature to be applied across market. To me, that's is not a robost system.

IMHO, A robut system makes very little assumptions about the nature of the market environment and use only very universal and general rules in its design

TK
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Choppy trends

The turtle /breakout system so robust because it 's so simple. Basically, the only assumption is making is that " if the market is going to trend up/down , It MUST make a new high/low" which is perfectly logical and this assumption is valid under any market and market environment.
While this is true, it does not automatically mean that all a trend-following system needs to make money is trends. Equally important is the quality of a trend. If there are huge corrections every now and then, a system may be stopped out many times and continually re-enter at less favourable prices. The market can go considerably higher/lower, and a trend-following system may still lose money.

So as far as I'm concerned, I would not worry if a system lost money in a choppy market, but I would, if it couldn't make advantage of a trend.

The question remains: how can you address the problem of "choppy trends"? Is it better to widen your stops or rather to trade multiple systems using different time frames?

damian
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

WILD system trades SP very well, intra-day. It can't trade ED very well at all. WILD is designed to profit from volatility. It has traded SP for a long time and is still strong. Real time results are much the same as backtesting results. WILD is simple and has few parameters, degrees of freedom, what evers. Optimisation shows that WILD works over a wide range of parameter values.

is WILD a robust even though it trades only one market?

Continuing the story......There is one catch to the WILD system that the trader is not aware of: if SP volatility drops below X, the system is useless. Last week such an event happened (hypothetical). Now it is 5 years later, 2008. Traders are talking about the swinging days of SP volatility that all ended in mid 2003. Since then SP has been tame and quite.....but it trends really well. Most traders added SP to their Aberration portfolio. WILD was ditched as a system after a 50% DD, this happened by late 2003. Some time in 2008 the trader of WILD reminds thinks about his old system and is happy that he no longer trades it. He reminds himself of the importance of a robust system.

Does a system have to work in all market conditions to be called robust?
can one say of a system "if <some market characteristic> remains high/low, this system is robust"

I have a very good surfboard, one of the best I have ever owned. This board rides so well.... so long as the waves are reasonably small and fast, other than that it is useless. A silly parallel, but a tool is a tool. Take a tool out of the purpose/environment it was designed for and it is no longer a relevant tool, just a bad idea.

Another angle of approach (after thought - I got a little carried away here, sorry folks):

(Creationists, please bear with me). It is some massive number of years ago. You are a monkey. You swing in the tress and eat ripe fruit generally safe from Saber Tooth Tigers and other such nasties. You have a few children monkeys and family life is well. As the years pass your forest, and all the ones around you begin to die as a result of bad weather. Not all is lost for you, but you are concerned for your children. You sadly die (sorry, it was in your sleep as an old content monkey). As your children grow up things get more bleak. By middle age they are in real trouble. No more fresh fruit and they have to search for water. They start to venture onto the plains. More time passes and generations come and go. During this period the ventures into the plains for food and water became more and more frequent. One day, your great, great, great,........, great grandchild is out in the plain with his family and decides not to go back to the trees that night. A Saber Tooth Tiger tries to attack him but he runs so fast he and his family escape. There is no way you could have escaped. Little does your descendent know, he is very different from you. His thumbs are in a different position, he stands nearly upright and that nice hair has gone. More centuries pass. The latest member of your family lives in a cave, has a fire and just noticed how useful a round object is to move heavy things. What has happened? You, as a monkey, were robust, but as the environment changed your descendents were not as robust, but it was not their fault. Luckily nature guided your descendents in an evolutionary path. They adapted.

Whilst you were in the trees, eating bananas and picking knits from your children fur, there was a funny creature swimming around in the ocean. You did not know this creature existed. Your descendent in the cave (the one who invented the wheel) had seen this creature. He didn't think anything of it besides the fact that it looked funny, tasted ok but was hard to catch. Today, your latest descendent (remember, you were the monkey) is sitting in front of a Pentium III flatscreen and have ADSL. It is a cold day he turns on the heater. Later, he turns on the TV and watches the nature channel. He is happy to find a special about his favorite creature, the turtle. This living dinosaur has not changed since the monkey at the top of your family tree was alive. Wow, what a robust animal...... so long as he is in the ocean.

(Side note: when I first read of 'The Turtles' I decided that the better reason for the naming was because they are slow, patient, wise trend followers who will last as long as markets trend. But no, it was some aqua farm in Malaysia).

bloom
Senior Member
Posts: 35
Joined: Thu Apr 17, 2003 12:45 am
Location: SARS infested HONG KONG..ahh

cockroach

damian's evolution metaphor is a very fitting and A zero sum game like futures is definitely a very likely environment where evolutionary force will apply. I hope there are some zoologist among us who can tell me what is the most "robust" living organism on planet earth. I would define a robust organism as one where its phenotype has't change in many millions of years because there is no evolutionary pressure for the "design" to change , the "design" is able to survive under the most hostile and diffcult environment. Was dinosaurs a robust organism? if it were. it wouldn't be extinct, right? It over-specialize in a certain area(namely, size) and environment and didin't consider catrasphoic times in its "design" The dinosaurs sort of reminds me of LTCM. \

In comparison, I think crockroach is a very "robust", its primitive design havn't change for 320 million years( got it from britannica online) , I have heard it can even survive nuclear wars. What makes them so robust? I am hoping someone can tell me. My point is that we should design system as robust as the crockroach, I know cockroach can eat virtually eat anything, in trading terms, it would be the ability to capture short, intermediate and long term trends. I also know they can survive in very hostile environment, there would be a very conservative money mangement plan to minimize the risk of ruin. They are also very primitive organism which does not over-specialize in a certain physical attribute. ( like the cheetah specilize in speed and humans specialize in mental abilities.) In a trading system, this would be a system that is simple enough to work in most market environment.

Come to think of it, I really wouldn't mind being a Cockroach Trader...

Sir G
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Good one!

Damian-

Great metaphor indeed! I enjoyed reading and will refer to it as it captures things very well.

It's great to see you express your thoughts in such a great way. Have you always been like this, or is this forum tapping into something?

Cheers. Sir G

Howard Brazzil
Roundtable Fellow
Posts: 54
Joined: Wed Apr 16, 2003 12:45 pm
Location: Houston, TX USA

been around for a while, too.

In fact, a rancher friend recently told me that scorpions and cockroaches
rarely inhabit the same locale because (beside having diametrically
opposed climatic preferences), scorpions will dine on roaches if they
are available.

A scorpion's gestation period can very over a very wide time range,
and modulates up and down with available food supply / hospitability

Scorpions prefer hot dry climes. So several years ago researchers
in the UK froze one in a block of ice. Three weeks later they carefully
thawed it with a blowtorch. In a few minutes, the scorpion scampered
off the table. Now that's robust!

Sir G
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah
I think NickR puts an important spin on it… I’m going to spin his spin a bit.

Here are two different sets of logics:
1. Let’s go to the beach and have a picnic when there is a snowstorm dumping 2 feet of snow on the beach.
2. Let’s go to the beach and have a picnic when it is 85 degrees and there is nothing but blue skies and white fluffy clouds as far as you can see.
Which one is “right?”

Well that depends on what you want the outcome to be. If you want a nice enjoyable time, choose the latter, if you want a hard time, then choose the former.

Look within your system logic the same way. Does it have the ability to direct you to good times or bad times?

Here are two more sets of logic:
1. A system has a great looking Equity Curve and is the product of optimizing 12 different sets of parameters and it works on only one market.
2. A system has a good looking Equity Curve and is the product of a few basic ideas and the system works pretty good across all the market data you have.
Which one is “right?”

Well that depends on what you want the outcome to be.

The logic that you use to build your systems need to make sense and stand on its own merits. Because it is going to produce the results that it was designed to produce.

Sir G