By What Measure? - How do You Know if a System is Good?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
yoyo2000
Roundtable Fellow
Roundtable Fellow
Posts: 58
Joined: Fri Jan 30, 2004 10:37 pm

From "Schwager On Futures Managed Trading:Myths & T

Post by yoyo2000 » Fri Dec 03, 2004 1:48 am

In Part II of "Schwager On Futures Managed Trading:Myths & Truths",he said,neither historic performence level nor relative historic performence ranking could predict future performence of a system,and the exams in the Part proved his argument.So how to know System A is superior than System B?

Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt » Fri Dec 03, 2004 7:17 am

Roscoe wrote:Would those figures be based on single-contract testing?
I don't ever test a single contract. Position sizing is too important to proper diversification and risk control, so the figures I quoted are for systems that use a position sizing algorithm for multiple contracts.
Would you consider saying a little about what sort/type of systems you trade?
I like to trade many systems that are simple and that work reasonably well over a long period of time. This is in contrast with systems that work very well over the last several years.

I believe that this is the major factor affecting the gap between tested performance and real-world performance; testing over too short a timeframe and trading overly complicated systems.

The systems are simple but very different. A mix of systems that are based on different principles or different mechanisms will perform more reliably than any single system.
yoyo2000 wrote:he said,neither historic performence level nor relative historic performence ranking could predict future performence of a system,and the exams in the Part proved his argument.So how to know System A is superior than System B?
In short, he's wrong. His statement is true when you look at short-term performance but wrong when you look at performance over many years.

- Forum Mgmnt

Hiramhon
Roundtable Fellow
Roundtable Fellow
Posts: 98
Joined: Fri May 09, 2003 12:45 am

Post by Hiramhon » Fri Dec 03, 2004 8:25 am

Schwager is the victim of his own experience. He was a high powered executive in the futures industry (Director of Research at Prudential). He personally interviewed three dozen of the finest traders in the world and got them to tell him about their trading approaches, in his books Market Wizards (I and II). Then he hired hotshot programmer/researcher Louis Lukac and they set up a CTA business called "Wizard Trading". How could he possibly lose?

Well, Jack and Louis cooked up a few mechanical trading systems that used the same logic and the same parameter values to trade every market. They traded their clients' money using these systems. And, lo & behold, Wizard Trading lost money. How could this possibly happen?

Certainly it wasn't because the geniuses Schwager and Lukac made a mistake or performed their research incorrectly. Heck no. NFW. Jack Schwager was a stud and Louis Lukac had a Ph.D. from Purdue, so you can forget about any kind of sloppiness or human error.

Clearly, therefore, the problem had to be with systems. Yeah that's the ticket. Sure. Systems suck and only fools trade systems. Shoot, Wizard Trading even has data that proves the unprofitability of systems. All their clients' statements showing losses, are evidence of the futility of systems trading. Everybody knows that!

WAI
Contributing Member
Contributing Member
Posts: 8
Joined: Tue Jan 06, 2004 3:18 am
Location: Zurich, Switzerland

Post by WAI » Fri Dec 03, 2004 8:52 am

And so a different approach was required...

(from Albourne VIllage)


Fortune Press Release - Markets Wizards ranked in top 20% after first 3 years
posted by fortune on Monday 29 Nov 2004 05:37 MET


The Market Wizards Fund www.marketwizardsfund.com, the flagship fund of funds offering of hedge fund advisory firm Fortune Group, celebrated its three year track record on Nov 1st 2004.


The Market Wizards original programme was launched as a US partnership in September 1999*, but the current programme was launched in Nov 2001 as a Cayman Island, Dublin listed structure with two classes. The flagship Market Wizards Fund Plus (or B class - for institutional investors CLICK HERE), uses the judicious application of leverage to increase the overall weighting of the very low volatility funds in the portfolio but unlike many so called unlevered funds, avoids higher levered underlying strategies, as well as traditional CTAs and global macro managers. It has now delivered 30 out of 36 months of positive returns. The B class has returned 26.33% since inception or 8.10% on an annualised basis and is ranked 82 percentile on performance alone in a peer group drawn from a database of over 600 funds (see attached study). It ranks top quartile versus all equivalent funds on return to risk. The annual standard deviation for the fund is now 3.2%.

On 1st April 2004 Fortune launched Market Wizards XL, which employs up to 2x leverage and invests into the Market Wizards Plus share class (a version of the Wizards Programme targeting private wealth managers and banks). A further innovative share class, Market Wizards Fund XLR, is structured to pay a 3% introducer fee that does not directly penalise the investor as it is funded through a higher management fee and redemption penalty.

MWF (B class, Plus and LLC) utilises leverage in a unique fashion to enhance returns while still controlling risk. Rather than leveraging the entire portfolio, as is typically done, the fund leverages only the lower risk holdings in the portfolio (a methodology we term "selective leverage"). In effect, this approach results in a portfolio rebalancing process, wherein the largest allocations are made to the lowest risk funds - a risk-mitigating effect that provides an offsetting factor to the risk increase due to the use of leverage.


And the beat goes on...

yoyo2000
Roundtable Fellow
Roundtable Fellow
Posts: 58
Joined: Fri Jan 30, 2004 10:37 pm

Post by yoyo2000 » Sat Dec 04, 2004 2:01 am

:shock: two different statement...
anyway,I got some tips from his book,the thought of RRR is very good.

below is a table of RRR:

http://www.okhere.net/upload/images.asp?id=353

could anyone code it,please?

and what's your opnion about rebalancing asset every month?
Is it really a good approach to manage money?

alfaman
Senior Member
Senior Member
Posts: 37
Joined: Tue Jun 08, 2004 8:08 am
Location: PENNSYLVANIA

What is a reasonable value for Lake Ratio

Post by alfaman » Tue Dec 07, 2004 9:36 am

I have been evaluating several long term trend following systems using a multitude of qualitative criteria. Most systems show strengths in different areas as one would expect. Following on the thread of this forum, I too find the MAR measure extremely narrow in focus. I do however like Seykota's Lake Ratio in particular, and was wondering if there is some benchmark value that is considered good, or if there is a rough break point between good/bad. Have looked around and can't seem to find any sort of "consensus" # (for whatever it is worth).

Thanks,
Steve

alfaman
Senior Member
Senior Member
Posts: 37
Joined: Tue Jun 08, 2004 8:08 am
Location: PENNSYLVANIA

Post by alfaman » Wed Dec 08, 2004 9:49 pm

Your idea is a good one for sure. The only problem is that if I show a twenty year equity curve (a data set that most in this forum deem a minimum), the scale would make the first 4-5 yrs appear flat, unless a logarithmic illustration is used, whereby the lake would be meaningless, at least visually. Most of the systems that I am looking at have very similar attributes in terms of MAR and CAGR (a 20 yr MAR of +/- 1.6 and a CAGR of roughly 50%) though the Lake Ratio varies from 10.5% to 7.5%.

I guess what I am after is someone who has long experience trading a long term trend following system that can give me a hint about pain, and whether there exists some correlation between pain and lake, and where that pain vs. lake might reach a sweet spot (highly subjective I know). So given that a system has to be in line with the trader's personality, is there anyone who can tell me what they shoot for?

(I reread the question and think to myself- what ambiguity!)

Thanks,

Steve

PruBear
Senior Member
Senior Member
Posts: 29
Joined: Sat Aug 23, 2003 11:03 am
Location: New Jersey

Post by PruBear » Fri Dec 17, 2004 6:07 am

- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks
Re: this standard of system performance, I have two questions:

1) Is this for a single system or a portfolio of systems designed to work together?
Using VT to test, I have had a hard time getting my MAR ratio near 2.0 on any single system that has decent returns.
To achieve a MAR ratio > 2.0, you're going to have to find a way of really limiting your max DD. I haven't been able to design any stand-alone system that can cut max DD w/o dramatically lowering returns.
A portfolio of systems, however, (some that follow the trend, others that make money in the chop), seems more likely to be able to do the trick.

2) How much optimization are we talking about with these kinds of statistics?
The best way I know of to generate knock 'em dead stats is to go back and curve fit your system to the data. Without this technique, the whole cottage industry of selling trading systems to the retail public would probably collapse, lol.
Are you achieving these test results with simple and robust systems with, say, 3 or fewer parameters?
Since you're testing over 20+ years, I would think that length of time would poke holes in any over-opimization.
And who knows, maybe there is a certain level of optimization that is good. I'm just tempermentally suspicious of any optimization.

I'm grateful for whatever thoughts you can share on these topics.

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Fri Dec 17, 2004 12:05 pm

And who knows, maybe there is a certain level of optimization that is good. I'm just tempermentally suspicious of any optimization.
Any time you make any choice based on past performance, you are performing an optimization. So for example when you choose mutual funds based on their 5-year and 10-year rankings, you are optimizing.

When you choose to trade Trendchannel and not trade Catscan and not trade Andromeda and not trade Aberration and not trade the Original Turtle System, because Trendchannel's performance is the best of the bunch, you are optimizing.

When you leave Pork bellies out of your portfolio because it generated huge losses in testing, you are optimizing. And when you include Japanese Yen and Crude Oil in your portfolio because they have produced big profits in the past for trend followers, you are optimizing.

Maybe one thing you can do is open several different accounts and trade them differently; one will use no optimization at all (choose a system at random, choose system parameter values at random; choose a portfolio at random). Another account will use "mild" optimization (optimize the system selection but not the parameter values or the portfolio). A third account will use "aggressive" optimization (optimize everything, to the nth degree).

If optimization is good, you've got an optimized plan as part of your method and it will generate super duper big profits, which will pay for the losses in the other accounts. On the other hand if optimization is bad, you've got a decidedly NON optimized plan as part of your method.

Your overall performance will be the average of optimized, partly-pptimized, and non-optimized performance. You won't make as big a fortune as the person who guessed right when picking only one approach. On the other hand you won't make as big a loss as the person who guessed wrong when picking only one approach. And, as an extra benefit, you will obtain actual data (not opinions, data!) about the real world performance of optimized vs non-optimized methods. Such data is next to impossible to procure at any price.

Old European
Roundtable Knight
Roundtable Knight
Posts: 144
Joined: Mon Apr 19, 2004 3:04 pm
Location: Old Europe

Post by Old European » Fri Dec 17, 2004 1:59 pm

PruBear,

It is definitively possible to come up with single systems with a MAR ratio of 2 or higher without optimizing too aggressively.

The procedure I use is the following. I first define what I call a 'generic market universe'. It is simply the universe that consists of all the futures markets that meet a number of basic selection criteria (like sufficient liquidity). It contains about 50 markets in my case. Then I choose a simple system and look for a set of parameters that lead to stable results (flat area in parameter space). It isn't too difficult to find robust systems for such generic market universe and tested over 25+ years (with reasonable assumptions for commissions and slippage) with a MAR of say 1.25 or so. It is then very easy to increase the MAR dramatically (even without touching the parameters) by cherry picking the best markets. This is however something that goes way too far in my opinion. Instead of choosing the best performing markets I just throw out the very worst markets (a small number of markets). In order to figure out whether a futures market is good or bad, I compare the test results on the generic universe with and without this specific market (and thus automatically take correlations with the other markets in the portfolio into account). Even by prudently optimizing the investment universe in this way (and still using the same parameters for all markets), it usually turns out that the MAR ratio increases easily from say 1.25 to 2+.

I don't think there is anything wrong with optimization (why would you after all prefer parameter values that lead to inferior historic test results over better parameter values?) if you don't do it too brutally and if you don't count on repeating the optimized results in real trading. To be on the safe side I have set my personal real life performance target (MAR) as only half the MAR of the historic tests.

Cheers,

Old European

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Fri Dec 17, 2004 7:29 pm

Old European wrote:The procedure I use is the following. <snip>
Thanks for that, I certainly appreciate your post. One question if I may: is your 'generic market universe' testing done with or without a position-sizing strategy? Thanks in advance.

Old European
Roundtable Knight
Roundtable Knight
Posts: 144
Joined: Mon Apr 19, 2004 3:04 pm
Location: Old Europe

Post by Old European » Sat Dec 18, 2004 2:55 am

Roscoe,

I never ever perform tests without dynamic position sizing at the market portfolio level.

Cheers,

Old European

Sir G
Moderator
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Post by Sir G » Sat Dec 18, 2004 12:26 pm

Optimization is in every aspect of our lives.

When we make a choice of yes or no… we have optimized based on analysis or feel, or maybe our negative impulses.

It’s just like the people we are attracted to…we are optimizing our desires our wants and needs. And that will decide if we are happily married and our lives are in balance, or if our life is full of heart ache. The question that should always be processed is…. Are our desires, wants and needs good for us beyond tomorrow?

With computers and software we can produce results which give us the image of the dream of robustness and success, and the performance summaries are simply a translation of our desires, hopes and dreams. In the end as Ed Seykota so simply stated.. we all get what we want. If all we want is simply the feel of success, then we will tweak parameters, statically change our portfolio and the like. But if we want more then just the feel of success, then we will look into the logics of what our systems really are and why they are.

Just as in the people we are attracted to, we decide the acceptable levels of shallowness and depth. Outer beauty will never tell us what is on the inside. We can imagine the appearance of our “physical dreamâ€

seale1
Senior Member
Senior Member
Posts: 45
Joined: Tue May 20, 2003 3:02 pm
Location: Vancouver, Canada

Post by seale1 » Sat Dec 18, 2004 5:32 pm

Gordon-I couldn't agree more.How many of us have thought "This (fill in the blank) would be so great if I could (again fill in the blank). Be it a relationship, job, event or whatever, the surface details, as we all find out sooner or later, make up maybe 5-10% of the total. However, we all too often only figure this out in retrospect. At least this has been my experience. :D

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Sat Dec 18, 2004 10:37 pm

Old European wrote:I never ever perform tests without dynamic position szing at the market portfolio level.
Thanks Old European, that was the missing piece of the puzzle for me.

PruBear
Senior Member
Senior Member
Posts: 29
Joined: Sat Aug 23, 2003 11:03 am
Location: New Jersey

Post by PruBear » Sun Dec 19, 2004 4:12 pm

Thanks to all for responding,

I'm not trying to harp on this point but there is something still bothering me:

Looking through the data on http://www.iasg.com there really is no CTA with a 20+ year record that supports a MAR ratio >2. All of the big boys (Campbell, Dunn, JWH, Eckhardt, etc.) have max DD's in the range of 30%-50% on their big systems that they have been trading the longest. If I've forgotten someone please let me know, though I'd be inclined to think that the exception proves the rule.

With these kinds of max DD's, you are going to need a CAGR in the range of 60%-100% for a MAR >2. What system trader has produced this kind of performance over 20 years? I think I read somewhere that Ed Seykota has produced returns in this range with the money he privately manages, but he doesn't report to any service (and anyway he's kind of a special case imho).

There appear to be a lot of newer CTA's with shorter records who have performance stats that might support a MAR >2. I'm inclined to think that they haven't hit their big DD yet. To compare apples to apple, you've got to look at the same length of time. A 20-year max DD is likely to be a lot worse than a 5-year max DD.

As far as Paul Tudor Jones goes, I have enormous respect for what he has achieved. He is one of the great ones. But I believe that his trading is basically discretionary, no? That's a whole other kettle of fish imho.

So how does one achieve a MAR > 2 in a historical test? My first thought would be either cherry picking (markets or time periods) or monkeying with profit targets. Is this what we're talking about, b/c I'm not really comfortable with either one?

Old European, I really enjoyed your post. I've read it several times and am still thinking it over. I'm conceptually uncomfortable with the idea of screening out the worse performing market(s) b/c of the obvious, (i.e., it may be the next big trender). However, you make some very good points and after all, trading takes place in the real world, not the ivory tower. I tend to design from the concept side of the table and I know that one must be careful not to let one's concepts overrule empirical data. If I did that I'd be no better than a fundamentalist, lol. :!:

From the concept side, I would think that systems that rely too heavily on cherry picking or profit targets would show their warts over a 20+ year period. Whether or not the data supports that conclusion is an entirely different (and much more important) question. :wink:

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Mon Dec 20, 2004 6:29 am

As has been pointed out a number of times, you have to add back in management and performance fees to these CTA performances which of course improves MAR. But I have come to exactly the same conclusion as you have. Over the long term I am doubtful that such high MARs are achievable in the real world as witnessed by the long term track record of JW Henry et al on iasg.com

You may also note various members of the futures trading community on other forums who built up accounts rapidly in the mid to late 90's using aggressive betsising on smaller accounts and then went much, much quieter for a couple of years. Until recently, when figures of 80 to 100% return for this year have been reported.

Nothing wrong with that at all. And backtesting from say 1994 up to then end of 2000 shows dramatic results on many generic systems. And then some record drawdowns set in - some of which we have not emerged from. Depending on system, betsising, portfolio of course.

What I think all this has taught me is that one can only win through based on one's own extensive and careful testing and trading. No good looking for a guru figure. The other factor which applies to trading as much as any endeavour in life, is that you are not going to get it all right all at once in a short period of time. I have only been looking at LTTF for 3 years and each passing day brings new insights. I am sure the learning process will continue as long as I trade.

Some one somewhere correctly pointed out that a surgeon takes 7 to 10 years to learn his trade and why should a futures trader expect to become an instant expert. I suspect one needs every bit as long with futures trading!

Sir G
Moderator
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Post by Sir G » Mon Dec 20, 2004 12:31 pm

The guys that you are looking at (Campbell, Dunn, JWH, Eckhardt) seem all to be LTTF type of guys. IMHO, Long Term traders are a different breed then Short Term traders.

Take a look at Toby Crabel, his record is ~12 years and his real time MAR is >2. I don’t classify traders as style, but in time frame. So while Tudor could very well be a discretionary trader, he is a Short term trader. The reason why I look at time frames before style is that the opportunities are based on the time frame traded.

I think you won’t find many real time track records >20years as this is still a young industry.

Cheers. Gordon

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Mon Dec 20, 2004 1:30 pm

Yes, sorry Gordon, quite right,very sloppy of me. I should have made plain I was talking about longer termn trend followers in general. I have indeed looked at Crabel's funds for my own investment purposes and while you are of course absolutely correct about the MAR, the CAGR is lowish (from memory) by LTTF standards. That is the trade off - lowever volatility of returns, lower max dd and lower CAGR.

I am sure that for a smaller private trader, short term trading can be made to appear/be far more profitable. If you day trade financial spreads your margin requirement is slim and your profits on that margin (even if you add a cushion) can presumably look very exciting.

I suppose what I was trying to say, as regards LTTF, is that it is interesting to have witnessed the publication of impressive sounding results by private traders for the mid to late 90's along with a slew of simple systems published and generously coded for others to look at.

And equally interesting to realise, post the event, that those few years represented a particularly profitable period for many LTTF systems.

So, it is important to test over many time frames and to recognise that that you may have the luck to start to trade at a propitious time, you may not. If you do strike lucky with your timing an aggressive betsizing approach on a small account can indeed turn $100/- into a few million. If you have the bad luck to start to trade at the beginning of a couple of trendless years I suspect you will either lose your stake if you bet too big or at least show no growth.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Mon Dec 20, 2004 1:57 pm

I was also talking purely about mechanical systems traders. I guess I can be forgiven for the mistake of not making that plain, given the forum. To talk of the trading records of Kovner, Tudor Jones, Luis Bacon and so on was not my intention. For one thing I imagine they are mostly discretionary with more than a hint of fundamental. For another, their funds being closed a) it is wishful thinking to want to invest unless you pay a premium on the rather illiquid secondary market and b) these people do not and have no requirement to publish their track record other than to thier own investors. Although I think I have seen Caxtons figures posted somewhere or other amongst the multitude of hedge fund websites.

Post Reply