By What Measure? - How do You Know if a System is Good?

[Forum Mgmnt]

One of the core issues in evaluating a trading system is determining exactly what makes a good system.

Each of us has different tolerances for risk, boredom, and volatility. Even these are interpreted differently by each of us.

So I am curious what each of you uses to determine what makes a good system. One that it good enough that you would trade it if it were the only system you had.

Here is my best guess at own personal threshold criteria for my own money given my personality:

- Drawdowns < 50%
- MAR Ratio > 1.2
- Returns > 35% for futures, 20% unleveraged stocks

I would trade systems that met the above if that was the best I could do.

Given that I know I can do better, I won't trade every system that meets the above. But I'd consider adding a system to the portfolio of active systems I traded if it was at least:

- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks

This would be tested over 20 years using the same system parameters for each year and each market.

- Forum Mgmnt

[leanhog]

Forum Mgmnt,

You bring about a good point. Thanks for sharing.
Every one of us has different tolerance as well as preference for certain
measure.

In my personal trading I always focus first on the (MAR) ratio.
The second is the quality of the MAR. What do I mean by that..
I look to see the frequency and lengh of the drawdown.
For example: I would rather have a 25 percent drawdown 10 times in twenty years than 2 such drawdowns and rest being 5 or 8 percent.
I want the distribution to be closer to random than having an occurence that is out of character with the system.
This is not something that can be quantified instantly but can be checked very easily visually when running tests.
So to sum it up MAR over 2, lenght of drawdown of 8 months or less, and distribution of drawdowns, would be my important factors when doing preliminary evaluation of a system.

[Forum Mgmnt]

MODERATORS NOTE: This topic from Sir G was split to separate the subject of computing skid. Appologies to Sir G.

ORIGINALLY FROM SIR G:
I’m in agreement with the performance numbers that tb offers. One thing that I plan on paying more attention to is Standard Deviation. I think that is a good gauge of risk... emotional risk... I believe Toby Crabel is found attractive by many because of the smooth sailing he offers. This at a time when risk has become a 4 letter word for some. Any thoughts or suggestions?

[damian]

can you call an equity curve with many horizontal sections (ie, no positions) smooth? If there are many horizontal sections, can you apply the same measures of equity curve smoothness?

I started to enjoy designing models that would produce EC's with many horizontal sections, even though the testing was based on a 10 market portfolio.

Part of me is comfortable with a system that is happy doing nothing.

ps - Hi Mark.

[Ted Annemann]

Trader Y's monthly equity values are
1 1 1 1 5 5 5 5 9 9 9 9 13 13 13 13 17

Trader Z's monthly equity values are
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Trader Y's "equity curve" has flat spots, during which he has no positions and is exposed to no risk.

Trader Z's "equity curve" changes every month. He has no flat spots.

Traders Y and Z have exactly the same profits: 16 units.

Both the Sharpe Ratio and Mark_Johnson's Rsquared smoothness measure, would say that Y is preferable to Z. However we human beings are free to make our own judgements. If we so desire, we can reject these Goodness Meters and invent our own. I am interested to learn how damian would calculate upon these equity numbers to give the result that Z is more desirable than Y.

TedA

[Ted Annemann]

whoops In haste I didn't proofread and I interchanged the examples while typing.

Sharpe and Rsquared prefer the 1 2 3 4 5 6 7 8.... equity curve. Sorry!

[Idea Man]

What I find interesting is that they prefer the:

1 2 3 4 5 6 7 8 9 10

curve to the:

1 2 5 6 8 9 12 13 15 16

curve.

Smoothness dominates return

[Kiwi]

Ted,

I'm not sure what Damien's reason for preferring the first curve will be but mine is simple. If the system is not in the market during the periods when it is flat and takes the same risk when it is in the market then the overall risk is considerably lower.

What I mean by that is that an unforseen event such as a nuke going off over Seoul will not damage its equity if it occurs when the system is out of the market. A system in the market is usually hurt by spike events as slippage becomes huge.

John

[Sir G]

In historical testing, what should one expect for a gross Average Profit Per Trade in both Short Term Systems & Long Term Systems?

Thank you.

[Robert Nio]

Sir G.

Why do you want to limit yourself thinking about a "minimum" average return?

Just choose the "best" place for your money. With limited funds, just swap the worst trade with a new one.

If you have a good system the return will show up all by itself.

PS: Trying to predict / expect a certain attributes of a trade will just set you up for failure. Let the market tell you the facts and be prepaired to react.

[damian]

regarding curve shape and flat sections:

In the hypothetical context of system X and system Y as posed, I agree with Kiwi's thoughts. If I could make the same money I would take curve Y as it makes it's money then leaves the market alone and then returns to make some more. I spent years in an old job being told that you can't make money being flat/square. I always responded with "square is a position". SirG said words to this effect in another post (I think).

SirG, regarding gross Average Profit Per Trade, what is the difference between gross average profit and average profit? (do you mean before deductions for brokerage?)

I am yet to reconcile my thoughts regarding average trade profit as it is driven in part by number of trades. A very low trade count system can produce a great average trade number. This is one of my problems with systems that produce equity curves with flat sections.

On the topic of what measures a good system, I am not a very precise person. If I were given a set of good results generated from hypothetical back testing, and asked to choose one system I would casually choose the one with teh best results, but the 'utility' I got from the choice would be no greater than if I was forced to choose the one with the weakest results. To be honest, any good system will do. Obviously a line needs to be drawn that delineates good and not-good. Beyond that I am happy to be vague and imprecise. I see trading as totally imprecise and I don't want to get bogged down looking for precision, particularly from back test data.

At the risk of just repeating myself, I am forced to be precise when I group systems as good or not-good. After that I don't differentiate between the systems in the set of good systems. That is the precision I avoid.

If the total population of systems that I reference (when creating my good, not-good sets) is increasing, which it is* then my line in the sand may move, but only to the right, making membership in the good set harder.

I often wonder if my lack of interest in precision is my way of dealing with lower system testing skills. I am binary up until a certain point, then I become fuzzy. When referencing the same topic very clever people will remain binary well after me... but eventually find comfort in vagueness/imprecision/fuzziness.


* or is it? sometimes I feel that I am seeing the same basic concept re-structured into a different system that has roughly the same performance that all systems have that are built on the same concept. The population of systems is huge, but the population of concepts on which they are based is not.

[Sir G]

I can see my question was way too broad.

Yes, the Gross Avg. Profit Per Trade (GAPPT) is the Average Profit Per Trade (APPT) with NO commish or skids figured into it. I’m asking as I’ve never really touched a Long Term system, so I wouldn’t know what to “Expect.â€

[drm7]

Your "minimum" performance criteria seem very ambitious to somebody from the outside looking in. 50% annual returns (even with a 35% drawdown) would blow away even legendary CTAs and hedge funds.

-John Henry has had about a 20-25% annual return with a 45% max drawdown
-William Eckhardt has a 25-30% return with similar drawdowns.

(Source: IASG top 100 CTA rankings)

To ask the question a different way, Do you trade the system with those measures because:

1. Your track record is better than that

-or-

2. You know the actual performance will be worse, and you want a good margin of safety.

If the answer is #1, is trend-following really that "easy"? (provided you don't deviate from the rules and can stand the drawdowns)

I do not have a lot of system development or trading experience, so please excuse my ignorance!

[chuckm_ca]

"If the answer is #1, is trend-following really that "easy"? (provided you don't deviate from the rules and can stand the drawdowns) "

I think what happens is that you discover trend following isn't as easy as it looks. On paper, that 50% return and 40% drawdown looks great, but can you personally trade it, or if you are managing money, can your clients handle that kind of drawdown. Most people can't. There is a lot of psychological pressure on you when you are down that much. A mechanical method removes the emotion from the decision, but it doesn't necessarily remove the emotion from the trade. This is where the discipline comes in, you have to follow the rules no matter what is going on in your head. This has been the down fall for many traders I've seen trying to use mechanical systems. They start to second guess signals and not take some of them. Inevitably, the hardest trade to make is the big winner, it also happens to be the trade you will most likely not take if you let your discipline slip.

Chuck

[Forum Mgmnt]

Chuck is right. Most people can't do anywhere near as well as the tests show.

The other thing is the effect of size. Trading $1 billion will take you out of most of the futures except in a token way.

I like to trade systems that have great results because next year might be the bad year.

If you are trading other people's money it is a different story. Most investors can't handle drawdowns of more than 15% to 20% without having serious emotional problems. This means toning down the trading size with a corresponding reduction in returns. It matters little if you can take the drawdown if your investors all leave when you get them.

Look at Toby Crable's growth recently. It is mostly because he offers a very smooth ride with good returns. If you are smooth you can raise as much money as you can handle.

#1 - Sort of. I can find systems that test that well so why look at worse systems. I certainly believe that I can trade in the same league as the best of the public guys, but I've only recently returned to trading after a 15 year hiatus in software development and entrepreneuring. It will take a few years before I can make any definitive statements.

#2 This is not necessarily the case, but certainly can be. Having some room in case there were some wrong assumptions seems prudent if you have the option.

[Moodaeng]

I have played around with the concept of Sharpe Ratio and posted this some time ago on another forum.

Suppose you have System1 with an annualized rate of return over riskless rate m1, and with an annualized standard deviation of s1. Then, your Sharpe Ratio is SR1 = m1/s1. Now, you also have System2 with a Sharpe Ratio of SR2 = m2/s2. SUPPOSE THE TWO SYSTEMS HAVE A SERIAL CORRELATION OF 0. Now, what is going to be the Sharpe Ratio of 1+2 ?

SR = (m1+ m2) / (SQRT(s1^2 + s2 ^ 2))

Let's put : s2 = k *s1
k is the weight you are going to give to system 2 if system1 has a weight of 1.

then, we have : m1 = SR1 * s1 and m2 = SR2 * k * s1, so

SR = (SR1 + k SR2) / (SQRT(1 + k^2)

We then ask the following question. On what condition is SR maximum? In other words, how much of System2 should we trade in order to have the best final Sharpe Ratio?

Answer : for kmax = SR2 / SR1 (i) (you can check by yourself )

When you think about it, the answer is pretty intuitive. The higher the Sharpe Ratio of your system, the more weight it should have in the final system.

Now, thanks to (i) we can calculate the final Sharpe Ratio :

SR = SR1 * SQRT(1 + (SR2 ^ 2 / SR1 ^ 2)) (ii)
We can check that SR > SR1 and that SR > SR2 , which shows the value of diversification.

Now let's illustrate what we've found. Suppose we trade n systems that all have a Sharpe Ratio of SR0. TAKEN 2 by 2, ALL THE SYSTEM PAIRS HAVE A PAIRWISE CORRELATION OF 0. Now what is going to be the Sharpe Ratio (SR(n)) of our mega system if we use an optimal weighting? Using (ii), we find :

1 system : SR(1) = SR0
2 systems : we trade one system with Sharpe Ratio = SR0, and one system with Sharpe Ratio SR0. Then, using (ii), we have SR(2) = SR0 * SQRT(2).
3 systems : we trade on system with Sharpe Ratio = SR0 * SQRT(2), and one system with Sharpe Ratio SR0. Then, using (2), we have SR(2) = SR0 * SQRT(3).
If you continue, you will find out (and can easily prove!) that :

n systems : SR(n) = SR0 * SQRT(n) (iii)

In other words, if :
- all your systems have the same Sharpe Ratio
- your systems have no 2 by 2 correlation
- you mix your systems at the optimal weighting (given by (i))

(a lot if ifs!)

then, YOUR SHARPE RATIO GROWS WITH THE SQUARE ROOT OF THE NUMBER OF THE SYSTEMS YOU TRADE.

I understand that this is just a simple case of the mean / variance optimization, but I found (iii) simple and easy to remember!

[Sir G]

c.f.-

Thanks for sharing your work on the g.c. Ratio. I also had thought that there must be a better way to compare apples to apples.

It was very good food for thought.

Gordon

[Jimmy]

Is there a metric for measuring consistency of a system? I seem to remember a trader in Market Wizards 2 that implied on his business card that he expected 20+% returns with 95% confidence. This is a pretty incredible ascertion to say the least. After analyzing the results posted on the VeriTrader website and running a few tests myself varying the Turtle rules, it’s obvious that long term trend trading with futures can be very profitable if one has the initial capital and mental fortitude to withstand the long drawdown periods.

For each instance with the turtle rules where I varied the parameters to try to decrease drawdown, whether it was changing the % to risk per trade or moving the stops, a decrease in drawdown was almost always associated with a decrease in returns. This exercise was extremely educational for me as a beginner just to feel the influence changing one parameter could have on the results of the system. I always knew this to be the case of risk/reward, but seeing it for myself really helped drive home the point.

Back to the point of my post. As noted by others in this forum, consistency or smoothness of returns is more of a concern for those managing investor money, but it’s definitely an area I’d like to study further. I guess rather than a metric of consistency, what I’m really looking for is a component of a system that improves consistency. Is this even possible? Is there no way to decrease the number of false signals, whether you’re using a breakout, MA, or any other indicator without losing return? Is a shorter term system with greater win/loss ratio and lower profit/win the key, especially for a smaller investor? If so, how would one classify a “goodâ€

[Kiwi]

Jimmy this is just one persons opinion.

I am not a better trader than c.f. or many of the money managers --- the biggest thing that seperates me or any other successful sole trader from a money manager is that we can take the drawdowns. So in return for 3x the return I can take 4x the drawdown.

As you suggested, Money Managers cant take drawdowns because of their customers.

I have tried quite a number of things for smoothing. Mixing system/portfolios to get different rules for exit & entry on a variety of contracts gives the best return (CAGR) for maxDD or worst4DDs or Standard Deviation of the returns (or whatever). I am always searching for a better measure but I dont let that stop me following the "multiple system / multiple commodity / multiple timeframe approach.

I spend a lot of time preparing to face drawdowns without falling apart and failing to follow my rules - which is what kills many sole traders.

I also segment my money so that if one approach blows up completely I can build up again

[djb]

Kiwi,
So just to clarify, if you are trading say three systems, over different portfolios, you seperate you total stake into three equal parts and trade each system/bet size combination independently ?.

I was thinking along the lines trading multiple systems,but calculating the bet size for each systems signal based on my total capital.
Which do you think is better ?


DB