By What Measure? - How do You Know if a System is Good?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Bondtrader
Roundtable Knight
Roundtable Knight
Posts: 101
Joined: Mon Apr 28, 2003 11:07 am
Location: Zimbabwe

Post by Bondtrader » Fri Feb 20, 2004 10:59 am

CAGR is correct, MAR is wrong (the error is in spreadsheet cell B18), didn't look at the other numbers.

ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

MAR and Geometric Mean

Post by ksberg » Sat Feb 21, 2004 12:30 am

Ross,

Looks like MAR is off by 1 decimal. I find it easiest to express CAGR as %, then dividing by Max DD % keeps the same units. The example with CAGR=9.16% and MaxDD=48.4% would yeild MAR=0.18.

Otherwise, looks good.

Cheers,

Kevin

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Sat Feb 21, 2004 4:42 am

Thanks Kevin, much appreciated!

Just posted the corrected spreadsheet (in my prev post above) for anyone who wants to use it.

While we are on this subject, does anyone have any other suggestions for useful metrics?

Roscoe

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Sat Feb 21, 2004 10:03 am

Any other suggestions for useful metrics? This exact thread, four screens long and entitled "How Do You Know if a System is Good?" is loaded with suggestions for useful metrics. That's its purpose.

Some additional suggestions already put forward and searchable:

viewtopic.php?p=882&highlight=ratio#882
viewtopic.php?p=1045&highlight=ratio#1045
viewtopic.php?p=26&highlight=ratio#26
viewtopic.php?p=20&highlight=ratio#20
viewtopic.php?p=137&highlight=ratio#137
viewtopic.php?p=954&highlight=ratio#954

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Sun Feb 22, 2004 4:56 pm

Thanks Ted, I appreciate the links, and I certainly appreciate this site - some great content presented with wisdom and dignity.

Thanks also to c.f. for making this possible!

Roscoe

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Mon Mar 22, 2004 8:29 pm

I just found this interesting composite measure of goodness. Comments?

Hiramhon
Roundtable Fellow
Roundtable Fellow
Posts: 98
Joined: Fri May 09, 2003 12:45 am

Post by Hiramhon » Mon Mar 22, 2004 8:56 pm

Roscoe, how about giving us your comments on the GPF at that link?

ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Kitchen Sink

Post by ksberg » Mon Mar 22, 2004 11:01 pm

Optimax GPF looks like the kitchen sink, and then some. I can understand the need enable a wide variety of fitness functions, but I would think the huge number of parameters make this cumbersome to use. Improvements? The classic "gang of four" software patterns book comes to mind.

Cheers,

Kevin

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Tue Mar 23, 2004 1:25 am

Hiramhon wrote:Roscoe, how about giving us your comments on the GPF at that link?
Hi Hiramhon,

I am just amazed that anyone could wrap so many components into one statistic! Still not real sure what to make of it - Kevin's "kitchen sink" comment sums it up rather well.

The obvious appeal of having one single measure of goodness is offset by the difficulty (to me at any rate) of grasping the actual meaning of that value. Having said that, the software seems to have been designed to allow the user to reflect their exact preferences.

Roscoe

Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt » Tue Mar 23, 2004 4:35 am

Genetic Optimization, the mechanism that Optimax uses to find the best set of values, requires a single "fitness function". Genetic Optimization works by running a bunch of different series of optimiztions and then changing those series slightly over time. New series start from the components of the most "fit" of the old series through a combination process called "mating". The idea is to mimic evolutionary biology.

A fitness funciton in this context gives a value for a set of parameters that is used to determine whether or not to keep pursuing a series of tests or to let that series die because other more "fit" series are available. In the evolutionary context, how successful that series is at propogating children. More fit series mate more and live longer thereby generating more children.

So the kitchen sink in this example comes more from their desire to let you make that fitness function conform to what you are personally looking for rather than a statement the one should use the entire kitchen sink.

ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Software Design

Post by ksberg » Tue Mar 23, 2004 11:37 am

So the kitchen sink in this example comes more from their desire to let you make that fitness function conform to what you are personally looking for rather than a statement the one should use the entire kitchen sink.
Understood. In fact, the whole notion of fitness function is to enable the user to define what "fit" really means. So more parameters = more choice for the user.

My comment is there are other ways in software to deal with variable inputs rather than force a user to user an unweildy API. For instance, a functor or call back could deliver/grab a narrow set of inputs instead of loading the stack with the kitchen sink. Another approach is to pass a facade that mediates access to subsequent values. In many cases the facade can be made as efficient as accessing original variable storage. These approaches are actually more flexible than a fixed set of parameters, since they enable extension after the API has been packaged. At any rate, given the number of parameters on this thing, I think any performance trade-offs with an alternate approach are probably acceptable.

Well, an opinion bit on software design. I digress.

Cheers,

Kevin

Arthur W. Cutten
Contributing Member
Contributing Member
Posts: 5
Joined: Mon Mar 29, 2004 4:48 pm

Re: By What Measure? - How do You Know if a System is Good?

Post by Arthur W. Cutten » Fri Apr 23, 2004 8:26 am

Forum Mgmnt wrote: - Drawdowns < 50%
- MAR Ratio > 1.2
- Returns > 35% for futures, 20% unleveraged stocks

- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks

This would be tested over 20 years using the same system parameters for each year and each market.

- Forum Mgmnt
A good system IMO is:

Drawdown <20%
Returns >25%

I would prefer to use:

Drawdown <15%
Returns >35%

ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Re: By What Measure? - How do You Know if a System is Good?

Post by ksberg » Fri Apr 23, 2004 12:40 pm

Arthur W. Cutten wrote: A good system IMO is:

Drawdown <20%
Returns >25%

I would prefer to use:

Drawdown <15%
Returns >35%
For the most part, both returns and draw down can be controlled by the amount of risk allocated for position sizing. For example, a system with 35% return and 15% drawdown has a MAR ratio of 2.3. If you're comfortable with 30% draw down, the same system should return roughly 70%. If 10% draw down is more acceptable, then you should expect roughly 23% returns from the same system.

Cheers,

Kevin
Last edited by ksberg on Fri Nov 05, 2004 9:16 pm, edited 1 time in total.

Moodaeng
Senior Member
Senior Member
Posts: 34
Joined: Thu Apr 17, 2003 4:23 am

Post by Moodaeng » Wed Nov 03, 2004 9:44 am


george chahine
Full Member
Full Member
Posts: 20
Joined: Sat Oct 23, 2004 2:37 pm
Location: Sunnyvale, CA

MAR

Post by george chahine » Wed Nov 03, 2004 11:00 am

Can someone tell me what does MAR stand for? How is it calculated?
Thanks.
GC

Tim Arnold
Site Admin
Site Admin
Posts: 8500
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Wed Nov 03, 2004 2:25 pm

Hi George --

MAR stands for Minimum Acceptable Return.

As for how it's calculated, there are a couple of threads that are useful.

viewtopic.php?t=36

Basically the % CAGR divided by the % maximum drawdown over your test period. Nice clean measure of reward vs. pain.

Hope that helps,

Tim

george chahine
Full Member
Full Member
Posts: 20
Joined: Sat Oct 23, 2004 2:37 pm
Location: Sunnyvale, CA

Thanks Tim

Post by george chahine » Wed Nov 03, 2004 3:58 pm

Thanks Tim.
GC

Murray Ruggiero
Senior Member
Senior Member
Posts: 41
Joined: Mon Oct 11, 2004 3:35 pm

How good is my system

Post by Murray Ruggiero » Thu Nov 11, 2004 11:05 pm

In an article I did several years ago, I tested how robust a given system was by optimize it over a large range of values and measuring the returns and standard deviation of returns across the range of parameters. We wanted to see an average positive expectation greater than the standard deviation at minimum. I never followed up on this idea much past the research I did for the article , but I just wanted to pass it on.

leonardo
Roundtable Knight
Roundtable Knight
Posts: 100
Joined: Wed Apr 30, 2003 2:27 pm
Location: Minneapolis, MN
Contact:

Re: How good is my system

Post by leonardo » Sat Nov 13, 2004 1:31 pm

Murray wrote:..., I tested how robust a given system was by optimize it over a large range of values and measuring the returns and standard deviation of returns across the range of parameters. We wanted to see an average positive expectation greater than the standard deviation at minimum.
I've applied this concept to my systems over the years also. It helps you see if the principle you're intent on trading actually has an advantage, and if it does---how much.

And, if you decide to trade the principle because you now know that there is this overwhelming advantage---then you can spread your money across the different parameters so as to smooth your equity curve.

Leonardo----

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Re: By What Measure? - How do You Know if a System is Good?

Post by Roscoe » Tue Nov 23, 2004 7:23 pm

Forum Mgmnt wrote:But I'd consider adding a system to the portfolio of active systems I traded if it was at least:

- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks

This would be tested over 20 years using the same system parameters for each year and each market.
To re-arrange the above slightly and focusing just on futures:
  • CAGR > 50%
  • MAR > 2.0
  • MDD% < 40%
The next question that I need to ask is: would those figures be based on single-contract testing or would they include the effect of position sizing (applied to just the single MarketSystem)? I find it difficult to get much better than:
  • CAGR > 10%
  • MAR > 0.5
  • MDD% < 30%
during single-contract testing, however I can get closer to c.f.' figures with the application of position sizing. If c.f.’ figures are based on single-contract testing then I’m obviously doing something very wrong! :?

At the risk of being cheeky: Forum Mgmnt, would you consider saying a little about what sort/type of systems you trade? I'm not asking for specifics or a handout, just some indication of which system types fare better than others in the real world.

Thanks in advance!

Post Reply