By What Measure? - How do You Know if a System is Good?
-
- Roundtable Knight
- Posts: 101
- Joined: Mon Apr 28, 2003 11:07 am
- Location: Zimbabwe
MAR and Geometric Mean
Ross,
Looks like MAR is off by 1 decimal. I find it easiest to express CAGR as %, then dividing by Max DD % keeps the same units. The example with CAGR=9.16% and MaxDD=48.4% would yeild MAR=0.18.
Otherwise, looks good.
Cheers,
Kevin
Looks like MAR is off by 1 decimal. I find it easiest to express CAGR as %, then dividing by Max DD % keeps the same units. The example with CAGR=9.16% and MaxDD=48.4% would yeild MAR=0.18.
Otherwise, looks good.
Cheers,
Kevin
-
- Roundtable Knight
- Posts: 118
- Joined: Tue Apr 15, 2003 7:44 pm
- Location: Arizona
Any other suggestions for useful metrics? This exact thread, four screens long and entitled "How Do You Know if a System is Good?" is loaded with suggestions for useful metrics. That's its purpose.
Some additional suggestions already put forward and searchable:
viewtopic.php?p=882&highlight=ratio#882
viewtopic.php?p=1045&highlight=ratio#1045
viewtopic.php?p=26&highlight=ratio#26
viewtopic.php?p=20&highlight=ratio#20
viewtopic.php?p=137&highlight=ratio#137
viewtopic.php?p=954&highlight=ratio#954
Some additional suggestions already put forward and searchable:
viewtopic.php?p=882&highlight=ratio#882
viewtopic.php?p=1045&highlight=ratio#1045
viewtopic.php?p=26&highlight=ratio#26
viewtopic.php?p=20&highlight=ratio#20
viewtopic.php?p=137&highlight=ratio#137
viewtopic.php?p=954&highlight=ratio#954
Kitchen Sink
Optimax GPF looks like the kitchen sink, and then some. I can understand the need enable a wide variety of fitness functions, but I would think the huge number of parameters make this cumbersome to use. Improvements? The classic "gang of four" software patterns book comes to mind.
Cheers,
Kevin
Cheers,
Kevin
Hi Hiramhon,Hiramhon wrote:Roscoe, how about giving us your comments on the GPF at that link?
I am just amazed that anyone could wrap so many components into one statistic! Still not real sure what to make of it - Kevin's "kitchen sink" comment sums it up rather well.
The obvious appeal of having one single measure of goodness is offset by the difficulty (to me at any rate) of grasping the actual meaning of that value. Having said that, the software seems to have been designed to allow the user to reflect their exact preferences.
Roscoe
-
- Roundtable Knight
- Posts: 1842
- Joined: Tue Apr 15, 2003 11:02 am
- Contact:
Genetic Optimization, the mechanism that Optimax uses to find the best set of values, requires a single "fitness function". Genetic Optimization works by running a bunch of different series of optimiztions and then changing those series slightly over time. New series start from the components of the most "fit" of the old series through a combination process called "mating". The idea is to mimic evolutionary biology.
A fitness funciton in this context gives a value for a set of parameters that is used to determine whether or not to keep pursuing a series of tests or to let that series die because other more "fit" series are available. In the evolutionary context, how successful that series is at propogating children. More fit series mate more and live longer thereby generating more children.
So the kitchen sink in this example comes more from their desire to let you make that fitness function conform to what you are personally looking for rather than a statement the one should use the entire kitchen sink.
A fitness funciton in this context gives a value for a set of parameters that is used to determine whether or not to keep pursuing a series of tests or to let that series die because other more "fit" series are available. In the evolutionary context, how successful that series is at propogating children. More fit series mate more and live longer thereby generating more children.
So the kitchen sink in this example comes more from their desire to let you make that fitness function conform to what you are personally looking for rather than a statement the one should use the entire kitchen sink.
Software Design
Understood. In fact, the whole notion of fitness function is to enable the user to define what "fit" really means. So more parameters = more choice for the user.So the kitchen sink in this example comes more from their desire to let you make that fitness function conform to what you are personally looking for rather than a statement the one should use the entire kitchen sink.
My comment is there are other ways in software to deal with variable inputs rather than force a user to user an unweildy API. For instance, a functor or call back could deliver/grab a narrow set of inputs instead of loading the stack with the kitchen sink. Another approach is to pass a facade that mediates access to subsequent values. In many cases the facade can be made as efficient as accessing original variable storage. These approaches are actually more flexible than a fixed set of parameters, since they enable extension after the API has been packaged. At any rate, given the number of parameters on this thing, I think any performance trade-offs with an alternate approach are probably acceptable.
Well, an opinion bit on software design. I digress.
Cheers,
Kevin
-
- Contributing Member
- Posts: 5
- Joined: Mon Mar 29, 2004 4:48 pm
Re: By What Measure? - How do You Know if a System is Good?
A good system IMO is:Forum Mgmnt wrote: - Drawdowns < 50%
- MAR Ratio > 1.2
- Returns > 35% for futures, 20% unleveraged stocks
- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks
This would be tested over 20 years using the same system parameters for each year and each market.
- Forum Mgmnt
Drawdown <20%
Returns >25%
I would prefer to use:
Drawdown <15%
Returns >35%
Re: By What Measure? - How do You Know if a System is Good?
For the most part, both returns and draw down can be controlled by the amount of risk allocated for position sizing. For example, a system with 35% return and 15% drawdown has a MAR ratio of 2.3. If you're comfortable with 30% draw down, the same system should return roughly 70%. If 10% draw down is more acceptable, then you should expect roughly 23% returns from the same system.Arthur W. Cutten wrote: A good system IMO is:
Drawdown <20%
Returns >25%
I would prefer to use:
Drawdown <15%
Returns >35%
Cheers,
Kevin
Last edited by ksberg on Fri Nov 05, 2004 9:16 pm, edited 1 time in total.
-
- Full Member
- Posts: 20
- Joined: Sat Oct 23, 2004 2:37 pm
- Location: Sunnyvale, CA
MAR
Can someone tell me what does MAR stand for? How is it calculated?
Thanks.
GC
Thanks.
GC
-
- Site Admin
- Posts: 9015
- Joined: Tue Apr 06, 2004 1:41 pm
- Location: Boston, MA
- Contact:
Hi George --
MAR stands for Minimum Acceptable Return.
As for how it's calculated, there are a couple of threads that are useful.
viewtopic.php?t=36
Basically the % CAGR divided by the % maximum drawdown over your test period. Nice clean measure of reward vs. pain.
Hope that helps,
Tim
MAR stands for Minimum Acceptable Return.
As for how it's calculated, there are a couple of threads that are useful.
viewtopic.php?t=36
Basically the % CAGR divided by the % maximum drawdown over your test period. Nice clean measure of reward vs. pain.
Hope that helps,
Tim
-
- Full Member
- Posts: 20
- Joined: Sat Oct 23, 2004 2:37 pm
- Location: Sunnyvale, CA
Thanks Tim
Thanks Tim.
GC
GC
-
- Senior Member
- Posts: 41
- Joined: Mon Oct 11, 2004 3:35 pm
How good is my system
In an article I did several years ago, I tested how robust a given system was by optimize it over a large range of values and measuring the returns and standard deviation of returns across the range of parameters. We wanted to see an average positive expectation greater than the standard deviation at minimum. I never followed up on this idea much past the research I did for the article , but I just wanted to pass it on.
-
- Roundtable Knight
- Posts: 100
- Joined: Wed Apr 30, 2003 2:27 pm
- Location: Minneapolis, MN
- Contact:
Re: How good is my system
I've applied this concept to my systems over the years also. It helps you see if the principle you're intent on trading actually has an advantage, and if it does---how much.Murray wrote:..., I tested how robust a given system was by optimize it over a large range of values and measuring the returns and standard deviation of returns across the range of parameters. We wanted to see an average positive expectation greater than the standard deviation at minimum.
And, if you decide to trade the principle because you now know that there is this overwhelming advantage---then you can spread your money across the different parameters so as to smooth your equity curve.
Leonardo----
Re: By What Measure? - How do You Know if a System is Good?
To re-arrange the above slightly and focusing just on futures:Forum Mgmnt wrote:But I'd consider adding a system to the portfolio of active systems I traded if it was at least:
- Drawdowns < 40%
- MAR Ratio > 2.0
- Returns > 50% for futures, 30% unleveraged stocks
This would be tested over 20 years using the same system parameters for each year and each market.
- CAGR > 50%
- MAR > 2.0
- MDD% < 40%
- CAGR > 10%
- MAR > 0.5
- MDD% < 30%
At the risk of being cheeky: Forum Mgmnt, would you consider saying a little about what sort/type of systems you trade? I'm not asking for specifics or a handout, just some indication of which system types fare better than others in the real world.
Thanks in advance!