c.f. obviously makes the point in his book that the MAR ratio and CAGR are not robust indicators of performance as they are heavily dependant on the start and end dates of the test
How do people feel about this given thats all we've had to work on until now.
In my opinion the global parameter of incremental start dates is a good way round this problem as it varies the start date and you can see how the system performs when you change the start and end dates - if you dont get much variation on your original test by doing this then I guess its a good indicator of robustness even using the traditional methods of performance measurement ?
Any thoughts
Regards
Chris
The Robustness of MAR and CAGR
Re: The Robustness of MAR and CAGR
Blox Builder and Blox Pro customers have had other "Measures of Goodness" for quite some time; they are in the customers-only section of this website, viewforum.php?f=61 . In November of 2005, for example, the Blox Marketplace provided free downloads ofChris67 wrote:given thats all we've had to work on until now. (MAR, CAGR)
- Ulcer Index
- Kestner K-Ratio
- Semideviation Ratio
- R-squared
- Return Retracement Ratio
to Sluggo for sharing the illumination...
System robustness...
Imho, all this variations and tradional concepts have an implicit methodic gap: they all show only a ratio for system robustness concerning the used historical test data.
But markets change and so you have to simulate also possible future time series of data (data scrambling, data simulation) and test your system in these new scenarios, to get a more complete answer concerning system robustness.
Only my two cents,
zentrader
But markets change and so you have to simulate also possible future time series of data (data scrambling, data simulation) and test your system in these new scenarios, to get a more complete answer concerning system robustness.
Only my two cents,
zentrader
zentrader,
you're mixing up two different things here I think. The usefulness/non-usefulness of scrambling up data has been discussed here before. Whatever your preference, that would still leave you with the question of what kind of measure of goodness to use.
That said, personally MAR is still my favourite because I feel that it adresses the most basic concern of every trader/investor: the price you have to pay (drawdown) for your performance (CAGR).
If I test my system on multiple time frames, stepping start-/end-dates as it is easily done in Trading Blox and if MAR is not too erratic, I consider the system robust.
However, other ratios such as R-Squared or Return Retracement Ratio are very useful (much more than MAR) to locate the shortcomings of a system and to help work on the actual coding of indicators and signals as opposed to pure parameter stepping.
you're mixing up two different things here I think. The usefulness/non-usefulness of scrambling up data has been discussed here before. Whatever your preference, that would still leave you with the question of what kind of measure of goodness to use.
That said, personally MAR is still my favourite because I feel that it adresses the most basic concern of every trader/investor: the price you have to pay (drawdown) for your performance (CAGR).
If I test my system on multiple time frames, stepping start-/end-dates as it is easily done in Trading Blox and if MAR is not too erratic, I consider the system robust.
However, other ratios such as R-Squared or Return Retracement Ratio are very useful (much more than MAR) to locate the shortcomings of a system and to help work on the actual coding of indicators and signals as opposed to pure parameter stepping.
personally i don't read much into MAR, except if it is High(say above 1.5), then i take it as a warning signal that the test is a "lucky fit". i.e. in an alternate universe the same system would not be so fortunate as to have such a small drawdown relative to it's returns. [i.e.not so fortunate with different data- i don't quite agree with Monte Carloing the equity curve, as that equity curve itself is only a subset of the different data universes]
I prefer annual Sharpe ratio. even though this is imperfect i have observed a strong correlation between this specific measure and my eyeballing a "good solid equity curve". The eyeball, while unscientific in method, is good at picking up detail that is missed by ratios.
Furthermore, i have observed that the best constructed portfolios with the smoothest equity curves often exhibit unremarkable MAR. e.g. mar of around 1.
to me MAR is a measure of how effectively you *might* have worked/leveraged your limited risk towards outstanding returns, but in the future it is more probable that you will not achieve that.
the annual sharpe ratio is no panacea though. depending on your choice of start date for the defining of "annual" the output is itself quite variable. tbh i don't like the modified sharpe ratio but that is because i don't like applying sharpe ratio logic to smaller units of time as the trend:noise drops. But Iwould like to suggest an improvement on annual sharpe ratio for inclusion in trading blox:
Namely, a Rolling Annual Sharpe Ratio, where the annual sharpe ratios are resampled/aggregated from the 12 different month start dates. This is not to say that it is constructed out of monthly returns, but just yearly returns starting on a different month than january. perhaps my logic is flawed but this would maybe give a more stable reading than Annual sharpe ratio.
I prefer annual Sharpe ratio. even though this is imperfect i have observed a strong correlation between this specific measure and my eyeballing a "good solid equity curve". The eyeball, while unscientific in method, is good at picking up detail that is missed by ratios.
Furthermore, i have observed that the best constructed portfolios with the smoothest equity curves often exhibit unremarkable MAR. e.g. mar of around 1.
to me MAR is a measure of how effectively you *might* have worked/leveraged your limited risk towards outstanding returns, but in the future it is more probable that you will not achieve that.
the annual sharpe ratio is no panacea though. depending on your choice of start date for the defining of "annual" the output is itself quite variable. tbh i don't like the modified sharpe ratio but that is because i don't like applying sharpe ratio logic to smaller units of time as the trend:noise drops. But Iwould like to suggest an improvement on annual sharpe ratio for inclusion in trading blox:
Namely, a Rolling Annual Sharpe Ratio, where the annual sharpe ratios are resampled/aggregated from the 12 different month start dates. This is not to say that it is constructed out of monthly returns, but just yearly returns starting on a different month than january. perhaps my logic is flawed but this would maybe give a more stable reading than Annual sharpe ratio.