System testing toolbox
Posted: Fri May 09, 2003 1:45 pm
good evening/morning/day,
There are so many ways of judging a systems performance. Some (to me) make sense only if the test is on a single market with constant position size. Even then they can be argued to be faulty as the $ value of a % change is different at different market levels.
I used to look at largest losing trade, but this really doesn't mean much when your position size is increasing over time (hopefully). In general, risk reward is a sound concept. I like to express each trade in terms of gain/loss as a % of initial risk (which is constant at x% over the test period). This allows me (I think) to evaluate one trade against another regardless of market, place in time or position size. However Profit/risk on each trade is only sound if you do not scale into the position.
What about DD? Do you think it is more useful in terms of $ or %? Discussions I have read so far suggest that it is limited value either way when not weighed against return. So far other threads cover equity curve profile v's return. I too like this class of indicators of goodness.
When it comes to portfolio testing, I still like to round 1 contract single market tests on each market in the portfolio so that I have on record some simple statistics regarding the performance of that market on my system. For example, the largest string of losing trades (but then we get into the curly topic of Montecarlo...)
In the case of portfolio backtesting with position sizing based on market volatility, what do you judge to be worthwhile metrics?
For every book I read I find a different preference. It is a little late and I am tired, however I do not want to evaluate one more system until I have identified a set orf useful metrics... and stick with them. In the past I have always ended up relying on % winners v's profit factor (or avg win/avg loss if profit factor was poor!!!) plus eye-balling the equity curve chart and the monthly % return bar chart. When it comes to evaluating the equity curve numerically, should I only look at the StDev of +ve moves? I read this somewhere but for the life of me can't find it again.
Sorry for all the questions and half questions asked here before really sitting down and working it all out my self. I am feeling decidedly amateur at the moment and want to do something about it.
Regardless of what my skill is in designing systems (FYI: average), I think it is more important to be able to evaluate a system. I might take a look at Futures Truth and see what they use as benchmarks. (I can hear you now: "What! He hasn't looked at Futures Truth yet?!")
cheers
damian
There are so many ways of judging a systems performance. Some (to me) make sense only if the test is on a single market with constant position size. Even then they can be argued to be faulty as the $ value of a % change is different at different market levels.
I used to look at largest losing trade, but this really doesn't mean much when your position size is increasing over time (hopefully). In general, risk reward is a sound concept. I like to express each trade in terms of gain/loss as a % of initial risk (which is constant at x% over the test period). This allows me (I think) to evaluate one trade against another regardless of market, place in time or position size. However Profit/risk on each trade is only sound if you do not scale into the position.
What about DD? Do you think it is more useful in terms of $ or %? Discussions I have read so far suggest that it is limited value either way when not weighed against return. So far other threads cover equity curve profile v's return. I too like this class of indicators of goodness.
When it comes to portfolio testing, I still like to round 1 contract single market tests on each market in the portfolio so that I have on record some simple statistics regarding the performance of that market on my system. For example, the largest string of losing trades (but then we get into the curly topic of Montecarlo...)
In the case of portfolio backtesting with position sizing based on market volatility, what do you judge to be worthwhile metrics?
For every book I read I find a different preference. It is a little late and I am tired, however I do not want to evaluate one more system until I have identified a set orf useful metrics... and stick with them. In the past I have always ended up relying on % winners v's profit factor (or avg win/avg loss if profit factor was poor!!!) plus eye-balling the equity curve chart and the monthly % return bar chart. When it comes to evaluating the equity curve numerically, should I only look at the StDev of +ve moves? I read this somewhere but for the life of me can't find it again.
Sorry for all the questions and half questions asked here before really sitting down and working it all out my self. I am feeling decidedly amateur at the moment and want to do something about it.
Regardless of what my skill is in designing systems (FYI: average), I think it is more important to be able to evaluate a system. I might take a look at Futures Truth and see what they use as benchmarks. (I can hear you now: "What! He hasn't looked at Futures Truth yet?!")
cheers
damian