System testing toolbox

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

System testing toolbox

Post by damian » Fri May 09, 2003 1:45 pm

good evening/morning/day,

There are so many ways of judging a systems performance. Some (to me) make sense only if the test is on a single market with constant position size. Even then they can be argued to be faulty as the $ value of a % change is different at different market levels.

I used to look at largest losing trade, but this really doesn't mean much when your position size is increasing over time (hopefully). In general, risk reward is a sound concept. I like to express each trade in terms of gain/loss as a % of initial risk (which is constant at x% over the test period). This allows me (I think) to evaluate one trade against another regardless of market, place in time or position size. However Profit/risk on each trade is only sound if you do not scale into the position.

What about DD? Do you think it is more useful in terms of $ or %? Discussions I have read so far suggest that it is limited value either way when not weighed against return. So far other threads cover equity curve profile v's return. I too like this class of indicators of goodness.

When it comes to portfolio testing, I still like to round 1 contract single market tests on each market in the portfolio so that I have on record some simple statistics regarding the performance of that market on my system. For example, the largest string of losing trades (but then we get into the curly topic of Montecarlo...)

In the case of portfolio backtesting with position sizing based on market volatility, what do you judge to be worthwhile metrics?

For every book I read I find a different preference. It is a little late and I am tired, however I do not want to evaluate one more system until I have identified a set orf useful metrics... and stick with them. In the past I have always ended up relying on % winners v's profit factor (or avg win/avg loss if profit factor was poor!!!) plus eye-balling the equity curve chart and the monthly % return bar chart. When it comes to evaluating the equity curve numerically, should I only look at the StDev of +ve moves? I read this somewhere but for the life of me can't find it again.

Sorry for all the questions and half questions asked here before really sitting down and working it all out my self. I am feeling decidedly amateur at the moment and want to do something about it.

Regardless of what my skill is in designing systems (FYI: average), I think it is more important to be able to evaluate a system. I might take a look at Futures Truth and see what they use as benchmarks. (I can hear you now: "What! He hasn't looked at Futures Truth yet?!")

cheers
damian

Bondtrader
Roundtable Knight
Roundtable Knight
Posts: 101
Joined: Mon Apr 28, 2003 11:07 am
Location: Zimbabwe

Post by Bondtrader » Fri May 09, 2003 4:32 pm

I firmly believe Robert Pardo's 13 year old book is still the most fertile source of ideas. He covers a dazzling array of measurements and indicators of goodness and figures of merit. Some of them are very clever and almost all of them are thought provoking.

A sample idea is Pardo's "Perfect Profit". Construct an equity curve that would have resulted if you were able to sell at the high of every day and buy at the low of every day. (if only!!) Now compare your system's equity curve against this Perfect Profit equity curve. A brilliant way to handle fluctuating volatility.

Yes, he puts in an annoying amount of "advertorial" stuff about his now-defunct product Advanced Trader. If you're able to ignore that and concentrate upon his ideas, I think you'll be pleased.

damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian » Fri May 09, 2003 11:42 pm

Thanks BT. I will investigate.

damian

Post Reply