Measuring a systems performance accurately

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
mtinsley
Contributor
Contributor
Posts: 1
Joined: Wed Jun 03, 2009 8:37 am

Measuring a systems performance accurately

Post by mtinsley » Sat Jun 06, 2009 3:26 pm

Let me pose the following question: "If you backtest using a series of parameters to attempt to identify the best parameters for the system, how do you actually measure which paremeter set is best?". In other words, what 'performance measure' should you use?

This is something that I have grappled with for a long time now and I am still unsure whether I am doing it as effectively as I could. I think that this is something that a lot of people don't give the relevant consideration. I started out by simply using the '% Return' that each parameter resulted in. But often, this system, although in the long run gave the best %, in the short term, often made you bankrupt becuase of significant drawdowns. Then I started to incorporate the drawdown figure into my 'performance measure' but I have never really been sure how to do it. What makes is more difficult is when you test against different size data sets. How do you get a fair performance measure that can be compared across different size data sets.

Any suggestions welcome.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Sat Jun 06, 2009 4:26 pm

There's a chapter in "Way of the Turtle" (link @ amazon) entitled By What Measure? that discusses the problem and puts forward a proposed solution: (sneak peek)

There are also quite a few threads on this Trader's Roundtable that discuss similar concepts. Plentiful search results would result from trying search strings like "Measure of Goodness", "By What Measure", "Gain To Pain ratio", and so forth.

viewtopic.php?t=17&highlight=goodness

viewtopic.php?t=1699&highlight=goodness

viewtopic.php?t=6367&highlight=goodness

babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader » Mon Jun 08, 2009 7:20 pm

I would recommend the following for a good read on this subject
http://www.amazon.com/Evidence-Based-Te ... ingblox-20

Post Reply