Measuring a systems performance accurately
Posted: Sat Jun 06, 2009 3:26 pm
Let me pose the following question: "If you backtest using a series of parameters to attempt to identify the best parameters for the system, how do you actually measure which paremeter set is best?". In other words, what 'performance measure' should you use?
This is something that I have grappled with for a long time now and I am still unsure whether I am doing it as effectively as I could. I think that this is something that a lot of people don't give the relevant consideration. I started out by simply using the '% Return' that each parameter resulted in. But often, this system, although in the long run gave the best %, in the short term, often made you bankrupt becuase of significant drawdowns. Then I started to incorporate the drawdown figure into my 'performance measure' but I have never really been sure how to do it. What makes is more difficult is when you test against different size data sets. How do you get a fair performance measure that can be compared across different size data sets.
Any suggestions welcome.
This is something that I have grappled with for a long time now and I am still unsure whether I am doing it as effectively as I could. I think that this is something that a lot of people don't give the relevant consideration. I started out by simply using the '% Return' that each parameter resulted in. But often, this system, although in the long run gave the best %, in the short term, often made you bankrupt becuase of significant drawdowns. Then I started to incorporate the drawdown figure into my 'performance measure' but I have never really been sure how to do it. What makes is more difficult is when you test against different size data sets. How do you get a fair performance measure that can be compared across different size data sets.
Any suggestions welcome.