Statistical significance of a new rule

LeviF · Post by **LeviF** » Thu Jan 29, 2009 3:14 pm

Say I add a new rule to a system that seems to improve results. How many instances should said rule come in to effect for it to be statistically significant? Obviously more is better, but is there any sort of rough guide?

Paul King · Post by **Paul King** » Thu Jan 29, 2009 5:16 pm

"... that seems to improve results"

Depends how you quantify this. I'd look at how the trade frequency changes when you apply the new rule, and also the average size of winners to losers. Also, I'd be suspicious if a new rule happened to significantly improve the win% since this may be eliminating losing trades by pure coincidence/curve fitting/date anomaly.

It also make a difference whether it's an entry rule, position sizing rule, or exit rule - they each should affect different aspects of your trading program in different ways (see attached model).

Hope this helps

LeviF · Post by **LeviF** » Thu Jan 29, 2009 9:49 pm

Here is a chart that summarizes the new "rule". It is a trade filter. The dark line is the R3 value and the pink line illustrates the number of instances the rule comes into effect.

Paul King · Post by **Paul King** » Fri Jan 30, 2009 8:45 am

What are the x-axis units (1-17)? Is that years, months, day, or something else?

LeviF · Post by **LeviF** » Fri Jan 30, 2009 9:00 am

lookback days.

Paul King · Post by **Paul King** » Fri Jan 30, 2009 10:15 am

How many trades are there when this rule is not implemented at all?

Just looking at the chart it appears to be a "robust" parameter with respect to R3 since it has a nice "bell curve" shape with the "sweet spot" lookback around 9 days, but only filtering out 100 trades.

If I was analyzing this I'd calculate System Value for each instance and see which one is the highest.

System Value = (Average R/STDEV(losing R) ) * number of trades per year (or day, week, month, depending on average trade duration).

Assuming you're still left with at least 100 trades in your sample after filtering i'd say the results were statistically significant, but it would be better to see a complete TBB report with the lookback stepped from 1 to 17.