Has anyone applied the ChiSquared tests as mentioned Kaufman's "Smarter Trading"? He describes how one can setup and monitor expected vs actual results; then test the probability of those results happening by chance. In other words, would those results be in line with expected system characteristics.
Now, the "normal" trader conditioning we seek is to ride out a draw down. But I must admit to trading a few systems where the right thing to do was bail when the system exhibited characteristics outside the norm. In each case, the system continued to plummet far beyond where I bailed and would have seriously damaged the account.
My trading plan includes conditions for emergency bailout. If the system shows evidence of not behaving as in the past, I'll step down position sizing all the way to single contract. If the evidence of failure continues to mount, then I pull the plug. I will continue monitoring theoretical system performance on new data.
So far, I haven't faced a case of bringing a system back yet. Some folks look for an upturn in the equity curve, which is certainly encouraging. I would really want to understand how the system went outofbounds, and reevaluate what I might expect to see in the future.
BTW: The systems were pattern based. One used intermarket relationships, and another traded coded price patterns. I've come to believe that patternbased systems are statistically sensitive to structural market changes.
Cheers,
Kevin
Dos and Donts of Drawdowns
I forgot to mention something that I had learnt:
Trading single contracts in most markets as your max allowable size means that your account is big enough to start trading but not big enough to allow you to manage drawdowns with any meaningful granularity in position sizing. This may be all well and good and it does not stop the game, but you need to ensure that you have the correct mind wiring before you expose yourself to such an eventuality.
With long term diversified portfolio trend following in mind: If your market timing is lucky and you have a >30% return each year for the first 2 years then USD100,000 is a big enough start account. Outside of a lucky run in your opening years I suggest that USD100,000 is not nearly large bullets to take into the arena.
Trading single contracts in most markets as your max allowable size means that your account is big enough to start trading but not big enough to allow you to manage drawdowns with any meaningful granularity in position sizing. This may be all well and good and it does not stop the game, but you need to ensure that you have the correct mind wiring before you expose yourself to such an eventuality.
With long term diversified portfolio trend following in mind: If your market timing is lucky and you have a >30% return each year for the first 2 years then USD100,000 is a big enough start account. Outside of a lucky run in your opening years I suggest that USD100,000 is not nearly large bullets to take into the arena.

 Contributing Member
 Posts: 8
 Joined: Fri Jun 06, 2003 8:57 pm
Kevin
Firstly, I'm no statistician and am still working on developing a trading system, just so you can give my comments whatever weight you think appropriate.
Regarding the Chisquared test, it assumes the sample variances are drawn from a normally distributed population, in this case the system's returns over two different time periods, if I understand correctly (I haven't yet read Kaufman). In fact, most standard statistical tests make the assumption that the underlying distribution(s) is normally distributed (or possibly lognormally). While this may be reasonable for many processes, it seems a dangerous one for marketrelated populations as it is known that market prices cannot generally be assumed to be lognormally distrubuted. I guess I'd want to have some reasonable assurance that the system returns were normally distributed before I would place much faith in the Chisquared test.
I am presently fooling around with methods of modeling the distrubution of volatility and this issue crops up, specifically: Is the measurement error of volatility normally distributed?
Anyway, I suppose it boils down to this: I am a bit gunshy of using standard statistical tests which are based on assumptions of normality.
If you have any insights, I'd be grateful to hear them.
RR
Firstly, I'm no statistician and am still working on developing a trading system, just so you can give my comments whatever weight you think appropriate.
Regarding the Chisquared test, it assumes the sample variances are drawn from a normally distributed population, in this case the system's returns over two different time periods, if I understand correctly (I haven't yet read Kaufman). In fact, most standard statistical tests make the assumption that the underlying distribution(s) is normally distributed (or possibly lognormally). While this may be reasonable for many processes, it seems a dangerous one for marketrelated populations as it is known that market prices cannot generally be assumed to be lognormally distrubuted. I guess I'd want to have some reasonable assurance that the system returns were normally distributed before I would place much faith in the Chisquared test.
I am presently fooling around with methods of modeling the distrubution of volatility and this issue crops up, specifically: Is the measurement error of volatility normally distributed?
Anyway, I suppose it boils down to this: I am a bit gunshy of using standard statistical tests which are based on assumptions of normality.
If you have any insights, I'd be grateful to hear them.
RR
Normal Distribution
RangeRover,
Excellent point. We'd have to really examine the distribution of what is being measured; it may or may not have a normal distribution.
In terms of variance between projected slippage and actual slippage, that may depend on how you're accounting for slippage. If, for instance, your fill function already accounted for changing market conditions, the projected vs. actual may stand a chance of being normally distributed.
I terms of variance of returns, we're likely to see some fat tailed, nonnormal distribution. I'm sure the math for equivalent Chisquare test gets more complex as well.
Kevin
Excellent point. We'd have to really examine the distribution of what is being measured; it may or may not have a normal distribution.
In terms of variance between projected slippage and actual slippage, that may depend on how you're accounting for slippage. If, for instance, your fill function already accounted for changing market conditions, the projected vs. actual may stand a chance of being normally distributed.
I terms of variance of returns, we're likely to see some fat tailed, nonnormal distribution. I'm sure the math for equivalent Chisquare test gets more complex as well.
Kevin