Reliable number of trades?
-
LeviF
- Roundtable Knight
- Posts: 1436
- Joined: Mon Dec 22, 2003 12:24 pm
- Location: Des Moines, IA
- Contact:
Reliable number of trades?
How does one know if a system generates enough trades to be statistically reliable into the future? Would 100 trades over 10 years on a basket of 30 futures be adequate?
Just some random thoughts on the matter.
Philosophical answer:
You never "know" for sure, because, in theory at least, the conditions that existed in the market during your test period may have been unique, and have changed in the current market. With one trade per contract every three years or so, I guess there's a real possibility of some kind of outlier in that data. Statistics involves assumptions. Not comforting, I know. So much of this is having confidence or a reason to believe we have an edge, without ever knowing for sure. Well, sometimes you almost maybe know for sure.
Short statistical answer: Maybe. More would be better.
Long statistical answer: Statistics involves assumptions, and with only 100 data points, you're not on super ground, but it depends on what kind of results those 100 data points have. Assume that a trade is a trade, regardless of contract or source. BIG assumption. Then with 100 trades you have a payout distribution that can be tested.
You could take the 100 trades, convert them into +/- percent of equity, and categorize them into a distribution that had a mean, standard deviation, skewness, kurtosis, etc., and perform some tests on that, too. With only 100 in the sample, unless it looks really promising from a profitability standpoint, you might want to proceed with some caution. Any big bad losses? Look at the losses and what the contracts were doing, and think if that's a common situation or what the cause might be, how to eliminate those big losses.
Winning percentage could be tested with the binomial approximation to the normal, which assumes some things, like a normal distribution. The population mean winning percentage is estimated by your sample of 100 trades, and the standard deviation is the square root of the product of win %, loss %, and number in the sample (in this case, 100). For example, if your sample was 35% winners, then statistically you can have about 98% confidence that your system has a future winning percentage between 25% and 45%. Assuming your assumptions are good.
Now if you take your average win and average loss amounts as percentages of equity, and *assume* that these distributions are solid, you could calculate your expectancy based on the lower bound (25% wins in the example), the sample amount (35% wins in the example), and the upper bound (45% wins in the example) of the winning percentage. Are all of those numbers "good" to you? Could you take that system at the lower bound of win % and turn over enough trades to make a good return? Is the system even profitable at that lower bound? What's the maximum number of losses in a row you could expect at that lower bound, and with ten trades a year, what are the odds that you go winless in a year? Would that make you stop trading?
Bottom line, we all have to make decisions with very imperfect data. You have to be comfortable with what your analysis of those 100 trades gives you. If you're not, could you backtest longer? More contracts? I don't know.
Philosophical answer:
You never "know" for sure, because, in theory at least, the conditions that existed in the market during your test period may have been unique, and have changed in the current market. With one trade per contract every three years or so, I guess there's a real possibility of some kind of outlier in that data. Statistics involves assumptions. Not comforting, I know. So much of this is having confidence or a reason to believe we have an edge, without ever knowing for sure. Well, sometimes you almost maybe know for sure.
Short statistical answer: Maybe. More would be better.
Long statistical answer: Statistics involves assumptions, and with only 100 data points, you're not on super ground, but it depends on what kind of results those 100 data points have. Assume that a trade is a trade, regardless of contract or source. BIG assumption. Then with 100 trades you have a payout distribution that can be tested.
You could take the 100 trades, convert them into +/- percent of equity, and categorize them into a distribution that had a mean, standard deviation, skewness, kurtosis, etc., and perform some tests on that, too. With only 100 in the sample, unless it looks really promising from a profitability standpoint, you might want to proceed with some caution. Any big bad losses? Look at the losses and what the contracts were doing, and think if that's a common situation or what the cause might be, how to eliminate those big losses.
Winning percentage could be tested with the binomial approximation to the normal, which assumes some things, like a normal distribution. The population mean winning percentage is estimated by your sample of 100 trades, and the standard deviation is the square root of the product of win %, loss %, and number in the sample (in this case, 100). For example, if your sample was 35% winners, then statistically you can have about 98% confidence that your system has a future winning percentage between 25% and 45%. Assuming your assumptions are good.
Now if you take your average win and average loss amounts as percentages of equity, and *assume* that these distributions are solid, you could calculate your expectancy based on the lower bound (25% wins in the example), the sample amount (35% wins in the example), and the upper bound (45% wins in the example) of the winning percentage. Are all of those numbers "good" to you? Could you take that system at the lower bound of win % and turn over enough trades to make a good return? Is the system even profitable at that lower bound? What's the maximum number of losses in a row you could expect at that lower bound, and with ten trades a year, what are the odds that you go winless in a year? Would that make you stop trading?
Bottom line, we all have to make decisions with very imperfect data. You have to be comfortable with what your analysis of those 100 trades gives you. If you're not, could you backtest longer? More contracts? I don't know.
A trading system exploits (profits from) a certain kind of price action. When market prices act this certain way, the system makes profitable trades. And when market prices don't act this certain way, the system makes losing trades.
If you decide to trade a system with real money, you are betting that future price action will be the kind that your system profits from. You are making a prediction: that prices will act a certain way (similar to the way they acted in the past, whose historical data gave you nice backtest results). However there is a problem. Your prediction might not come true. The markets might change (see (this thread) for an example) and if they do, your system can turn unprofitable.
So I don't know of any way to guarantee that a system will be profitable in the future. Nor do I know of any way to prove that a system will be "statistically reliable into the future" as our Original Poster asks. The makets can change. Backtests, performed on past prices, won't help you predict whether or not the markets will change and turn your system into a loser. If the markets change, it doesn't matter how many trades were in your backtest; your backtest becomes irrelevant when future prices don't resemble past prices.
I think about all you can do and all you can hope for is to get comfortable with a system, which is a highly personal & subjective judgement. What would make one person feel comfortable, might not satisfy someone else.
Some things that might or might not make you feel comfortable, include
If you decide to trade a system with real money, you are betting that future price action will be the kind that your system profits from. You are making a prediction: that prices will act a certain way (similar to the way they acted in the past, whose historical data gave you nice backtest results). However there is a problem. Your prediction might not come true. The markets might change (see (this thread) for an example) and if they do, your system can turn unprofitable.
So I don't know of any way to guarantee that a system will be profitable in the future. Nor do I know of any way to prove that a system will be "statistically reliable into the future" as our Original Poster asks. The makets can change. Backtests, performed on past prices, won't help you predict whether or not the markets will change and turn your system into a loser. If the markets change, it doesn't matter how many trades were in your backtest; your backtest becomes irrelevant when future prices don't resemble past prices.
I think about all you can do and all you can hope for is to get comfortable with a system, which is a highly personal & subjective judgement. What would make one person feel comfortable, might not satisfy someone else.
Some things that might or might not make you feel comfortable, include
- Seeing that the system's performance is "good" (according to your judgment) over a wide range of parameter values
- Seeing that the system's performance is "good" across many different tradeable instruments
- Seeing that the system's performance is "good" over a large piece of past history that includes many different market behaviors
- Seeing that the system's performance is "good" across many different market sectors
- Seeing that the system's performance is "better" (according to your judgment) than well known reference systems such as buy-and-hold, or 200 day MA, or 20 day Donchian channel breakout, or whatever is the appropriate reference system
- Seeing that Monte Carlo resampling of your sysem's longterm equity curve, produces an "acceptable" distribution of outcomes (according to your judgment)