Excellent idea. Since individual markets are often in different states, trending or not, volatile or not, etc. scrambling the portfolio gives you a better sense of the range of potential outcomes which might occur.
This too is a form of Monte-Carlo simulation and a good addition to the robustness checking Arsenal.
The reason I don't like synthetic data is the same reason I don't like Monte Carlo simulation of trades, it hides important relationships between the days that will tend to make the synthetic data kinder to your equity than the markets will be when one considers multi-market trading.
sluggo covered most of the issues in some detail above. First when things go bad, they tend to go bad all at once, and for a few days or months at a time, how do you preserve this with synthetic data? How does synthetic data account for the fact that on the day after the crash of 1987, I had my Eurodollar, Gold, Silver, Bonds, TBills and Sugar positions all gap against me in a massive way?
As sluggo mentioned, this might not matter if you are testing using single-market-at-a-time tools like TradeStation but I certainly don't recommend doing that either.
So I prefer ideas like nickmars and like what we do with Monte Carlo simulation in Trading Blox, we randomize while preserving the underylying information rooted in the human interactions in the markets.
On a related note, nickmar's suggestion gave me an interesting idea for extending Trading Blox's Monte Carlo analysis. In essence his portfolio manager lets you determine what would happen if you picked a different set of the available market data.
This got me thinking about what would happen if you did the same thing over time instead of markets. If you picked say only two or three years of data out of the many that were available. This is somewhat like what we do with Monte Carlo analysis but with one important difference: the simulation timeframe would be less than the underlying test from which the simulation data is drawn.
Currently we scramble chunks of the equity curve over the same length as the underlying test. So if you test for 20 years we scramble those 20 years of data. This gives you some idea how the 20 years could have been different.
This process does not actually answer the question that is on most people's minds when the decide to run a Monte Carlo test. I suspect most people are concerned with what the next few years might bring, or perhaps the first year of actual trading.
I suspect many people would like to use the 20 years of data as a pool from which to draw a test on a much shorter period, say 1 or 2 years instead of the entire 20 years. There will be much more variability in the results of a two-year extraction from a 20 year simulation. This increased variability is reflective of the actual increased variability one is likely to encounter when considering only one or two years of trading.
An improvement to Trading Blox's Monte Carlo simulation system like this should help traders get a more realistic sense of the possibilities for a given year of trading.
Finally, since I too have done many of these random tests using parameters whose sole purpose was to control the number of iterations in the test, it struck me that these are just another Monte Carlo test and it would be nice if we allowed you to specify that the test was a Monte Carlo test so that you could get the same output graphs, i.e. the distribution of max drawdowns, MAR, CAGR%, etc. that we get when we turn on the Monte Carlo reports.
I'm putting this idea into the suggestion box.
Thanks again everyone for an interesting dicussion, and nickmar in particular for stimulating my thinking here.
- Forum Mgmnt