Is Monte Carlo Analysis Useful?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Is Monte Carlo Analysis Useful?

Post by Forum Mgmnt »

[MODERATOR'S NOTE: This topic was split from one in the Trading Blox Support Area discussing the new Monte Carlo feature in Trading Blox.]
Old European wrote:I first want to say that I'm convinced that c.f. (and sluggo) is doing a great job in implementing Monte Carlo in TBB. The graphs presented so far look really really nice.

I also think I understand why he is adding this Monte Carlo functionality: the system developers' marketplace asks for it and competing products allow also for MC testing.

I however have a fundamental theoretical problem with Monte Carlo and system testing.

As sluggo described very well above, it's all about looking at trading results in parallel universes. These universes can be created in two (or more) different ways. Or one randomizes the order in which trades are generated or one randomizes the daily (or ...) returns.

I am not convinced that this makes a lot of sense. While backtesting I always note that a lot of the time when I'm long USD against EUR I am also long USD against CHF ... when I'm short LC I am also short FC ... and so on. There are inter-market correlations. When one randomizes the trade order, one doesn't take these correlations into account and the newly created parallel universe has nothing to do with the real world.

The same thing applies to the randomization of daily returns. Again when backtesting I always observe that there are regularly periods of mostly positive daily returns. Why? Because there are strong trends in a number of markets and a market trends until it finally bends. When the daily returns are randomized, all of this gets lost and the newly create world is very ir-real.

Ideally one should only look at parallel universes and virtual market data time series that show similar correlations and trending behavior as the real world. These are however extremely difficult to construct.

Cheers,

Old European
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

I had the same gut reaction to Monte Carlo analysis of Trading Systems when I first examined the idea. However, I now see it and its output like any other statistic from any simulation, more information useful for some things but with limitations or shortcomings; If we examined any statistic: MAR, Sharpe, Maximum Drawdown, etc. we could make an argument for it being incomplete, innaccurate, misleading...

I see three major shortcomings with Monte Carlo analysis when performed in the traditional ways:
  1. No accounting for serial correlation of returns.
  2. No accounting for intermarket correlation.
  3. No accounting for the psycholigical effects of the market.
Some have attempted to overcome some of these limitations by altering the algorithm for the Monte Carlo simulations.

Indeed, we do this partially with Trading Blox' upcoming Monte Carlo feature. We allow users to group days together to preserve the effects of multi-day market movements and to partially account for serial correlation (sometimes referred to as autocorrelation).

What we are left with, after our best attempts at improvement, is imperfect. It does not have the same meaning as might an analysis of failure rates for parts coming off an assembly line. A 95% confidence level does not mean that there is really a mathematical 95% chance that something will or will not happen.

Nevertheless, randomly scrambling our equity curves does show us how sensitive our outputs are to the specific ordering of events. It also, in the process, erodes our faith in the numbers we get as outputs to a specific simulation. This is a good thing.

In the beginning especially, we tend to think of test results as authoritative and weighty; e.g. this system is better than that because MAR of 1.9 is better than 1.75 , or Sharpe of 1.4 is better than 1.7 etc. This "precision" is misleading as the likely margin of error often exceeds all but the largest differences in performance testing output. So to the extent that looking at the outputs of Monte Carlo analysis erodes our "faith" in the specific outputs of a single test, this is a good thing.

Also if we can alter/lower our expectations as a result of seeing graphs showing a 20% chance of catastrophe based only on randomly changing the equity each day, that too is a good thing. It will be far easier to ride out a drawdown if we expected it because of Monte Carlo modeling than if it freaks us out because it exceeded our historical test by some specific but arbitrary margin.

- Forum Mgmnt
Old European
Roundtable Knight
Roundtable Knight
Posts: 144
Joined: Mon Apr 19, 2004 3:04 pm
Location: Old Europe

Post by Old European »

Forum Mgmnt,

I couldn't agree more with what you say. MC, although theoretically flawed, can indeed help remind us of the intrinsic unstability of mechanical trading systems (despite all the effort we put into improving their robustness).

Cheers,

Old European
Christian Smart
Roundtable Fellow
Roundtable Fellow
Posts: 50
Joined: Fri Apr 18, 2003 8:53 pm
Location: Huntsville, AL
Contact:

Post by Christian Smart »

Faulting Monte Carlo by saying that it cannot account for correlation is like faulting Calculus because it is limited to single variables. That's an incorrect claim. Correlation can definitely be accounted for in Monte Carlo analysis. There are standard techniques of accounting for correlation and lots of published research, dating back at least 20 years. One type of correlation has been implemented in commerical Excel add-in packages including Crystal Ball and @Risk.

And there is an even easier technique that is simpler and can be calculated on a hand calculator. The method of moments can be used to aggregate the mean and standard deviations of the returns for each market (correlation is easy to account for in this approach). Given enough markets (and a couple of other conditions that most likely will hold), the two top-level moments can be fit to a Normal distribution. In my work, I have found that this analytic approximation is as accurate as Monte Carlo simulation.
marriot
Roundtable Knight
Roundtable Knight
Posts: 365
Joined: Thu Nov 20, 2008 3:02 am

MonteCarlo

Post by marriot »

Still trying to understand montecarlo analysys, i came up with these differents montecarlo lines.
One is on the center, the other is almost on the lowest.
Does it mean something?
thank you
Attachments
mc low.jpg
mc low.jpg (47.6 KiB) Viewed 5425 times
mc medium.jpg
mc medium.jpg (52.19 KiB) Viewed 5426 times
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Re: MonteCarlo

Post by LeviF »

marriot wrote:Does it mean something?
No.
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Re: MonteCarlo

Post by RedRock »

Levi wrote:
marriot wrote:Does it mean something?
No.
A prediction of the future? Prolly not. But I do infer aspects of 'health and character' from these displays. Some may find scientific value within (or not). For me, the value is more right brain.
Post Reply