How to interpret Monte Carlo results

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

How to interpret Monte Carlo results

Post by fab1usa1 »

My apologies in advance if this is not the proper forum for this question...

I am trying to understand the difference between the CAGR that is reported in the "Stepped Parameter Performance Summary" at the top of the report and the "90% Return" statistic report at the end of the report.

I am backtesting my long-only stock system from 1/1/2000 through 6/30/2000 and arrive at a CAGR of 21.55%. Am I correct in assuming that that would have been my return if I followed the order generator exactly?

Now when I look at the Monte Carlo simulation I see that the return is 11.00% with 90% confidence. Am I correct in assuming that this will probably be my return going forward into the future?

Thanks for all replies.
fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Post by fab1usa1 »

Sorry, typo, that date range was misquoted. It should have read: 1/1/2000 to 6/30/2011.
Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon »

The typical way a monte carlo simulation works under these circumstances is to take the equity curve from your backtest and break it into smaller chunks (monthly returns, for example). Then you assume this represents the population of returns you can expect from your system.

You then re-create alternate equity curves by drawing with replacement from the population at random. You do this over and over again either a fixed number of times (naive method) or until the simulation achieves some level of accuracy you specify.

This gives you a population of equity curves, each with its own CAGR and drawdown characteristics. Obviously, path-dependent characteristics like max equity drawdown can come out wildly different from your original equity curve. But also, because you are not simply shuffling the order of returns (the simulated equity curves will contain repeats of some of the original months, and some of the original months will not appear), non-path-dependent characteristics like CAGR will also come out different.

The reported MC results will tell you, for instance, that, of all the equity curves generated using this method, 90% of them generated a CAGR (max drawdown, etc) of 11% or greater. I believe it is incorrect to interpret the results along the more familiar "there is a 90% probability that my future returns will be greater than 11%".

I am not sure of the EXACT details of how TB "chunks" the equity curve (for example what the time slices are, if they are equal, if there is any serial correlation used), but the above explanation whould be correct in broad brush strokes.

edit: grammar / typos
fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Post by fab1usa1 »

Thank you, EventHorizon, I understand better now.

Is it typical to have your "actual" CAGR greater than the 90% MC CAGR, as in my case where 21.55% > 11.00% ?

Should the objective of my trading system be to make the MC CAGR approximately equal to the actual CAGR? Is that the sign of a good design?

Thanks again.
Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon »

Yes, you would expect that your actual back-test CAGR would be higher than the worst 10% (1 - 90%) of the simulated equity curve CAGRs.

Consider an equity curve from a back-test that consists of 3 monthly returns: 0%, -2%, +5%. Over the 3 months the Cumulative Monthly Growth Rate was 0.957% (cubed root of 1 x 0.98 x 1.05). Drawing from this population at random with replacement we could have gotten the following combinations that would be worse than our back test: -2, -2, -2; -2, -2, 0; -2, -2, 5; -2, 0, 0; 0, 0, 0. each of these combinations would have turned up 3.7% (1/3 cubed) of the time in our MC simulation. The total possible orderings of these combinations are 1, 3, 3, 3, 1 respectively.

So the actual equity curve from the backtest is better than 40.7% (1 + 3 + 3 + 3 + 1) x 3.7% of all the possible equity curves from the MC simulation. Further, 85.2% (the closest I can get to 90%) of the equity curves deliver better than -1.34% (-2%, -2%, 0). The 0.957% achieved in the original backtest is a lot better than -1.34% (just as your +21.55% is a lot better than 11%).

You might need to check my math - I just had a couple Avery Maharaja Dry-Hopped double IPA's at the Avery Tap Room!
fab1usa1
Roundtable Knight
Roundtable Knight
Posts: 383
Joined: Sat May 21, 2011 8:30 pm
Location: New York

Post by fab1usa1 »

Very interesting but definately NOT stupid.

So the math suggests that in order to make MC CAGR approximately equal to Backtest CAGR, that a system should have:
1. Zero maximum drawdown.
2. A constant-slope equity curve, logarithmically speaking.

One of the things I really like about TB are the stepped parameters. EventHorizon, I feel awful asking you this, but can you point me in the direction of a specific set of parameters to concentrate on?

I am a Stella Artois man. (It's 9am and you are making me thirsty, not a good sign!)
Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon »

fab1usa1 wrote:So the math suggests that in order to make MC CAGR approximately equal to Backtest CAGR

Hmmm ... The back test CAGR is what it is. The MC CAGR isn't a single CAGR it is a stochastic function. It is not like a single valued function (y = x^2), the result is a distribution of values. So there isn't really a question of the MC CAGR equaling the back-test CAGR.

Under very special circumstances (an equal return for every period in the sample) you would get a constant slope logarithmic equity curve. In addition, as long as each period return was positive, the maximum draw-down would indeed be zero - as it would if each and every period return was > 0.

The likelihood that the back test CAGR is greater than, say, the median CAGR of the MC simulation is a function of the distribution of returns. My guess is that if the back-test CAGR had a small number of very big winning months, then the MC CAGR would likely have a median BELOW the back-test. OTOH if the back-test had a small number of big losers then the MC CAGR would likely have a median ABOVE then back-test CAGR.

Try setting up some simple examples using a small population of monthly returns (e.g. 2) and enumerate all the possible combinations to see what happens.

In terms of parameters to focus on there's not much I can say other than depends on your system! There is a wealth of information on these bulletin boards from issues such as how many parameters to use to scaling in and out of positions, etc. Get your reading glasses on and have at it!!
Post Reply