In your experience as a system trader... 80/40 or 40/40 ?

[RedRock]

Which is the more realistic system to expect a repete performance into the future.

System 1 earns 80%cagr over twenty years with a 40% max DD.

System 2 earms a mear 40% and has drawdowns of 46% max over the same period.

All things being equal, which would you rather trade with your own money??

My own answer is system two. It seems risk usually ends up equaling reward over a real trading period. The larger spreads seem wonderful, but in my history dont seem to live up to expectations or 'hopes'.

Curious about your experiences / thoughts...

Thanks, rr

[zentrader]

@RedRock,

give me the following information for both systems and I can give you a "monte carlo simulation" based answer:

1. number of winning trades
2. number of losing trades
3. average profit of a winning trade
4. average loss of a losing trade
5. highest loss of a losing trade
6. number of price bars for that system test
7. what kind of data? EOD or intraday (minute, 5 minutes, 15 minutes ...)

...you see "The Magnificent Seven" are important also for trading and analysis...

bye,
zentrader

[RedRock]

@RedRock,

...you see "The Magnificent Seven" are important also for trading and analysis...

bye,
zentrader

Wonderful film!

This (7) falls into the all things being equal category.

Ive run plenty of MC Sims which say I should have many happy stress free days ahead. But it always works out that the draw down is twice what it "should "be. [Given enough market time] And profits generally are less than advertised. I presume this is from good ol curve fitting. This has happened sooo many times, I now look for balanced risk and reward. A consistent 35% now seems pretty good. Am I just getting old and whipped\/\ /\/ or what?

[drip007]

Zen trader...can you offer advice about how to take actual historical data and scramble it to create an unlimited supply of synthetic data? Would like to use this to import into tradingblox and perform stress tests this way.

[drip007]

Are these hypotheticals using a diversified basket of instruments? If you are running this on a diversified portfolio and havent optimized your parameter settings, I would go with the first one. But if this is on one commoditity or one sector (eg grains) then I would make sure the entries and exits were not fitted in anyway before trading.

Just my two cents. I have been a systems trader for several years now.

[sluggo]

can you offer advice about how to take actual historical data and scramble it to create an unlimited supply of synthetic data? Would like to use this to import into tradingblox and perform stress tests this way.

viewtopic.php?t=2220
viewtopic.php?t=1670

[RedRock]

Are these hypotheticals using a diversified basket of instruments? If you are running this on a diversified portfolio and havent optimized your parameter settings, I would go with the first one. But if this is on one commoditity or one sector (eg grains) then I would make sure the entries and exits were not fitted in anyway before trading.

Just my two cents. I have been a systems trader for several years now.

Yes very broad and globally diversified portfolio of futures selecting liquidity leaders from each group. Basic trend following concepts.

But Drip, TUNING is a good thing, to a point. Are you saying just pull a number, or portfolio members from the air? No, I'm sure that wasn't it. Unless we are trading the Sluggo 100+ (all futures, period), in fact, its already curve fit in a way.

[zentrader]

@RedRock,

you know the words of Albert Einstein:
"Make everything as simple as possible, but not simpler..."

You are right, that mcs based system simulations also have their weakness: they are only valid under given market conditions (the underlying of the existing backtest results). But nevertheless it's a first step to compare the two systems. Do you have the information for the two systems or not?

A second step would then be, to use synthetic data (see below) to run new system backtests and new system simulations under other (hypothetique possible) market conditions...


@drip007,

in my software I implement the "data scrambling" in preparing the data using existing historical data , build a temporal dataset with relative relations of the OHLC data and then scramble this using random components. Additionally the user can select different options to customize the data generation (e.g. single price bars, blocks of price bars etc.)...see screenshot example (gif)

bye,
zentrader

[sluggo]

give me the following information for both systems and I can give you a "monte carlo simulation" based answer:

1. number of winning trades
2. number of losing trades
3. average profit of a winning trade
4. average loss of a losing trade
5. highest loss of a losing trade
6. number of price bars for that system test
7. what kind of data? EOD or intraday (minute, 5 minutes, 15 minutes ...)

Here are the seven numbers you request, for two different systems. Profits and losses are measured in R-multiples. So for example when it says that System 1's avg profit of winners = 2.374, that means that the average winning trade was 2.374 R's. On average, winning trades received a profit of 2.374 times the amount risked on the trade. The maximum loss for system 1 was 2.68 R's.

The two systems were run on the same slice of history, using the same portfolio, and using fixed fractional position sizing. The portfolio is reasonably well balanced, with 4 grain markets, 3 meats, 4 "softs", 3 metals, 3 energies, 3 stock indexes, 4 currencies, 4 long-term bonds, and 3 short-term interest rates. CSI symbols of these markets are overlaid on the equity curves shown below. The two trading systems are among the 10 built-in systems that ship with Trading Blox.

I have also attached Excel spreadsheets showing each individual trade, and its profit in R's. (Losing trades have a minus sign, i.e., their profit was a negative number of R's).

Everything you say you need, is provided here. I'm curious to see the results of your analysis.

+SLUGGO+

[zentrader]

@sluggo,

thank you very much for the information.

It's possible to show the simulation with the "R"-values but the result may be a little bit cryptic to understand.

I normally use values in real (future) points or currency.

For example:

If the basic instrument is the DAX Future, e.g. the average win is 75 points, the average loss is 50 points (1 point in the german DAX Future is 25 Euro, that is currently > 30 Dollar).

If the basic instrument are stocks or baskets it will be useful to know the average wins and losses in dollars !!!

Or tell me please, how much R is (in dollars!), than I can compute it exactly for a selected timeframe (profit/year etc.).

Thanks and bye,
zentrader

[sluggo]

It's possible to show the simulation with the "R"-values but the result may be a little bit cryptic to understand.

Good, let's stick to R-values. That's what TBB customers are used to thinking about. TBB gives lots of helpful information in terms of R-multiples (see two example TBB plots below, just grabbed at random, not from either sys1 or sys2).

R-multiples are ideal for situations where the system trades a portfolio of instruments (31 futures markets in this case), using fixed fractional position sizing, and numerous trades are ongoing at the same time. The simple-simon case of one system and one market and one-contract position sizes might be particularly easy to analyze, but the real world of real trading with real portfolios and real positionsizing doesn't work that way. Not at all.

System 1 and system 2 risk 1% of total account equity on each trade. (Which means that 1R = 1.0% of equity.) As you can see from the equity curve plots, over this particular span of 21 years System 1 has grown its equity by a factor of 500X. Which means that using fixed fractional position sizing, new entry signals were traded with 500X more contracts in Nov 1993, than they were in Feb 1973.

So please proceed ahead with the Monte Carlo work using R-multiples. Single-contract position sizes would be just plain silly.

[zentrader]

@sluggo,

"...Good, let's stick to R-values. That's what TBB customers are used to thinking about...Single-contract position sizes would be just plain silly..."

Perhaps I live on another planet, but in Europe you don't trade with R-values, but with Future points and/or real currency values (Euro, $ etc.). But if you are not able to give a ptractical input - anyway, let's try a "R-value" based simulation...

I ran 10,000 simulation runs with the number of the whole time period of each system (system1: > 10,000,000 trades, system2: > 5,000,000 trades), which last ca. 1-2 seconds...

I've attached the result screens for the system simulation as system1.gif and system2.gif.

bye,
zentrader

[BARLI]

RR, I wouldn't trade system 2.... return has to be at least 3 times bigger than the DD, so system 2 doesn't qualify for me

[sluggo]

zentrader, that's wonderful, thank you! I haven't read the manual of that software so a few things are a little confusing to me.

I suppose that "252 bars per time unit" means that you have defined "time unit" to be a year? (since these are EOD bars and since there are usually 252 EOD bars in a year). Is that correct?

I also suppose that the numbers in Green, such as Profit/time unit, should be interpreted in this situation to mean "R-multiples per time unit", which is "R-multiples per year"? Since we risked 1% per trade, 1R = 1% of account equity, and so am I right in concluding that the Monte Carlo estimate for System 1 is a profit somewhere between 28% per year (min) and 45% per year (max), with an average value of 36% per year? Do I interpret the green numbers correctly?

If so then the Monte Carlo estimates for System 2 would appear to be an average profit per year between 7R's per time unit (i.e. 7% per year), (min) and 22R's per time unit (i.e. 22% per year) (max). System 1's green numbers are much larger than System 2's green numbers.

I suppose the numbers in Red, the Drawdowns, are absolute magnitudes of drawdown, again in R's? System 1 seems to have dramatically smaller red numbers (MC drawdowns) than system 2.

Then a final pair of boxes, "Estimated Profit / time unit, VaR" and "Estimated drawdown, VaR" are shown at the bottom. Again System 1 has a much bigger number in the green (profit) box, along with a much smaller number in the red (drawdown) box.

I am especially curious to know, how to interpret these results in a way that helps me forecast the FUTURE performance of these two systems. Is your software's Monte Carlo analysis designed to assist in forecasting future performance? For example can I interpret the bottom green number "Estimated Profit / time unit, VaR" as a forecast of the system's profitability in the future?

Or how about relative comparisons among systems? Since System 1 seems to have received much better Monte Carlo results, in every category, than System 2, does this Monte Carlo analysis forecast that System 1 will outperform System 2 in the future? Is it mathematically proper to conclude "With a confidence level of 99% we can forecast that System 1 will outperform System 2 by approximately 19 R's per year (22 - 3 = 19)?"

Thanks, sluggo

[zentrader]

...I suppose that "252 bars per time unit" means that you have defined "time unit" to be a year? (since these are EOD bars and since there are usually 252 EOD bars in a year). Is that correct?...

Yes, exactly. That's correct. It may be little bit strange, but so the software can be used for intraday, EOD or other timeframes.

...I also suppose that the numbers in Green, such as Profit/time unit, should be interpreted in this situation to mean "R-multiples per time unit", which is "R-multiples per year"? Since we risked 1% per trade, 1R = 1% of account equity, and so am I right in concluding that the Monte Carlo estimate for System 1 is a profit somewhere between 28% per year (min) and 45% per year (max), with an average value of 36% per year? Do I interpret the green numbers correctly?...

I think so, but that's the point, that is a little bit confusing for me and the reason for my intention to use "hard" dollars or future pointe etc.

...If so then the Monte Carlo estimates for System 2 would appear to be an average profit per year between 7R's per time unit (i.e. 7% per year), (min) and 22R's per time unit (i.e. 22% per year) (max). System 1's green numbers are much larger than System 2's green numbers...

If R is equal for both systems, then it's correct.

...I suppose the numbers in Red, the Drawdowns, are absolute magnitudes of drawdown, again in R's? System 1 seems to have dramatically smaller red numbers (MC drawdowns) than system 2...

Yes, that's the point. You can compare two systems easily with my special ratios ACR and WCR. Here smaller ratios are better...

...Then a final pair of boxes, "Estimated Profit / time unit, VaR" and "Estimated drawdown, VaR" are shown at the bottom. Again System 1 has a much bigger number in the green (profit) box, along with a much smaller number in the red (drawdown) box....

The VaR-Estimation is to be seen additional to the absolute result of the mcs, we have discussed before. Here the estimated profit(VaR) shows the minimal expected profit in a simulation run with a statistical reliability of 99%. The estimated drawdown shows the maximum (negative) account drawdown which will be expceted with a security of 99%.

...I am especially curious to know, how to interpret these results in a way that helps me forecast the FUTURE performance of these two systems. Is your software's Monte Carlo analysis designed to assist in forecasting future performance? For example can I interpret the bottom green number "Estimated Profit / time unit, VaR" as a forecast of the system's profitability in the future?...

MCS based system simulations can always only be interpreted concerning the data which is used. In this case (my assumption) historical data is the basis so the results show a system charecteristics under these historical market conditions. If these conditions remain, the results may be valid also in future...

But for this important problem of changing market conditions i've implemented the second main mcs based functionality, the data simulation (I've posted the principle and a screenshot above in another forum contribution). So you can design future market conditions and test your systems again and again with that new data. You will get new backtest results and can start new system simulations - that's the process.

...Or how about relative comparisons among systems? Since System 1 seems to have received much better Monte Carlo results, in every category, than System 2, does this Monte Carlo analysis forecast that System 1 will outperform System 2 in the future? Is it mathematically proper to conclude "With a confidence level of 99% we can forecast that System 1 will outperform System 2 by approximately 19 R's per year (22 - 3 = 19)?"...

With system simulation imho definitely no, see the explanation before!
But if you use changed scenarios und you get new system simulation results you can compare systems with the absolute data of profits and account drawdowns or with the little helper ratios ACR and WCR.

I thank you also for this practical system test example!

bye,
zentrader

[sluggo]

I'm glad to hear that you believe Monte Carlo analysis cannot be used to forecast system performance in the future.

Although System 1 had a nice smooth equity curve during 1973-1994 (21 years!) and although it outperformed System 2 during that period, the markets changed. System 2 outperformed System 1 quite dramatically in 1994-2007(Q1), despite System 1's vastly nicer-looking Monte Carlo results for 1973-1994.

System 1 follows intermediate-term trends, and makes good profits when intermediate-term trends are smooth. These type of trends were plentiful before 1994 and not so plentiful afterwards. System 2 follows long-term trends, which were present the entire time from 1973 to 2007(Q1), but not quite as profitable as smooth intermediate-term trends. The full equity curves, from 1973 to 2007(Q1) are contained in the original .zip archive, see picture below. Their password is "Roundtable". I've copied those equity curve pictures here, and added a marker at 1994, showing the date where the MC simulations were run.

Obviously no software could predict that the nature of trends was going to change in the future. No software could predict that the type of trends which profit System 2 were going to keep occuring in the future, while the type of trends which profit System 1 were going to stop occurring in the future. Keep in mind that Systems 1 and 2 traded the identical same portfolio of markets for the entire period.

System 1 was not a "bad system"; it was merely a machine that profited handsomely when the markets behaved a certain way. The markets behaved that certain way for 21 years, and then they stopped. No analysis of the data of those 21 years, could have predicted the change. Not even Monte Carlo analysis.

And now we have 34 (!!) years worth of profits for System 2. Now that we see what happened from 1994 to 2007, System 2 looks like an excellent machine that profits quite nicely when the markets behave a (different) certain way. Especially in comparison to System 1 !! But: can we predict whether or not the markets will continue to behave that way in the future? No. Can Monte Carlo analysis predict whether or not System 2 will continue to be profitable in the future? No. Can we use any of this data to predict whether the markets will revert to their old behavior, such that System 1 starts to be profitable again? (e.g.(this post) )? No.

Too bad.

[RedRock]

)? No.

Too bad.

Now thats funny...

Thanks for the educational demonstrations. Any new (or not so) trader should take this experience to heart!

Perhaps a synthetic data generator would be a useful feature in TBB to 'stress test' the little darlings we create with backtesting...

rr

[zentrader]

...I'm glad to hear that you believe Monte Carlo analysis cannot be used to forecast system performance in the future...

D'accord. No one has the crystal ball. And if there is one who believes he has one, he should better read Taleb's "Fooled by randomness"...

Despite of that I think, with a two-step monte carlo simulation based addition to your "normal" system development process you can nevertheless reduce the possibilities of failure. It's a toolset to improve your development process.

I've described this in an article in Traders' Germany which was also translated for Traders' UK magazine February 2007:
http://www.zentrader.de/mcs_article_traders2007.pdf

bye,
zentrader

[sluggo]

Perhaps a synthetic data generator would be a useful feature in TBB to 'stress test' the little darlings we create with backtesting...

Where would you obtain the settings to operate the synthetic data generator? The synthetic data cannot be perfectly identical to the original, if it were then it is no help whatsoever. The synthetic data also cannot be pure white noise with infinite bandwidth and infinite amplitude; that would be infinitely different from market price behavior and it would be no help either.

What you want from synthetic data is to be "somewhat" different from historical data, but not "too much" different.

Sounds like a human-adjustable parameter to me. Who gets to decide how different is "somewhat" different, and how different is "too much" different? A human being. Either you, the user of the synthetic data generator (bad), or else the software developer who wrote the synthetic data generator (worse).

What is the goal of synthetic data? Perhaps, to stress-test trading systems with "somewhat realistic" price streams that were unseen during the system development process. Perhaps also, to stress-test trading systems with price streams that are "somewhat more challenging" than historical prices. Maybe throw some not-so-smooth intermediate term trends at System 1, for example.

Who gets to decide where the border line is, between useful "somewhat realistic" price streams, and non-useful "not very realistic" price streams? A human being. Either the software user or the software developer. And they can guess wrong.

To generate synthetic data, is to predict the future. Inherently you must make a prediction of "How different is the future likely to be, than the past?" You can predict too optimistically, too pesimistically, or just right. If you are too optimistic, you and your system will be surprised by price events in the future, events you are unprepared for. If you are too pessimistic, you will develop systems with needlessly excessive "armor plating" (for example, stops that are far too wide) and you will choose a position size far too small (in preparation for a nuclear winter of unfavorable market behavior ... that never comes) and so you will miss out on profits.

But, how do you know what is too optimistic, vs. too pessimistic, vs. just right? You don't know. You must predict the future, and hope that you have predicted well.

Maybe it will make you feel good to run your system on synthetic data with the "difference from the past" knob set to "10". Maybe your system performs well. Maybe you create another wad of synthetic data with the difference knob set to 20, and maybe your system still performs well. Maybe you do this two dozen times and discover that your system performs well for "difference from the past" knob settings anywhere from 0 to 30. Then at settings from 31-40 your system is still profitable but not nearly as nice as before. Then at settings above 40 your system is uprofitable. This might make you feel very good; your system seems capable of handling the past prices, AND synthetic prices that are 40 units of "somewhat" different from the past. Great. Now you want to trade it in the future. How many units of "somewhat" will the future be, than the past? Nobody knows. You must guess, um, I mean, you must forecast.

[drip007]

Zentrader,

How can I use your services for stress testing? My first choice is to have a way of generating synthetic data on the fly that I can import into tblox and use as a possible future out of sample data set.

I am not technical enough nor am i inclined to bother with all of the details in setting up something like this on my own or through reading an instruction manual.

THoughts?