In your experience as a system trader... 80/40 or 40/40 ?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

In your experience as a system trader... 80/40 or 40/40 ?

Post by RedRock »

Which is the more realistic system to expect a repete performance into the future.

System 1 earns 80%cagr over twenty years with a 40% max DD.

System 2 earms a mear 40% and has drawdowns of 46% max over the same period.

All things being equal, which would you rather trade with your own money??

My own answer is system two. It seems risk usually ends up equaling reward over a real trading period. The larger spreads seem wonderful, but in my history dont seem to live up to expectations or 'hopes'.

Curious about your experiences / thoughts... :?:

Thanks, rr
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

System1 or System2?

Post by zentrader »

@RedRock,

give me the following information for both systems and I can give you a "monte carlo simulation" based answer:

1. number of winning trades
2. number of losing trades
3. average profit of a winning trade
4. average loss of a losing trade
5. highest loss of a losing trade
6. number of price bars for that system test
7. what kind of data? EOD or intraday (minute, 5 minutes, 15 minutes ...)

...you see "The Magnificent Seven" are important also for trading and analysis... :-)

bye,
zentrader
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Re: System1 or System2?

Post by RedRock »

zentrader wrote:@RedRock,

...you see "The Magnificent Seven" are important also for trading and analysis... :-)

bye,
zentrader
Wonderful film!

This (7) falls into the all things being equal category.

Ive run plenty of MC Sims which say I should have many happy stress free days ahead. But it always works out that the draw down is twice what it "should "be. [Given enough market time] And profits generally are less than advertised. I presume this is from good ol curve fitting. This has happened sooo many times, I now look for balanced risk and reward. A consistent 35% now seems pretty good. Am I just getting old and whipped\/\ :shock: /\/ or what?
drip007
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Mon Nov 27, 2006 2:54 pm

simulating data

Post by drip007 »

Zen trader...can you offer advice about how to take actual historical data and scramble it to create an unlimited supply of synthetic data? Would like to use this to import into tradingblox and perform stress tests this way.
drip007
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Mon Nov 27, 2006 2:54 pm

Post by drip007 »

Are these hypotheticals using a diversified basket of instruments? If you are running this on a diversified portfolio and havent optimized your parameter settings, I would go with the first one. But if this is on one commoditity or one sector (eg grains) then I would make sure the entries and exits were not fitted in anyway before trading.

Just my two cents. I have been a systems trader for several years now.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Re: simulating data

Post by sluggo »

drip007 wrote:can you offer advice about how to take actual historical data and scramble it to create an unlimited supply of synthetic data? Would like to use this to import into tradingblox and perform stress tests this way.
viewtopic.php?t=2220
viewtopic.php?t=1670
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

drip007 wrote:Are these hypotheticals using a diversified basket of instruments? If you are running this on a diversified portfolio and havent optimized your parameter settings, I would go with the first one. But if this is on one commoditity or one sector (eg grains) then I would make sure the entries and exits were not fitted in anyway before trading.

Just my two cents. I have been a systems trader for several years now.
Yes very broad and globally diversified portfolio of futures selecting liquidity leaders from each group. Basic trend following concepts.

But Drip, TUNING is a good thing, to a point. Are you saying just pull a number, or portfolio members from the air? No, I'm sure that wasn't it. Unless we are trading the Sluggo 100+ (all futures, period), in fact, its already curve fit in a way.
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

MCS...

Post by zentrader »

@RedRock,

you know the words of Albert Einstein:
"Make everything as simple as possible, but not simpler..." :-)

You are right, that mcs based system simulations also have their weakness: they are only valid under given market conditions (the underlying of the existing backtest results). But nevertheless it's a first step to compare the two systems. Do you have the information for the two systems or not?

A second step would then be, to use synthetic data (see below) to run new system backtests and new system simulations under other (hypothetique possible) market conditions...


@drip007,

in my software I implement the "data scrambling" in preparing the data using existing historical data , build a temporal dataset with relative relations of the OHLC data and then scramble this using random components. Additionally the user can select different options to customize the data generation (e.g. single price bars, blocks of price bars etc.)...see screenshot example (gif)

bye,
zentrader
Attachments
mcs_data_simulation.gif
mcs_data_simulation.gif (16.54 KiB) Viewed 22475 times
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Re: System1 or System2?

Post by sluggo »

zentrader wrote:give me the following information for both systems and I can give you a "monte carlo simulation" based answer:

1. number of winning trades
2. number of losing trades
3. average profit of a winning trade
4. average loss of a losing trade
5. highest loss of a losing trade
6. number of price bars for that system test
7. what kind of data? EOD or intraday (minute, 5 minutes, 15 minutes ...)
Here are the seven numbers you request, for two different systems. Profits and losses are measured in R-multiples. So for example when it says that System 1's avg profit of winners = 2.374, that means that the average winning trade was 2.374 R's. On average, winning trades received a profit of 2.374 times the amount risked on the trade. The maximum loss for system 1 was 2.68 R's.

The two systems were run on the same slice of history, using the same portfolio, and using fixed fractional position sizing. The portfolio is reasonably well balanced, with 4 grain markets, 3 meats, 4 "softs", 3 metals, 3 energies, 3 stock indexes, 4 currencies, 4 long-term bonds, and 3 short-term interest rates. CSI symbols of these markets are overlaid on the equity curves shown below. The two trading systems are among the 10 built-in systems that ship with Trading Blox.

I have also attached Excel spreadsheets showing each individual trade, and its profit in R's. (Losing trades have a minus sign, i.e., their profit was a negative number of R's).

Everything you say you need, is provided here. I'm curious to see the results of your analysis.

+SLUGGO+
Attachments
testData_for_ZT.zip
Excel spreadsheets of trade outcomes, plus all equity curves and other image files
(244.36 KiB) Downloaded 918 times
System 1 and System 2 equity curves using Fixed Fractional position sizing
System 1 and System 2 equity curves using Fixed Fractional position sizing
eqcurves_thru_1994.png (63.19 KiB) Viewed 22431 times
The seven pieces of data needed by ZT software, for Systems 1 and 2
The seven pieces of data needed by ZT software, for Systems 1 and 2
the_stats.png (11.27 KiB) Viewed 22426 times
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

mcs example...

Post by zentrader »

@sluggo,

thank you very much for the information.

It's possible to show the simulation with the "R"-values but the result may be a little bit cryptic to understand.

I normally use values in real (future) points or currency.

For example:

If the basic instrument is the DAX Future, e.g. the average win is 75 points, the average loss is 50 points (1 point in the german DAX Future is 25 Euro, that is currently > 30 Dollar).

If the basic instrument are stocks or baskets it will be useful to know the average wins and losses in dollars !!!

Or tell me please, how much R is (in dollars!), than I can compute it exactly for a selected timeframe (profit/year etc.).

Thanks and bye,
zentrader
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Re: mcs example...

Post by sluggo »

zentrader wrote:It's possible to show the simulation with the "R"-values but the result may be a little bit cryptic to understand.
Good, let's stick to R-values. That's what TBB customers are used to thinking about. TBB gives lots of helpful information in terms of R-multiples (see two example TBB plots below, just grabbed at random, not from either sys1 or sys2).

R-multiples are ideal for situations where the system trades a portfolio of instruments (31 futures markets in this case), using fixed fractional position sizing, and numerous trades are ongoing at the same time. The simple-simon case of one system and one market and one-contract position sizes might be particularly easy to analyze, but the real world of real trading with real portfolios and real positionsizing doesn't work that way. Not at all.

System 1 and system 2 risk 1% of total account equity on each trade. (Which means that 1R = 1.0% of equity.) As you can see from the equity curve plots, over this particular span of 21 years System 1 has grown its equity by a factor of 500X. Which means that using fixed fractional position sizing, new entry signals were traded with 500X more contracts in Nov 1993, than they were in Feb 1973.

So please proceed ahead with the Monte Carlo work using R-multiples. Single-contract position sizes would be just plain silly.
Attachments
frequency distribution graph of R-multiples from TBB
frequency distribution graph of R-multiples from TBB
plot2.png (6.31 KiB) Viewed 22370 times
Total profits vs. Trade R-multiples graph from TBB
Total profits vs. Trade R-multiples graph from TBB
plot1.png (6.78 KiB) Viewed 22370 times
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Post by zentrader »

@sluggo,

"...Good, let's stick to R-values. That's what TBB customers are used to thinking about...Single-contract position sizes would be just plain silly..."

Perhaps I live on another planet, but in Europe you don't trade with R-values, but with Future points and/or real currency values (Euro, $ etc.). But if you are not able to give a ptractical input - anyway, let's try a "R-value" based simulation...

I ran 10,000 simulation runs with the number of the whole time period of each system (system1: > 10,000,000 trades, system2: > 5,000,000 trades), which last ca. 1-2 seconds...

I've attached the result screens for the system simulation as system1.gif and system2.gif.

bye,
zentrader
Attachments
system2.gif
system2.gif (17.39 KiB) Viewed 22349 times
system1.gif
system1.gif (17.34 KiB) Viewed 22349 times
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

RR, I wouldn't trade system 2.... return has to be at least 3 times bigger than the DD, so system 2 doesn't qualify for me
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

zentrader, that's wonderful, thank you! I haven't read the manual of that software so a few things are a little confusing to me.

I suppose that "252 bars per time unit" means that you have defined "time unit" to be a year? (since these are EOD bars and since there are usually 252 EOD bars in a year). Is that correct?

I also suppose that the numbers in Green, such as Profit/time unit, should be interpreted in this situation to mean "R-multiples per time unit", which is "R-multiples per year"? Since we risked 1% per trade, 1R = 1% of account equity, and so am I right in concluding that the Monte Carlo estimate for System 1 is a profit somewhere between 28% per year (min) and 45% per year (max), with an average value of 36% per year? Do I interpret the green numbers correctly?

If so then the Monte Carlo estimates for System 2 would appear to be an average profit per year between 7R's per time unit (i.e. 7% per year), (min) and 22R's per time unit (i.e. 22% per year) (max). System 1's green numbers are much larger than System 2's green numbers.

I suppose the numbers in Red, the Drawdowns, are absolute magnitudes of drawdown, again in R's? System 1 seems to have dramatically smaller red numbers (MC drawdowns) than system 2.

Then a final pair of boxes, "Estimated Profit / time unit, VaR" and "Estimated drawdown, VaR" are shown at the bottom. Again System 1 has a much bigger number in the green (profit) box, along with a much smaller number in the red (drawdown) box.

I am especially curious to know, how to interpret these results in a way that helps me forecast the FUTURE performance of these two systems. Is your software's Monte Carlo analysis designed to assist in forecasting future performance? For example can I interpret the bottom green number "Estimated Profit / time unit, VaR" as a forecast of the system's profitability in the future?

Or how about relative comparisons among systems? Since System 1 seems to have received much better Monte Carlo results, in every category, than System 2, does this Monte Carlo analysis forecast that System 1 will outperform System 2 in the future? Is it mathematically proper to conclude "With a confidence level of 99% we can forecast that System 1 will outperform System 2 by approximately 19 R's per year (22 - 3 = 19)?"

Thanks, sluggo
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Post by zentrader »

sluggo wrote: ...I suppose that "252 bars per time unit" means that you have defined "time unit" to be a year? (since these are EOD bars and since there are usually 252 EOD bars in a year). Is that correct?...
Yes, exactly. That's correct. It may be little bit strange, but so the software can be used for intraday, EOD or other timeframes.
sluggo wrote: ...I also suppose that the numbers in Green, such as Profit/time unit, should be interpreted in this situation to mean "R-multiples per time unit", which is "R-multiples per year"? Since we risked 1% per trade, 1R = 1% of account equity, and so am I right in concluding that the Monte Carlo estimate for System 1 is a profit somewhere between 28% per year (min) and 45% per year (max), with an average value of 36% per year? Do I interpret the green numbers correctly?...
I think so, but that's the point, that is a little bit confusing for me and the reason for my intention to use "hard" dollars or future pointe etc. :-)
sluggo wrote: ...If so then the Monte Carlo estimates for System 2 would appear to be an average profit per year between 7R's per time unit (i.e. 7% per year), (min) and 22R's per time unit (i.e. 22% per year) (max). System 1's green numbers are much larger than System 2's green numbers...
If R is equal for both systems, then it's correct.
sluggo wrote:
...I suppose the numbers in Red, the Drawdowns, are absolute magnitudes of drawdown, again in R's? System 1 seems to have dramatically smaller red numbers (MC drawdowns) than system 2...
Yes, that's the point. You can compare two systems easily with my special ratios ACR and WCR. Here smaller ratios are better...
sluggo wrote:
...Then a final pair of boxes, "Estimated Profit / time unit, VaR" and "Estimated drawdown, VaR" are shown at the bottom. Again System 1 has a much bigger number in the green (profit) box, along with a much smaller number in the red (drawdown) box....
The VaR-Estimation is to be seen additional to the absolute result of the mcs, we have discussed before. Here the estimated profit(VaR) shows the minimal expected profit in a simulation run with a statistical reliability of 99%. The estimated drawdown shows the maximum (negative) account drawdown which will be expceted with a security of 99%.
sluggo wrote:
...I am especially curious to know, how to interpret these results in a way that helps me forecast the FUTURE performance of these two systems. Is your software's Monte Carlo analysis designed to assist in forecasting future performance? For example can I interpret the bottom green number "Estimated Profit / time unit, VaR" as a forecast of the system's profitability in the future?...
MCS based system simulations can always only be interpreted concerning the data which is used. In this case (my assumption) historical data is the basis so the results show a system charecteristics under these historical market conditions. If these conditions remain, the results may be valid also in future...

But for this important problem of changing market conditions i've implemented the second main mcs based functionality, the data simulation (I've posted the principle and a screenshot above in another forum contribution). So you can design future market conditions and test your systems again and again with that new data. You will get new backtest results and can start new system simulations - that's the process.
sluggo wrote: ...Or how about relative comparisons among systems? Since System 1 seems to have received much better Monte Carlo results, in every category, than System 2, does this Monte Carlo analysis forecast that System 1 will outperform System 2 in the future? Is it mathematically proper to conclude "With a confidence level of 99% we can forecast that System 1 will outperform System 2 by approximately 19 R's per year (22 - 3 = 19)?"...
With system simulation imho definitely no, see the explanation before!
But if you use changed scenarios und you get new system simulation results you can compare systems with the absolute data of profits and account drawdowns or with the little helper ratios ACR and WCR.

I thank you also for this practical system test example!

bye,
zentrader
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

I'm glad to hear that you believe Monte Carlo analysis cannot be used to forecast system performance in the future.

Although System 1 had a nice smooth equity curve during 1973-1994 (21 years!) and although it outperformed System 2 during that period, the markets changed. System 2 outperformed System 1 quite dramatically in 1994-2007(Q1), despite System 1's vastly nicer-looking Monte Carlo results for 1973-1994.

System 1 follows intermediate-term trends, and makes good profits when intermediate-term trends are smooth. These type of trends were plentiful before 1994 and not so plentiful afterwards. System 2 follows long-term trends, which were present the entire time from 1973 to 2007(Q1), but not quite as profitable as smooth intermediate-term trends. The full equity curves, from 1973 to 2007(Q1) are contained in the original .zip archive, see picture below. Their password is "Roundtable". I've copied those equity curve pictures here, and added a marker at 1994, showing the date where the MC simulations were run.

Obviously no software could predict that the nature of trends was going to change in the future. No software could predict that the type of trends which profit System 2 were going to keep occuring in the future, while the type of trends which profit System 1 were going to stop occurring in the future. Keep in mind that Systems 1 and 2 traded the identical same portfolio of markets for the entire period.

System 1 was not a "bad system"; it was merely a machine that profited handsomely when the markets behaved a certain way. The markets behaved that certain way for 21 years, and then they stopped. No analysis of the data of those 21 years, could have predicted the change. Not even Monte Carlo analysis.

And now we have 34 (!!) years worth of profits for System 2. Now that we see what happened from 1994 to 2007, System 2 looks like an excellent machine that profits quite nicely when the markets behave a (different) certain way. Especially in comparison to System 1 !! But: can we predict whether or not the markets will continue to behave that way in the future? No. Can Monte Carlo analysis predict whether or not System 2 will continue to be profitable in the future? No. Can we use any of this data to predict whether the markets will revert to their old behavior, such that System 1 starts to be profitable again? (e.g.(this post) )? No.

Too bad.
Attachments
System 2 from 1973 to Feb 2007
System 2 from 1973 to Feb 2007
after2.png (32.3 KiB) Viewed 22238 times
System 1 from 1973 to Feb 2007
System 1 from 1973 to Feb 2007
after1.png (32.12 KiB) Viewed 22239 times
Zip archive contains equity curves for 1973 to 2007
Zip archive contains equity curves for 1973 to 2007
pic.png (13.55 KiB) Viewed 22235 times
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

sluggo wrote:)? No.

Too bad.
Now thats funny...

Thanks for the educational demonstrations. Any new (or not so) trader should take this experience to heart!

Perhaps a synthetic data generator would be a useful feature in TBB to 'stress test' the little darlings we create with backtesting...

rr
Last edited by RedRock on Wed Feb 28, 2007 10:47 am, edited 1 time in total.
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Monte Carlo Simulation...

Post by zentrader »

sluggo wrote: ...I'm glad to hear that you believe Monte Carlo analysis cannot be used to forecast system performance in the future...
D'accord. No one has the crystal ball. And if there is one who believes he has one, he should better read Taleb's "Fooled by randomness"... ;-)

Despite of that I think, with a two-step monte carlo simulation based addition to your "normal" system development process you can nevertheless reduce the possibilities of failure. It's a toolset to improve your development process.

I've described this in an article in Traders' Germany which was also translated for Traders' UK magazine February 2007:
http://www.zentrader.de/mcs_article_traders2007.pdf

bye,
zentrader
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

RedRock wrote:Perhaps a synthetic data generator would be a useful feature in TBB to 'stress test' the little darlings we create with backtesting...
Where would you obtain the settings to operate the synthetic data generator? The synthetic data cannot be perfectly identical to the original, if it were then it is no help whatsoever. The synthetic data also cannot be pure white noise with infinite bandwidth and infinite amplitude; that would be infinitely different from market price behavior and it would be no help either.

What you want from synthetic data is to be "somewhat" different from historical data, but not "too much" different.

Sounds like a human-adjustable parameter to me. Who gets to decide how different is "somewhat" different, and how different is "too much" different? A human being. Either you, the user of the synthetic data generator (bad), or else the software developer who wrote the synthetic data generator (worse).

What is the goal of synthetic data? Perhaps, to stress-test trading systems with "somewhat realistic" price streams that were unseen during the system development process. Perhaps also, to stress-test trading systems with price streams that are "somewhat more challenging" than historical prices. Maybe throw some not-so-smooth intermediate term trends at System 1, for example.

Who gets to decide where the border line is, between useful "somewhat realistic" price streams, and non-useful "not very realistic" price streams? A human being. Either the software user or the software developer. And they can guess wrong.

To generate synthetic data, is to predict the future. Inherently you must make a prediction of "How different is the future likely to be, than the past?" You can predict too optimistically, too pesimistically, or just right. If you are too optimistic, you and your system will be surprised by price events in the future, events you are unprepared for. If you are too pessimistic, you will develop systems with needlessly excessive "armor plating" (for example, stops that are far too wide) and you will choose a position size far too small (in preparation for a nuclear winter of unfavorable market behavior ... that never comes) and so you will miss out on profits.

But, how do you know what is too optimistic, vs. too pessimistic, vs. just right? You don't know. You must predict the future, and hope that you have predicted well.

Maybe it will make you feel good to run your system on synthetic data with the "difference from the past" knob set to "10". Maybe your system performs well. Maybe you create another wad of synthetic data with the difference knob set to 20, and maybe your system still performs well. Maybe you do this two dozen times and discover that your system performs well for "difference from the past" knob settings anywhere from 0 to 30. Then at settings from 31-40 your system is still profitable but not nearly as nice as before. Then at settings above 40 your system is uprofitable. This might make you feel very good; your system seems capable of handling the past prices, AND synthetic prices that are 40 units of "somewhat" different from the past. Great. Now you want to trade it in the future. How many units of "somewhat" will the future be, than the past? Nobody knows. You must guess, um, I mean, you must forecast.
drip007
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Mon Nov 27, 2006 2:54 pm

Post by drip007 »

Zentrader,

How can I use your services for stress testing? My first choice is to have a way of generating synthetic data on the fly that I can import into tblox and use as a possible future out of sample data set.

I am not technical enough nor am i inclined to bother with all of the details in setting up something like this on my own or through reading an instruction manual.

THoughts?
Post Reply