By What Measure? - How do You Know if a System is Good?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

It depends.

The conservative approach is to set up each independently.

The more profitable approach is to take them all together and work out maxDD etc on the full portfolio. If you do this and watch the equity on system 1 say then instead of a maxDD of 30% you might have 40% while the total for all 3 systems shouldnt fall below 30%. The risk is that they may be more correlated in future than you thought - I liked c.f.'s response to the questions I was asking in this area. If you are comfortable then ok.

Personally I fall somewhere in the middle. The reason being that my uncorrelated systems are uncorrelated from luck not because one is designed to make money when the other loses so I don't want to push it too far. Art not science on the final choice :shock:

John
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

Just in case I wasnt absolutely clear in the final answer.

I divide capital roughly in 3.

I trade two of the systems by determining bet size as though it was one pool of capital. I used the combined historical results to determine a satisfactory return and corresponding expected drawdown. When I analyse the results though I treat them as though I began the period by allocating X$ to one and Y$ to the other and watch the performance independently for failure or normal drawdowns.

I trade one system absolutely independently with no pooling.

As c.f. said:

"What to do... what to do...

Yes, this is a messy business with no clearly defined right answers to many of the questions we most want to have answered."


So, in the absence of any certainty I fall exactly down the middle and await my fate :D

John
MCT
Roundtable Knight
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm

Post by MCT »

[quote]“For unleveraged investments this is a pretty good indicator of the exposure to the risk of crazy unpredictable events like 9/11, the 87 crash, etc. If you only have 20% of your money in the market on average, you will only lose 10% of your account if the market drops 50% overnight. If you use 80% you could lose 40%.â€
Last edited by MCT on Wed Sep 17, 2003 7:59 pm, edited 1 time in total.
drm7
Roundtable Fellow
Roundtable Fellow
Posts: 96
Joined: Sun Apr 20, 2003 9:02 pm
Location: Richmond, VA

Sudden events

Post by drm7 »

I agree with the previous post, Zeno.

You really can't quantify the direction of every "black swan" event, such as a crop failure, terrorist attack, etc. The best you can do is make sure that one doesn't wipe you out. You accomplish that with a combination of position sizing and diversification (across markets, systems, time horizons, etc.)

That doesn't mean that Trend Following is "bad." It just means that Trend Followers can get hit with an adverse event just like anybody else. c.f. described an event in his original turtle rules that went massively against all of the Turtles' interest rate positions - the 1987 crash.

-Doug
rs

Post by rs »

This is an interesting aspect of trading with the trend that I have often wondered about. From my empirical observation only, it seems that Zeno has a point. Most surprises seem to occur with the trend.

Of course, there are exceptions but trend following seems to insure you against really getting hammered, at least in terms of closed equity drawdown. Going with the trend also seems to protect against surprises in the stock market I have noticed.

rs
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

I'm afraid that 9/11 wasnt favourable to TFs.

Those systems/markets on the wrong side got stopped out very unfavourably.

Those on the right side got a sharp positive equity shift.

But just like the recent war, things shifted back quickly and the positive equity shift disappeared quickly to leave the bad stops behind and increased volatility without much trend for quite a period :cry:

Black Swans are something you just have to live with and realise that the stop unlike an option will often disadvantage you when it happens. But if it was easy then my Grandma would be doing it.

On a related subject I often get a laugh from people who say "if the system didnt perform well then "just fade it"". They seem to miss the fact that chopping into stops is frequently the cause .... try and fade that for fun some time :twisted:
rs

Post by rs »

Kiwi,

I am not sure that I would agree. The stockmarkets and interest rate futures markets both continued to trend quite well. Their trends were simply accentuated by 9/11. Stockmarkets were in massive downtrends and short term interest rate futures were trending up and both continued to do so.

That of course does not cover all markets or timeframes or types of system. It is just my observation. I suppose it depends alot on what markets you were trading at the time and your system.

rs
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

I think we agree completely. 8)

I was too broad when I included all trend followers in my last posting. You've illustrated another aspect of the black swan experience RS. Your experience was of markets that didn't see swans flying on 911. Perhaps just a bunch of trendfollowing sheep flying by :D

My perception came from currencies, oil and gold which got shocked and then fell into a period of higher volatility. This was negative for the systems I traded but could well have been positive for a countertrend system like Catscan.

This seems to be a practical illustration of the value of trading multiple types of systems on multiple markets. If the shadow of Taleb's black swan only darkens some then survival is much more certain.

Thanks for the insight.

John
MCT
Roundtable Knight
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm

Post by MCT »

Western philosophy regarding event causality seems to have inverted causal relationships. Social mood, reflected by the aggregate record of stock market prices, “leads eventsâ€
Last edited by MCT on Wed Sep 17, 2003 10:52 am, edited 1 time in total.
Vince
Roundtable Fellow
Roundtable Fellow
Posts: 61
Joined: Wed Jun 11, 2003 12:40 pm

Merits of various trading systems

Post by Vince »

From my own limited, system development experience, I would say that the essential components of a trend following system are:

1. Signal Generator.
2. MM module.
3. Risk Analysis Unit.
4. Equity Curve trading module.

Therefore, once traders have a basic system, their research should centre around trying different combinations of components. Does anyone have experience of testing such combinations? If you did, did you find that results vary wildly, depending on the combinations? In essence, I am trying to guage how stable are the overall systems, based on the above units (it might be possible that I have missed an essential component from my list, in which case, I will soon find out when I get my first big DD!).
Dan G

Surprise

Post by Dan G »

I am a little surprised there was no discussion of Monte Carlo Simulations, as they are great for determining max drawdowns, especially if you are going to use 20 years of data to find out trade distributions as c.f. recommends.

20 years of data will provide you with a rather robust distribution of trades. With that much information, You'll have an excellent idea exactly how many 20R winners you can expect, and how many 5R losses you can expect per 1000 trades. Plug this into a Monte Carlo, and bingo - you'll have a good idea about what to expect as a max drawdown over a given time period. I would have lots of confidence in an Avg. Max Drawdown + 1 STD number, calculated over a 1000 simulated 20 year periods (or whatever time period is important to you).

Also, There was no discussion of using dowside deviation, where only your returns below a given level are used in your Standard Devaition calculation. It's pretty good for trend followers as it doesn't penalize the good months. It is good for traders generally, but you have to watch how to use it.

Mark Jurik gave a formula in one of his books that compared trading systems, and it works pretty well. It generates a single number, from the output of your trading results, that directly compares trading systems. I don't have it here, and I don't know if I should even post it - I don't know if it is really in the public domain. It doesn't use any drawdown measure, so it is limited in some ways.
MCT
Roundtable Knight
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm

Post by MCT »

Western philosophy regarding event causality seems to have inverted causal relationships. Social mood, reflected by the aggregate record of stock market prices, “leads eventsâ€
Sir G
Moderator
Moderator
Posts: 243
Joined: Wed Apr 16, 2003 12:21 am
Location: Salt Lake City, Utah

Post by Sir G »

Hi MT-

I came across a headline earlier today that I think also shows your point.

Breast Implants Linked to Suicides
Increased Suicide Risk Shown Among Women Who Get Breast Implants

http://content.health.msn.com/content/a ... /88985.htm

When I first read the headline, what came of my mind is the women who have breast implants are a selected group who have found that they are in a certain way dissatisfied with themselves. The degree of that dissatisfaction will run the spectrum, so higher degrees of certain types of behavior should be found in that group.

It's like comparing the shoppers at a 7-11 to a health food store. You will find a much different element in both of these stores.

I can almost see the headlines...

Shopping at 7-11 Linked to Lung Cancer
Increase Smoking Risk Shown In People Who Shop At 7-11

Gordon
Edit by Gordon: I should make note for those that don’t know what 7-11 is… it is a convenience store, where you can quickly run in and make a purchase. Many smokers find it very convenient to purchase their cigarettes there. [/quote]
Last edited by Sir G on Fri Sep 19, 2003 12:19 pm, edited 1 time in total.
Toni
Senior Member
Senior Member
Posts: 31
Joined: Mon Apr 21, 2003 3:10 am
Location: Ukraine
Contact:

just mine 0,2$

Post by Toni »

Forum Mgmnt wrote:So I am curious what each of you uses to determine what makes a good system. - Forum Mgmnt
Hi, c.f.
Every measure that you said, came from inside of system , with "MM-optimization" i can make any numbers :cry:

So we need something that not depend from system, I now only one thing - ZigZag system or MPS (MaximumProfitSystem), if corellation beetween Log of "positive time profit" MTS and "MPS on Zig-Zag" is high, system good, it`s takes what it have to take from the market.

MPS on Zig-Zag - inputs can be change to suite type of system (trendf..,contrtrend etc).

Here digest about it:
digest


PS:Just mine 0,2$ :D
Toni
Senior Member
Senior Member
Posts: 31
Joined: Mon Apr 21, 2003 3:10 am
Location: Ukraine
Contact:

Re: Surprise

Post by Toni »

mickslam wrote:I am a little surprised there was no discussion of Monte Carlo Simulations, as they are great for determining max drawdowns, especially if you are going to use 20 years of data to find out trade distributions as c.f. recommends.
freeware MonteCarlo (BootStrap) program for Excel- BootStrapIt!. :roll:
Dan G

Post by Dan G »

Thanks Toni - I am new here and somehow missed the giant forum about MC simulation. :oops:
Toni
Senior Member
Senior Member
Posts: 31
Joined: Mon Apr 21, 2003 3:10 am
Location: Ukraine
Contact:

Post by Toni »

mickslam wrote:Thanks Toni - I am new here and somehow missed the giant forum about MC simulation. :oops:
where you find about "giant forum about MC simulation" ? Give me link ;-)
I placed link to program and article (in russian ;-) ) by mine friend (Moysha) - measure probability of profit,drawdown etc.

Very good program, only for traders (specific) ;-)
MCT
Roundtable Knight
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm

Post by MCT »

By What Measure?

My take on the Problems Associated with Induction
title Edited: from "The problems of induction.

The very heart of the problem is a problem of logical induction, where conception and reasoning begins from the “system parts,â€
Last edited by MCT on Fri Sep 26, 2003 2:35 pm, edited 2 times in total.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

MODERATOR'S NOTE: The amusing computer tranlations were split into a separate topic:

viewtopic.php?t=539
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

Attached is an Excel spreadsheet that I have put together to test a few performance calculations as discussed in the thread "Identifying Efficiency in a 3D Matrix", which, by common concensus, has been moved to this thread.

Thanks to Kevin for putting me straight on the MAR calc.

Contains:
CAGR and MAR
Geometric Mean (Vince)
Pessimistic Rate of Return (Vince)
Pessimistic Return on Account

I think we can say that these calcs can now be used.
Attachments
Stray Calcs.xls
Sample performance calcs (revised version with corrected MAR)
(21 KiB) Downloaded 1581 times
Last edited by Roscoe on Sat Feb 21, 2004 4:35 am, edited 2 times in total.
Post Reply