how to "rate" a system?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
maildigger
Contributing Member
Contributing Member
Posts: 7
Joined: Mon Dec 15, 2008 11:41 am

how to "rate" a system?

Post by maildigger »

A system is trading the index unleveraged long & short each day (only a position during regular trading hours) with a 3% stop (if the market goes against you). The average return is 4,5% / month.

Would you consider this a good system? I would think yes ... but it occured that the system was wrong several days in a row what would result in a 12% drawdown. Is this too much? Who can help me?
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

Hi Maildigger,

return alone is not enough to rate a system. So the question whether a system making 70% a year is a good system does not have an answer. For example, if the expected draw down is 100%, then the answer is: no.

At least you need to look at the risk together with the return. A 12% draw down in relation to 70% CAGR still sounds great. However, the ugly truth in this business is that risks and returns are correlated. There are no exeptions, as the executive officers of all the major banks currently are learning. So it might be possible that your expected draw down is only 12%, when going for 70% CAGR. I have read on the net that such systems exist, though personally I have yet to see one. Therefore the other possibility is that your expected risk is far greater than you have realized in the past.

Preferably one uses a lot other measured to rate system. Like the average length of the largest draw downs, the distribution of monthly returns, win percentage, ... The blox software for example provides several dozens of measures, which tackle precisely this subject from different angles.

Regards,
Asamat
maildigger
Contributing Member
Contributing Member
Posts: 7
Joined: Mon Dec 15, 2008 11:41 am

Post by maildigger »

Asamat thanks for your reply.

I have a problem with evaluating al the data of a system. This system has 56% winners. Expectancy of a trade = +0.3%. I have no idea what my expected risk is. How do you calculate this? 12% is the biggest drawdown I have experienced untill now after a losing streak.
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

With the data given so far it's not possible to know the risk.

How do you know the win percentage and the other numbers? Do these numbers come from backtests, or from your real trades? If they come from real trades, how many trades over which timespan? If from backtests, how long back was your data?

Do I read your first post correctly, that it's a single market system?

Is it a pure mechanical system or is there a discretionary element?
maildigger
Contributing Member
Contributing Member
Posts: 7
Joined: Mon Dec 15, 2008 11:41 am

Post by maildigger »

I have been tracking this system for almost a year now (forward testing). It took 164 trades on the S&P500 index. It's mechanical (if than else).

Thanks for you input!
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

howdy. you need to code that system up into blox and run it over lots of history. at the very least 10 years. then you can answer your questions about its worthiness to trade. one year inst enough. Test on many markets as well.
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

I'm sorry to have to repeat: this information is not enough to know your risk. We could calculate a few things from these 11 months, but in reality that timespan is too short to give you sufficient confidence in your system to continue trading in case the 12% draw down go to 20% or 30%. Generally it is much more difficult to measure or judge risk than return.

But if your system is purely mechanical, the solution to your problem is simple:
see that you get S&P500 data for the past 20 or 30 years, and suitable software will tell you whether the 12% are an exception, or will happen regularly, or whether you can expect them to grow to 25% or 50%. If resulting from that you know, for example, that the largest DD of your system in 30 years was 20% and the 5 largest DDs were all between 15% and 20%, you can confidently go on trading. If you learn on the other hand that in the past 30 years your system would have gone bankrupt five times, you will be much more careful. That would not mean you need to abandon it, you can reduce the risk to an acceptable level.

As an additional plus you possibly could improve the system, or identify certain conditions where it works, or when it doesn't.
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

Hi maildigger, and welcome aboard. You have just asked the BIG question. I found my answer in the book Mathematics of Money Management by Ralph Vince, specifically the Weighted Geometric Mean. That, along with quite a few statistical checks, can get you a handle on the goodness (or otherwise) of a given system that you can then test over many markets. IMO any system that only trades a single market is a ticket to pain, but that is outside the scope of your question.
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

Roscoe wrote: IMO any system that only trades a single market is a ticket to pain, but that is outside the scope of your question.
Just to be difficult... I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.

Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.

There are always exceptions to "rules"!
alp
Roundtable Knight
Roundtable Knight
Posts: 309
Joined: Mon Aug 27, 2007 8:09 pm

Post by alp »

RedRock wrote:I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.

Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.
Do you know whether they're trading a mechanical or discretionary system, intraday or in the pits? Do they have other businesses or have they made themselves only out of trading SP and soybeans?
babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader »

I would suggest that some serious Monte Carlo permutation and bootstrap testing of your actual results might give you the answers you're looking for.
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

babelproofreader wrote:I would suggest that some serious Monte Carlo permutation and bootstrap testing of your actual results might give you the answers you're looking for.
Sorry, I have to disagree. His problem is not about how to extract more information from the limited set of trades he has. His problem is that his system has seen the market only in a specific condition, and he does not know whether and how it works in other conditions. The answer to that can not come from permutations of existing trades. The answer can only come from exposing the system to data from previous times, when other conditions existed.
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

alp wrote:
RedRock wrote:I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.

Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.
Do you know whether they're trading a mechanical or discretionary system, intraday or in the pits? Do they have other businesses or have they made themselves only out of trading SP and soybeans?
They were both lions in their pits. the SP guy does trade his primary market from a screen now, but I don't know in what fashion. The beans guys? He made his dough in the pit, started a clearing house. He's now retired. Last I knew sons were trading spreads on screen in grains and cbot rates.

For most of us, portfolio theory is the magic, though I wouldn't say a one market specialist couldn't succeed.

several systems on one market for example.
babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader »

Asamat,

The original question was "Would you consider this a good system?" Your suggestion that back testing in other market conditions, while laudable, does not address the question at hand. Back testing can answer questions such as "Is this a robust system?" "Would I be happy trading this system under different market conditions?" "How does the system perform given different market conditions?" "Is it best to only trade this system when market conditions are deemed suitable for it?" etc.

I would paraphrase the original question as: given the results I have, obtained from testing in a specific market environment, how confident can I be that draw down of 12% is normal, or is over or understated for these market conditions?

Just for argument's sake consider that you have the results of a long only system tested in a bull market, would it make sense to judge the quality of these results by looking at the results obtained from the same system applied to a bear market?
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

babelproofreader wrote:I would paraphrase the original question as: given the results I have, obtained from testing in a specific market environment, how confident can I be that draw down of 12% is normal, or is over or understated for these market conditions?
IMO and given the limited information available my confidence level would be very low and I would expect a much larger DD over time, but I stress that the foregoing is merely my personal opinion, and, as others have tried to point out, the OP must consider the whole subject of testing (what and how) thoroughly.
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

babelproofreader wrote:The original question was "Would you consider this a good system?" Your suggestion that back testing in other market conditions, while laudable, does not address the question at hand.
I'm sorry, but I have to disagree again. His numbers come from a very specific situation, look at the S&P from the last 11-12 month, where his numbers come from.
Attachments
SandP.png
SandP.png (35.65 KiB) Viewed 10427 times
Asamat
Roundtable Knight
Roundtable Knight
Posts: 175
Joined: Fri Jun 03, 2005 7:50 am
Location: Walldorf, Germany

Post by Asamat »

His system does not include discretionary judgement, whether this situation is over and he should stop trading it. That's why he asks and is concerned. And if I look very closely at the right end of the chart where his biggest DD so far occurred, I say he is rightly concerned. It's possible that his system is good only when the index falls, but the index will clearly not fall again as much.
babelproofreader wrote:Just for argument's sake consider that you have the results of a long only system tested in a bull market, would it make sense to judge the quality of these results by looking at the results obtained from the same system applied to a bear market?
There is no need for "arguments sake". He has the results of a system from a period where the index fell in a way which will not repeat in the coming years. His question is not "were the results good in the last year", the question is "will they be good in the coming year, can I continue to use it". And for that he has to look at sideway and upway markets, not interpolate the situation of the past year.
maildigger
Contributing Member
Contributing Member
Posts: 7
Joined: Mon Dec 15, 2008 11:41 am

Post by maildigger »

I still don't found an answer but thank you all for the input!

:lol:
Post Reply