how to "rate" a system?
-
- Contributing Member
- Posts: 7
- Joined: Mon Dec 15, 2008 11:41 am
how to "rate" a system?
A system is trading the index unleveraged long & short each day (only a position during regular trading hours) with a 3% stop (if the market goes against you). The average return is 4,5% / month.
Would you consider this a good system? I would think yes ... but it occured that the system was wrong several days in a row what would result in a 12% drawdown. Is this too much? Who can help me?
Would you consider this a good system? I would think yes ... but it occured that the system was wrong several days in a row what would result in a 12% drawdown. Is this too much? Who can help me?
Hi Maildigger,
return alone is not enough to rate a system. So the question whether a system making 70% a year is a good system does not have an answer. For example, if the expected draw down is 100%, then the answer is: no.
At least you need to look at the risk together with the return. A 12% draw down in relation to 70% CAGR still sounds great. However, the ugly truth in this business is that risks and returns are correlated. There are no exeptions, as the executive officers of all the major banks currently are learning. So it might be possible that your expected draw down is only 12%, when going for 70% CAGR. I have read on the net that such systems exist, though personally I have yet to see one. Therefore the other possibility is that your expected risk is far greater than you have realized in the past.
Preferably one uses a lot other measured to rate system. Like the average length of the largest draw downs, the distribution of monthly returns, win percentage, ... The blox software for example provides several dozens of measures, which tackle precisely this subject from different angles.
Regards,
Asamat
return alone is not enough to rate a system. So the question whether a system making 70% a year is a good system does not have an answer. For example, if the expected draw down is 100%, then the answer is: no.
At least you need to look at the risk together with the return. A 12% draw down in relation to 70% CAGR still sounds great. However, the ugly truth in this business is that risks and returns are correlated. There are no exeptions, as the executive officers of all the major banks currently are learning. So it might be possible that your expected draw down is only 12%, when going for 70% CAGR. I have read on the net that such systems exist, though personally I have yet to see one. Therefore the other possibility is that your expected risk is far greater than you have realized in the past.
Preferably one uses a lot other measured to rate system. Like the average length of the largest draw downs, the distribution of monthly returns, win percentage, ... The blox software for example provides several dozens of measures, which tackle precisely this subject from different angles.
Regards,
Asamat
-
- Contributing Member
- Posts: 7
- Joined: Mon Dec 15, 2008 11:41 am
With the data given so far it's not possible to know the risk.
How do you know the win percentage and the other numbers? Do these numbers come from backtests, or from your real trades? If they come from real trades, how many trades over which timespan? If from backtests, how long back was your data?
Do I read your first post correctly, that it's a single market system?
Is it a pure mechanical system or is there a discretionary element?
How do you know the win percentage and the other numbers? Do these numbers come from backtests, or from your real trades? If they come from real trades, how many trades over which timespan? If from backtests, how long back was your data?
Do I read your first post correctly, that it's a single market system?
Is it a pure mechanical system or is there a discretionary element?
-
- Contributing Member
- Posts: 7
- Joined: Mon Dec 15, 2008 11:41 am
I'm sorry to have to repeat: this information is not enough to know your risk. We could calculate a few things from these 11 months, but in reality that timespan is too short to give you sufficient confidence in your system to continue trading in case the 12% draw down go to 20% or 30%. Generally it is much more difficult to measure or judge risk than return.
But if your system is purely mechanical, the solution to your problem is simple:
see that you get S&P500 data for the past 20 or 30 years, and suitable software will tell you whether the 12% are an exception, or will happen regularly, or whether you can expect them to grow to 25% or 50%. If resulting from that you know, for example, that the largest DD of your system in 30 years was 20% and the 5 largest DDs were all between 15% and 20%, you can confidently go on trading. If you learn on the other hand that in the past 30 years your system would have gone bankrupt five times, you will be much more careful. That would not mean you need to abandon it, you can reduce the risk to an acceptable level.
As an additional plus you possibly could improve the system, or identify certain conditions where it works, or when it doesn't.
But if your system is purely mechanical, the solution to your problem is simple:
see that you get S&P500 data for the past 20 or 30 years, and suitable software will tell you whether the 12% are an exception, or will happen regularly, or whether you can expect them to grow to 25% or 50%. If resulting from that you know, for example, that the largest DD of your system in 30 years was 20% and the 5 largest DDs were all between 15% and 20%, you can confidently go on trading. If you learn on the other hand that in the past 30 years your system would have gone bankrupt five times, you will be much more careful. That would not mean you need to abandon it, you can reduce the risk to an acceptable level.
As an additional plus you possibly could improve the system, or identify certain conditions where it works, or when it doesn't.
Hi maildigger, and welcome aboard. You have just asked the BIG question. I found my answer in the book Mathematics of Money Management by Ralph Vince, specifically the Weighted Geometric Mean. That, along with quite a few statistical checks, can get you a handle on the goodness (or otherwise) of a given system that you can then test over many markets. IMO any system that only trades a single market is a ticket to pain, but that is outside the scope of your question.
Just to be difficult... I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.Roscoe wrote: IMO any system that only trades a single market is a ticket to pain, but that is outside the scope of your question.
Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.
There are always exceptions to "rules"!
Do you know whether they're trading a mechanical or discretionary system, intraday or in the pits? Do they have other businesses or have they made themselves only out of trading SP and soybeans?RedRock wrote:I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.
Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.
-
- Roundtable Knight
- Posts: 138
- Joined: Wed Nov 10, 2004 4:36 pm
Sorry, I have to disagree. His problem is not about how to extract more information from the limited set of trades he has. His problem is that his system has seen the market only in a specific condition, and he does not know whether and how it works in other conditions. The answer to that can not come from permutations of existing trades. The answer can only come from exposing the system to data from previous times, when other conditions existed.babelproofreader wrote:I would suggest that some serious Monte Carlo permutation and bootstrap testing of your actual results might give you the answers you're looking for.
They were both lions in their pits. the SP guy does trade his primary market from a screen now, but I don't know in what fashion. The beans guys? He made his dough in the pit, started a clearing house. He's now retired. Last I knew sons were trading spreads on screen in grains and cbot rates.alp wrote:Do you know whether they're trading a mechanical or discretionary system, intraday or in the pits? Do they have other businesses or have they made themselves only out of trading SP and soybeans?RedRock wrote:I know one guy who lives in a waterfront mansion, started with next to nothing, and only traded the SP.
Oh wait, how about that guy who lives up high in a penthouse off the ave, who only traded soybeans. And taught his two sons to do the same.
For most of us, portfolio theory is the magic, though I wouldn't say a one market specialist couldn't succeed.
several systems on one market for example.
-
- Roundtable Knight
- Posts: 138
- Joined: Wed Nov 10, 2004 4:36 pm
Asamat,
The original question was "Would you consider this a good system?" Your suggestion that back testing in other market conditions, while laudable, does not address the question at hand. Back testing can answer questions such as "Is this a robust system?" "Would I be happy trading this system under different market conditions?" "How does the system perform given different market conditions?" "Is it best to only trade this system when market conditions are deemed suitable for it?" etc.
I would paraphrase the original question as: given the results I have, obtained from testing in a specific market environment, how confident can I be that draw down of 12% is normal, or is over or understated for these market conditions?
Just for argument's sake consider that you have the results of a long only system tested in a bull market, would it make sense to judge the quality of these results by looking at the results obtained from the same system applied to a bear market?
The original question was "Would you consider this a good system?" Your suggestion that back testing in other market conditions, while laudable, does not address the question at hand. Back testing can answer questions such as "Is this a robust system?" "Would I be happy trading this system under different market conditions?" "How does the system perform given different market conditions?" "Is it best to only trade this system when market conditions are deemed suitable for it?" etc.
I would paraphrase the original question as: given the results I have, obtained from testing in a specific market environment, how confident can I be that draw down of 12% is normal, or is over or understated for these market conditions?
Just for argument's sake consider that you have the results of a long only system tested in a bull market, would it make sense to judge the quality of these results by looking at the results obtained from the same system applied to a bear market?
IMO and given the limited information available my confidence level would be very low and I would expect a much larger DD over time, but I stress that the foregoing is merely my personal opinion, and, as others have tried to point out, the OP must consider the whole subject of testing (what and how) thoroughly.babelproofreader wrote:I would paraphrase the original question as: given the results I have, obtained from testing in a specific market environment, how confident can I be that draw down of 12% is normal, or is over or understated for these market conditions?
I'm sorry, but I have to disagree again. His numbers come from a very specific situation, look at the S&P from the last 11-12 month, where his numbers come from.babelproofreader wrote:The original question was "Would you consider this a good system?" Your suggestion that back testing in other market conditions, while laudable, does not address the question at hand.
- Attachments
-
- SandP.png (35.65 KiB) Viewed 10465 times
His system does not include discretionary judgement, whether this situation is over and he should stop trading it. That's why he asks and is concerned. And if I look very closely at the right end of the chart where his biggest DD so far occurred, I say he is rightly concerned. It's possible that his system is good only when the index falls, but the index will clearly not fall again as much.
There is no need for "arguments sake". He has the results of a system from a period where the index fell in a way which will not repeat in the coming years. His question is not "were the results good in the last year", the question is "will they be good in the coming year, can I continue to use it". And for that he has to look at sideway and upway markets, not interpolate the situation of the past year.babelproofreader wrote:Just for argument's sake consider that you have the results of a long only system tested in a bull market, would it make sense to judge the quality of these results by looking at the results obtained from the same system applied to a bear market?
-
- Contributing Member
- Posts: 7
- Joined: Mon Dec 15, 2008 11:41 am