Dealing with a drawdown

Discussions about personal psychology for the individual trader.
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Dealing with a drawdown

Post by Chris67 »

I spent about 6 months designing and backtesting and came up with a system with 20 year max drawdown of about 24 % . Am down 13.4 % in 11 days ... any advice ..
I am well up on the concepts of optimisation and all the pitfalls of backtesting and am extremely confident we have made no major errors in design.
However it does strike me that 2004 is a bad year and after afew bad months that August is as bad as I can ever remember for trendfollowing ... any comments warmly welcomed.
stancramer
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Sat Feb 07, 2004 10:34 am
Location: Washington DC

Post by stancramer »

What historical returns and drawdowns have you seen for "anti-trendfollowing" systems and/or "non-trendfollowing" sysems? If you blend some trendfollowing systems with some anti-trendfollowing systems, do the results shift in a direction which pleases you?

For example, with Veritrader, how does "Bollinger Countertrend" look to you? What happens when you trade Bollinger Countertrend and the Turtle System simultaneously?
richard
Roundtable Fellow
Roundtable Fellow
Posts: 94
Joined: Tue May 18, 2004 10:17 pm

Post by richard »

I am very inexperienced but have this to suggest with that in mind:

Why not stop trading for awhile. Paper trade your system and do not trade real money. Take a breather. Collect your thoughts.
leonardo
Roundtable Knight
Roundtable Knight
Posts: 100
Joined: Wed Apr 30, 2003 2:27 pm
Location: Minneapolis, MN
Contact:

Post by leonardo »

Chris67 wrote:I spent about 6 months designing and backtesting and came up with a system with 20 year max drawdown of about 24 % . Am down 13.4 % in 11 days ... any advice ..
It possible that your system having this quick drawdown is completely within its normal probabilities. You can do an existence check on all drawdowns of X days or 13.4% drawdowns or greater to see how often they can occur. It is likely more often than you might expect.

I'm sure you have heard too of the fact "your greatest drawdown is ahead of you" in any system of trading. Thankfully, the balancer to this is that your greatest "draw-ups" are also ahead of you.

When you are designing your own system, it is very useful to be aware of the rule sets of a number of known viable systems-- you need to know if it makes sense that you are having a drawdown while most other durable systems aren't. And vice versa.

If your system is a trend follower type, it will tend to have equity swings; at least on a monthly basis, which are very similar to other decent systems. By that guage, your system should be in a drawdown right now.

Leonardo---
MikeL
Contributor
Contributor
Posts: 4
Joined: Wed Aug 11, 2004 2:25 pm

Post by MikeL »

As I understand LTTF, your system is performing within the parameters that you initially established. Therefore, it would be the exact wrong thing to do at this point to do anything other than continue to follow your system. The moment you decide to stop will likely be the instant the equity curve turns and starts to make up for the losses. Of course, if your system continues to draw down past your initial drawdown parameter, then you have to question the validity of your system.

Mike
richard
Roundtable Fellow
Roundtable Fellow
Posts: 94
Joined: Tue May 18, 2004 10:17 pm

Post by richard »

all I can say is that if it were me, my confidence would be very shaken and I would want to stop and regroup for awhile.

This is what the traders almost to a person said they did in The Market Wizards and The New Market Wizards including the trend following adherents. All of them had limits where if their losses exceeded these, they gave trading a break.
TC
Roundtable Knight
Roundtable Knight
Posts: 101
Joined: Thu Jul 08, 2004 2:12 pm
Location: Toronto, Canada

Post by TC »

Relax !! :D

According to your backtesting results you're only just over half way, it could get a whole lot worse and still be within the bounds of what you considered satisfactory.

As I too am experiencing now, a large prolonged drawdown is a lot tougher to live through than looking at the same DD on a 20 year chart.

It sounds like this is your first experience with such a drawdown and like all new experiences it can seem daunting at first.

The next time this happens you'll hardly bat an eyelid, it is after all what your backtesting was designed to show you and what you expected would occur reasonable frequently over your 20 year time frame.

You probably wont get the jitters again until your next DD pushes you even closer to your 20-yr MaxDD :shock:

By all means take some time to review the process by which you designed and tested you LTTF system. If the logic of your entries and exits still make sense to you and you derived your parameter values from a broad range of markets then you are probably still in pretty good shape, and in the same boat as the rest of us.

Traders who have been around a lot longer than me claim this is the worst year for trend followers since 1998. Maybe it is and maybe it isn't. It really doesn't matter. Have confidence in yourself and your methodology and look on the bright side, your account is still intact, you are still in the game and this experience will have helped make you a more resilient and better trader.

This period has been a great object lesson in the importance of both MM & position sizing. Even with small bet sizing, the leverage provided by futures can still blow a sizeable hole in your account and without volatility-adjusted portfolio diversification the damage would have been a whole lot worse.

To fully appreciate these two aspects of futures trading is why I'm glad that my first year with futures has been down instead of up. I might have been tempted into more aggressive trading had I been enjoying the robust performance of say last year and would likely have imploded with the bust that occurred in April this year.

Whatever you do, now is NOT the time to stop trading

Good luck and hang in there !!

Tom
leonardo
Roundtable Knight
Roundtable Knight
Posts: 100
Joined: Wed Apr 30, 2003 2:27 pm
Location: Minneapolis, MN
Contact:

Post by leonardo »

Just a note:

Whatever a person uses to cause them to enter a trade is a system. Obviously, some systems better lend themselves to testing than others. That doesn't necessarily mean they are better because they're easier to test.

Systems don't care at all if they're in drawdown, or if they're making new equity highs.

Only the people who choose to use them may care.

System trading is like a marriage, commitment is required despite the "gotcha" moments. If you commit yourself to trading a system, then you must trade the system.

Never decide to trade a system that you are not willing to "trade through" with. If you don't "trade through" to get to the profits, you will never have profits.

You are much better off trading a simple system you absolutely can have confidence in, perhaps with poorer trading statistics; than some elegant "supposedly" high profit/high MAR system which doesn't have the seasoning needed for absolute confidence; either personally, mathematically or both.

15 years ago I obtained a system called Old Reliable from Larry Williams. He allegedly used it to get out of trading holes that he got into because of testing other systems which would invariably blow up and cause him great losses.

It's a swing trading system which depends on the inevitable significant breakouts to generate big money trades with very little position risk. With a few reasonable losses in between the big winning trades. Average trade duration is about 10 days. It works as well and as badly as it did 15 years ago, as I have continued following it. Testing accross 30 markets and over 15,000 trades, it has approximately a 1.45 profit factor with a MAR on the weak side.

It doesn't have extreme strings of large losses when trading 4-5 markets like Williams suggested trading it. He even mentioned that he was so confident of its eventual success in whichever market he was trading with it, that after he had a few losses in a market with the system he would tend to increase his position size.

I would trade it anytime instead of most systems promoted today for $1000's. It is durable and reliable.

I wouldn't be surprised if Larry still took some of the Old Reliable trades today. I do.

Leonardo
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Post by William »

http://www.iasg.com/


The above website has a list of the performance numbers for CTA's this year. If you look at a few of the biggies in LTTF, i think you note that they are having a tough year as well.

While this doesnt take the pain away or erase the losses, sometimes its good to know you are not alone.
Asterix
Senior Member
Senior Member
Posts: 44
Joined: Mon Apr 05, 2004 11:16 pm
Location: San Diego

Post by Asterix »

I think this thread points to some room for possible improvement in system testing software. Knowing the size (or %) of the maximum drawdown isn't sufficient data. It would be much better to take all the drawdowns encountered in the system test calculations and report the statistics of the entire set.

So, for example, if you tested the system over a 20 year timeframe of data and went thru 15 drawdown periods, you should be able to extract the following data from these 15 drawdown periods:

Average drawdown
Max drawdown
Std. dev of drawdowns
Average length of drawdown
Max length of drawdown
Std dev of length of drawdowns

Using this data, you could develop a better statistical understanding of what to expect in terms of both size of drawdown and frequency of occurence. If you find yourself in a situation where you begin to question your system, this data will be valuable.
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

New Measures

Post by ksberg »

Asterix wrote:I think this thread points to some room for possible improvement in system testing software. Knowing the size (or %) of the maximum drawdown isn't sufficient data. It would be much better to take all the drawdowns encountered in the system test calculations and report the statistics of the entire set.

So, for example, if you tested the system over a 20 year timeframe of data and went thru 15 drawdown periods, you should be able to extract the following data from these 15 drawdown periods:

Average drawdown
Max drawdown
Std. dev of drawdowns
Average length of drawdown
Max length of drawdown
Std dev of length of drawdowns

Using this data, you could develop a better statistical understanding of what to expect in terms of both size of drawdown and frequency of occurence. If you find yourself in a situation where you begin to question your system, this data will be valuable.
I whole-heartedly agree.

If you have access to TR or system that will do Worst Case/Start Trade Analysis, I'd recommend taking a look at the statistical probabilities and distributions for MaxDD and other statistics. This is the direction I've moved in my own software.

One important thing to note is that drawdowns are not normally distributed, so things like Std Deviation need to be modified in order to get meaningful results. To build a confidence interval, I usually take median value and walk percentage quantiles to either side (e.g. 2 StdDevs covers 95%). Taking cummulative values from either tail will give probability greater or less than a given amount.

As an example, I've attached quantiles and histogram for MaxDD trial over a 14 year study. This study examined over 3500 one-year portfolios and their resulting MaxDDs. You can see how much MaxDD varies depending on start date. You also get a sense of where the distribution is centered, and a taste of black-swan events way out on the tails (pretty sobering!). The quantiles example shows the probability of MaxDD being less than a given threshold. In this case, there was a 3 in 4 chance that MaxDD didn't exceed 42.3% (or following, a 1 in 4 chance that MaxDD is greater than 42.3%).

Why do this? Well, I don't get this level of insight from existing trading tools. Also if you can understand the information, the results are pretty sobering.

Cheers,

Kevin
Attachments
MaxDD Distribution Example
MaxDD Distribution Example
ExampleDistributionMaxDD.jpg (31.87 KiB) Viewed 32134 times
Max DD Quantiles
Max DD Quantiles
ExampleQuantilesMaxDD.jpg (36.94 KiB) Viewed 32134 times
richard
Roundtable Fellow
Roundtable Fellow
Posts: 94
Joined: Tue May 18, 2004 10:17 pm

Post by richard »

ksberg for the benefit of someone like me who is a know-nothing, what does the first chart represent?
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Post by ksberg »

richard wrote:ksberg for the benefit of someone like me who is a know-nothing, what does the first chart represent?
Literally each increment in the first chart represents a 1% group in the sample space. So, adding 75 x 1% groupings from the LHS gives 75% probability, or the cummulative probability.

Here is an easy way to understand quantiles: Collect all your results and sort them from low to high. This gives the curve (thick black line). Then divide the number of results by 100 to get 1% groupings. For this example, 3500 results / 100 increments = 35 trials in each 1% increment. Finally, we add up 75 increments, or 35 x 75 = 2625 trials, which is exactly 2625/3000 = 75%.

The way you read the quantile graph is to find a spot on the curve, then go down to X axis to read the probability, and read right to the Y axis to read the outcome. In the example, the total area to the left represents all events less than 42.3% MaxDD. Now we know there were 2625 trials that generated 42.3% MaxDD or less.

People gravitate toward distributions, but quantiles are pretty simple to understand once you get the hang of it. I like to look at both quantiles and distribution. Every trading system will give a different signature.

Cheers,

Kevin
Attachments
How to read quantiles
How to read quantiles
ExampleQuantilesMaxDD2.jpg (33.42 KiB) Viewed 32119 times
richard
Roundtable Fellow
Roundtable Fellow
Posts: 94
Joined: Tue May 18, 2004 10:17 pm

Post by richard »

thank you very much!
TK
Roundtable Knight
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Re: New Measures

Post by TK »

ksberg wrote:In this case, there was a 3 in 4 chance that MaxDD didn't exceed 42.3% (or following, a 1 in 4 chance that MaxDD is greater than 42.3%).
The question remains: what do you do with the information that the probability of MaxDD greater than 42.3% is 25%? Does it boost your confidence in the system? Does it make you believe that the MaxDD of 60% is highly improbable?

I remember myself performing once a Monte Carlo simulation of a LTTF system results using data up until the end of 2002. The result I got was that the chance that MaxDD would reach 60% or more was below 10% (if my memory serves me well). Sure it felt good, as if it just couldn't happen. Then I tested the system on the 2003 data and, sure enough, the equity dropped by 60%. I have not performed a Monte Carlo simulation ever since. But then, I consider myself a newbie in the Monte Carlo world.
Asterix
Senior Member
Senior Member
Posts: 44
Joined: Mon Apr 05, 2004 11:16 pm
Location: San Diego

Post by Asterix »

TK wrote:
I remember myself performing once a Monte Carlo simulation of a LTTF system results using data up until the end of 2002. The result I got was that the chance that MaxDD would reach 60% or more was below 10% (if my memory serves me well). Sure it felt good, as if it just couldn't happen. Then I tested the system on the 2003 data and, sure enough, the equity dropped by 60%.
My opinion on using Monte Carlo methods for testing trading system performance is that the Monte Carlo results may be misleading. My reason for stating this is that Monte Carlo simulations assume independence of the individual trades. (i.e. each new trade is independent from the previous trades.)

There has been a lot of academic research in this area in the last few years, and my general impression is that a lot of the academics seem to agree that that there is some kind of memory in market prices. In other words, tomorrow's prices are somewhat dependent on what has happened in the recent past. I believe that trends in particular can be a result of this market memory. Consequently, I'm not convinced that Monte Carlo analysis of your results will give you a true feel for risk.

Let's not get too far into the pros and cons of Monte Carlo on this thread though - there are some other places in this forum where this topic is discussed in detail.

What ksberg plotted was the real distribution of drawdowns that may or may not be independent. One way you can use a distribution is to take the known data and extrapolate a general distribution that you think represents everything including the unknown data. (i.e. the drawdowns in the future.) You can use this to at least estimate your probability of seeing a future drawdown of any specific size.

But, it is also important to remember that the word is probability not certainty.

Like the well known disclaimer always states, "Past performance is not a guarantee of future results."
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Post by ksberg »

TK wrote:
ksberg wrote:In this case, there was a 3 in 4 chance that MaxDD didn't exceed 42.3% (or following, a 1 in 4 chance that MaxDD is greater than 42.3%).
The question remains: what do you do with the information that the probability of MaxDD greater than 42.3% is 25%? Does it boost your confidence in the system? Does it make you believe that the MaxDD of 60% is highly improbable?
A very good question TK. First and foremost it's sobering to realize that a good system can and will misbehave, no matter how great the single-run statistics. Do I believe that 60% MaxDD is less probable? No, in fact, I can see that 60% occured a little under 4% of the time, say roughly a 1 in 25 chance. Think of it like a negative lottery with pretty decent odds of actually happening (did I mention it was pretty sobering?).

The most significant thing I can do with the information is prepare and plan for the downside, like providing appropriate funding, or reducing bet size. It may also prompt me to change the market selection or system blend. So, for one, I use the information to help guide system and portfolio decisions and setting expectations around performance.

I use the information when comparing two systems. As an example, I placed the graph from Mark Johnson's public domain system Thirteen atop the existing graph. Note how the probability of drawdown is higher at every single quantile. While the test system has a 96% chance of having a drawdown less than 60%, there is only a 20% chance of seeing less than 60% drawdown with Thirteen. Or ... 4 in 5 chance that MaxDD in the comparison system will be greater than 60%. (BTW: MaxDD can be greater than 100% if and when you get a margin call and the liquidated positions place the account at negative equity). In terms of confidence, I place more confidence in trading system #1 than system #2.

I also use the information to provide insight when developing and choosing trade-offs in system behavior. For example, in LTTF systems, most of MaxDD comes from what you give back from new equity highs. So, the graph and information is also useful for evaluating exit efficiency.

Cheers,

Kevin
Attachments
MaxDD Comparisons
MaxDD Comparisons
ExampleQuantilesMaxDD3.jpg (39.72 KiB) Viewed 32458 times
Last edited by ksberg on Wed Aug 25, 2004 8:29 pm, edited 1 time in total.
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Chance

Post by ksberg »

Asterix wrote:My opinion on using Monte Carlo methods for testing trading system performance is that the Monte Carlo results may be misleading. My reason for stating this is that Monte Carlo simulations assume independence of the individual trades. (i.e. each new trade is independent from the previous trades.)

Let's not get too far into the pros and cons of Monte Carlo on this thread though - there are some other places in this forum where this topic is discussed in detail.

What ksberg plotted was the real distribution of drawdowns that may or may not be independent. One way you can use a distribution is to take the known data and extrapolate a general distribution that you think represents everything including the unknown data. (i.e. the drawdowns in the future.) You can use this to at least estimate your probability of seeing a future drawdown of any specific size.

But, it is also important to remember that the word is probability not certainty.

Like the well known disclaimer always states, "Past performance is not a guarantee of future results."
Asterix, agreed on Monte Carlo. I'm still working to find or define meaningful Monte Carlo results. Look at how many explicit dependencies exist within the Turtle system: correlations, last-trade rule, even multiple units. Monte Carlo totally ignores all of those. Not a very realistic simulation IMO. Thanks for mentioning other threads for Monte Carlo.

I think it's extremely important to stress probability over certainty. That is yet another reason I use the quantiles and distributions. From the single-run statistics there is an inclination to believe that System X will return Y% with a Maximum DD of Z%. The trader expectation then revolves around those statistics and s/he believes there's something amiss when the numbers turn out differently. In my experience, I hear these type of comments very frequently. In fact, it's actually led me to believe that single-run stats may be more misleading than helpful. Oh well, that is current state.

Cheers,

Kevin
leonardo
Roundtable Knight
Roundtable Knight
Posts: 100
Joined: Wed Apr 30, 2003 2:27 pm
Location: Minneapolis, MN
Contact:

Post by leonardo »

ksberg wrote: As an example, I've attached quantiles and histogram for MaxDD trial over a 14 year study.... You also get a sense of where the distribution is centered, and a taste of black-swan events way out on the tails (pretty sobering!)....

Why do this? Well, I don't get this level of insight from existing trading tools. Also if you can understand the information, the results are pretty sobering.
Thanks for all the other information and explanations given. Your results are thought provoking and should be sobering!!!

I've had this type of analysis done on my systems. My actual drawdowns have always been bigger the longer I've traded them. The sobering point is that you can have the "black swan" drawdown from the moment you start. And the way things work out, very few people (virtually none) start out trading a system which has been in a major drawdown. Which increases the actual average drawdown which would be experienced by a real trader.

Without going into the studies which have proven this, suffice it to say that few investors get into a trading programs at random times. They usually enter after the equity curve hits new highs or after a tiny setback. Not during a HUGE setback. Ask any CTA. Even if the system is believed to be a winning concept, the nagging doubts that maybe the system is blowing up right now and statements like "things are different now," may be true.

Extremely true example: Look at the following chart. Used with permission, this is the Aberration equity curve of a well diversified small fund (28 markets---trading domestic and foreign futures) that was traded in real time (real profits/real losses) by a friend of mine. He started trading at the beginning of the chart after doing all of the due dilligence that has been suggested in the previous posts, and then some. According to his triple-checked analysis, he should have had less than 5% chance of exceeding 40% drawdown. He stopped trading in September-2003 and the equity curve shown since then is the theoretical position since his account quit trading the model; after exceeding every known parameter of stability that his original due dilligence indicated was virtually impossible to experience.

Remember, the result below was by trading perfectly---no missed trades, errors, incorrect position sizing. Imagine how bad the result could have been if trading were done according to Optimal F?

How many people REALLY are interested in starting to trade Aberration right now? It probably is the ideal time.


Leonardo---

Chart below based on 2% Risk, 28 Markets, $150 comm/slip per contract
Attachments
Aberration-28 market portfolio
Aberration-28 market portfolio
X1.jpg (147.41 KiB) Viewed 32436 times
William
Roundtable Knight
Roundtable Knight
Posts: 238
Joined: Sun May 04, 2003 4:41 pm
Location: Manhattan, New York

Post by William »

One of the things i have noticed in my testing is that there are some systems that routinely suffer drawdowns in the 55 - 60% range, but there are also systems that normally suffer 30 - 40% drawdowns but on one occasion have suffered a drawdown in the 55 - 60%. Using the simple math of multiplying your max drawdowns by 1.5 or something similar i would simply discard that system. But considering what we have seen in the past 5 years, namely that many, many systems have their dog days, i am wondering aloud if max drawdown is the best way to go about measuring a system's worth. I have read elsewhere on this forum that it is and that it isn't, but this thread and my observations in testing makes me requestion my faith in that stat...(obviously there are many, but in the past i have given this stat an overweighting) Perhaps if you start throwing away all systems that have drawdowns in excess of say 50% in historical testing, you can miss a potentially rewarding system, that in your testing hit a rough patch. In the past i have used that rough patch as a sign that it has the potential to really slap me in the face and probably harder in the future and therefore a system not to be trusted. But given the possible observation that many, many systems are going to slap you in the face eventually, maybe its not the best way to judge a system. I think a telling indicator would be how it has performed since that time, was that drawdown an anomaly or the start of a new trend.

In a way this hints of the arbitray nature and blind luck of back testing time periods. While you might be comfortable with a 25 year back test that had a really low max drawdowns, who knows, if you used 26 years that great system might have been totally blown out... :cry: :evil:

Perhaps this is obvious to all, but it came as a potential "AHA!" moment to me... :)
Post Reply