The Mirage

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

The Mirage

Post by AFJ Garner »

In an effort to gaze into the crystal ball I once again looked at the simple BB BO system I traded with a "well diversified" 25 instrument portfolio. 80/2/0 parameters, profit taking 50% of positions at 2 std dev and using a ridiculously high 2% fixed fractional bet size with small capital of $300,000. Trading both long and short equally. I commenced in March 2003 having fooled myself using the then state of the art Trading Recipes product.

Below I show how the back test looked up to the end of 2002 - just before I started trading. I then show the draw down chart of its performance through to 2012. I stopped trading it sometime in 2006 having got my money back (difficult to tell from the chart).

Just the period Dunn and JWH ran into maximum trouble.

Yes, the system worked well for over 30 years and then clapped out the day I started trading. How was I to know? With hindsight, what should I have done? What should I have learnt from the exercise? Hmmm............. :?

Well, I'll let you guys figure out your own answers. I have posted the story before but I found a particular focus today looking at this very long term chart and noting the alluring possibility of huge returns at bearable drawdowns which appeared a reality to me a decade ago. I thought a bit of honesty appropriate as a counterbalance to the ridiculous salesman hype out there on the net from non trading "trading gurus" and assorted scum. Merry chortle.
Attachments
To 2012.JPG
To 2012.JPG (55.18 KiB) Viewed 11097 times
To 2002.JPG
To 2002.JPG (88.28 KiB) Viewed 11097 times
squaredQ
Full Member
Full Member
Posts: 23
Joined: Wed Feb 01, 2012 5:27 pm

Post by squaredQ »

Hi and thanks for sharing that useful cautionary insight. I have a few questions if you don't mind.

1) Were all 25 instruments active over that entire period? If not, any more coloring on the breakdown of instrument/timeframe would be appreciated (or even something like most were 10years or less, 25 yrs or more, etc).
2) How many parameters were optimized on the 1st set?
3) was the same parameter set used across all instruments, or was each optimized independently?
4) was same fitness used to optimize all? ie.. cagr/dd for instance, and what was the actual fitness used.
5) Was the entire history used to optimize parameters up to trading the first day, or was any partitioning done (IS/OOS)
prior to live.
6) I saw the 80/2/0 parameters, but I'm not sure what that represents
Could you elaborate?

It would even be more surprising to see it performed all the way up to some IS set with identical performance on pure OOS data for a year or more, and then to have it immediately crash like that.

Thank you.
rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Re: The Mirage

Post by rhc »

AFJ Garner wrote: How was I to know? With hindsight, what should I have done? What should I have learnt from the exercise? Hmmm............. :?
Daniel Kahneman, in his book, “Thinking fast & slowâ€
marriot
Roundtable Knight
Roundtable Knight
Posts: 365
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot »

>If, therefore, you are running a system from 1965 to date with static parameters using a portfolio of (say) 190 instruments, it is hardly surprising if max DD increases as more instruments come on stream.

:roll:

So,also a portfolio with fixed instrument traded well before 2000 is not the answer.


If i find 400 yr of Soyabean data, i will throw out of the window. :D
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

BUT THEN AGAIN - Maybe as more instruments come on line the draw downs should get smaller ? that is one theory and a theory that many large TF's trade on such as Bluetrend who trade a whopping 200+ instruments - the smoothing effect ?
However correlation changes as we all know and that can have dramatic effects
Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Re: The Mirage

Post by Moto moto »

rhc wrote: He suggests that this is a good technique to perform to counter the ‘Optimism bias’ (which is emotional in nature and causes a person to believe they are less at risk of experiencing a negative event compared to others)
LOL - I have this inbuilt - people say I am a pessimist - I reply I am an optimist who works out what can happen, prepare for those risks and if its still worth while, you become an optimist working from a low risk base.

My partner calls it the Tsunami effect -"oh no....what happens if a tsunami comes through".....to me its all about a process that works for you.

I pretty much gave up too much relying on testing similar to AFJ Gardner reasonably quickly because I felt it would not work for me for various reasons. Now I am not dismissing it (i tried it, i traded a few things, I invested with some others) - Along with other reasons it just was not going to suit me, and the Tsunami effect said "find another place to have a seaside house - rent instead"

My testing now is more about probabilities, and individual ideas is more bespoke and then is a guide....using many trend following principles - but its not a one turn key solution whereby number of instruments, data time frames or such play such a crucial part.
(I will let you know in 30 years if it works :) - so far its been ok for the last 20 years, except for those black swan individuals I have met/encountered who have stolen money. My Tsunami counter is clearly poor at that).
rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric »

Moto moto wrote: My testing now is more about probabilities, and individual ideas is more bespoke and then is a guide....using many trend following principles - but its not a one turn key solution whereby number of instruments, data time frames or such play such a crucial part.
Agreed

The trap is this: thinking trading and economics is some kind of physics based complex phenomenon with higher order traits that can be isolated and exploited with a particular set of fixed techniques.

In actual fact it is more like a game, warfare or an ecosystem. Any successful technique or tactic will inevitably become a victim of it's own success; Firstly because once you have taken all the losers money, they have to stop, cutting off your meal ticket to some extent; and secondly because the recent winning technique becomes predictable and that knowledge can be exploited by others who adapt their tactics.

Of course the million dollar question is how exactly to go about assessing and adapting to changing opportunities. Well that is where the thought and graft comes in....

In the last 5 years both broad diversification and some medium-longterm momentum following techniques have fallen victim to their own prior success. They may arise like phoenixes in future years, who is to say, but I think it is better to just try and figure out what is working now, and what might work for the next few weeks and months, than to hold out in hope of some amazing resurrection. It may come, but will you still be in the game by then? who knows. Doesn't seem like a great career idea imho

Aided by big commercial advances in financial data processing and fidelity between 2003-2009 my trading philosphy was based around riding out pain others didn't want to take and doing what they found difficult or inconvenient(mixing LTTF techniques with high resolution intraday ST data, but holding winners like a weekly swing trader). I was quite signed up to Econo-physics as my dominant ideology at the time.
The core of my whole trading philosophy since 2008 has adapted back to my pre-2003 hit and run style of being more focussed on what the "other guys" might be thinking and doing and how i can exploit that on shorter frames. I am still mechanical in my trading techniques(enty,exit stop etc), but quite discretionary in my selection.

Took a couple of years in the wilderness to get right(2009-2010) but has been working well since then.
Don't be the majority, but don't be in too small a minority either, the sweetspot is in between.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Here is the without doubt curve fit portfolio I was trading with start dates:

30/12/1966 Corn-CBT(Floor and Electronic)
30/12/1966 CopperHG-COMEX(Combined)
30/12/1966 Wheat-Kansas City-KCBT
30/12/1966 Hogs-Lean(Floor+Electronic Combined)-CME
30/12/1966 Sugar #11
30/12/1966 Silver-COMEX(Combined)
18/07/1967 Cotton #2
28/01/1970 Lumber-CME (Floor Trading Only)
08/09/1972 FX-Euro(Floor+Electronic Combined)-CME
08/09/1972 FX-Japanese Yen-CME(Floor+Electronic Combined)
08/09/1972 FX-Swiss Franc-CME-(Floor+Electronic Combined)
13/12/1972 Coffee
29/04/1975 Gold-COMEX(Floor Trading Only)
27/04/1977 Palladium-NYMEX (Combined)
13/03/1979 Petroleum-Heating Oil #2-NYMEX(Combined)
05/04/1982 Eurodollar-3 Mth-CME-Globex(Floor+Electronic Combined)
25/08/1982 T-Note-U.S. 10 Yr
15/03/1983 Gilt-Long(8.75-13yr)-LIFFE
25/07/1983 Petroleum-Crude Oil Light-NYMEX(Combined)
01/04/1985 Petroleum-Gasoline Reformulated Blendstock
18/03/1986 Index-U.S. Dollar
27/07/1990 Gas-Natural Henry Hub-NYMEX(Combined)
16/10/1990 T-Note-U.S. 2 Yr
22/03/1991 Euro German Bund-EUREX
21/09/1992 Govt Bond-Swiss(10Yr)-EUREX
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Here is the definition of the Bollinger Band Breakout System copied from the TB User Guide:
The center of the Bollinger Band is defined by an Simple Moving Average of the closing prices using a number of days defined by the parameter Close Average Days. The top and bottom of the Bollinger Band are defined using a fixed-multiple of the standard deviation from the moving average specified by the parameter Entry Threshold.

The system enters at the open following a day that closes over the top of the Bollinger Band or below the bottom of the Bollinger Band. The system exits following a close below the Exit Band which is defined using a fixed-multiple of the standard deviation from the moving average specified by the parameter Exit Threshold.

The value of the Exit Band on the day of entry is used as the stop for the purpose of determining position size using the standard Fixed Fractional position sizing algorithm.
80/2/0
80 = the number of days used to calculate the moving average
2 = the number of standard deviations used
0 = the exit is at the moving average.
rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric »

I have to say that (with the exception of hogs, eugh) that looks like a great bunch of markets!
8)
I still think the core problem is not the portfolio composition -i would trade any of those markets today- but the static and "gamesmanship-less" philosphy used on them in your example.
(not a personal criticism at all Anthony, at the time who knew? we had these fresh new computer based data analysis tools and were figuring out how to use them, they did uncover prior inefficiencies, but the act of mass implementation slayed the golden goose of course)
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Applying 50 different randomly chosen portfolios to the system between 1967 and 2002, each consisting of 25 instruments, should have given me a more reliable picture of what to expect in real life. Each portfolio was (roughly!) equally distributed between 10 different sectors.

MAR ranged from 0.20 to 0.95
CAGR ranged from 10.14% to 22.01%
Drawdown ranged from 19.2% to 50.8%
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

rabidric wrote:
Not a personal criticism at all Anthony, at the time who knew?
I think personal criticism would be fully justified anyway, but I think this sort of "hang your dirty knickers out in public" is a useful and cathartic exercise. Many of the points you and others make have great validity. I have gained enormously in experience since that date over many, many hours of thought and back testing but I am no closer in being able to see the future.

The game has definitely changed (deteriorated) over the last decade and many cogent reasons have been suggested for this. I continue to believe in TF but it is best to prepare for the worst and be pleasantly surprised if it does not happen.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Extending the random portfolio tests to 2012 (again starting in 1967) and using another 50 randomly generated portfolios each divided by sector in the same manner, produces the following figures:

CAGR ranged from 7.84 to 19.99%
MAR ranged from 0.13 to 0.57
Max DD ranged from 29.3% to 65.8%

It is worth noting that in the great majority of these tests the real rot set in and from around the year 2000.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

You could of course carry on this game until you had exhausted every single possibility but for what it is worth, here are some figures for 100 randomly generated 25 future portfolios, again evenly split between sectors. This time the tests end on 31st December 1999:

CAGR ranges from 5.85% to 23.57%
MAR ranges from 0.20 to 1.27
Max DD ranges from 15.5% to 40.4%

Despite the opinion of some, I also find Monte Carlo simulations useful and interesting and such tests as I have conducted in these various ways fit in rather well with the figures provided by Dean Hoffman on CTA performance in general. I can't remember whether his figures did or did not include closed programs but he came up with an average MAR of 0.40 I must revisit his post and check it out.

There are still many CTAs out there who adopt a traditional trend following approach. Their systems are not static since they have had to continually adapt over the years (despite denial of this by an asinine and much publicized non trading commentator on TF).

What does the future hold for traditional TF? Again, I suspect it will continue to work in some shape or form but it probably pays to keep one's expectations within reasonable and realistic bounds.
marriot
Roundtable Knight
Roundtable Knight
Posts: 365
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot »

>CAGR ranged from 7.84 to 19.99%
MAR ranged from 0.13 to 0.57
Max DD ranged from 29.3% to 65.8%

Now i would take the 65.8 %DD portfolio and i would try to play
on with parameters, stops or whatever.

But would not be easier do these test on all 160 futures ?
( i do not count Lme)
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

marriot wrote:
Now i would take the 65.8 %DD portfolio and i would try to play
on with parameters, stops or whatever.

But would not be easier do these test on all 160 futures ?
( i do not count Lme)
The total portfolio was indeed a large one out of which 25 were chosen at random and spread equally between sectors. By all means use a big portfolio and govern risk by risk limits rather than portfolio size if you prefer. Or any combination of these approaches together with a long term filter. I think it is perfectly valid to optimize the parameters and doing this could indeed have greatly improved your back tests for the past 12 years. I doubt that going forward this will prevent a trend follower from suffering unpleasant surprises however.
marriot
Roundtable Knight
Roundtable Knight
Posts: 365
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot »

>Alp wrote:

I would just add that Pardo points to a sober way of dealing with this issue: "If market conditions remain the same, a new reoptimization of the trading system will most likely arrive at the same parameter values that were identified by the previous optimization. Conversely, if market conditions change, especially dramatically, reoptimization will most likely identify new parameter values." After all, there might have been errors in the design and optimization process of the first system which, as a matter of fact, is not quite robust; or else, market history available at the inception was not representative enough of general and recurring market conditions.
Imperium
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Tue May 17, 2005 1:04 pm
Location: London

Post by Imperium »

marriot wrote: If i find 400 yr of Soyabean data, i will throw out of the window. :D
I read an article recently where Winton (i think) said they have wheat price data going back to Babylonian times. Also, they send people to libraries to memorise price data in old manuscripts/records that cant be photographed/removed and model it afterwards!

..... I didnt check to see if the date on the article was April 1st... maybe I should have :?
Imperium
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Tue May 17, 2005 1:04 pm
Location: London

Post by Imperium »

Here is the link to the article mentioned:

http://www.canada.com/mobile/iphone/tec ... story.html

It was barley and sesame prices, not wheat.

This is what you're up against.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

I am very glad this thread has not, after all, been deleted in Management's latest purge.

I had put a lot of thought into my post and was about to reveal what appeared to me to be a possible answer to the questions I raised when suddenly the shutters came down. Hey presto: 10 years of usually pleasant and productive exchanges with the trend following community at large came to an end. And this thread had been deleted.

Oh well, perhaps I will get around to finishing my arguments tomorrow.
Post Reply