Most of us are using insufficient data for testing

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Trading Leech
Senior Member
Senior Member
Posts: 48
Joined: Mon Nov 29, 2010 2:07 pm

Most of us are using insufficient data for testing

Post by Trading Leech »

Are most of us using sufficient data for testing? I'm starting to doubt it.

Even if we are testing against data going back to the early 1970s, it may not be enough. Why? Because it was in 1971 that the US severed the dollar's link to gold for international parties and the whole world went on a new monetary system of fiat debt-based money.

Word's on the street that this system is very fragile, and is on its deathbed. What comes next? I don't know. It could be some kind of a gold-based currency system.

Have most of us in here tested our systems against such possibilities? Probably not. To do that properly, one would need data going prior to 1933, which is when the dollar's link to gold was severed for US citizens.

This is my concern. Most of us have way too little data. Maybe Winton Capital has some densely packed data from the 19th century, but I doubt most of us do. And if the monetary system changes, the markets could indeed change, which could lead to a disaster for most of us who have only tested our systems against data from decades of a completely different environment.

My questions, therefore, are: Should we continue to trade, or should we back out of the markets for the next five years or so to just observe how things develop? Could the end of an era -- the era of trend followers -- be on the horizon?
Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Post by Moto moto »

Sluggo makes this point in this thread about trend forecasting v trend following
viewtopic.php?p=54029#54029

and maybe I am drawing a long bow here and I am sure its all been said before...... to a large extent having too much data will not necessarily improve your tests, it may in fact show that your tests over the long term merely work in some times better than others. ..... are you forecasting here?
Either way could it (I suspect so as my corns are throbbing) be argued that either looking to participate in trends --- or forecast the trends (hence the link to the thread) does not necessarily need a lot of data.
If you find that your average holding period for your system is only 6-9 months, then will having more data massively change that.....or if what you believe is that the markets are about to massivle y change from a fundamental point of view then will the trends change that much - it seems its more about how the trends are entered and exited which causes a lot of problems with whipsaws etc.
does the extra data help or hinder, are you better optimising on the most recent data and present data rather than giving massively historical data an equal weighting???

(disclaimer ; I am not a massively rabid tester and this is all just speculative musing)
Toosday
Roundtable Fellow
Roundtable Fellow
Posts: 70
Joined: Fri Jun 03, 2011 11:21 am
Location: Austin, TX

Post by Toosday »

This is an interesting subject. Based on my experience the volume data involved in testing should be related to the "return period" of the edge your system has. Meaning that if you depend on the extreme tail of the return distribution to generate returns then you need more data. Take the following (made up) examples:

30 year test 1:
5 years -20% return
20 years +10% return
5 years +25% return

30 year test 2:
2 years -50% return
26 years +5% return
2 years +100% return

I would say that test 2 should be run over more years to get a robust estimation. The two tests above have roughly the same return but different return profiles.

This concept is common in earthquake or hurricane prediction. If you are trying to get a handle on the 1 in 100 event, you need more data than if you are trying to estimate the 1 in 10. The exact amount of data can be calculated using stats but assumptions must be made. Nonetheless is you make some consistent assumptions, you can figure the relative amount of data needed to make you comfortable. As with most things trend trading, comfort is an individual decision.
BuyHigh SellLow
Roundtable Fellow
Roundtable Fellow
Posts: 50
Joined: Wed Apr 27, 2011 12:46 pm
Location: U.S.

Post by BuyHigh SellLow »

Toosday wrote:This is an interesting subject. Based on my experience the volume data involved in testing should be related to the "return period" of the edge your system has. Meaning that if you depend on the extreme tail of the return distribution to generate returns then you need more data. Take the following (made up) examples:

30 year test 1:
5 years -20% return
20 years +10% return
5 years +25% return

30 year test 2:
2 years -50% return
26 years +5% return
2 years +100% return

I would say that test 2 should be run over more years to get a robust estimation. The two tests above have roughly the same return but different return profiles.

This concept is common in earthquake or hurricane prediction. If you are trying to get a handle on the 1 in 100 event, you need more data than if you are trying to estimate the 1 in 10. The exact amount of data can be calculated using stats but assumptions must be made. Nonetheless is you make some consistent assumptions, you can figure the relative amount of data needed to make you comfortable. As with most things trend trading, comfort is an individual decision.
If we're really putting on our black swan hats, no amount of data can tell us what to expect to be realistic.
Algonquin
Roundtable Knight
Roundtable Knight
Posts: 158
Joined: Sun Jan 28, 2007 9:38 pm
Location: Greenwich, CT

Post by Algonquin »

After testing over more than forty years of data, I am confident that trend following works. From that testing, I am also confident that trend following returns are in a state of terminal decline, and that ever greater research efforts must be expended to maintain the same results. I suspect that the emergence of trend following mutual funds and ETFs isn't going to help matters.
rajivm
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Fri Jul 25, 2003 12:50 am
Location: New Delhi, India

Post by rajivm »

Algonquin wrote:After testing over more than forty years of data, I am confident that trend following works. From that testing, I am also confident that trend following returns are in a state of terminal decline, and that ever greater research efforts must be expended to maintain the same results. I suspect that the emergence of trend following mutual funds and ETFs isn't going to help matters.
Last few years ---
Gold gone from 200 to 2000 dollars
silver from 5 to 50 dollars
crude from 10 to 150 dollars - back to 35 then back to 100 +
priceline gone from 4 or 5 dollar stock to 600 dollar stock
apple from 20 to 500 dollars
goog has huge price move
Then how can you claim that trendfollowing returns would be in decline...::))
Maybe your definition of trendfollowing needs a relook
:D
Algonquin
Roundtable Knight
Roundtable Knight
Posts: 158
Joined: Sun Jan 28, 2007 9:38 pm
Location: Greenwich, CT

Post by Algonquin »

I wonder whether you have looked at the performance of professional trend followers over these "last few years?" I wonder whether you have considered what their poor performance might mean given the very large moves we've seen in various markets.
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

I think this could prove to be one of the most valuable posts ever put on this forum - I'm not sure everybody will understand why

Without doubt we do not have enough data - 30 years is a miniscule time frame in the history of what has gone before and what is to come and it may well be that Trend Following takes a 5 - 10 year break from success - the results of teh trend followers over the last few years indicates that something has changed from about 2009 onwards - ok only 3 years - but if you have investors thats way too long
We are without doubt witnessing changing markets - they are manipulated and do not move on underlying supply and demand factors necessarily. The information age means everybody has teh same information instantly and it means there are thousands of bedroom trend followers who could not have accessed the markets 10-20 years ago.

This doesnt mean it wont work again or in fact that it isnt working now / immediate future - but people who dont realise just how manipulated markets are at present really need to take a rain check

Also I believe - and its my personal opinion that we need to see one or two high profile trend followers run into problems to scare people away before we take off again

If you read teh circulars doing the rounds from compliance consultants at present about the imminent limits on speculative positions in commodities - now in Europe - as well as the USA and if you work the maths backwards - its only going to get harder for any TF with a billion plus to make money going forward - if you have 20 billion plus I think there are serius clouds on teh horizon - nobody seems to be thinking ahead re this situation
ALL IMHO
C
babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader »

If you don't have enough data for back testing, why not create it? On my blog there are two posts, here and here (both with code), which show different approaches to creating synthetic data. By adjusting the input parameters you could simulate different market environments and stress test your trading strategy.
rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Post by rhc »

Further to user 'babelproofreader's' comments above, Tuschar Chande in his book "Beyond Technical analysis" discusses the creation of synthetic data (see image below)

There is also a decent discussion (from the olden days) on this topic right here on this forum . . . for those that are interested.
viewtopic.php?t=1670&highlight=synthetic+data
Attachments
chande.jpg
chande.jpg (35.46 KiB) Viewed 12313 times
Trading Leech
Senior Member
Senior Member
Posts: 48
Joined: Mon Nov 29, 2010 2:07 pm

Post by Trading Leech »

Chris67 wrote:its my personal opinion that we need to see one or two high profile trend followers run into problems to scare people away before we take off again
A very good observation indeed. It will be interesting to see whether this happens in the coming years.
babelproofreader wrote:If you don't have enough data for back testing, why not create it?
I hadn't thought of that. Thanks!
rhc wrote:Tuschar Chande in his book "Beyond Technical analysis" discusses the creation of synthetic data
Thanks for the book recommendation!
rajivm
Roundtable Knight
Roundtable Knight
Posts: 131
Joined: Fri Jul 25, 2003 12:50 am
Location: New Delhi, India

Post by rajivm »

Algonquin wrote:I wonder whether you have looked at the performance of professional trend followers over these "last few years?" I wonder whether you have considered what their poor performance might mean given the very large moves we've seen in various markets.
Those professional trend follower might be Just happy getting 2% management fees off their clients.. Who cares about those stupid clients so long Professional trend follower can get his 2% :D
Sheep will always be slaugtered.. :D

And the " professional TREND FOLLWER " Must have mastered the art of negative returns to give negative returns in months when NASDAQ is hitting a new 10 year high on a daily basis
:lol:
Chelonia
Roundtable Knight
Roundtable Knight
Posts: 525
Joined: Mon Apr 30, 2007 3:37 pm

Post by Chelonia »

There a link here related to the very same issue.

Trend following at the brink of monetary failure

viewtopic.php?p=51469&highlight=#51469
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

40 years ago you could probably trade just 2 or 3 unrelated futures contracts like bonds and corn and oil and have a good return/risk profile. 30 years ago we have the turtles trading a portfolio of US futures with good results on a short/medium term system. 20 years ago we need to start going longer term. 10 years ago we need to start adding global futures. 5 years ago we need to start adding profit targets...

This is all simple to see in hindsight. How to deal with it in real time is another story...
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

LeviF wrote:40 years ago you could probably trade just 2 or 3 unrelated futures contracts like bonds and corn and oil and have a good return/risk profile. 30 years ago we have the turtles trading a portfolio of US futures with good results on a short/medium term system. 20 years ago we need to start going longer term. 10 years ago we need to start adding global futures. 5 years ago we need to start adding profit targets...

This is all simple to see in hindsight. How to deal with it in real time is another story...
I wholeheartedly agree with what you say. Like you I have no real answers and certainly no crystal ball. One thing I'm very sure about - I could not live through a Dunn style draw-down. I take my hat off to the man for his extraordinary fortitude. In personal terms it leads me back to something DPH said and which I agree with: from hereon I intend to trend trade with an insignificant portion of my funds so that I can survive the worst in emotional terms and live to learn from my many mistakes. For the past 4 or 5 years I have been too narrowly focused on one form of trading and, emotionally and psychologically speaking this proved to be a great mistake last year. I have now redressed the balance.
Chelonia
Roundtable Knight
Roundtable Knight
Posts: 525
Joined: Mon Apr 30, 2007 3:37 pm

Post by Chelonia »

I find this quite remarkable. So are you saying you re-balanced away from trend following? If so then what other strategies are used if any.
LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog »

Chelonia wrote:I find this quite remarkable. So are you saying you re-balanced away from trend following? If so then what other strategies are used if any.
Of course, I'm sure you know, there are a whole host of trading strategies out there aside from "trend following". I'd venture to say that "most traders" don't use the infamous LTTF technique. "Most traders" being all your "retail" traders, and all those smarty pants Ph.D. traders at the big investment banks, and all those dumber pants no degree traders at regular commercal banks, and a whole bunch of traders at various types of funds.

What I dare say all these traders have in common is that 95+ percent of them fall 100 percent into the category of traders that don't make money. This raises the more interesting question, for which there is no established answer, of which trading strategies make money.

We know the ones that the people working for Jim Simmons use makes a ton of money, but they aren't telling anyone what they are. Sure as hell it isn't a trend following set of stategies.
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

AFJ Garner wrote:I intend to trend trade with an insignificant portion of my funds...
So then why even bother with it?
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

LeviF wrote:
AFJ Garner wrote:I intend to trend trade with an insignificant portion of my funds...
So then why even bother with it?

I greatly enjoyed The Zurich Axioms by Max Gunther. Despite the breathless "shhh, this is a secret" writing style, I thought it contained several ideas of substance. The very first chapter says
In The Zurich Axioms, Max Gunther wrote:
AXIOM I: Worry is not a sickness but a sign of health. If you are not worried, you are not risking enough.

COROLLARY: Always play for meaningful stakes.
Attachments
Since Amazon.com doesn't show the cover of the book, here is a scan
Since Amazon.com doesn't show the cover of the book, here is a scan
zurichaxiom.jpg (128.43 KiB) Viewed 11916 times
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Worry can lead to severe illness and depression. I have often talked of meditation and like practices and these are designed largely to free the human mind from fear and eventual disaster. To actively court "worry" has never been a part of my strategy.

By insignificant I am talking perhaps up to 10% of my investable free capital. A 10% return on the amount devoted to my own TF trading would make a reasonable contribution to my annual living costs (particularly as I intend to trade tax free through spread betting) but is insignificant to the extent that a 35% draw down would be water off a duck's back. Last year's 30% DD plus the MF Global fiasco was not water off this particular duck's back. In fact I will let it roll up rather than taking money out and that way it may or may not become a rather more significant factor.

The big mistake I made was adding capital on several occasions. I had suffered a couple 30 to 35 % DDs before which were unpleasant but perfectly acceptable. Each time I added capital. But then the third 30% DD proved too much to stomach on the increased capital. Dumb.

I enjoy TF and the research and programming enormously and will remain very active. But it was foolish of me to be so un-diversified - I should at least have spread some money around other TF programs and may still do so, although a couple of my favourites also suffered significantly last year.

As to other trading strategies the returns on trading new issues are very much less volatile and in terms of MAR ratio very much more interesting. It is not a capital intensive business. I have restarted in earnest.

Other hedge fund strategies perhaps don’t differ much from TF in terms of risk reward ratios – somewhere else I posted the MAR ratios on a bunch of HF indices. Very similar to the CTA MARs posted by DPH – an average of 0.4. But don’t let’s forget that diversification into those other strategies at least gives you a chance of some of them being up while TF is down. You will get similar drawdowns but hopefully at different times.

I would not want to give the impression I am knocking TF – I am not. But there are a number of factors which I have come to recognise over the years which only became apparent with time. Back testing is a useful tool but is inevitably curve fit. If you don’t agree, hey that’s fine with me but that is my view. Its predictive power is strictly limited. It is indeed useful to ascertain what strategy and parameters and portfolio broadly work; and TF does broadly work. But let’s be honest: the system you have lovingly devised with a smooth 45 degree upward sloping equity curve with and fantastic risk reward metrics ain’t gonna hold up. Look at the CTA records out there; look at Dunn, at JWH’s original program, at Dave Druz’s blow up …blah blah blah. Look at the systems and portfolios myself and others have devised and published over the years…and look what has become of the risk reward ratios on many of them.

I just think we have to face reality as trend traders: whatever you devise is not going to look nearly as good going forward as your lovingly devised back tests would suggest. You are going to have to make continual adjustments down the line and you won’t know it’s f***ed up until it has. Mind you, I’d have pulled the plug rather sooner than either Dunn or JWH and gone back to the drawing board long before they did.

So for me personally this means that I will remain fascinated by TF, I will continue to trade it, I will continue my research and I will constantly try to achieve the impossible – a robust system with an ever lasting shelf life. But at the same time I will trade the higher risk reward returns available in the new issues business, I will invest in a few other strategies, I will but a few apartments in London. At least the latter should be safe from Corzine – I hope.

That MFG blow up and the fact that Corzine is going to get away with it have also sharpened my mind. I don’t want significant sums of money with any of these scum. I have opened numerous bank accounts now in the UK with banks and building societies to take account of the government guaranty of £170,000 per joint account. And so forth and so on.

But I very much hope that any one new to TF and mechanical systems will take something from my posts and not be fooled into thinking they have discovered El Dorado after they have read a few ludicrous websites and the salesmen’s ridiculous hype. There is no such thing as mechanical trading – there are continuous discretionary decisions and changes to be made and endless continuous tinkering necessary.

Anyone who believes that you wind up a mechanical system and let it run like clockwork to produce a vast pot of gold at the end of the rainbow is p***ing in the wind.

Here endeth the sermon from a very grumpy old man.
Post Reply