CURVE FIT OR GENIUS?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
mpok8
Roundtable Fellow
Roundtable Fellow
Posts: 82
Joined: Wed Nov 26, 2008 6:00 am

CURVE FIT OR GENIUS?

Post by mpok8 » Mon May 31, 2010 2:53 am

I've read a lot of books about how the 'best' systems are those which are simple with as few parameters as possible and work over the widest range of parameter settings... This seems to be the accepted wisdom and the 'secret' to trading success which is regurgitated by gurus and newbies alike...

However, I have come across a system that seems to go against this wisdom and does very well in doing so (on past data at least and in the last month since i have been watching it out of sample)... I would like to hear opinions about whether this system could work going forward....

The system was basically discovered by plotting various indicator values against returns for a certain commodity. E.g. if an oscillator is at 65 for a certain commodity then how on average does the commodity behave for the next month...? This was done for an oscillator (as above) and a couple of other indicators to find the range of values for which the commodity tends to trend in a single direction... Then the system basically waits for all 3 (for example) indicators to be in trend up (or down) mode and goes long or short... it gets out when one of the indicators is no longer in trend up (or down) mode...

Immediately you can see that this is a system created by curvefitting and will only work with the parameter values given by the test to find those parameter values... i.e. if you change the ranges for which you define the commodity to be trending then the system will not work... also this is commodity specific: the ranges are discovered over all the data available for the commodity...

I know this challenges the common wisdom of how systems should be... but can anyone tell me why it shouldn't work? Assuming that the commodity behaves in a way that isn't totally random then why isn't it possible to find instances in the past when on average if a number of indicators are aligned in a certain way the price is likely to do one thing more than another? And then to make the assumption that this behaviour is likely to continue into the future?

And by the way: there are plenty of people I know who spout the common wisdoms out there but have never made money in the markets (maybe they make money selling books and courses though)... The most successful hedgefund manager I know who made 200 mill last year with starting capital of 360 mill never uses stops... and he has been making similar returns for the last 10 years, having only 1 down year in taht whole period....

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Mon May 31, 2010 9:29 am

Great post from Ted Annemann about such things: (REF)
There's absolutely nothing wrong with trading that way. And there's absolutely nothing wrong with getting your trading signals from a Coke bottle that allows you to communicate with super-intelligent beings on Mars. If you haven't heard the (true) story, check it out: http://www.traderclub.com/index.php?opt ... &Itemid=58

The big question is, how well does your proposed trading method fit YOU? Will you be able to faithfully carry out its signals? Or will you find it irresistable to "jump" the system? Does it trade too often for you? Not often enough? Does it require you to constantly place new stop orders at new stop prices, every single day, for every single market in your portfolio? If so will that be a problem for you?

Are you interested to know how your system performed in past history, using a computer backtest? Are you interested to know how a bunch of other systems performed in past history, so you can compare your system's backtest results against them? Are you interested enough to spend the time and effort necessary to obtain the software and historical price data, then put these systems into the software and run the tests? Are you interested enough to spend the money that this kind of testing will cost?

It all gets down to you and who you are (as a trader) and what you want. For that reason, we the other readers of your post, can't offer too much specific advice. We're not you.

techtrade89
Senior Member
Senior Member
Posts: 28
Joined: Tue Dec 30, 2008 5:36 pm

Post by techtrade89 » Mon May 31, 2010 12:06 pm

My thoughts:

A month real time watching after testing on past data isn't enough watching.

I suggest you go back to the system conceptually and reserve a few recent years as "out of sample" data. At least 3 probably, more if you can spare it.

Curve fit the system on the in sample data. How much data is used in sample is somewhat up to you. 30 years? 20? 7? Normally more is better but this sounds like the kind of system where the optimal parameters might drift somewhat over time, so probably you need to take this into account.

So, you build your system on in sample data, say 7 years (e.g, 2000-2006), then once that is done test it with one final run on the next 3 years of out of sample data (2007-2010).

How does it do then on the out of sample data . . . ?

Good luck!

mpok8
Roundtable Fellow
Roundtable Fellow
Posts: 82
Joined: Wed Nov 26, 2008 6:00 am

Post by mpok8 » Tue Jun 01, 2010 2:59 am

Hi

Sluggo -- your answer does not address my question: I am not asking whether this system fits my personality or whether I have the right software/historical data...

Techtrader89 -- I have already done as you suggested, i.e. used out of sample data to find my parameter settings and in sample data to test whether it still works.... and yes it does...

however, i'm still suspicious... you gotta remember that this system is right 75% of the time and on average makes twice the amount on a winner compared to a loser... is this some knid of holy grail???

Mike

jasonp111
Contributor
Contributor
Posts: 1
Joined: Thu Mar 25, 2010 4:48 pm

Re: CURVE FIT OR GENIUS?

Post by jasonp111 » Tue Jun 01, 2010 6:00 am

Hi Mike
mpok8 wrote: the ranges are discovered over all the data available for the commodity...
Does it not follow that the system will work with all the test data in the series if you have fitted the ranges to the data?

I would remove some test years or random chunks of data, recalculate indicator ranges and then test just on the missing years. Hopefully it will produce similar results.

Good luck.

mpok8
Roundtable Fellow
Roundtable Fellow
Posts: 82
Joined: Wed Nov 26, 2008 6:00 am

Post by mpok8 » Tue Jun 01, 2010 6:49 am

Yes -- that has been done... i.e. the parameter values were found on out of sample data (1985-2003) and tested on in-sample data (2003-today)... the system still works fine and gives roughly the same values as using data 1985-today in order to find the values...

7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 » Tue Jun 01, 2010 10:00 am

I can think of at least 2 dozen extremely curve fit parameter sets for a trading system that I've used and people I know still use for trading different markets. the sets are curve fit to specific markets, time periods, price, volume, season....
the big issue is directly related to sluggo's post of ted annemann's comment, do you really believe in the system? if you don't, once the system starts losing money doubts about the curve fit nature of the system will creep in and you might stop trading it.
my 2 cents would be to trade it as part of a strategy so when you start thinking about the curve fit nature of this system, the income from other sources will help cushion the losses and enable you to keep trading it.

bobsyd
Roundtable Knight
Roundtable Knight
Posts: 361
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd » Tue Jun 01, 2010 3:55 pm

Have you tried walk forward analysis? Various combinations - eg: In sample for 10 years, followed by out of sample for 2 years, etc?

mpok8
Roundtable Fellow
Roundtable Fellow
Posts: 82
Joined: Wed Nov 26, 2008 6:00 am

Post by mpok8 » Wed Jun 02, 2010 2:11 am

I have tried in-sample followed by out-of-sample tests and it has passed those... so i guess this is just a really good system then...

td80
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Thu Mar 08, 2007 8:38 pm
Location: Irvine, CA

Post by td80 » Thu Jun 03, 2010 8:07 pm

I don't mean to speak for sluggo but I'm not sure you will see him much more on this thread, he tends to offer (valuable) bits of wisdom and doesn't seem to like to wrestle in the mud too much unless there is some new aspect that is brought to light. I'm afraid I haven't reached that level of zen yet. :wink:

This topic of curve-fitting is something that I sort of obsess over. I am totally against the type of trading you describe, but I can't be sure that my thinking is absolutely the way to go for others (see sluggo's quote of Ted above)...

I personally think trading something in the manner you describe is intellectually dishonest and your subconscious is saying DANGER DANGER (and hence you come here asking to be saved), but your results look so damn good that they are leading you down a path of a potential abdication of discipline!

I think this happens to all of us. It has certainly cost me time and money more than once (usually working with day trading systems that actually do have an edge until you get into the nightmare of execution). The get-rich slow scheme with all of its potential pitfalls, and worse its pedestrian (almost dumb as wood common sense) approach is just not that exciting or intellectually rewarding (on a regular basis).

All I ask you to consider is if I showed you the Turtle system in say, 1985, with a solid 25-30 years of data, and you know what you do today about systems with many parameters, unstable edges, etc. Would you have traded the Turtle for 10-15 years as the returns went haywire? Do you think this system you have come up with for instrument XYZ is less or more likely to be curve fitted than the original turtle? Just something to think about...

techtrade89
Senior Member
Senior Member
Posts: 28
Joined: Tue Dec 30, 2008 5:36 pm

Post by techtrade89 » Thu Jun 03, 2010 8:32 pm

Here are further thoughts, which like free advice may be worth what you pay for it . . .

There is a right way and a wrong way to use out of sample data.

Here is a wrong way:

Say you have in sample data, data set A, and out of sample data set, data set B. So you do the following:

#1. Find/fit a working system to data A, test it on data B, it doesn’t work, so you . . .
#2. Find/fit a new working system to data A, test it on data B, it doesn’t work, so you . . .
Repeat about N times . . .
#N. Find/fit a new working system to data A, test it on data B, it works – declare victory!

The problem here is that this process is essentially curve fitting (albeit manually and laboriously) to what should be out of sample data, data set B. To put it another way, if N is 100, you have generated a statistical ensemble of 100 systems which you know work on data A and then selected the top 1% of these, with hindsight, as regards to working with data set B. It can be a bit insidious because it sneaks up on you – if you do this testing over a long period of time you can accidentally end up in the situation without realizing it.

The right way to do it is more along the lines of the following:

Come up with ideas for a trading system and work with it for a few months on data A and subsets of data A. Learn how and why some parts of it work and/or come to terms with not fully understanding other parts. Once complete, get ready to trade tomorrow – but today run a dress rehearsal on data set B. If it works, start trading tomorrow. If not, back to the drawing board for a few months.

If this is too psychologically hard to do in practice, then you need data set A, B and C. A is for testing, B is for quasi-out of sample evaluation and C is for the one dress rehearsal before opening night.


Or perhaps you already did thing the right way and things are still looking up . . . ?

babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader » Fri Jun 04, 2010 6:55 am

Following on from techtrade89's post, another "right way" to test might be

i) start with a hypothesis about how markets behave, drawn from observation, theory or experience

ii) use Monte Carlo techniques to create synthetic data that have this property and then develop, test and optimise your system(s) on this data

iii) test your system on real, out of sample market data

If step iii) is successful, do MC randomisation with this data to determine whether your results are significant or not. If they are significant, congratulations, you've developed a non random system that profitably exploits a defined market behaviour.

7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 » Fri Jun 04, 2010 9:55 am

some people refer to what techtrader89 has written as the 'wrong way' as data burning. a very serious issue for a one person trading operation.
once you've tested out of sample its tough to go back to testing in sample with out knowing what happened out of sample and adjusting your testing.

in fact if you have been carrying positions over the last 5 or 10 years its tough to consider the data out of sample even if you haven't used it for testing.
you know what happened during the commodity bubble from late 07 to 08 and without realizing it you might test for parameters that have wider exits so you will catch the full move in energy.
or possibly expand your entries so you wouldn't get chopped up in stocks during 04-05.

techtrade89
Senior Member
Senior Member
Posts: 28
Joined: Tue Dec 30, 2008 5:36 pm

how did it go?

Post by techtrade89 » Sun Mar 11, 2012 4:58 pm

Hello mpok8. It's been over a year and half since the questions arose on your system.

How did it go? Did you trade it? Did it perform well?

Thanks!

trackstar
Roundtable Fellow
Roundtable Fellow
Posts: 76
Joined: Sun Mar 09, 2008 11:03 pm
Location: Houston Texas

Post by trackstar » Thu Mar 15, 2012 11:49 am

I am adding a bump, very interested in his reply!

Demon

Post by Demon » Thu Mar 15, 2012 1:04 pm

Perhaps his final post gives us a clue...

viewtopic.php?p=44143&highlight=#44143

Post Reply