Curve Fitting

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
jgh2020
Full Member
Full Member
Posts: 20
Joined: Wed Jan 02, 2008 6:31 pm

Curve Fitting

Post by jgh2020 »

Consider the following: When backtesting my long term trend system, I have run individual market tests for all liquid US futures using multiple time frames. Several futures have very low or negative expectancies. These futures have been eliminated from this particular system. I'm curious to see if traders on this forum would consider this curve fitting.

Thanks for any input
Eventhorizon
Roundtable Knight
Roundtable Knight
Posts: 229
Joined: Thu Jul 08, 2010 2:36 pm
Location: Boulder, CO
Contact:

Post by Eventhorizon »

In short, yes - though I would not call it curve fitting: Curve fitting amounts to trying out so many things that you find something that appears to work but is only coincidence.

What you are proposing is worse than curve-fitting it is using hindsight to ignore unfavorable outcomes.

So how might you address the challenge (it's something I have been working on too!) of identifying whether a given instrument trades in a way that we can predict whether a given system can extract a profit from it?

Let's say you have a pool of 100 futures contracts. Divide them randomly into 2 sets of 50. Run your system on one set of 50 to separate that group into, say, n that you would want to eliminate and (50 - n) you would want to keep. Now, you need to devise some test, independent of your trading system, that discriminates between the two groups i.e. you endeavor to identify some characteristic in the data for each contract that "tells" you whether it is a good candidate for your system or not.

Now run your discrimination test on the other 50 to separate that set into "contracts I expect to be good" and "contracts I expect to be bad". Then test those contracts using your system, are they actually "good" and "bad" contracts (in a statistically meaningful way, with an acceptable level of type 1 and type 2 errors)?

If not, put all the contracts back into a group of 100 and start again randomly dividing them into 2 groups. Repeat until you run out of ideas for tests to discriminate between contracts.

Better yet, you would want to divide up your data into random time periods to see if your "bad" contracts are always bad, or just bad some of the time, and to see if your discrimination test accurately predicts when they will be bad and when good. This increases your sample set and reduces the likelihood of curve-fitting.

If, and only if you are successful, then I think you can justifiably filter your portfolio using your discrimination test, NOT using the trading performance of your system (there will be some Type 1 and 2 errors).

Now, here's the bit that bakes my noodle ... what if my test is the performance of a trading system (The Discriminator System)? What if Discriminator effectively discriminates among the universe of contracts and accurately predicts the likelihood that my "real" system (RealDeal) will perform well when applied to a given contract? How different would Discriminator and RealDeal have to be to give me confidence that I can use Discriminator as a test? I don't know - I told you my noodle's baked!

edit: typos
Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Post by Moto moto »

plus just because an instrument has a low or negative expectancy over the total period of a test, it does not mean that it did not contribute positively at some stage during the test.
rabidric
Roundtable Knight
Roundtable Knight
Posts: 243
Joined: Mon Jan 09, 2006 7:45 am

Post by rabidric »

but if you are trying to maximize bang for the buck when it comes to something like returns on margin invested, then you are onto something jgh2020.

If you believe there are persistent and sound fundamental and structural reasons why your "black sheep" markets may not suit the particular trading algorithm you use, and you find a different trading algorithm that does suit them better. Well you might want to look into that...

You won't get much agreement on this forum, as it is a line of reasoning that appears to fly in the face of the espoused doctrine....which has started to turn into dogma here i'm afraid to say, due to its popular endorsement by many forum users.

This is probably due to the various subtle pitfalls you can encouter by digressing from the standard "one size fits all" approach. It is difficult to succesfully tune in alternative subsets, and you should always beware of going to too small a subset portfolio(e.g. single market fitting). But it is a viable approach, if you are aware of what you are getting into, and practice moderation.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

rabidric wrote: You won't get much agreement on this forum, as it is a line of reasoning that appears to fly in the face of the espoused doctrine....which has started to turn into dogma here i'm afraid to say, due to its popular endorsement by many forum users.
Ouch. In any event you do not need "agreement" from anybody about anything. Surely the point of such a Forum as this is for people to express their trading ideas, their opinions, their queries, their research. The more varied the ideas the better. There is no inquisition. In point of fact few have actually published a system here which captures pattern recognition along the lines of traditional technical analysis formations. By way of a "for instance". I don't think anyone would scream heresy, or laugh or cry if someone cared to attempt to describe "head and shoulders" to TB to capture profits that way. Or anything else. Yes, most/many of us seem to be LTTF but that does not mean that any of us actually have a closed mind.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Re: Curve Fitting

Post by sluggo »

I failed to notice an important phrase (highlighted in red) until just now
jgh2020 wrote: ... When backtesting my long term trend system, I have run individual market tests for all liquid US futures using multiple time frames. Several futures have very low or negative expectancies. These futures have been eliminated from this particular ...
I think it may be useful to glance at a prior thread (here) for a piece of data on non-US futures. One of the figures from that thread is copied below.
Attachments
other_thread.png
other_thread.png (31.23 KiB) Viewed 4245 times
Post Reply