Curve Fitting or appropriate market selection

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
ColdFact
Senior Member
Senior Member
Posts: 31
Joined: Wed Oct 27, 2010 2:07 pm

Curve Fitting or appropriate market selection

Post by ColdFact »

I have been able to dramatically improve both the live (but only recent past) performance, and backtested performance of my system, by not trading as many markets as possible. I use a very simple system:
IF 45/150 MACD is positive, go long on a new month high
IF 45/150 MACD is negative, go short on a new month low
Initial stop set to 1ATR, next stop set to new low / high for the month
Risk per trade .75%

My system is up 10%ish on the year (after a 20% drawdown) - but would have been down 20% had I simply traded all markets

What I have tweaked is the following:
1)After testing every 10 year period from 1970 to now (1970-1980, 1971-1981, 1994-2004, etc) - I have found that using these parameters, you are better off not shorting grains (over 95% of the time). Hence, I don't short grains.
2)After the same tests mentioned above, I've found that with a similar 90+% of the time, this set of rules does not trade Cocoa, British Pound, Stock Indices, Coffee, Oats, or Orange juice profitably. When they are profitable, its ever so slightly.
3)After looking at every year of trading since the Vix has existed, and particularly since the contango has gone crazy - it makes no sense to go long the Vix - hence I only short the Vix.
4) I don't take long signals in any market with month over month contango>10%, and vice versa for backwardation

This leaves me with 23 tradeable markets (using OptionsXpress currently, switching to tradestation):
Aussie, Corn, Crude, Cotton, Dollar index, Bunds, Euro, Eurodollar, Feeder Cattle, Gold, Copper, Heating Oil, Gasoline, Nat Gas, Rough Rice, Soybeans, Sugar, Swiss Franc, Silver, 10 year, Vix, Wheat

My first instinct in reading this message is to slap myself for curve fitting, but at what point does it make sense to not do things that never seem to work out? Further, the results improve in almost all time periods, not just recently. Cocoa, BP, etc are not profitable using these parameters in hardly any 10 year period. I understand that hindsight bias in my market selection, and only taking longs in the grains would imply curve fitting is going on - but the markets i've eliminated are consistently in almost all time periods not profitable.

I'd very much be interesting in hearing/discussing when it makes sense to not trade a given market with a given system, when it makes sense to only go long - as well as the case for only running systems that can trade all markets profitably.

-ColdFact
Attachments
70s.png
70s.png (9.75 KiB) Viewed 12431 times
80s.png
80s.png (10.15 KiB) Viewed 12431 times
90s.png
90s.png (10.92 KiB) Viewed 12431 times
2000.2009.png
2000.2009.png (10.67 KiB) Viewed 12431 times
2002 to current.png
2002 to current.png (10.32 KiB) Viewed 12431 times
Chris67
Roundtable Knight
Roundtable Knight
Posts: 1052
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 »

thanks for sharing your system / analysis
results obviously speak louder than words and you have good results so thats a good start

Obviously you are breaking some "classically accepted" rules of optimisation with regards to leaving out markets that have not been profitable - but then your reasons for thsi appear quite sound i.e. backwardation / contango issues

Maybe shorting grains but only above a given threshold ?
Also cocoa and pound have gone nowhere for years - oarticularly the pound - but that could change i guess ?

I think your predicamnet is one we all face - we kind of all know certain things just dont make sense but we do them anyway as its accepted that we must - no easy answers i think
good luck
marriot
Roundtable Knight
Roundtable Knight
Posts: 365
Joined: Thu Nov 20, 2008 3:02 am

Post by marriot »

What if today is 30 dec 2010.
Are your best parameters 45 /150?
If not, look at results on the last 24 month with new parameters.
You are testing with one million as starting equity.
Try also 500.0000 and 1.500.000.

regards
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

I'd say you are asking for trouble, but what do I know.

I will note that if you were a proponent of the scientific method you would not perform an experiment in this manner.
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 »

I like marriot's comment, but I would go back to a different fiscal period.
pretend it is 1985 and run your tests from 70-75, 75-80 and 80-85.
would you have left out the pound and ny cocoa?
I actually don't think it is a big deal to cherry pick markets, we are after all cherry picking systems to trade based on past performance.
ColdFact
Senior Member
Senior Member
Posts: 31
Joined: Wed Oct 27, 2010 2:07 pm

Post by ColdFact »

Marriot: I have altered the starting equity, it has no impact. I have also altered the 45/150 - you can move either parameter by 25-50%, and it has very little impact on the performance of the system. I am not afraid at all that the curve fitting has to do with the parameters, I am afraid that it's not valid to cherry pick the markets.

7432: I have done testing in every 10 year window from 1970-2012. I.E. 1971-1981, 1982-1992, 2002-2012 etc - so some 40 odd 10 year periods. What I found, was that certain markets are consistently, in multiple decades, deleterious to the performance. What I'm wondering is, when the characteristics of a market have made it unprofitable to trade for 20 straight years, does it make sense to keep on trading it?

For example, the following describes Cocoa:
1985-1990: Profit Factor 1.06
1990-1995: Profit Factor .47
1995-2000: Profit Factor 1.01
2000-2005: Profit Factor .62
2005-2012: Profit Factor .45

So...Cocoa is either neutral, or highly negative, in its impact on the portfolio. It has consistently been neutral to negative for the past 30 years, why would I add it in the portfolio given this to be the case?
LeviF
Roundtable Knight
Roundtable Knight
Posts: 1436
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF »

You should test this in a walk-forward simulation. Start 40 years ago, select parameters for entire portfolio, throw out worst markets, walk forward a year or 5, rinse & repeat.

I did this not too long ago and unfortunately, my conclusion was that past market performance is not indicative of future success.
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 »

I only mentioned the 75-85 because it was a good trading period for the pound. if you see no added value just leave it out. the pound could be range bound for 10yrs like 93 to 03 and only good for selling atm straddles.

there are some posts on this forum about trading seemingly bad markets only for their negative correlation to your equity curve. so on a short term look, say 1 month, bad market X will make money while everything else loses money.
in the end the trade in market X is a loser, but for that 1 month period the pain of the DD was decreased.
stopsareforwimps
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Sun Oct 10, 2010 1:47 am
Location: Melbourne Australia

Re: Curve Fitting or appropriate market selection

Post by stopsareforwimps »

ColdFact wrote:I have been able to dramatically improve both the live (but only recent past) performance, and backtested performance of my system, by not trading as many markets as possible.
What I have tweaked is the following:
1)After testing every 10 year period from 1970 to now (1970-1980, 1971-1981, 1994-2004, etc) - I have found that using these parameters, you are better off not shorting grains (over 95% of the time). Hence, I don't short grains.
2)After the same tests mentioned above, I've found that with a similar 90+% of the time, this set of rules does not trade Cocoa, British Pound, Stock Indices, Coffee, Oats, or Orange juice profitably. When they are profitable, its ever so slightly.
How are you making these decisions to exclude markets? It is very trivial to have a massively profitable system, in retrospect, if you omit trading in markets that proved to be unprofitable.

It is a lot harder to do this prospectively. If you are using 1970s data to decide which markets to trade in the 1980s, that may be OK. But if you are using 1980s data to decide which markets to trade in the 1980s and 1970s you are probably deluding yourself.
fisu
Contributing Member
Contributing Member
Posts: 8
Joined: Thu Aug 07, 2008 4:12 pm

Re: Curve Fitting or appropriate market selection

Post by fisu »

ColdFact wrote: 4) I don't take long signals in any market with month over month contango>10%, and vice versa for backwardation
How do you test this condition?
ColdFact
Senior Member
Senior Member
Posts: 31
Joined: Wed Oct 27, 2010 2:07 pm

Post by ColdFact »

I haven't coded that, I'm simply to doing it because i looked at situations when the system would have, and it never ended well. I understand that in hindsight I cannot remove unprofitable markets, and expect performance to be the same going forward.

What I'm interested in learning though, is if a market has NOT been profitable for 30 straight years, or 20 straight years, or even 10 straight years, then shouldn't I assume some characteristic about that market makes it unlikely to trade profitably in the future?

The parameters I am using, were the ones that I found to be the most robust over the past 40 years. I.e. they are the mid point, from where I can change any parameter, or any combination of parameters by 40-50% (for example: instead of 45/150 for the MACD, 25/200 will work fine) and still have good results. What crushes the results, are trading markets that never seem to be profitable, and shorting markets using the same parameters as going long, in markets that always tank differently than they rise.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

I strongly suggest you investigate the superb work undertaken by various Forum Members on the following topics with code generously provided by said members:

1) Testing a given system with randomly chosen portfolios.
2) Testing random entries with trend following exits.
3) Testing trend following entries with random exits.

viewtopic.php?t=3779&postdays=0&postorder=asc&start=0
viewtopic.php?t=3796
viewtopic.php?t=8703&highlight=random+portfolio

For good measure see my never fully developed post on curve fitting portfolios:

viewtopic.php?t=9607&highlight=random+portfolio

I am as fallible as the next guy but I am increasingly wary of any clever clogs approaches and prefer....cockroach systems, few as possible parameters, wary of different standards applied to different markets or trade direction. I believe the above posts should give the trader a great deal of comfort for the longer term.

Note: Some of the above posts are in Customers Only sections of the Trader's Roundtable Forum -- Moderator
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2071
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner »

Also look at the following post by Sluggo and my suggested amendments to his code:
viewtopic.php?p=56364#56364

Note - the above post is in a Customers Only section of the Trader's Roundtable forum -- Moderator
mirec79
Full Member
Full Member
Posts: 16
Joined: Fri Sep 02, 2011 6:41 am

Post by mirec79 »

AFJ the link doesnt work for me...

M.
babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader »

if a market has NOT been profitable for 30 straight years, or 20 straight years, or even 10 straight years, then shouldn't I assume some characteristic about that market makes it unlikely to trade profitably in the future?
Well, yeah! You have a lot of evidence that shows no matter how you tweak the parameters of your system you can't get it to show a profit on some markets, so why should you assume that it''ll be profitable to trade it on these markets? So what if not including these markets in your back test improves the overall results at the expense of breaking some rigorous statistical "no go" such as hindsight bias.

It's worthwhile remembering that a lot of the advice for sound back testing, such as walk forward testing, out of sample testing, randomisation, boot straps etc. are simply attempts to impose good practice from the world of statistics to back tests, with the hopeful intent of avoiding fooling oneself with over optimistic results that lead one to take real money to the market only to see it disappear in an horrendous drawdown/blow up that the back test results didn't suggest was likely. However, you're not in this position. You're not taking some idea to market with some over optimistic hopes of future profitability with low/no draw down - you're just dumping a dog of an idea that has no evidence to support it.

I remember that on a statistical blog I regularly follow someone asked a question about a machine learning algorithm they were trying to use on n data with n plus a lot more parameters, and someone else answered that you shouldn't do this because statistically blah, blah, blah it's bad practice, which then received the comment "This is machine learning, not classical statistics, so you can do that," or words to that effect. Well, in a similar vein you're not writing a research paper for a learned society where the deliberate omission of data would be a cardinal sin; you're simply not including markets, which you have no inclination to trade, in your back test, and if you want you can do that.
Moto moto
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Mon Jun 01, 2009 4:12 am
Location: once again in the UK

Post by Moto moto »

in other words - do more of what works and less of what does not :)

Deal with reality and not some purist idea - unless of course that a product you have sold to your investors.....
Also - in my brief reading i assume that even if an instrument has been unprofitable over 10-20 years, does that mean it did not help at all.
Could there have been an example when instrument A loses over 10-20 years, but in year 6, it was the only one that contributed positively helping contribute to the overall combined compounded return of the 20 years?
Post Reply