portfolio cherry picking

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

portfolio cherry picking

Post by 7432 »

is portfolio cherry picking curve fitting?

I've always tried to trade as many markets as IB gives me access, but recently I've considered tossing a few that I've used for years.
one example is Gilts which loses money in both my systems and both my time frames and doesn't seem to smooth out the equity curve at all.
this seems to be curve fitting, but then again I don't trade s&p's, and there are many markets I don't trade just because IB doesn't offer them.

anyone want to comment on this? does it matter if I toss out a handfull of markets that haven't performed in any time period?
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

Will the future resemble the past?

Will markets that performed "good in the past" also be "good in the future"?

Will markets that performed "lousy in the past" also be "lousy in the future"?

If your honest answer is "I don't have any idea" then perhaps it is unwise for you to include or exclude markets based on past history. Perhaps you'll want to either trade ALL markets, or trade a randomly chosen subset of all markets. If you exclude any markets, do so at random, since you don't have the faintest idea which ones will be good or lousy, in the future.

I do have a piece of happy news though: it doesn't take an infinitely large account to trade an infinitely large portfolio. Search this site for "Thermal BBBO" to see a proof and a working example.

On the other hand, if you are CERTAIN that (Collection A) of markets will perform well in the future, and CERTAIN that (Collection B) of markets will perform lousy in the future, then by all means include Collection A in your portfolio and exclude Collection B from your portfolio. Is it cherry picking to act on this certain knowledge? Is it curve fitting? I myself don't think so -- as long as you are are CERTAIN that A will be good and B will be lousy, in the future.

And finally, what about the gray area between (I don't have any idea) and (I am absolutely CERTAIN)? It is the zone most ripe for thoughtful experimentation. "If I had used procedure X to choose a portfolio in 2005, how would that portfolio have done for me in 2006, 2007, and 2008? How would that portfolio have done for me, compared to a trade-them-all portfolio? How would that portfolio have done for me, compared to a portfolio containing the two highest volume markets in each commodity group?"
Paul King
Roundtable Knight
Roundtable Knight
Posts: 207
Joined: Mon Feb 23, 2004 9:13 am
Location: Vermont, USA
Contact:

Post by Paul King »

"compared to a portfolio containing the two highest volume markets in each commodity group" is the real pearl of wisdom at the end of sluggo's (very informative) post.

Maybe you're asking the wrong question when you talk about cherry picking in relation to particular markets. Maybe a better approach would be to ask:

1 Am I trading enough different markets to be diversified i.e. not too much correlated risk?
2 Is any particular market liquid enough to trade with low implementation costs at my desired position sizes?
3 Is any particular market moving enough to present current opportunities for profit (assuming you're not trading some sort of short-volatility system that depends on prices not moving to make a profit)?
4 Does the overall profile of my trading program using a particular portfolio look how I want it to regardless of which individual markets make or lose money over any particular time period?

Excluding or including a market based on an unanswerable question (i.e. what's it going to do in the future) doesn't make much sense to me.

Hope this helps

Paul
Last edited by Paul King on Wed Nov 19, 2008 7:24 pm, edited 1 time in total.
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 »

thank you gentlemen.
helpful and thoughtful responses as usual.
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

I am biased that each market has its own personality..there are markets I dont like even during backtesting stage, not because its losing or winning but because I know I cant get enough size on it once the system will start accumulate profits and get a bigger size... There are markets that trend and those that rarely trend.. I'll never trade them with same system..its not cherry picking..you just throw away the crap that shouldn't be in your portfolio, Gilts -crap, rough rice- illiquid, oats - illiquid... by the way, I am in Utah for Christmas... good weather so far
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 »

golf, skiing and the gun show this weekend.
utah is the place to be.
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

no snow in the mountains though!
7432
Roundtable Knight
Roundtable Knight
Posts: 199
Joined: Wed Dec 31, 1969 7:00 pm
Location: sandy, ut

Post by 7432 »

so I finally got around to doing the obvious with this question.

I pretended it was late 2000 and I was getting ready to trade. I tossed out any market that had not performed over the last 16 years or so(14 of 62 were tossed) and retested getting better CAGR, MAR and other stats.
at this point it was kind of obvious what would happen since some of what I considered 'good' markets had been discarded.

I traded with the cherry picked portfolio for the last 8 years, then added the markets back to the portfolio and ran the test again.
not surprisingly sluggo is right about not being able to predict what markets will be 'good' in the future based on past results.

by cherry picking the portfolio I would have missed out on quite a bit of upside with very little extra drawdown. and very important to me, I would have experienced an almost 20% longer drawdown.
if I had cherry picked even more, say gone with just the top 20 producers of the earlier period, my results for the 2000's would have suffered greatly. an example being the mexican peso which was a late 90's star making it a top ten producer, but very poor reults lately.

I used the instument performance summary for this very unscientific experiment rather than the tradelog because I don't like to sit at my desk for very long.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

7432 wrote:I pretended it was late 2000 and I was getting ready to trade. I tossed out any market that had not performed over the last 16 years or so(14 of 62 were tossed) and retested getting better CAGR, MAR and other stats.

I traded with the cherry picked portfolio for the last 8 years, then added the markets back to the portfolio and ran the test again.

Very well done, congratulations! It's thoroughly satisfying to come up against a difficult question, formulate a plan of experiments that you think will shed some light, perform those experiments, analyze your results, and Bam! realize that you've got the answer. Hats off to you.

If you plan to do any further exploration, it might be interesting to find out what happens if the trader updates his portfolio more frequently than once every eight years. Maybe the trader picks a portfolio on 12/31/1999 and trades it for 3 years (2000, 2001, 2002). Then picks a second portfolio on 12/31/2002 and trades that second portfolio for 3 years (2003, 2004, 2005). Finally on 12/31/2005 the trader chooses a third portfolio and trades it for 3 years (2006, 2007, 2008).

This would perhaps address any concerns you might have, that freezing the portfolio and leaving it unchanged for eight years doesn't reflect what real traders would do in real money accounts.

I doubt that it will change the final conclusion very much, but that's just opinion. In this case I think data is probably a lot more valuable than opinion.
Post Reply