Back Adjusting Futures Data

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
enigma
Roundtable Fellow
Roundtable Fellow
Posts: 99
Joined: Tue Apr 29, 2003 6:56 am
Location: UK

Post by enigma »

Which is the preferred method for linking historical futures prices (i.e. rollover) for backtesting? Thanks.
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

This is a huge subject ... the real answer is that you roll them over in the way that reflects how you are going to trade them.

This means that the simple answer is to backadjust them (old contracts matched to current price at rollover date) usually after the open interest of the new contract has exceeded the old contract for two days. Stridsman recommends proportional back adjustment but I disagree with him (because of the way systems perform on data).

If you will trade a contract on a different basis (like ED where you may want to look for the most volatile contract which probably isnt the front month) then you need to roll this way for your test.

A full discussion of methods which increased my understanding considerably is in Bob Fulk's paper at:
http://www.traders2traders.com/linkinde ... tegoryID=4

John
enigma
Roundtable Fellow
Roundtable Fellow
Posts: 99
Joined: Tue Apr 29, 2003 6:56 am
Location: UK

Post by enigma »

Thanks for the article John and also for your comments in the other threads.
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

I have been looking at differences in performance between two files of teh same commodity where each file uses a different set of expiries.

For example,

File 1: March June Sep Dec
File 2: June Dec.

It amazes me how much difference it makes and neither of them is 'right'.

Is it fooling ones self to use the file that gives the better return?
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

I'd suggest that it is fooling oneself if one would not have worked out ahead of time that this was the correct way to roll a contract.

If you now work out that ED is likely to have more volatility in December than in the intervening months then a rule where you roll when the OI on december reaches X would be valid. You could retrospectively test the difference to find out what would have happened if you traded that way in the past to decide if it would be more profitable if it continued to be valid in future.

John
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

John: Thanks for your thoughts on my question. I also agree that this would be fooling ones self.

For what it is worth: In real trading I try to roll an open trade into the most distant contract that has a reasonable level of V and OI. This changes depending on market and also current events. It is not uncommon for me to be long XYZ June, then roll into XYZ sep after which the position is closed at my trailing stop. As Dec comes to expiry I might decide that Sep next year is the contract that I want to follow, so I will skip all the other months and back adjust dec next year onto teh time series. I now have quite a few market files (on which I run my trading system in real time) which are not composed of a regular sequence of expiries. So it is difficult to replicate this when generating a back adjusted historical file for testing purposes.

cheers
damian
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

How does it compare with just adjusting on OI profitwise?
kmulford
Senior Member
Senior Member
Posts: 41
Joined: Mon May 12, 2003 5:59 pm
Location: Setauket, New York

Backtesting with backadjusted time series

Post by kmulford »

How does one handle commission costs in historical testing across backadjusted time series? Presumably, for long-term trend following systems, a long (short) position may carry over a number of different contract months/years. Commissions are charged on the rollover. Should one simply goose-up the commission/slippage charge against the sytem? Or, should one review each trade's duration and estimate the added commission expense?

Additionally, has anyone performed any studies regarding relative liquidity for various commodity and financial futures contracts across each calendar month? Has anyone gleened any insights from this sort of study?

-- Ken
Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

commissions+slippage of long duration trades

Post by Ted Annemann »

I've been long Eudollars for almost a year, rolled over the position 4 times and hoping for more. (Bollinger's %B oscillator is at +1.60 so this still looks great!). I've paid 4 commissions & incurred 4 slippages (on "spread trades" used to roll over). How to handle these long duration trades in testing?

A simplistic method that makes some simplistic assumptions:
1. Assume 252 trading days per year
2. Assume 4 rollovers per year (or handle each commodity separately)
3. Assume trade entries occur midway between rollovers
4. Assume it's okay to assess all (commission+slippage) costs on the day of trade exit

Then when a position is exited, you can estimate the number of times it was rolled over based purely upon the NumberOfDaysInTrade ("NODIT"). Rollovers occur on day# 31, 94, 157, 220, 283, etc.

Code: Select all

if(NODIT < 31) then NumRollovers = 0; else(NumRollovers = roundDown((NODIT - 31) / 63);
CommAndSlpg = NumContracts * NumRollovers * CSperContract;
edward kim
Roundtable Knight
Roundtable Knight
Posts: 344
Joined: Sun Apr 20, 2003 2:42 pm
Location: Silicon Valley / San Jose, CA USA
Contact:

Post by edward kim »

This topic was also discussed here:

viewtopic.php?t=112&highlight=roll

Fulks said that there is no perfect answer to the rolling problem. If it's a market where there is usually a constant carry charge going from forward to back months (like gold), then you MIGHT be ok with those techniques (I never thought the S&P would go inverted, but look at what happened last year.)

For markets where each contract month can do the unthinkable, it's best to have a test mechanism where EVERY contract is evaluated at the same time.

Just because you are still bullish on the front month on expiration day, doesn't mean you have to roll to the next month. I remember about 11 or 12 years ago, there was a couple of days where it was freezing cold in the midwest. There were news reports how the hogs refused to move out of their pens to the slaughterhouses because of the cold, and the live hogs farmers couldn't make it in time for contract delivery (whether or not those news stories were exaggerated, it was pretty funny :shock: ) Anyway, the current contract was moving upward towards expiration, while the next month stayed flat.

If you were long the front month and used any of the contract splicing/bridging techniques, that would have put you long into the next month upon rollover. If you evaluated the contracts separately, you would have exited the current month, and stayed out of the next month which was chopping.

Most people test on the front month, and roll over to the next month when liquidity starts rising. When you roll over to a new contract, you have to consider the trade-off between liquidity and how strong a trend is going (for trend-following systems, that is.) If you are long the June contract, and July has a weak choppy trend while November is persistent, I might go for November. Then you have to factor in more slip/skid into your calculations, but it might be the better contract to roll into.

Edward
Chuck B
Roundtable Knight
Roundtable Knight
Posts: 481
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B »

How to handle these long duration trades in testing?
In Trading Recipes and Athena you can have it use the rollover schedule, by market, that you used to create the continuous contracts and they will charge the commission/slip at each roll point in testing. Since roll slippage is much less than trade entry/exit slippage (when using stops), Athena allows you to input slippage charges by symbol for both rollovers and trades separately.
kmulford
Senior Member
Senior Member
Posts: 41
Joined: Mon May 12, 2003 5:59 pm
Location: Setauket, New York

Detrending

Post by kmulford »

Detrending methods can be used to remove an upward (or downward) bias in time series of prices, including continuous, back-adjusted series. This technique may remove any inflationary bias, for example, that might falsely imply that buying is better than selling.

Does anyone on this forum use detrending in their simulations? How useful has it been? Are there any pitfalls to avoid?

Are there any good resources of which you might be aware (papers, etc.) that I might read to understand this better?

Thanks,
Ken
TradingCoach
Roundtable Knight
Roundtable Knight
Posts: 176
Joined: Thu Apr 17, 2003 9:52 am
Location: Sacramento, CA
Contact:

How does CSI data deal with the rollover?

Post by TradingCoach »

Doesn't CSI sell contiguous prices? Why would you bother with any other data than CSI ?
kmulford
Senior Member
Senior Member
Posts: 41
Joined: Mon May 12, 2003 5:59 pm
Location: Setauket, New York

Re: How does CSI data deal with the rollover?

Post by kmulford »

TradingCoach wrote:Doesn't CSI sell contiguous prices? Why would you bother with any other data than CSI ?
I use CSI. My concern is with the problem of biased time series (inflation, for example) that may overstate profitability of the trading sytems I test over past data. CSI linearly adjusts prices via its detrend method. This is not, I understand, the only way to accomplish this. There are more sophisticated means, and I was looking for some guidance to find resources to explore the topic.

I also know that CSI allows one to proportionally adjust continuous, back-adjusted price series. Is this a better method than its detrend method?

So, in short, it's not the data, but the manipulation of it about which I have questions.

Thanks,
Ken
Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi »

Ken,

I've only used it to solve the occasional problem of a price series that goes to zero. It seems to me that the problem to answer is:

- does it matter (=significant equity/dd impact) if you don't use a more sophisticated method?

If I was trying to answer this question I would take a couple of examples of contracts that needed detrending and manually check the impact. If it was small or the impact of more sophisticated detrending was likely to be small I'd relax; if not ...

John
SysTrader
Contributing Member
Contributing Member
Posts: 9
Joined: Mon Nov 24, 2003 4:15 pm

Synthetic Futures Contract

Post by SysTrader »

I have built a synthetic futures contract, which is based on the tick by tick return (yield) of the real individual contracts. On Rollover (known by Date/Time) i use the return between the previous tick and the current tick of the next contract. (so the gap does not effect the synthetic)

Yield = CurrentTick / PreviousTick - 1.0

SynPrice = SynPrice + SynPrice*Yield (SynPrice starts with 1000.0)

The advantage is, that i can use it for calculating Signals for Long Term Systems (Calculation of MovAvg using 200 Days of Data)

So the Signals are calculated on the "Synthetic" futures contract, while the Profit/Loss calculation uses the entry/exit/current prices of the real individual contract. Rollovers are also done as Trades on the real individual contracts (including Slippage and Commision).

What Do you think of this idea? Is it better than using back-adjusted futures data?
TK
Roundtable Knight
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Post by TK »

It seems that you have built a ratio-adjusted contract as proposed by Thomas Stridsman. I think it is the only way to go if you want to get your signals right in systems such as this one:

Code: Select all

if ATR/C<0.35 then buy
from Damian's example.
So the Signals are calculated on the "Synthetic" futures contract, while the Profit/Loss calculation uses the entry/exit/current prices of the real individual contract. Rollovers are also done as Trades on the real individual contracts (including Slippage and Commision).
Yes, ratio-adjusted contracts can be used for signal generation only, and you still need individual contracts or point-based back-adjusted contracts for P/L calculations. So in such systems as above you need two price series: signals are generated on one, and you trade / back-test on the other. One program capable of using primary and secondary prices series is Wealth-Lab.
amasnic
Senior Member
Senior Member
Posts: 38
Joined: Mon Oct 06, 2003 7:19 pm

Back Adjusted Splicing

Post by amasnic »

Could anyone of you confirm if it would be prudent for me to use Back Adjusted splicing option when selecting data series in CSI for testing in VT? Png Files are attached.

Also, is this BP graph for real, assuming I have never seen it trading at 1.32 levels over the last 3 yrs?

Many Tnx
A
Attachments
BP.png
BP.png (54.13 KiB) Viewed 17105 times
BackAdjusted2.png
BackAdjusted2.png (100.75 KiB) Viewed 17106 times
BackAdjusted1.png
BackAdjusted1.png (109.32 KiB) Viewed 17105 times
kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

bp

Post by kianti »

Interesting topic; I attached a monthly continuos chart of British Pound futures (BP) versus British Pound cash (BPY) available on www.futuresource.com
BP never traded down to 1.28 and there's obviously a difference on the back-ajusting method used by futuresource. I know from traders using CSI that their data are reliable, i.e. the rollover are correctly taken into account.
I would like to know if someone is using futuresource as a provider with medium-long term trendfollowing systems or if someone is more familiar with their continuos contracts construction method.
I used futuresource in the past for intra-day trading and although quite expensive they provided EOD data since contract inception.

Also it would be interesting to share some knowledge about contracts such as Crude Oil. When using CSI back-adjusting you get negative prices !?!
Attachments
fscom2.png
fscom2.png (19.77 KiB) Viewed 17086 times
CL.png
CL.png (13.92 KiB) Viewed 17086 times
Post Reply