Back Adjusting Futures Data
This is a huge subject ... the real answer is that you roll them over in the way that reflects how you are going to trade them.
This means that the simple answer is to backadjust them (old contracts matched to current price at rollover date) usually after the open interest of the new contract has exceeded the old contract for two days. Stridsman recommends proportional back adjustment but I disagree with him (because of the way systems perform on data).
If you will trade a contract on a different basis (like ED where you may want to look for the most volatile contract which probably isnt the front month) then you need to roll this way for your test.
A full discussion of methods which increased my understanding considerably is in Bob Fulk's paper at:
http://www.traders2traders.com/linkinde ... tegoryID=4
John
This means that the simple answer is to backadjust them (old contracts matched to current price at rollover date) usually after the open interest of the new contract has exceeded the old contract for two days. Stridsman recommends proportional back adjustment but I disagree with him (because of the way systems perform on data).
If you will trade a contract on a different basis (like ED where you may want to look for the most volatile contract which probably isnt the front month) then you need to roll this way for your test.
A full discussion of methods which increased my understanding considerably is in Bob Fulk's paper at:
http://www.traders2traders.com/linkinde ... tegoryID=4
John
I have been looking at differences in performance between two files of teh same commodity where each file uses a different set of expiries.
For example,
File 1: March June Sep Dec
File 2: June Dec.
It amazes me how much difference it makes and neither of them is 'right'.
Is it fooling ones self to use the file that gives the better return?
For example,
File 1: March June Sep Dec
File 2: June Dec.
It amazes me how much difference it makes and neither of them is 'right'.
Is it fooling ones self to use the file that gives the better return?
I'd suggest that it is fooling oneself if one would not have worked out ahead of time that this was the correct way to roll a contract.
If you now work out that ED is likely to have more volatility in December than in the intervening months then a rule where you roll when the OI on december reaches X would be valid. You could retrospectively test the difference to find out what would have happened if you traded that way in the past to decide if it would be more profitable if it continued to be valid in future.
John
If you now work out that ED is likely to have more volatility in December than in the intervening months then a rule where you roll when the OI on december reaches X would be valid. You could retrospectively test the difference to find out what would have happened if you traded that way in the past to decide if it would be more profitable if it continued to be valid in future.
John
John: Thanks for your thoughts on my question. I also agree that this would be fooling ones self.
For what it is worth: In real trading I try to roll an open trade into the most distant contract that has a reasonable level of V and OI. This changes depending on market and also current events. It is not uncommon for me to be long XYZ June, then roll into XYZ sep after which the position is closed at my trailing stop. As Dec comes to expiry I might decide that Sep next year is the contract that I want to follow, so I will skip all the other months and back adjust dec next year onto teh time series. I now have quite a few market files (on which I run my trading system in real time) which are not composed of a regular sequence of expiries. So it is difficult to replicate this when generating a back adjusted historical file for testing purposes.
cheers
damian
For what it is worth: In real trading I try to roll an open trade into the most distant contract that has a reasonable level of V and OI. This changes depending on market and also current events. It is not uncommon for me to be long XYZ June, then roll into XYZ sep after which the position is closed at my trailing stop. As Dec comes to expiry I might decide that Sep next year is the contract that I want to follow, so I will skip all the other months and back adjust dec next year onto teh time series. I now have quite a few market files (on which I run my trading system in real time) which are not composed of a regular sequence of expiries. So it is difficult to replicate this when generating a back adjusted historical file for testing purposes.
cheers
damian
Backtesting with backadjusted time series
How does one handle commission costs in historical testing across backadjusted time series? Presumably, for long-term trend following systems, a long (short) position may carry over a number of different contract months/years. Commissions are charged on the rollover. Should one simply goose-up the commission/slippage charge against the sytem? Or, should one review each trade's duration and estimate the added commission expense?
Additionally, has anyone performed any studies regarding relative liquidity for various commodity and financial futures contracts across each calendar month? Has anyone gleened any insights from this sort of study?
-- Ken
Additionally, has anyone performed any studies regarding relative liquidity for various commodity and financial futures contracts across each calendar month? Has anyone gleened any insights from this sort of study?
-- Ken
-
- Roundtable Knight
- Posts: 118
- Joined: Tue Apr 15, 2003 7:44 pm
- Location: Arizona
commissions+slippage of long duration trades
I've been long Eudollars for almost a year, rolled over the position 4 times and hoping for more. (Bollinger's %B oscillator is at +1.60 so this still looks great!). I've paid 4 commissions & incurred 4 slippages (on "spread trades" used to roll over). How to handle these long duration trades in testing?
A simplistic method that makes some simplistic assumptions:
1. Assume 252 trading days per year
2. Assume 4 rollovers per year (or handle each commodity separately)
3. Assume trade entries occur midway between rollovers
4. Assume it's okay to assess all (commission+slippage) costs on the day of trade exit
Then when a position is exited, you can estimate the number of times it was rolled over based purely upon the NumberOfDaysInTrade ("NODIT"). Rollovers occur on day# 31, 94, 157, 220, 283, etc.
A simplistic method that makes some simplistic assumptions:
1. Assume 252 trading days per year
2. Assume 4 rollovers per year (or handle each commodity separately)
3. Assume trade entries occur midway between rollovers
4. Assume it's okay to assess all (commission+slippage) costs on the day of trade exit
Then when a position is exited, you can estimate the number of times it was rolled over based purely upon the NumberOfDaysInTrade ("NODIT"). Rollovers occur on day# 31, 94, 157, 220, 283, etc.
Code: Select all
if(NODIT < 31) then NumRollovers = 0; else(NumRollovers = roundDown((NODIT - 31) / 63);
CommAndSlpg = NumContracts * NumRollovers * CSperContract;
-
- Roundtable Knight
- Posts: 344
- Joined: Sun Apr 20, 2003 2:42 pm
- Location: Silicon Valley / San Jose, CA USA
- Contact:
This topic was also discussed here:
viewtopic.php?t=112&highlight=roll
Fulks said that there is no perfect answer to the rolling problem. If it's a market where there is usually a constant carry charge going from forward to back months (like gold), then you MIGHT be ok with those techniques (I never thought the S&P would go inverted, but look at what happened last year.)
For markets where each contract month can do the unthinkable, it's best to have a test mechanism where EVERY contract is evaluated at the same time.
Just because you are still bullish on the front month on expiration day, doesn't mean you have to roll to the next month. I remember about 11 or 12 years ago, there was a couple of days where it was freezing cold in the midwest. There were news reports how the hogs refused to move out of their pens to the slaughterhouses because of the cold, and the live hogs farmers couldn't make it in time for contract delivery (whether or not those news stories were exaggerated, it was pretty funny ) Anyway, the current contract was moving upward towards expiration, while the next month stayed flat.
If you were long the front month and used any of the contract splicing/bridging techniques, that would have put you long into the next month upon rollover. If you evaluated the contracts separately, you would have exited the current month, and stayed out of the next month which was chopping.
Most people test on the front month, and roll over to the next month when liquidity starts rising. When you roll over to a new contract, you have to consider the trade-off between liquidity and how strong a trend is going (for trend-following systems, that is.) If you are long the June contract, and July has a weak choppy trend while November is persistent, I might go for November. Then you have to factor in more slip/skid into your calculations, but it might be the better contract to roll into.
Edward
viewtopic.php?t=112&highlight=roll
Fulks said that there is no perfect answer to the rolling problem. If it's a market where there is usually a constant carry charge going from forward to back months (like gold), then you MIGHT be ok with those techniques (I never thought the S&P would go inverted, but look at what happened last year.)
For markets where each contract month can do the unthinkable, it's best to have a test mechanism where EVERY contract is evaluated at the same time.
Just because you are still bullish on the front month on expiration day, doesn't mean you have to roll to the next month. I remember about 11 or 12 years ago, there was a couple of days where it was freezing cold in the midwest. There were news reports how the hogs refused to move out of their pens to the slaughterhouses because of the cold, and the live hogs farmers couldn't make it in time for contract delivery (whether or not those news stories were exaggerated, it was pretty funny ) Anyway, the current contract was moving upward towards expiration, while the next month stayed flat.
If you were long the front month and used any of the contract splicing/bridging techniques, that would have put you long into the next month upon rollover. If you evaluated the contracts separately, you would have exited the current month, and stayed out of the next month which was chopping.
Most people test on the front month, and roll over to the next month when liquidity starts rising. When you roll over to a new contract, you have to consider the trade-off between liquidity and how strong a trend is going (for trend-following systems, that is.) If you are long the June contract, and July has a weak choppy trend while November is persistent, I might go for November. Then you have to factor in more slip/skid into your calculations, but it might be the better contract to roll into.
Edward
In Trading Recipes and Athena you can have it use the rollover schedule, by market, that you used to create the continuous contracts and they will charge the commission/slip at each roll point in testing. Since roll slippage is much less than trade entry/exit slippage (when using stops), Athena allows you to input slippage charges by symbol for both rollovers and trades separately.How to handle these long duration trades in testing?
Detrending
Detrending methods can be used to remove an upward (or downward) bias in time series of prices, including continuous, back-adjusted series. This technique may remove any inflationary bias, for example, that might falsely imply that buying is better than selling.
Does anyone on this forum use detrending in their simulations? How useful has it been? Are there any pitfalls to avoid?
Are there any good resources of which you might be aware (papers, etc.) that I might read to understand this better?
Thanks,
Ken
Does anyone on this forum use detrending in their simulations? How useful has it been? Are there any pitfalls to avoid?
Are there any good resources of which you might be aware (papers, etc.) that I might read to understand this better?
Thanks,
Ken
-
- Roundtable Knight
- Posts: 176
- Joined: Thu Apr 17, 2003 9:52 am
- Location: Sacramento, CA
- Contact:
How does CSI data deal with the rollover?
Doesn't CSI sell contiguous prices? Why would you bother with any other data than CSI ?
Re: How does CSI data deal with the rollover?
I use CSI. My concern is with the problem of biased time series (inflation, for example) that may overstate profitability of the trading sytems I test over past data. CSI linearly adjusts prices via its detrend method. This is not, I understand, the only way to accomplish this. There are more sophisticated means, and I was looking for some guidance to find resources to explore the topic.TradingCoach wrote:Doesn't CSI sell contiguous prices? Why would you bother with any other data than CSI ?
I also know that CSI allows one to proportionally adjust continuous, back-adjusted price series. Is this a better method than its detrend method?
So, in short, it's not the data, but the manipulation of it about which I have questions.
Thanks,
Ken
Ken,
I've only used it to solve the occasional problem of a price series that goes to zero. It seems to me that the problem to answer is:
- does it matter (=significant equity/dd impact) if you don't use a more sophisticated method?
If I was trying to answer this question I would take a couple of examples of contracts that needed detrending and manually check the impact. If it was small or the impact of more sophisticated detrending was likely to be small I'd relax; if not ...
John
I've only used it to solve the occasional problem of a price series that goes to zero. It seems to me that the problem to answer is:
- does it matter (=significant equity/dd impact) if you don't use a more sophisticated method?
If I was trying to answer this question I would take a couple of examples of contracts that needed detrending and manually check the impact. If it was small or the impact of more sophisticated detrending was likely to be small I'd relax; if not ...
John
Synthetic Futures Contract
I have built a synthetic futures contract, which is based on the tick by tick return (yield) of the real individual contracts. On Rollover (known by Date/Time) i use the return between the previous tick and the current tick of the next contract. (so the gap does not effect the synthetic)
Yield = CurrentTick / PreviousTick - 1.0
SynPrice = SynPrice + SynPrice*Yield (SynPrice starts with 1000.0)
The advantage is, that i can use it for calculating Signals for Long Term Systems (Calculation of MovAvg using 200 Days of Data)
So the Signals are calculated on the "Synthetic" futures contract, while the Profit/Loss calculation uses the entry/exit/current prices of the real individual contract. Rollovers are also done as Trades on the real individual contracts (including Slippage and Commision).
What Do you think of this idea? Is it better than using back-adjusted futures data?
Yield = CurrentTick / PreviousTick - 1.0
SynPrice = SynPrice + SynPrice*Yield (SynPrice starts with 1000.0)
The advantage is, that i can use it for calculating Signals for Long Term Systems (Calculation of MovAvg using 200 Days of Data)
So the Signals are calculated on the "Synthetic" futures contract, while the Profit/Loss calculation uses the entry/exit/current prices of the real individual contract. Rollovers are also done as Trades on the real individual contracts (including Slippage and Commision).
What Do you think of this idea? Is it better than using back-adjusted futures data?
It seems that you have built a ratio-adjusted contract as proposed by Thomas Stridsman. I think it is the only way to go if you want to get your signals right in systems such as this one:
from Damian's example.
Code: Select all
if ATR/C<0.35 then buy
Yes, ratio-adjusted contracts can be used for signal generation only, and you still need individual contracts or point-based back-adjusted contracts for P/L calculations. So in such systems as above you need two price series: signals are generated on one, and you trade / back-test on the other. One program capable of using primary and secondary prices series is Wealth-Lab.So the Signals are calculated on the "Synthetic" futures contract, while the Profit/Loss calculation uses the entry/exit/current prices of the real individual contract. Rollovers are also done as Trades on the real individual contracts (including Slippage and Commision).
Back Adjusted Splicing
Could anyone of you confirm if it would be prudent for me to use Back Adjusted splicing option when selecting data series in CSI for testing in VT? Png Files are attached.
Also, is this BP graph for real, assuming I have never seen it trading at 1.32 levels over the last 3 yrs?
Many Tnx
A
Also, is this BP graph for real, assuming I have never seen it trading at 1.32 levels over the last 3 yrs?
Many Tnx
A
- Attachments
-
- BP.png (54.13 KiB) Viewed 17180 times
-
- BackAdjusted2.png (100.75 KiB) Viewed 17181 times
-
- BackAdjusted1.png (109.32 KiB) Viewed 17180 times
bp
Interesting topic; I attached a monthly continuos chart of British Pound futures (BP) versus British Pound cash (BPY) available on www.futuresource.com
BP never traded down to 1.28 and there's obviously a difference on the back-ajusting method used by futuresource. I know from traders using CSI that their data are reliable, i.e. the rollover are correctly taken into account.
I would like to know if someone is using futuresource as a provider with medium-long term trendfollowing systems or if someone is more familiar with their continuos contracts construction method.
I used futuresource in the past for intra-day trading and although quite expensive they provided EOD data since contract inception.
Also it would be interesting to share some knowledge about contracts such as Crude Oil. When using CSI back-adjusting you get negative prices !?!
BP never traded down to 1.28 and there's obviously a difference on the back-ajusting method used by futuresource. I know from traders using CSI that their data are reliable, i.e. the rollover are correctly taken into account.
I would like to know if someone is using futuresource as a provider with medium-long term trendfollowing systems or if someone is more familiar with their continuos contracts construction method.
I used futuresource in the past for intra-day trading and although quite expensive they provided EOD data since contract inception.
Also it would be interesting to share some knowledge about contracts such as Crude Oil. When using CSI back-adjusting you get negative prices !?!
- Attachments
-
- fscom2.png (19.77 KiB) Viewed 17161 times
-
- CL.png (13.92 KiB) Viewed 17161 times