Stitching data together manually

General discussions about futures.
Post Reply
corvus
Senior Member
Senior Member
Posts: 31
Joined: Sat Sep 01, 2007 11:41 pm

Stitching data together manually

Post by corvus »

My sense is from reading STIRS Roll in CSI is that I may be biting off more than I can chew but I also think what I’ll learn will be worth the work. That said:

I’ve downloaded a large set of historical data so I run a few simple tests such as correlation, R-square and the like in Excel. My rationale for this comes in large part from Sluggo’s trade everything, plus the kitchen sink philosophy and partly to understand what a realistic baseline would result in.

My understanding is I can easily do this in TB but I‘ve sent that machine home. Stitching the data together manually seems to be the next best idea, especially since upon looking at the data, I see that the prices dates doesn’t match which illustrates the fact that I don’t really know how the data gets stitched together in the first place.

My questions are twofold. How is data stiched together and how can this be done manually?

If the topic has already been addressed (other than in the above thread), please accept my apologies and point me to the right thread.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

Attachments
Fulks_Back_Adjusting.pdf
Fulks paper
(38.12 KiB) Downloaded 783 times
corvus
Senior Member
Senior Member
Posts: 31
Joined: Sat Sep 01, 2007 11:41 pm

Post by corvus »

Thanks, Sluggo. I really appreciate it.
Jez Liberty
Roundtable Knight
Roundtable Knight
Posts: 123
Joined: Tue Nov 03, 2009 8:49 am
Location: London
Contact:

Post by Jez Liberty »

When you start thinking about Continuous Futures contracts (and read CSI docs or similar articles that Sluggo just posted) you realise that there are quite a few options all with different drawbacks, etc. However I believe you ned to choose one method that is the best compromise (even if some argue that only raw data is valid - a first pass using continuous futures contracts should still offer practicality and usefulness)

I have some resources on the different methods (back-adjusted, proportional, perpetual contracts, Gann Series, etc.) that you might find useful:
An overview of the different aggregation options
A closer look to the least worst method (in my opinion).

And some code using the CSI API to extract the data and build a continuous contract "on the fly" - together with the documented CSI API method used for this to be able to control all the options.
fc
Full Member
Full Member
Posts: 13
Joined: Mon Dec 12, 2011 8:49 am

Post by fc »

Can someone explain to me how futures traders actually roll between contracts when they trade in reality?

Because I don't know this, I'm having a hard time understanding the purpose of all these different methods of stitching the data together. Shouldn't the stitching simply be done in the exact same way as the traders do when they actually trade? I don't get it.
rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Post by rhc »

This link might help
viewtopic.php?t=7978
fc
Full Member
Full Member
Posts: 13
Joined: Mon Dec 12, 2011 8:49 am

Post by fc »

rhc wrote:This link might help
viewtopic.php?t=7978
Thanks for the link. According to that link, there are two alternatives that people can take to roll over in reality. One is simply selling the contract nearing expiry and buying the next contract. The other one is using some "calendar spread" technique, which can sometimes result in a bit of a discount.

But here is the thing in any case. In real treading, people see sudden jumps in their prices. So why wouldn't they want to use historical data with jumps?

Why would anyone want to create some artificial back-adjusted data with jumps removed, if this is not how the actual trades are placed?

I don't see this properly explained anywhere...
rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Post by rhc »

fc wrote:Why would anyone want to create some artificial back-adjusted data with jumps removed, if this is not how the actual trades are placed?

I don't see this properly explained anywhere...
This link might help;
http://www.premiumdata.net/support/futu ... inuous.php

For additional information you could also refer to the pdf file attached in post no.2 of this thread above (i.e. Fulks_Back_Adjusting.pdf)

For additional-extra information you could also refer to the two links in post no.4 of this thread above.
http://www.automated-trading-system.com ... contracts/
and
http://www.automated-trading-system.com ... t-options/
altomas

Post by altomas »

In real treading, people see sudden jumps in their prices
fc,

The essence of the problem is that in real trading when one contract expires and you have to roll to another, you do not participate in the price discrepancy between the two contracts.

i.e. contract A is trading at 50 and contract B is trading at 60. You are long contract A and want to roll your position to B. You sell out A at 50, buy B at 60. The difference is 10, but you don't earn or lose this 10, for you it is essentially 0 (assuming no slippage and commissions).

If you just stitch all the contracts together for a historical simulation then you would be simulating a world where you participate in the difference between A and B.

Exactly how you adjust for this fact, which contracts you want to participate in, what triggers you to roll and whether you want to treat A and B as separate commodities etc... are just complications around this basic problem.
fc
Full Member
Full Member
Posts: 13
Joined: Mon Dec 12, 2011 8:49 am

Post by fc »

altomas wrote:The essence of the problem is that in real trading when one contract expires and you have to roll to another, you do not participate in the price discrepancy between the two contracts.

i.e. contract A is trading at 50 and contract B is trading at 60. You are long contract A and want to roll your position to B. You sell out A at 50, buy B at 60. The difference is 10, but you don't earn or lose this 10, for you it is essentially 0 (assuming no slippage and commissions).
Thank you for this great insight. I wasn't aware that real trading worked like this. Most if not all of the material I read (including the links above) didn't mention this. I guess they just took it as obvious, and rightfully so since they are targeting people who have actually traded. Since I haven't, I wasn't aware.

However, I have a hard time understanding how this can possibly work. So you have your contract A at 50, and contract B at 60. These are two different contracts on the marketplace. B is more expensive than A. You want to sell A and buy B. How can you possibly avoid having to pay the difference of 10? Does the broker cover that? That sounds strange. It sounds to me like if you can avoid paying the difference of 10, you have somehow managed to obtain "free money," in some sense. Can you show me the error in my thinking?
LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog »

fc, one way you could possibly think about this which may make it more palatable, is to remember that buying or selling a futures contract is not like buying an asset. You are entering into a contract that gives you a right to buy or sell a fixed item or derivative sometime in the future.

The quantity or derivative does not change from one contract month to another.

So if you are rolling from Dec to Mar, and you are long Dec, you will sell Dec and buy Mar to do your roll. To the extent there is a price difference, you have simply moved the peg by which differentials from that peg are determined when it comes time to take or give delivery (or close out the position).

Does this help?
rhc
Roundtable Knight
Roundtable Knight
Posts: 464
Joined: Tue Nov 18, 2008 8:46 pm
Location: Oz

Post by rhc »

fc wrote:However, I have a hard time understanding how this can possibly work. So you have your contract A at 50, and contract B at 60. These are two different contracts on the marketplace. B is more expensive than A. You want to sell A and buy B. How can you possibly avoid having to pay the difference of 10? Does the broker cover that? That sounds strange. It sounds to me like if you can avoid paying the difference of 10, you have somehow managed to obtain "free money," in some sense. Can you show me the error in my thinking?
From the reference link mentioned above (listed here again for your convenience);
http://www.premiumdata.net/support/futu ... inuous.php

We have the following,
In Part 1 of this article, it was argued that the different contracts in a futures market are not homogenous. Each delivery month points to where cash prices might stand at a particular date in the future. Therefore each contract will tend to trade at its own level. This leads to a problem when individual contracts are spliced together. When one contract is replaced in the series by another, the difference in price between the two will manifest itself as a gap. This explains the gaps evident in the Lean Hogs chart above. For instance, the first big gap in the chart, when prices appear to "jump" from around 62.5 to 70 early in 2000, was caused by the following sequence of events:

23/3/2000 - the April 2000 contract closes at 62.525. It is now abandoned.
On the same day, the June 2000 contract closes at 69.950
24/3/2000 - the June 2000 contract, which is now the current contract in the series, closes at 70.525.

The spliced chart suggests that the Lean Hogs market jumped about 7 basis points from the 23rd to the 24th. But the June contract itself jumped only 1.25 points. It was already trading well above the level of April on the day that April was abandoned. So when did the market jump those 7 points? The answer is that it never did. The "market" is actually comprised of individual contracts that run their own course.
I hope that helps
fc
Full Member
Full Member
Posts: 13
Joined: Mon Dec 12, 2011 8:49 am

Post by fc »

Thanks for your replies, but I'm still just as confused. I already know that the "market" is comprised of different contracts, as rightly pointed out by rhc and all the other articles I read. I also am aware of what LeapFrog mentioned, namely that you just own the right to buy or sell something in the future (which is what a futures contract is). The confusion, however, still remains.

What altomas said was that one can somehow avoid paying the difference of 10 (in his example) in real trading. That is the source of my confusion. If this is the case, why wouldn't someone simply buy contract A for 50, somehow "magically" roll over to contract B for free (which seemingly is claimed by altomas to be possible), and sell it immediately for 60, thus make a free profit of 10?

If possible, such operations would of course quickly converge the prices of the two contracts (remove the arbitrage). This however, doesn't happen, which seems to imply that somehow "magically" rolling between contracts for free in actual trading is not possible in the first place.

The above "magic" is the source of my confusion. If someone can point out the errors of my ways (I know I'm the one making all the mistakes here), I'd be much grateful.
altomas

Post by altomas »

fc,

I think what LeapFrog is getting at is that with a futures contract (aside from margin) you do not pay for anything upfront.

This means that in the prior example you are not receiving 50 and then paying 60, you are merely resetting the price at which the mark-to-market process will begin.

Had you held contract A, you make 1 point when it rises to 51 and lose 1 when it falls to 49.

After rolling to B, you make 1 point when it rises to 61 and lose 1 when it falls to 59.

These two situations are equivalent and you have outlaid 0 cash upfront to be in either one.

The key to what I was saying in the earlier post is that you do not participate in the difference between A and B.
fc
Full Member
Full Member
Posts: 13
Joined: Mon Dec 12, 2011 8:49 am

Post by fc »

altomas wrote:The key to what I was saying in the earlier post is that you do not participate in the difference between A and B.
altomas: Thank you very much! Now I get this, finally. I completely forgot the way futures markets actually work. You provided a wonderful explanation of it. Thanks!
Post Reply