Continuous Data Building Software

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Continuous Data Building Software

Post by Roscoe » Thu Oct 14, 2010 5:47 pm

Every once in a while I come back to this topic - I get sidetracked onto something else and lose myself in that and then I come up for air and look at my TODO list and think "Damn, forgot about that". So here I am again - I want to acquire or build software that would allow me to construct my own continuous back-adjusted contracts from individual contracts.

Klatt_attack was doing something in Matlab here but that seemed to kind of fade from view many months ago. Klatt, did you ever finish it? If so please PM me.

So my questions are:
  1. Does anybody here know of any suitable standalone software that can be purchased that will allow me to construct my own continuous contracts, rolled in any way that I choose?
  2. Or, failing that, are there any C++ programmers here with a similar desire who would be interested in working with me to build such software? If so please PM me.
Thanks in advance. I have been Googling about the intertubes with very little (read "no") success so far, and as this forum has by far the highest proportion of civilized members of any trading forum I am hopeful of being able to achieve a useful result.

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Post by kianti » Fri Oct 15, 2010 1:37 am


Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Fri Oct 15, 2010 2:04 am

Thanks kianti, but I was hoping for something that doesn't require face paint and a trance state to operate (sorry Ed).

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Post by kianti » Fri Oct 15, 2010 2:20 am

Roscoe wrote:.....I was hoping for something that doesn't require face paint and a trance state to operate.....
No need to stay 70 years without eating :D :lol: :D

But perhaps we can start a Roll-over Tribe on this forum; we have illuminated :idea: :idea: :idea: members and we can get some programming skills on Guru Employer (Find freelancers at the World's Largest Online Marketplace)

Best regards, as ever
Attachments
trance.png
trance.png (85.84 KiB) Viewed 7931 times

Chris67
Roundtable Knight
Roundtable Knight
Posts: 1046
Joined: Tue Dec 16, 2003 2:12 pm
Location: London

Post by Chris67 » Fri Oct 15, 2010 3:22 am

I have to say that the TT website is full of nutters ? Is it like a 12 step program of making money - sure looks like it ??
Has Ed Sekota got a published track record ?
C

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Fri Oct 15, 2010 3:33 am

kianti wrote:But perhaps we can start a Roll-over Tribe on this forum; we have illuminated :idea: :idea: :idea: members
I'm in. But who shall lead us? Brother AFJ is a sage-like figure, while Brother Sluggo speaks oft times in parables. Perhaps you Brother Kianti? Or the venerable Brother RedRock? We will need a secret greeting, perhaps where we each lie down and roll over (rolling ever to the right of course)? No-one would ever work that one out...

kianti
Roundtable Knight
Roundtable Knight
Posts: 335
Joined: Fri May 02, 2003 6:10 am
Location: Florence - Italy

Post by kianti » Fri Oct 15, 2010 4:13 am

Roscoe wrote:But who shall lead us? Brother AFJ is a sage-like figure, while Brother Sluggo speaks oft times in parables. Perhaps you Brother Kianti? Or the venerable Brother RedRock? We will need a secret greeting, perhaps where we each lie down and roll over (rolling ever to the right of course)? No-one would ever work that one out...
Great idea :shock:
Abbot AFJ, Father Sluggo as preacher, Brother RedRock running the Chest..I could take care of the cellar :D :lol: :D
Amen
Attachments
The Date Of The Roll.png
The Date Of The Roll.png (257.32 KiB) Viewed 7892 times

Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe » Sun Oct 17, 2010 11:48 pm

Bump.

C'mon guys, no serious answers at all? Nobody seen anything that might be worth a look at? No C++ coders looking far a small job? Maybe this will have to go onto the back burner again.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Mon Oct 18, 2010 12:07 am

Keep repeating your request.

Jez Liberty
Roundtable Knight
Roundtable Knight
Posts: 123
Joined: Tue Nov 03, 2009 8:49 am
Location: London
Contact:

Post by Jez Liberty » Mon Oct 18, 2010 6:51 am

just pm'ed you - was meant to do so earlier but had a busy weekend...

davidib
Contributing Member
Contributing Member
Posts: 9
Joined: Mon Jul 18, 2005 6:09 pm
Location: Chicago

Post by davidib » Tue Oct 19, 2010 5:16 pm

Hi Roscoe,
THE GOOD:
I do have code to build continuous contracts. The basic design is that you provide a set of roll rules (encoded as text like 'roll_on_last_trading_day=-5') and a set of contracts into the algorithm, and it spits out a continuous contract as well as the list of rules that actually caused a specific roll. If you have multiple rules to use when building a specific contract, you can define the precedence of them (i.e. a 'roll_on_volume=.95%' rule can take priority over a 'roll_on_last_trading_day=-5' rule)

You can provide as many rules as you like. In addition, you can explicitly filter out contracts via a simple rule (i.e. 'filter_out_contract=20100700').

The rules are specific to a market, i.e. the roll rules for spus are different than the roll rules for corn.

You can also build different types of continuous contracts, i.e. backward absolute adjusted and forward ratio adjusted by passing a parameter to the code (i.e. 'BAA' for backward absolute adjusted)

The price data input is organized by contract, it can be loaded from either a flat file or a database. Also, there are data adapters to normalize the data from different vendors, i.e. you can build a single continuous contract for bonds using data say 1980-2000 from Tradestation, and 2001-2010 from CSI

THE NOT SO GOODS

This code is implemented in c#, not C++.

The roll rules are implemented as a custom designed, pretty easy to understand RollRule class. So to add additional roll rules, you have to know a bit of C#/object orientated programming. Of course the roll rules as described above (i.e. the 'roll_on_last_trading_day=-5') allow one to put their own parameters in, so instead of rolling on 5 days before the last trading day, you could encode the rule like 'roll_on_last_trading_day=-10'.

I have about 4 roll rules developed so far (i.e. wrote code to parse/enforce 4 different rules). But if you have the skill, you can write 100s of different types of rules.

The code is part of a larger software system I'm working on. So I wouldn't want to distribute the entire system, just the continuous contract subsystem if you will. Not too much a problem to distribute just that however.


If you have any questions, comments etc, please let me know. FWIW I've been pretty much a lurker on this board for a while, never contributed much :cry: . Been working in the managed futures industry for sometime, worked for a bunch of turtles and currently work for a small (70mill AUM) short term CPO/CTA


David

lperepol
Senior Member
Senior Member
Posts: 30
Joined: Thu Feb 26, 2004 12:50 pm
Location: Castlegar, BC, Canada
Contact:

Generating Continous Contracts

Post by lperepol » Sun Nov 14, 2010 11:55 pm

Hi,

I am just starting out in generating continuous contracts.
I am writing in C++ and QT. So far, I have
reversed engineered a few of CSI's back adjusting methods.
I am using CRB and CSI data. I can replicate one of ED's
methods. Ed's continuous gold contract is a bit strange in the
way it generates continuous contract. I have not replicated his
method for the gold splicing method yet.

The application's purpose is to assist me in examining and
verifying different methods for continuous contract generation.

If anyone is interested let me know. I am not interested in just
generating continuous contracts but am interested in determining
the suitability of a method for a particular instrument.

I did notice that the parameters that CSI uses to generating a continuous contract do not match the parameters that I use; however, the end results do match -- go figure?

There are also a few discrepancies in price observations between CSI and CRB data. Eye Witness accounts of an event seldom match 100%.

Please let me know if this issue is still active and if there is any interest.

I would also welcome hearing from those who have completed ED's TSP projects. I would like to compare some results I am seeing on the equities side of things.

michaelt
Senior Member
Senior Member
Posts: 37
Joined: Wed Feb 03, 2010 10:53 am
Location: princeton,nj

Post by michaelt » Tue Nov 16, 2010 8:39 am

UPDATE TO THE BELOW: I heard from CSI and they say it's a database refresh issue. If I hadn't visually inspected the file I wouldn't have found the error (although my trading account would be impacted nonetheless). If the refresh doesn't repair this issue I'll update this post. Update: That fixed the data.


Hi,

I just posted a problem I found with the individual contracts for Copper (HG20) from CSI.

Naturally if the underlying data has gaps, the continuous contract will too.

What source are others using for individual contracts?

viewtopic.php?t=8054&highlight=
Post subject: Copper missing data; CSI individual & continuous contrac


-Michael

lperepol
Senior Member
Senior Member
Posts: 30
Joined: Thu Feb 26, 2004 12:50 pm
Location: Castlegar, BC, Canada
Contact:

What source are others using for individual contracts?

Post by lperepol » Tue Nov 16, 2010 1:11 pm

I have both CRB and CSI feeds.
I did not like Unfair Advantage when I first started using it, but
have been able to tolerate it. It works OK, has some extra functions.
CRB is more to my liking but does not have all the bells and whistles as CSI.
Using CSI's Unfair Advantage I manage to shoot myself in the foot once in a while.

I have not seen any issues with the CRB data so far.
In comparing individual contracts I did notice a few holes in CSI data.
Nothing too alarming.
I believe some of the discrepancies are due to when data is repaired -- exchange data may not be correct. CRB and CSI have different policies in fixing data. CRB keeps notes when data is suspect from the exchanges.

I had some instruments with holes from CSI. I called them and they asked me to reload the database. This resolved the blatant ones. However, discrepancies still exist between the two providers.

lperepol
Senior Member
Senior Member
Posts: 30
Joined: Thu Feb 26, 2004 12:50 pm
Location: Castlegar, BC, Canada
Contact:

Contango and Back Adjusting

Post by lperepol » Thu Nov 18, 2010 11:43 am

I ran into this post on http://www.bigmiketrading.com. This is the link:
http://www.bigmiketrading.com/commoditi ... #post77686

This is the content of the post:

This is an article showing the real problem of WTI rollover. The choice of the delivery loaction in Cushing Oklahoma leads to a significant distortion in the prices of the front month contracts.

http://ftalphaville.ft.com/blog/2008/12 ... t-so-easy/
FT Alphaville Rollover, rollover WTI
FT Alphaville It’s all about Cushing

(1) Usually Cushing is full and cannot take delivery. That means that the longs are pressed to get rid of their contracts. This nearly always creates a significant contango situation between the front month and the following month. If you compare the roll returns between WTI and Brent, the difference was -22% in 2009.

(2) Occasionally there is a shortage in Cushing - mostly due to technical reasons such as pipeline maintenance - and then there is an awful short squeeze.

To summarize: CL is a stupid and useless contract that should have been replaced since long.


Backadjusting CL

If you backadjust CL, you will have this distortion built into your contract. Because of that permanent contango situation for the front month, the offsets are oversized, and if you go back long enough, you may even find negative prices in your contract. So a longer term trader would better not backadjust the contracts, but use perpetuals.

The situation also affects long only investments funds. As they have to bear the negative return on rollovers, the return of the funds will be ways below the increase in commodities prices. Some of the funds already use back months instead of the front month to minimize the impact of the Cushing Contango.

For a short term trader, this is no problem. If you only need the current and the last contract's data, you can easily use the backadjusted contract with the standard offset that compensates for the contango.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Thu Nov 18, 2010 12:06 pm

I suppose that if someone was REALLY interested in this situation, they could perform some experiments with CSI UA + Blox, to test out some trading tactics. Maybe they might want to study ideas like
  • Use CSI UA to build five different backadjusted continuous contracts, which roll over 2, 9, 16, 23, and 30 days before expiration. Then run some trading systems on each of these contracts in Blox. As your rollover day moves farther and farther away from expiration, do the system test results differ from each other? Which is "best" and which is "worst"? Is there a simple explanation of why this might happen?
  • Use CSI UA to build six different backadjusted continuous contracts, which roll over 12, 6, 4, 3, 2, and 1 times per year. Then run some trading systems on each of these contracts in Blox. As you roll over less and less frequently, do the system test results differ from each other? Which is "best" and which is "worst"? Is there a simple explanation of why this might happen?
  • Use CSI UA to build three different backadjusted continuous contracts, which roll over on the 2nd calendar day of the month, 0, 1, and 2 months before expiration month. Then run some trading systems on each of these contracts in Blox. As you roll over larger and larger numbers of months before expiration, do the system test results differ from each other? Which is "best" and which is "worst"? Is there a simple explanation of why this might happen?
I'm sure an inquisitive and clever person could think up a dozen more cute strategems to test (and then to trade).

lperepol
Senior Member
Senior Member
Posts: 30
Joined: Thu Feb 26, 2004 12:50 pm
Location: Castlegar, BC, Canada
Contact:

Post by lperepol » Thu Nov 18, 2010 7:33 pm

Excellent ideas Sluggo,

In addition, I would also like a metric that measures the sensitivity of an instrument that is subjected to an adjustable roll trigger/strategy. Another project to put on the books.

michaelt
Senior Member
Senior Member
Posts: 37
Joined: Wed Feb 03, 2010 10:53 am
Location: princeton,nj

Post by michaelt » Fri Nov 19, 2010 9:55 am

re: building continuous contracts:

What is the problem with switching to the next contract that has both a higher volume and open interest, and on the day that both occur?

lperepol
Senior Member
Senior Member
Posts: 30
Joined: Thu Feb 26, 2004 12:50 pm
Location: Castlegar, BC, Canada
Contact:

Post by lperepol » Fri Nov 19, 2010 11:17 am

michaelt,

I do not know if there is an issue or not. If there is an issue it would be with price. There is chatter on the web about rolling and contango. ETF's tracking commodities have to roll at some point. The CL contract has also been sighted as being sensitive to contango.

Creating continuous contracts masks some of the issues that one would experience in real trading. Running a simulator on a continuous contract may reveal results and trading parameters that are not achievable in the real world. The worst case would be that the simulator creates an illusion of profitability.

>> What is the problem with switching to the next contract that has both a higher volume and open interest, and on the day that both occur?

I do not know.

Similarly, what is the problem with closing out the current contract and then waiting a few days before switching to the next contract that has both a higher volume and open interest?
Something like a discontinuous contract and how would one model it?

These are just thoughts rolling around in my brain right now.

michaelt
Senior Member
Senior Member
Posts: 37
Joined: Wed Feb 03, 2010 10:53 am
Location: princeton,nj

Post by michaelt » Sun Dec 12, 2010 2:18 am

Hi,

You wrote:

"Similarly, what is the problem with closing out the current contract and then waiting a few days before switching to the next contract that has both a higher volume and open interest? "

Under what basis would you exit? Why exit? Wait how long? And why that length of time?

I think any trading move needs a full rationale, an understanding of why it makes sense to do so.

Post Reply