STIRS Roll in CSI

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
ADMP
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Wed Jul 04, 2007 9:04 am
Location: Germany

STIRS Roll in CSI

Post by ADMP » Tue Feb 05, 2008 10:06 am

Hi,

is there a way in CSI to roll the STIRS more than 11 months before the end? The Eurodollar is quite liquid until 2 years maturity, and I would like to test different maturities.

Alex

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Tue Feb 05, 2008 11:09 am

Nope, you'll need to export the raw contracts from CSI and connect them together yourself using non-CSI rollover software.

(I only tested CSI UA 2.9.1; other releases may behave differently)

There is a very small but feverish group of people who believe that continuous contracts, perpetual contracts, Panama contracts, backadjusted contracts, and all other similar contracts & contrivances are misleading and evil. They believe the ONLY way to get proper and correct and accurate and righteous results, is to simulate using raw contracts (and explicitly simulate roll-over trades from one raw contract to the next). I'm not in this cult myself, but I've come across people who are, and they're certainly, uh, quite confident in their beliefs and opinions. Maybe a search for these kinds of people will turn up fresh new leads in the hunt for non-CSI contract manager software.

ADMP
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Wed Jul 04, 2007 9:04 am
Location: Germany

Post by ADMP » Wed Feb 06, 2008 2:00 am

Thanks Sluggo.
It's a pity. Will be more work.

davidib
Contributing Member
Contributing Member
Posts: 9
Joined: Mon Jul 18, 2005 6:09 pm
Location: Chicago

Post by davidib » Wed Feb 06, 2008 6:48 am

ADMP,

I'm in the camp that sluggo mentioned regarding continous contracts. Well I'm not a fanatic and there is a time and place for such contracts, but in general they leave alot to be desired.

I've recently (2003) worked for a couple turtles here in Chicago. One turtle used standard, absolute back adjusted contracts ala CSI. The other was adamant about the need for using actual prices and handling the rolls in their proprietary backtesting engine (FWIW the real price turtle manages over 500 mil, the adjusted price turtle less than half of that) I personally us actual prices for testing; when I need a long term time series I use proportionally, forward adjusted contracts.

FWIW, as I'm sure forum members are aware there has been alot of research recently on roll yields and such (google 'Hilary Till'). The research suggests much of what I would label institutional type returns in managed futures can be attributed to the roll yield.

In any case, I've code to calculate rolls and such. It's not as sophisticated as CSI (i.e. right now it just rolls on x days before expiry) but I don't need it to be. My system is in C#, but the code is such that it would be pretty straightforward to port it to another language (I'd vote for perl just because it's non prop, runs on any OS, and is easy enough to allow anyone with some programming knowledge and a text editor to modify to their needs).

I always thought there was a need for a simple suite of software tools for rolling and was always frustrated in that such common functionality was ether non existant or embedded in someone elses software. I'd be interested in developing such a tool and open sourcing it for everyones use (or maybe just everyone in this forum).

I'd solicit the forum for suggestions regarding the types of rolls they are looking for.

Forum?

David

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Feb 06, 2008 8:07 am

David, best wishes on this project. I've written software to do a little bit of what you're attempting, and found many ways to screw it up. Permit me to suggest that you look at the various types of calendar date based rollover algorithms used by Pinnacle Data Corp and shown in their table (here). Ultimately these "roll over on a certain calendar date or, if that date is a market holiday, on the first market day thereafter". But there are at least four different ways to express that certain calendar date
  • Roll over on the 3rd calendar day of the delivery month
  • Roll over on the 22nd calendar day, 1 month prior to delivery month
  • Roll over on the 2nd Thursday of the delivery month
  • Roll over on the Thursday before the 2nd Friday of the delivery month
and so forth. When you read through Pinnacle's list of rollover algorithms, you'll notice that the complicated ones are mostly used for stock index futures. (Fun fact: the latter two are not supported by the current release of CSI Unfair Advantage).

Calendar date rollovers are fixed and predetermined, they can be scheduled years in advance, just by looking at a calendar. Then there are the variable or non-predetermined types of rollovers, such as roll on Volume Crossover or roll on Open Interest Crossover. You don't know in advance, when these might occur (you don't even know whether they will occur). You have to search through the data. A major decision to be made is, will you perform this search by marching forward through time (CSI UA option checkbox "Generate Forward"), or will you search by marching backward through time. They give different answers, and one is easier to program than the other :-) What will you do if crossover NEVER occurs? Will you exclude a contract, jumping straight from the December to the February and skipping the January, if Jan's open interest is always less than Feb's?

Then there's the question of which underlying contracts to include or exclude when building the continuous contract. For example, some people want their continuous contract for Crude Oil to include twelve rollovers per year. Others like me prefer to roll Crude six times per year. Still others want 4 rolls per year. So you may want to copy CSI's feature that lets people include or exclude contracts as they choose. Some people trade October Cotton and want it in their continuous contract. Others don't. Same goes for September soybeans.

Then there's the question of what time-of-the-day you want the roll to happen. Do you splice based on exiting the old contract and entering the new contract on the Open? On the Close? One on the Open and the other on the Close? How do these influence the way you choose to present the "Unadjusted Close" field?

Then there's the question of intraday data, which Blox supports and which many Blox customers use. The time-of-day selected to perform the rollover is especially important here. And how do you define "roll on Volume" or "roll on Open Interest" for intraday data?

THEN there's the question of what input and output formats you'll choose to support. Space delimited ASCII? Comma delimited ASCII? Metastock? What directory structure do you assume the raw contracts will be stored in? What file naming convention will you select/demand for the input data and the output continuous contracts?

I've posted CSI UA's menu of rollover options, below. This might be a good resource to help you decide what options you want your software to support. CSI doesn't support intraday data yet, so use your imagination for those kinds of options.
Attachments
UA_options.png
UA_options.png (18.84 KiB) Viewed 8106 times

davidib
Contributing Member
Contributing Member
Posts: 9
Joined: Mon Jul 18, 2005 6:09 pm
Location: Chicago

Post by davidib » Wed Feb 06, 2008 8:27 am

Hi Sluggo,

the devil is in the details with this stuff. There are so many nuances, gotchas, cases were all your roll rules fails that this is not an easy task by any means. For example, a while back I started in earnest on such a project and ended up having to allow for the user to be able to specify an explict roll date in some cases because none of the roll rules 'hit'. Steve Davis, the head programmer at CSI (circa 2001) had mentioned in several conversations the headaches associated with coding the roll rules.

So what I was/am proposing is an easing into such a project, i.e. start with something simple and go from there. I would imagine that a simple process to take all the csi/csv/ascii files in a single directory and do a basic absolute back adjustement would cover the majority of peoples needs. Nothing exotic, but a start.

But it seems that maybe such a system exists already? Take a look at this thread
viewtopic.php?t=1790

Well I don't want to reinvent the wheel and if someone has such software and it works I'd say go with it, Maybe Roger has more info on such a tool?

David

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Post by Roger Rines » Wed Feb 06, 2008 11:32 am

davidib wrote: [snip]
But it seems that maybe such a system exists already? Take a look at this thread
viewtopic.php?t=1790

Well I don't want to reinvent the wheel and if someone has such software and it works I'd say go with it, Maybe Roger has more info on such a tool?

In the above link "Davidib" references there is discussion that points to our Windows Data Manager program in use daily. That software provides an interface for manually adding data to contracts and for making backwards adjustments. This image shows the main screen:
http://tinyurl.com/3dkhz4

In this data entry screen I don't add Volume or Open Interest, but that is going to change along with the ability to record Contract Information and Unadjusted Close values.

When working in this screen, the data input is manual, punch it in with the keyboard, that will change as well, but the feature that has been really helpful has been the ability to enter information as fractions or decimals depending upon how the data trades.

In another part of the software is a section for manually stitching contracts together:
http://tinyurl.com/2th6lw

In this Join-File screen File#1 is usually a Continuous Back-Adjusted file. To find a matching date in the second file, the user just clicks a row and the matching date appears so the user can decide if they want to move on that date. Here again I've got some changes in the works to make this part even easier and automated so that the stitching process isn't so time consuming when the timing of the stitch doesn't have any account impact.

Helping to make some of this manual stuff easier is a database that keeps track of the information for all the ASCII files it loads. Some of the details stored reflect the file name and path location along with the normal market data like this:
http://tinyurl.com/3xpkha

Along with maintaining any data is to know when there isn't any data. This is important in this software because it checks the date records and price relationships in each file to keep the data as clean as possible. To know what trades when is this screen:
http://tinyurl.com/2urgvy

Unfortunately this Days-Off process is localized to the US so international markets are being handled with more effort. Maybe some day I'll find the time to figure out a process for managing this better, but for now it is all lookup for markets not traded in the US.

As I moved towards automation all the issues Sluggo mentions had to be addressed and the process for automating must have a fail-safe date trigger for when things are just not normal. Here are a couple of screen images for setting automated roll timing triggers:
http://tinyurl.com/2hrvl9

http://tinyurl.com/2e6jp9

Right now these are two separate screens, but there is another version of this process in the works that will combine the process so the user will better understand that the process can be combined.

All of the above is moving towards this kind of interface where the data is presented in a more visual manner:
OLD: http://tinyurl.com/243pub
Current: http://tinyurl.com/ylqo44b

In this screen layout the user gets to see the raw data coming in from CSI and gets to look ahead in a visual manner to see what is happening with Prices, Volume and Open Interest. For example here are the Volume and Open Interest expanded detail screens:
http://tinyurl.com/yvfpzh

http://tinyurl.com/22tlvm

In this screen set we are only looking ahead 2-contracts, but the process will look ahead 6-contracts for markets like the Eurodollar Interest Rate.

There are still some problems in how this is working that need to be worked on, but the insight I get and the time I save to feed my need to add skill to the roll timing is making progress and it goes a long way towards satisfying my need to be aware of what is happening in the market everyday.

One of the areas not working well enough is one of the issues Sluggo mentions above:
http://tinyurl.com/3a2eqk

This screen controls which markets are excluded from consideration and it isn't part of the roll timing setup screen. This screen shows a disjoint that is making the process clumsy. When this screen was designed I didn't have all the mechanics of how to handle the date fail-safe process so I disabled the roll trigger sections on this screen, but left out the contract exclusion process because I didn't have the time back then to get it all stuffed into on better laid out dialog.

All of this effort competes with my trading and system development efforts. Over these last 8-months Trading Blox has been consuming any time not being taken by the markets so the work on this gets to run at near Glacial speeds, even with the recent speed increase the ice flows are demonstrating.

What isn't in any of the tools I use is the ability to do any of the other types of data stitching. Mostly because I find that Back-Adjusting works very well at paying the rent, but to some extent I've never tried to go down those other paths because of early 90's period disappointments with other approaches. For certain I've not tried everything, but then I've never seen anything meaningful to elevate my interest.

NOTE:
20100326 - Added updated image of Quote Board.
Last edited by Roger Rines on Fri Mar 26, 2010 6:37 pm, edited 1 time in total.

davidib
Contributing Member
Contributing Member
Posts: 9
Joined: Mon Jul 18, 2005 6:09 pm
Location: Chicago

Post by davidib » Wed Feb 06, 2008 12:44 pm

Hi Roger,

thanks for the reply. Your tool (:oops:) looks very good. You've got a nice GUI on it and it seems to do the job for you.

I agree with you regarding your last statement about different approaches not being worthwhile for you to follow up with. Aside from my belief that a ratio adjustment is 'better' than an absolute, the only reason I use the forward adjustment is that once adjusted, the prices never change. There is no need to go back and recalculate all the price adjustments when a new front month comes on board, I can rerun simulations that I ran a year ago and everything (i.e. entry/exit points, indicator values, etc) comes out the same (make it much easier to check reuslts, etc.)

I can get away with this because my simulator trades using the actual price vs the adjusted price. If I was simulating off of an adjusted value I'd use back adjusted ratio continuous contract.

Regardless of all the above, I occasionally find back adjusted prices handy and to that end would personally find it very useful to have an uncomplicated tool to do the various types of adjustments. Is your software for sale? Would it be easy to modify it to do various other types of adjustments?

Thanks
David

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Post by Roger Rines » Wed Feb 06, 2008 3:05 pm

davidib wrote:Hi Roger,
thanks for the reply. Your tool (:oops:) looks very good. You've got a nice GUI on it and it seems to do the job for you.
Hi David,
Thank you for the kind words.
It does work for us because I have this need to know what is happening while it is happening, and I'm always afraid that bad data can sneak into the signal process.

As time goes forward and the "Beta" software becomes more developed, our software will eventually allow viewing and editing of the data as needed, and it will also allow the user on a symbol basis to toggle between the scheduled auto update or handle that day manually forcing a change that gets picked up by the scheduler for the next day. In other words, most of the time the automated update of each contract can be allowed , but sometimes intervention is needed so both must be able to coexist to allow a user to be smarter when life isn’t following our plan.

What is important here is to have a data review process that allows changes or automation. Our plan is to allow the user to view newly downloaded data values, and then see the historical and projected paths of price, volume and open interest in a future contract comparison graphic that also shows the scheduled date-signal. I find this type of information generates more meaningful market information and better decisions. All this is so I’m allowed to intervene when I want to override the roll timing process and keep bad prices from happening to my signals. Just as important is to have a daily timing strategy in place and published to the broker so they are allowed to emulate the trade handling to be in line with the data handling strategy.
davidib wrote: I agree with you regarding your last statement about different approaches not being worthwhile for you to follow up with. Aside from my belief that a ratio adjustment is 'better' than an absolute, the only reason I use the forward adjustment is that once adjusted, the prices never change. There is no need to go back and recalculate all the price adjustments when a new front month comes on board, I can rerun simulations that I ran a year ago and everything (i.e. entry/exit points, indicator values, etc) comes out the same (make it much easier to check results, etc.)
I hope I didn't say it wouldn't be worthwhile. If that was the impression given than I apologize as that wasn't my thinking. My intended message was trying to say I didn't have any intelligence that was motivating me enough to spend time trying to see if a different approach would make any system performance difference. However when I say it like that, that might be the best reason to go down that path and find out.
davidib wrote: I can get away with this because my simulator trades using the actual price vs. the adjusted price. If I was simulating off of an adjusted value I'd use back adjusted ratio continuous contract.
Most of what we send to the market are signals from long-term systems. We have some systems that trade a shorter time period and they have sometimes been opposite the long-term system at times, but in all cases they use the same set of back-adjusted data files.

In addition, it is critical for me to not make any price adjustments to signal prices that come out of the system. I don’t want to adjust prices because the error probability would be too high. In simple terms, the prices used when the system signals are generated must always be the prices that are put into the faxes and emails sent to the broker so that we have a written record that can be tied back to the system’s original control documents. When it comes time to check on how things went the net change is the critical component for us because that is how the money is decided.
davidib wrote: Regardless of all the above, I occasionally find back adjusted prices handy and to that end would personally find it very useful to have an uncomplicated tool to do the various types of adjustments. Is your software for sale? Would it be easy to modify it to do various other types of adjustments?
I haven't wanted to sell the software as that would have been a distraction should the need to deal with support surface. I've given copies out on request, but I don't think most of the people who have asked for it use it like it we do, so I suspect most of what has been given out isn’t used at all. When I've asked some of them about their use, they mostly seem to want something more automatic. I'll have automatic some day, but I don't fly on autopilot and roam around the cabin, and that is safer than trading on automatic without knowing how the data is going to surprise the accounts.

“Can it be modified?â€

ADMP
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Wed Jul 04, 2007 9:04 am
Location: Germany

Post by ADMP » Thu Feb 07, 2008 2:13 am

The reply of CSI:

"At one time we had rolls prior to 11 months. Unfortunately it didn't
work well. We decided not to spend any more resources on this."

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Thu Feb 07, 2008 11:37 am

sluggo wrote: ... Calendar date rollovers are fixed and predetermined, they can be scheduled years in advance, ...
As an illustration of this, here are my rollovers for the next three weeks. Sorry for the poor quality image, I'm not much of a photographer.
Attachments
calendar.jpg
calendar.jpg (38.35 KiB) Viewed 7970 times

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Fri Sep 25, 2009 10:42 am

As long as we're revealing our high tech trading environments, here's a pic of my workstation...
Attachments
Trading Workstation.jpg
Should I just jump to Win 7?
Trading Workstation.jpg (81.22 KiB) Viewed 6070 times

Post Reply