CSI Data Corrections

Use this forum to discuss data providers like CSI, charting, or other non testing software.
Mowgli
Contributor
Contributor
Posts: 4
Joined: Sat Feb 21, 2004 10:48 am
Location: Columbus, OH

CSI Data Corrections

Post by Mowgli » Tue Mar 28, 2006 11:08 pm

I am a new customer with CSI. Recently, on more than one occasion, I have noticed lots of data corrections in many contracts after I perform a daily update. Strangely, none of these corrections are to the Close and Unadjusted Close prices. I'm comparing data from the same delivery month, so no spread/backadjusting issues there.

This ofcourse changes a lot of my backtesting results. Is anyone else here facing these issues? I emailed CSI Tech Support but no response so far. Any help would be greatly appreciated.

Below are few sample data lines for 'AD' contract before and after my last update:


Date Open High Low Close Vol OpenInt DeliveryMonth UnadjustedClose
19960201, 0.6495, 0.6538, 0.6493, 0.6531, 4697, 7534, 199603, 0.7463
19960202, 0.6611, 0.6638, 0.6596, 0.6637, 5765, 9416, 199603, 0.7569
19960205, 0.6600, 0.6612, 0.6591, 0.6606, 3689, 10977, 199603, 0.7538
19960206, 0.6588, 0.6602, 0.6576, 0.6592, 1511, 11359, 199603, 0.7524
19960207, 0.6603, 0.6610, 0.6560, 0.6566, 2844, 11322, 199603, 0.7498
19960208, 0.6638, 0.6643, 0.6620, 0.6625, 864, 11390, 199603, 0.7557
19960209, 0.6608, 0.6613, 0.6594, 0.6601, 366, 11341, 199603, 0.7533
19960212, 0.6613, 0.6623, 0.6591, 0.6595, 320, 11337, 199603, 0.7527
19960213, 0.6589, 0.6608, 0.6578, 0.6607, 130, 11360, 199603, 0.7539
19960214, 0.6599, 0.6630, 0.6594, 0.6603, 425, 11184, 199603, 0.7535
19960215, 0.6614, 0.6618, 0.6602, 0.6605, 365, 11253, 199603, 0.7537


Date Open High Low Close Vol OpenInt DeliveryMonth UnadjustedClose
19960201, 0.6507, 0.6538, 0.6493, 0.6531, 4718, 7534, 199603, 0.7463
19960202, 0.6539, 0.6638, 0.6534, 0.6637, 5786, 9416, 199603, 0.7569
19960205, 0.6615, 0.6650, 0.6580, 0.6606, 3703, 10977, 199603, 0.7538
19960206, 0.6588, 0.6602, 0.6576, 0.6592, 1511, 11359, 199603, 0.7524
19960207, 0.6603, 0.6610, 0.6560, 0.6566, 2844, 11322, 199603, 0.7498
19960208, 0.6594, 0.6643, 0.6594, 0.6625, 883, 11390, 199603, 0.7557
19960209, 0.6646, 0.6656, 0.6594, 0.6601, 387, 11341, 199603, 0.7533
19960212, 0.6613, 0.6623, 0.6591, 0.6595, 320, 11337, 199603, 0.7527
19960213, 0.6593, 0.6609, 0.6578, 0.6607, 134, 11360, 199603, 0.7539
19960214, 0.6598, 0.6630, 0.6594, 0.6603, 429, 11184, 199603, 0.7535
19960215, 0.6594, 0.6618, 0.6594, 0.6605, 365, 11253, 199603, 0.7537

Thanks!

PaulZ
Roundtable Fellow
Roundtable Fellow
Posts: 95
Joined: Thu Oct 06, 2005 9:59 am
Location: Philadelphia Area

Re: CSI Data Corrections

Post by PaulZ » Wed Mar 29, 2006 9:54 am

I've been a CSI customer for years (I have no other affiliation). I've occasionally found missing or incorrect data in my historical testing. Communication with CSI has always led to corrections in the historical data. Every daily update has data corrections, some are historical (such as the ones I - and others - found during historical testing) and some are recent.

I feel that CSI is really committed to achieving accurate data, but they also face multiple original data sources, conflicts between data sources, and deadlines in their daily updates. For me, I feel they do a very good job in balancing the tradeoff between timeliness of the updates and accurate data. To their credit they don't stop analyzing the daily data once they post the daily update. When they find discrepancies later - even years later, they issue corrections via the next daily update.

Please let us know what kind of response you get from CSI on your email. If you don't hear from them in a few days, I suggest you send them another email.

damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian » Wed Mar 29, 2006 7:30 pm

Mowgli, I have often seen the same thing - data corrections from 3 or 5 or 7 or 10 (!!!) years ago. Such data corrections make little sense and in no way support the gushing claim that CSI are " really committed to achieving accurate data" as PaulZ asserts in his apparently non-affiliated support post.

If, in my employment, I claimed to be really committed to accurate data, I certainly wouldn't then take 10 years to find my mistakes.

Actually now I think about it, the situation seems so absurd on the surface that there simply must be more to the story than we are seeing. For example, perhaps the IMM pit at the CME have identified an error and today are correcting the previously reported open high and low from trading in Feb of '96? I think not.

And Mowgli - good luck with obtaining satisfactory customer support.

(disclaimer or something like it: I freely admit to having a bias against CSI. They are accepted as good by default.... only because there is nothing better. We eat that which we are given).

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Sun Apr 23, 2006 2:24 pm

Paul and other CSI " veteran" users can you please suggest what "standard options" you check when collecting backadjusted data for commodities? For istance, do you leave checked "detrend/invert"? I was playing with it and found that I got different trading results, its confusing. Whats the difference between SF0_I0B and SF20_I0B ? whats better and how it influences the backtesting?

thanks in advance

mike168
Roundtable Knight
Roundtable Knight
Posts: 142
Joined: Mon Jan 23, 2006 10:24 pm
Location: Australia

Post by mike168 » Sun Jun 18, 2006 12:42 pm

The following is my personal experience of how CSI is "committed" to provide accurate and clean data:
In early this month I found that the CSI commodity number 119 (HSI) has several MONTHS of missing contracts between Mar 1989 to Jan 1991. I sent an email to CSI for investigation. Their answer was their database department found "no records of these contracts to have ever traded".
I looked up the Hong Kong Exchange web site myself and found that the exchange themselves sell the whole database of HSI for only HK$600. I emailed CSI again and direct them to the HKEX website for the data. After several days still no action was taken. I sent another email to CSI to ask them what happened. Their answer was they will not add the missing data, without any further explanation.
If you look up the CSI data for HSI, you could see a whole lot of missing data and large gaps within this period.
The conclusion: CSI no longer provides clean data. Period.

Turbowagon
Roundtable Fellow
Roundtable Fellow
Posts: 82
Joined: Wed Oct 08, 2003 1:19 pm
Location: Washington, DC

Post by Turbowagon » Mon Jun 18, 2007 9:36 pm

ANyone get bad data for the Euro BOBL tonight for the date of 06-06-2007, the decimal point of off by a factor of 10. Instead of 106.57 that day shows 1065.7. All the rest of the days are normal. Weird.

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Tue Jun 19, 2007 3:07 am

I can't remember the exact date but it was immediately after the alteration by the exchange to settling the instrument in half ticks. My broker made the same mistake resulting in my short position showing an alarming large loss running into many millions of dollars.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Tue Jun 19, 2007 7:47 am

Turbowagon wrote:ANyone get bad data for the Euro BOBL tonight for the date of 06-06-2007, the decimal point of off by a factor of 10. Instead of 106.57 that day shows 1065.7. All the rest of the days are normal. Weird.
Yes I got that too. Twice. Once on my "UA 2.9.2" computer and again on my "UA 2.9.1" computer. Thus I don't believe the problem is in the software code; I think the downloaded data packets from the Boca Raton mothership were the source of the problem.

As with much else in the USA, the squeaky wheel gets the grease. The more customers who send complaints to CSI, the sooner problems will get fixed. So, email to database@csidata.com before posting here. An angry cabal of guerrilla DJ EuroSTOXX traders raised holy hell on 17 May 2007 after (this) UA hiccup, and it got fixed right away

equa
Full Member
Full Member
Posts: 12
Joined: Thu Mar 22, 2007 9:18 am

Post by equa » Tue Jun 19, 2007 11:52 am

they promise that it will be fixed with tonight's update

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Jun 20, 2007 11:45 am

Wed 6/20/2007 13:01 Eastern time, performed "Download replacement Fact files" followed by "Get Data", problem has gotten worse instead of better. Now all days before 6/6/2007 are wrong. See image.
Attachments
BOBL.png
CSI data for EuroBOBL on 6/20/2007 at 13:01 Eastern
BOBL.png (18.18 KiB) Viewed 5127 times

equa
Full Member
Full Member
Posts: 12
Joined: Thu Mar 22, 2007 9:18 am

Post by equa » Wed Jun 20, 2007 11:53 am

the reply i got from Rudi at CSI which worked is as follows:
Download replacement fact files with "download new stock mkt specs" UNCHECKED. Then Refresh Price history on the 2 bobl cts. nºs 996 and 553. Atferwards rebuild files.
This worked for me.

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Wed Jun 20, 2007 12:16 pm

I gotta say, seeing these sorts of issues that come up with CSI data now and then, does not encourage me to switch over to them.

As a Pinnacle Data user, I seem to struggle along each day just fine - even though I'm limited to only about 40 markets, including some nice International ones. Data corrections (which don't happen often) are automatically corrected in the next day's update.

Now that TB shows the last price in the Generated Orders print screen, the motivation to catch curent active month detail is less also (thanks Tim!).

Guess I'm still confused why CSI is worth 3 to 4 times the price of Pinnacle.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Jun 20, 2007 12:38 pm

equa wrote:Download replacement fact files with "download new stock mkt specs" UNCHECKED. Then Refresh Price history on the 2 bobl cts. nºs 996 and 553. Atferwards rebuild files.
This worked for me.
It worked for me too, thanks! Actually it is not NECESSARY to un-check "download new stock mkt specs", this was only recommended because it saves lots of time. (The stock market specs are a huge file). What makes this procedure succeed is (A) download new commodity facts; followed by (B) Refresh price history on #553 and #996. Image below.
Attachments
BOBL2.png
fixer upper
BOBL2.png (7.13 KiB) Viewed 5113 times

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Jun 20, 2007 1:47 pm

LeapFrog wrote:cut

Guess I'm still confused why CSI is worth 3 to 4 times the price of Pinnacle.
see graph attached

Note interestingly, Dial Data was once somehow affiliated with TradeStation.
Attachments
comparison.jpg
comparison.jpg (27.59 KiB) Viewed 5107 times

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Wed Jun 20, 2007 1:54 pm

I like it!

And the independent research organization that produced this table was...?

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Jun 20, 2007 2:12 pm

LeapFrog wrote:I like it!

And the independent research organization that produced this table was...?
Futures Magazine

Tim Arnold
Site Admin
Site Admin
Posts: 8540
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Wed Jun 20, 2007 2:38 pm

http://www.csidata.com/csi/reviews/reprintsep99.html

Note that this study was done in 1999. The data used was from 1994 to 1998.

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Jun 20, 2007 11:10 pm

Yes. We should all drop a note to the editor suggesting an updated review of that accuracy study.

Here's a link to the contact info. I myself will drop a note tomorrow with that request / suggestion. Also a longer period of testing seems in order. Even from 97-07 would be most interesting.

I'm all for keeping csi on its toes... Seems validating the accuracy would [at best] not be a simple task...

http://www.futuresmag.com/cms/futures/w ... al+contact

EDIT: My Letter:

Dear Editors of Futures Magazine,

Your article, - Futures Magazine 09/99
How Clean is Your End of Day Data?
Has come up in a discussion at Traders Roundtable Forum.

I feel that it would be appropriate for Futures to run an updated study on data accuracy. CSI data, claims to have the best database based upon the results of your 1999 article. Being that almost ten years have elapsed, I would like to request or suggest, that a new scientific study be completed and published.

This would be a wonderful service to your readership and of great importance to the trading community at large.

Cordially,

Mark Johnson
Roundtable Knight
Roundtable Knight
Posts: 122
Joined: Thu Apr 17, 2003 9:49 am

Problem with Euro / Aussie Dollar cross rate futures @ FINEX

Post by Mark Johnson » Sat Oct 27, 2007 12:37 pm

Whoops, another data problem in CSI Unfair Advantage. This time on the currency cross-rate futures at FINEX. CSI Commodity #623, Euro over Australian Dollar futures.

Looks like a digit problem. Found with Futures Data Investigator.
Attachments
CSI_No_623.png
data error
CSI_No_623.png (10.47 KiB) Viewed 4982 times

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Sat Oct 27, 2007 12:54 pm

Whatever the problems with CSI (and they can indeed be exasperating) where else can you get what you want? Unless like Mark you write your own software.

Take stock data: can anyone point me to a reliable source for the European markets where they backadjust for dividends? Unfortunately CSI's coverage is limited to very partial coverage of the LSE.

Post Reply