Where to get accurate historical tick data?

MutantNinjaTurtle · Post by **MutantNinjaTurtle** » Sun Feb 12, 2006 4:46 pm

Hi,

I was wondering if anyone here has a any recommendations on vendors or sources of accurate historical tick data? It seems that most historical data that is sold by vendors is end of day data. I wanted write a program to back test the Original Turtle's Trading System. After reading about the trading system, it seems that decisions were made intraday and on the tick granularity. I have looked at CSI but they only provide tick data for a small portion of contracts. The only vendor that I have found that offers tick data is http://www.tickdata.com/. Does anyone have experience with their data?

TrendMonkey · Post by **TrendMonkey** » Tue Feb 14, 2006 5:11 pm

When I checked into it, the cost of TickData was way out of the reach of an individual trader, even one who (like me) pays thousands of $$ per year in commission. So if you find one I'd sure be interested

MutantNinjaTurtle · Post by **MutantNinjaTurtle** » Tue Feb 14, 2006 10:56 pm

$900 for 30 years of tickdata doesn't seem that unreasonable... But I guess that is substantially more expensive than EOD data. I'm surprised that no one here has responded about intraday/tick data.

mike168 · Post by **mike168** » Wed Feb 15, 2006 3:13 am

A few years back I have bought S&P and ND tick data but I have nothing to report. I don't know whether they are clean or anything. They are just that - a bunch of data for back-testing.

Paul King · Post by **Paul King** » Wed Feb 15, 2006 9:31 am

I have some experience in historical data and symbology from my previous career as a business analyst (my last 'real' job before I started my own company was at an ECN)

In my opinion 'accurate historical tick data' is an oxymoron (at any price). Sorry to disappoint you. People in this forum may have experience with particular data vendors that have historical tick data, but if you want 'accurate' you will have to 'clean' the data yourself or subscribe to many different databases and manually reconcile differences to come up with a 'best you can do' database.

The sheer volume of data involved if you look back any significant amount of time is usually difficult to cope with and ends up making any solution virtually unworkable.

My advice would be to test a concept on end-of-day data historically, and then test it on paper with tick-data going forward and seeing whether there are any major differences that would cause you to revise the system.

Paul

rabidric · Post by **rabidric** » Wed Sep 20, 2006 2:03 pm

i use cqg for backtesting on intraday data

the software has massive portfolio limitations, but, the minute bars data is of good quality(it is clean) going back over 5 years on normal subscription.

you can purchase further backdata at exorbitant prices....

I promise TB, that i will purchase TBB the moment they have the ability to handle intraday bars(not necessarily tick data- 1min bars is good, and 30minute bars will do even)

the EoD limitation is the final hurdle for TB to overcome- you appear to have all the other user configurable stuff sorted out!

ecritt · Post by **ecritt** » Wed Sep 20, 2006 2:21 pm

I used tickdata.com back in 2002-2003 for data on several indexes. I found it to be clean and their tickwrite program to be functional.

ramramsab · Post by **ramramsab** » Fri Sep 14, 2007 6:36 pm

Mr MutantNinjaTurtle (or anyone else), any feedback on your experience with tickdata.com?

Honeycomb · Post by **Honeycomb** » Tue Oct 02, 2007 10:34 am

I'm about to spring for the 15yr package. Data is cheap compared to the cost of being Wrong. They also have a .pdf on their scrubbing algorithm which is a good overview of some of the issues that come with high-freq data.

Chuck B · Post by **Chuck B** » Mon Oct 08, 2007 8:48 am

Honeycomb wrote:I'm about to spring for the 15yr package. Data is cheap compared to the cost of being Wrong. They also have a .pdf on their scrubbing algorithm which is a good overview of some of the issues that come with high-freq data.

Three years back when comparing Tickdata data versus CQG historical data (purchased CQG database for huge sum), many errors were found in the Tickdata data on the 20 or so futures markets analyzed. CQG has the most accurate historical tick data I have found, but you have to write your own program to extract the data and shape the output you want (i.e. specific contract months, continuous contracts, etc.).

One quick and dirty method of data checking to get a sense of the accuracy any tick data source is to form daily bars from the tickdata and then check the high and low versus known good historical daily data. It is amazing what this simple check will reveal about many vendorsâ€™ tick data...no matter what their claims of accuracy.

Honeycomb · Post by **Honeycomb** » Mon Oct 08, 2007 7:28 pm

Chuck B wrote:One quick and dirty method of data checking to get a sense of the accuracy any tick data source is to form daily bars from the tickdata and then check the high and low versus known good historical daily data. It is amazing what this simple check will reveal about many vendorsâ€™ tick data...no matter what their claims of accuracy.

Yeah, that seems workable provided you're careful to ensure that both data sets are using the same definition of day.

Looks like I've got more research to do

danZman · Post by **danZman** » Tue Oct 09, 2007 4:57 am

I have to assume this route is being taken for contra-trend following? There is definitely an edge there...but not for large positions. I've seen lots of these on collective2.com. Interesting how the MAR on VST systems is so high while LTTF is so crappy. But will LTTF take the last laugh when money chases these low drawdown systems?

I have most of my money in LTTF because it's proven. How are other MMs positioned? Anyone?

D

sluggo · Post by **sluggo** » Tue Oct 09, 2007 7:27 am

Short term systems with an edge, if they exist, do allow account equity to compound much faster. They close out many more trades per week, month, year than their longterm bretheren. Since profit per unit time is roughly (1+edge)^#trades, the increased exponent is, well, exponentially beneficial.

Short term systems can be traded at much lower total portfolio heat than longterm systems, which translates to much lower drawdown and lower volatility of returns, hence high Sharpe ratios and high MAR ratios.

However, short term systems tend to be "high maintenance girlfriends." You must feed them expensive, high quality intraday data. They give you a constant never ending stream of errands to perform: place this order, move this stop, cancel and replace this, reverse that, add this number to the opening tick and place these two orders, etc. They keep you up late at night, and make you arise from bed early, so you can lavish another daily dose of your attention upon them. They demand that you execute many transactions, so you need to have very low per-transactions costs. Some people are attracted to these kinds of girlfriends; others, not. I think it's mostly a personal preference. Short term systems come in trend-following, trend-anticipating, trend-fading ("counter trend"), and other flavors. This too is a personal preference.

AFJ Garner · Post by **AFJ Garner** » Tue Oct 09, 2007 10:01 am

Short term systems with an edge, if they exist

Coming from Sluggo, the caveat "if they exist" must be taken seriously. Although, reading between the lines of his posts I don't think Sluggo has ever moved to testing intra day data?

I have never ventured into this area nor ever will, unless I can greatly improve my coding skills and general IT savvy.

And yet we must assume that such an edge does exist. I assume that Chuck B is a very short term trader in addition to his longer term stuff.

We can of course discount 99% of the tales of derring do by the multitude of mostly mentally sub-normal posters on elitetrader.com but the fact remains that there do appear to exist intelligent and presumably successful traders who make their daily crust in this way.

And then of course there is the realm of the fully automated monster machine outfit which employs 200 IT specialists and whose incredible black boxes are physically situated within a few hundred yards of the relevant exchanges, so that they can exchange information within nanoseconds via fibre optic cable.

Such automation is beyond the reach of many of us and yet surely some degree of automation is the way to convert a â€œhigh maintenance girlfriendâ€

danZman · Post by **danZman** » Tue Oct 09, 2007 3:24 pm

AFJ Garner wrote:
Short term systems with an edge, if they exist

Coming from Sluggo, the caveat "if they exist" must be taken seriously. Although, reading between the lines of his posts I don't think Sluggo has ever moved to testing intra day data?

I have never ventured into this area nor ever will, unless I can greatly improve my coding skills and general IT savvy.

And yet we must assume that such an edge does exist. I assume that Chuck B is a very short term trader in addition to his longer term stuff.

We can of course discount 99% of the tales of derring do by the multitude of mostly mentally sub-normal posters on elitetrader.com but the fact remains that there do appear to exist intelligent and presumably successful traders who make their daily crust in this way.

And then of course there is the realm of the fully automated monster machine outfit which employs 200 IT specialists and whose incredible black boxes are physically situated within a few hundred yards of the relevant exchanges, so that they can exchange information within nanoseconds via fibre optic cable.

Such automation is beyond the reach of many of us and yet surely some degree of automation is the way to convert a â€œhigh maintenance girlfriendâ€

pallbritton · Post by **pallbritton** » Tue Oct 09, 2007 3:55 pm

As far as tick data goes, you might want to check out opentick (http://www.opentick.com/)

I've used it very briefly and can say that you'll need to scrub it yourself, but the price is right.

Can't comment on the quality as i havent compared it to any other sources, but it would probably be fine for 'proof of concept' type work.

Bounty · Post by **Bounty** » Fri May 29, 2009 9:06 am

I have been wrestling with the issue of intraday (1 min) data for years. I subscribe to IQFeed and IB and use QColletor. That allows me to download as much data as I want. The IQFeed data only goes back about a year and a half, so I've tried to come up with ways that I can create an intraday data warehouse of all the stock/futures symbols (mainly equities for the current testing I'm doing). I also use MLDownloader to get free EOD data.

The issues that have to be dealt with are similar to those in the tickdata data scrubbing pdf. For instance, how to accurately adjust for dividends/splits? Supposedly, the MLDownloader EOD data is split/dividend adjusted.

I have had some software developed that will help with that but it still is not easy. Basically, it compares the intraday data to the adjusted MLDownloader data to determine when an adjustment needs to be made.

Two questions:

1) Is anyone else adjusting intraday equity data for splits/dividends? How?
2) I assume that I can avoid this issue completely when I begin using futures?