Traders' Roundtable

There are a lot of people testing here on this forum. What kind of historical data do they use to test? Where is the data coming from?

Hi Melinda-

I use CQG intraday data, would I do it again? No, while the quality is very good, it cost a small fortune and I had to write my own data manager program to create continuous contracts and to compress the data into different intervals.

That detour was costly as it removed my focus from more important issues. That falls into the 20/80 category of I received 20% of the overall benefit by exerting 80% of the overall effort. See this link for more on that topic: viewtopic.php?t=41

Sir G

Anyone use Tick Data (intraday). The Prices are good, but I couldn't get a satifactory answer pertaining to the rollover method and configuring continous backadjusted contracts for testing. I only interested in the S&P's, Bonds, Gold, and the Nazdaq. Thanks

I've used CSI Data http://www.csidata.com for futures. This is an expensive option but they provide "the best" end of day data around. They they do stocks as well.

For stocks and indexes (end of day) I've used yahoos historical data (downloads as an excel csv spreadsheet).

For short term you can find samples on the net. Try http://www.traders2traders.com for a wide range of options in their data section.

I havent been able to find a cost effective way to get DAX (german) intraday data so I have been forced to record it myself. If anyone knows a source (or just has a couple of years of it ) for 15min or lower Dax data I'd be eternally grateful.

John

I can only offer what I use for end of day stock trading and futures, I dont use intraday so this only applies to eod traders (http://www.tradinggame.com.au/data/packages.asp)

I am now looking for 3 minute data on the ESTX50 futures so I can test another idea. Can anyone help please - either data collected or where to get it. Data covering more volatile times would be very helpful!

John

I think it's worth considering also

www.futuresource.com

The pros are:

continuos historical data from the start of the contracts
Very easy to download
Ready to import in TradeStation
They offer intra-day data

Cons:
Only future data
Cost

It would be interesting to compare continuos CSI and Futuresource data.


P.S.
I was trying to attach a file of EOD data for S from 1968, but I'm not allowed to attach files.
Would the forum moderator be kind enough to let me know when I will be allowed?

The first question asked what type of data and where from.

Where from, I have tried a variety of user pays providers for both stocks and futures and found most of them satisfactory for both data and histories, if they are popular its probably because they are ok.

What type, is by far the more important of the two questions. Why, because what data you use will determine how you set the parameters that make your system work.
For example, if you go to the Wealth Lab Club you will see hundreds of systems that were designed mainly on the Dow30 and the Nsd 100, as a result they look good enough on those stocks, but if you put them over Australian and NZ stocks or futures, the success rate is a different story, that is, where the testing done was done and where they are used. A lot of vendors sold sytems that seemed to perform well on the data from the 80's and 90's but now they don't work any more, that is times have changed. Long term capital management with the wisdom of hindsight after their costly failure said " we should have used the latin currency failures in our volatility models , then we would have been prepared for the baht". Others got caught with the bonds in 87 when the rates were slashed. Stock traders didn't learn from 87 when they designed their Nsd systems and 99-2000 took a lot of money back.
Why should we consider these statistical outliers and extreme tails. As a trader I use volatility breakouts as an entry signal, that is when the price change isn't normal its the entry sign. I am a trend follower, that is, the money is made when the prices continue in one direction for a long time, when usually because of regression the normal behaviour is for prices to return to their usual distributions. If a trend follower is to make good then they must make their aim to be there when the prices hit the tails or statistical outliers. If traders need extreme situations to do well then they must logically include extreme situations in their models. Failure to do so in the past has proved repeatedly costly.

Anyone had any problems using historical data from Bloomberg on indices and commodities/financials?? is this generally viewed as a good source??

I tried to obtain historical futures prices from Bloomberg a month ago and found that most of the data do not go back very far. Moreover, for some futures, there are a lot of missing data in the series. Does anyone have the same experience?

Has any one tried out C.I.S intraday/tick futures data?

They seem so cheap compared to CGQ that I can't help but question the quality of the data.

http://moss.bayou.com/smcg/

Sabrinian

I bought the entire S&P history a couple of years ago from C.I.S., the data was good, I didn't find any obvious errors. It served my purpose well. Don't know about their updates or live data though.

Sir G,

Sir G wrote:I use CQG intraday data, would I do it again? No...

We use CQG intraday data! What would you do if you could do it all again?

Sabrinian

Ahhhh yes…. Sabrinian.. the 100th member!!

Hi Sabrinian-

Sir G wrote:
I use CQG intraday data, would I do it again? No...

What would you do if you could do it all again?

Very good question. I should also say when I did my original purchases from CQG, I had also tried Tickdata Inc and Genesis Data. I was horrified with them. At the time Tickdata inc’s own program couldn’t run their data! And Genesis came across as a hodgepodge of data collected via satellite with all the gaps associated with data from those days. Genesis also placed a series of zeros in their data to keep their time stamps constant. That was then though and It appears Tick Data inc under new ownership is a much more viable solution.

They offer stocks and Commodities and their own data management software.

If I were in the same position in current times, I would take a serious look at Tickdata Inc. That is not be construed that I have taken a serious look at them and I’m recommending them.

A downside that I perceive they have is their data management software only does one type of rollover logic. In my case, this would mean that I would need to generate 1 minute bars of actual data and run the data through my own roll over algorithms. And I believe they don’t furnish you with contract Vol & OI.

Some examples of my idea of a serious look and I suggest everyone with data do this is, check to make sure that the data flows with the session time changes that have occurred in the past. Also one should take the time to check the data for any gaps in dates. I did this simple test on the CQG data and found that 3 months were missing in Ten Years and Corn. The salesperson was amazed that no one else every caught that as she had sold that data often over the years, upper management knew the data was corrupted in their database but didn’t bother telling the sales force and the prior customers! That is a bozo no-no in my book. This is one of the reasons why I wouldn’t deal with them again. Another test you should run your intraday data through is against a good quality daily data. Make sure that the highest highs and lowest lows match the daily high and low, zero tolerance on this test. They must match! Also be sure that the opens and close are within say 10% of the daily range of each other. If you end up with a list of occurrences of these things, investigate and contact the companies to find out why. If there are gaps in the dates, investigate them and contact the company to rectify them. If you find out that the data is all within the current session times, investigate and make the proper contacts to rectify these things. If your highs and lows don’t match an you have many Opens and Closes not near each other, again investigate and complain. And keep an eye on their fix, they might just add or delete a tick randomly. This comparision against daily data test will flush out among other things missing session times. Complain loudly whenever you need to.

Bad data and there is much of it, is like sub-par nuts & bolts holding an airplane together. There is certainly more to researching then just the buys & sells! Buyer Beware.

Gordon

Gordon,

As always, your responses are well thought out, informative, and, as I had hoped, filled with practical suggestions!

Thanks.

Sabrinian

Following is a link to a very good example of checking the data integrity prior to testing with that data.

https://www.tradestationworld.com/discu ... C_ID=16540

A user from the TradeStation forum ran a series of tests on the new stock data that they just introduced. I believe the user looked at 6 different stocks going back 4 years. The results are startling! 30% of the time the highs or lows of the intraday database don't match the highs & lows of their own daily data!!

That isn't a good thing.

I'm attaching the results file here incase you aren't a registered user at their forum.

It's a great example of why you need to test the data, before you test on the data!

Wow!

Gordon

“Slowly I turn, step by step, inch by inchâ€

Hello,

I'm a helper and my job is to find if

free, end of day data, with (open , close, high, and low ) exist for this stuff below.

Reading this board I see some folks find data and the data is no good. So I learned that you can check it.

For now though

I'm looking for free end of day data for Gold, Silver, Palladium, Platinum, Copper, Zinc and Euro-Dollar Yen-Dollar CAD-USD, ZAR-USD

I also hope it will go back longer in time than 5 five years, thank you in advance for your time to respond if you are able?

Is anyone using Bloomberg professional for historical (or live) data?

Hi weatherman, I use Bloomberg data, historical and real time.