Traders' Roundtable

I'd like us to come up with a standard benchmark set of test data which we could use for the purposes of making comparisons between different systems/approaches. This data will also be used for the Trading System Contest.

Does anyone have any suggestions for what data would be a good choice (source, markets, years)? It would obviously have to be data that comes from a provider that doesn't prohibit redistribution of daily data or that offers the data for free.

Unfortunately, the data I have can't be redisributed.

Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?

Another thought(s).

For commodities EOD would CSI, say, allow release of a subset (20 commodities backadjusted from 1980-1999 only) along with acknowledgement of where it came from.

For stocks & indexes EOD everyone could be referred to yahoo.

For intraday there are a number of samples on the web but perhaps someone would permit a subset to be made available with acknowledgement?

I like CSI and I think a lot of people use CSI.

By the way, Forum Mgmnt, Can Veritrader be used with intraday data too?

You can access and update 30 years worth of stock and commodity data
from the history bank it has C++ and VB API.
You can also get real time quotes into the GlobalServer via 4-5 vendors.
Just an idea....

I just visited Mark Johnson's site and he has a backadjusted data set for 20 commodities at:

http://www.mjohnson.com/rescap_lecture/lectop.htm

CSI proofed to be the best. So that must be the benchmark> most logical I think personally. It gives better insights what the differences are in testing results with different data providers.

Marc

I would like to see CSI data adopted for the standard data set. I think the 20 commodities listed on Mark Johnson's site would be a good place to start. Another good market group would be the 25 markets that make up the MLM and CMI. ( http://www.mtlucas.com/MLM_const.htm )

The next step would be to fully specify the CSI roll timing settings for each market. As shown in the attached image.

The CSI back adjusting options annoy me. If you stipulate roll on OI, it will do so... and then jump between contracts depending on OI. Say you roll from June03 to Dec03 on OI and then come August the Oct03 contract is trading at higher OI. In this case the continous contract will stop using Dec03 data and start using Oct03 data on the OI basis. As a trend follower, I do not trade like this.

No back adjusting engine is of any use to me until it let me say "roll into the most distant contract that has the required liquidity".

It used to do this but (i think last year) CSI changed to hold the later contract even if the OI in an earlier one popped up. I checked with Rudi and he confirmed that they had changed because most traders preferred this approach.

John

thanks john.

this goes to show that I should read that stuff they send me in the mail rather than complaining on an internet forum

damian wrote:Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?

Just throw in fatter tails and skewed distribution. And whatever you may say about prices in general. That would be a "statistical" benchmark .

Jakub

steady_jake wrote:

damian wrote:Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?

Just throw in fatter tails and skewed distribution. And whatever you may say about prices in general. That would be a "statistical" benchmark .

Jakub

If you just randomized the selection of the block of time series - I mean they have to be contiguous, of course, isn't that a valid test?

Glen

Glen,

It is a valid test ... but only of the series you have created. I'm not a statistician so someone correct me if I'm inaccurate.

Real world data has characteristics that a random series won't give you. The fat tails just means that where a normal distribution slopes nicely down to zero price data doesnt trail off so nicely - so you get more prices at the extreme than you would expect. Skew means that instead of being distributed evenly on both sides of the mean (average price) the data will be skewed to one side.

So when you test trading systems you need to use data with these characteristics or you won't see the extreme behaviours you can get in real markets.

John

I agree with John. Moreover, how could meaningful random data be generated from what is an inherently unknowable distribution. Sure- it's "similar" to lognormal, stable Pareto,etc. but is "similar" generated data any better than the actual sample of historical data.

Data is the heart of the whole issue for me.
I took statistically acceptable sample sizes* from a wide variety of data, including from different times, commodities, currencies,American and Australian stocks and randomly generated from the spreadsheet.
When I compared them in an unadulterated form they all exhibited specific characteristics, ie, class types, eg, futures, national stocks or random. These classes were correlated in their basic behaviour and how they responded to different statistical tests. There was variation in the way they responded over time, as much as how they differed from others in a similar class. In fact many of a similar class varied in a similar fashion over time. Barring random which performed much the same as the books say it will.
If you accept that data like most things has its own character, then whatever is chosen as your backtesting data will simply result in unconscious curve fitting. We can then produce great results in that frame, but if you switch over to a database of a different character or time frame then the same system produces different results.
Robustness is a fashionable topic, the dictionary calls it "strength, straight foward not confused by subtleties." A tough call, to be sure. The devil is in the detail and the subtle differences in the detail is what affects the outcome. Experience has shown me what suits one data class or given time period may not suit another. Is that robust or is it that it is only robust in that class. If so then chose the class that will be the one used for trading.

* Mendenhall. Introduction to probability and statistics.
________________________________________
Doubts are healthy