Benchmark Test Data
-
- Roundtable Knight
- Posts: 1842
- Joined: Tue Apr 15, 2003 11:02 am
- Contact:
Benchmark Test Data
I'd like us to come up with a standard benchmark set of test data which we could use for the purposes of making comparisons between different systems/approaches. This data will also be used for the Trading System Contest.
Does anyone have any suggestions for what data would be a good choice (source, markets, years)? It would obviously have to be data that comes from a provider that doesn't prohibit redistribution of daily data or that offers the data for free.
Unfortunately, the data I have can't be redisributed.
Does anyone have any suggestions for what data would be a good choice (source, markets, years)? It would obviously have to be data that comes from a provider that doesn't prohibit redistribution of daily data or that offers the data for free.
Unfortunately, the data I have can't be redisributed.
Another thought(s).
For commodities EOD would CSI, say, allow release of a subset (20 commodities backadjusted from 1980-1999 only) along with acknowledgement of where it came from.
For stocks & indexes EOD everyone could be referred to yahoo.
For intraday there are a number of samples on the web but perhaps someone would permit a subset to be made available with acknowledgement?
For commodities EOD would CSI, say, allow release of a subset (20 commodities backadjusted from 1980-1999 only) along with acknowledgement of where it came from.
For stocks & indexes EOD everyone could be referred to yahoo.
For intraday there are a number of samples on the web but perhaps someone would permit a subset to be made available with acknowledgement?
-
- Roundtable Knight
- Posts: 176
- Joined: Thu Apr 17, 2003 9:52 am
- Location: Sacramento, CA
- Contact:
I use TS 2000i GlobalServer (open system)
You can access and update 30 years worth of stock and commodity data
from the history bank it has C++ and VB API.
You can also get real time quotes into the GlobalServer via 4-5 vendors.
Just an idea....
from the history bank it has C++ and VB API.
You can also get real time quotes into the GlobalServer via 4-5 vendors.
Just an idea....
I just visited Mark Johnson's site and he has a backadjusted data set for 20 commodities at:
http://www.mjohnson.com/rescap_lecture/lectop.htm
http://www.mjohnson.com/rescap_lecture/lectop.htm
-
- Roundtable Fellow
- Posts: 58
- Joined: Wed Apr 30, 2003 4:35 pm
- Location: Netherlands
CSI as benchmark
CSI proofed to be the best. So that must be the benchmark> most logical I think personally. It gives better insights what the differences are in testing results with different data providers.
Marc
Marc
I would like to see CSI data adopted for the standard data set. I think the 20 commodities listed on Mark Johnson's site would be a good place to start. Another good market group would be the 25 markets that make up the MLM and CMI. ( http://www.mtlucas.com/MLM_const.htm )
The next step would be to fully specify the CSI roll timing settings for each market. As shown in the attached image.
The next step would be to fully specify the CSI roll timing settings for each market. As shown in the attached image.
- Attachments
-
- CSI.jpg (76.13 KiB) Viewed 15035 times
The CSI back adjusting options annoy me. If you stipulate roll on OI, it will do so... and then jump between contracts depending on OI. Say you roll from June03 to Dec03 on OI and then come August the Oct03 contract is trading at higher OI. In this case the continous contract will stop using Dec03 data and start using Oct03 data on the OI basis. As a trend follower, I do not trade like this.
No back adjusting engine is of any use to me until it let me say "roll into the most distant contract that has the required liquidity".
No back adjusting engine is of any use to me until it let me say "roll into the most distant contract that has the required liquidity".
Last edited by damian on Tue Jul 08, 2003 10:49 pm, edited 1 time in total.
-
- Full Member
- Posts: 22
- Joined: Fri Jun 27, 2003 8:05 pm
- Location: Warsaw, Poland
- Contact:
If you just randomized the selection of the block of time series - I mean they have to be contiguous, of course, isn't that a valid test?steady_jake wrote:Just throw in fatter tails and skewed distribution. And whatever you may say about prices in general. That would be a "statistical" benchmark .damian wrote:Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?
Jakub
Glen
Glen,
It is a valid test ... but only of the series you have created. I'm not a statistician so someone correct me if I'm inaccurate.
Real world data has characteristics that a random series won't give you. The fat tails just means that where a normal distribution slopes nicely down to zero price data doesnt trail off so nicely - so you get more prices at the extreme than you would expect. Skew means that instead of being distributed evenly on both sides of the mean (average price) the data will be skewed to one side.
So when you test trading systems you need to use data with these characteristics or you won't see the extreme behaviours you can get in real markets.
John
It is a valid test ... but only of the series you have created. I'm not a statistician so someone correct me if I'm inaccurate.
Real world data has characteristics that a random series won't give you. The fat tails just means that where a normal distribution slopes nicely down to zero price data doesnt trail off so nicely - so you get more prices at the extreme than you would expect. Skew means that instead of being distributed evenly on both sides of the mean (average price) the data will be skewed to one side.
So when you test trading systems you need to use data with these characteristics or you won't see the extreme behaviours you can get in real markets.
John
-
- Roundtable Fellow
- Posts: 81
- Joined: Tue May 13, 2003 12:21 am
Data
Data is the heart of the whole issue for me.
I took statistically acceptable sample sizes* from a wide variety of data, including from different times, commodities, currencies,American and Australian stocks and randomly generated from the spreadsheet.
When I compared them in an unadulterated form they all exhibited specific characteristics, ie, class types, eg, futures, national stocks or random. These classes were correlated in their basic behaviour and how they responded to different statistical tests. There was variation in the way they responded over time, as much as how they differed from others in a similar class. In fact many of a similar class varied in a similar fashion over time. Barring random which performed much the same as the books say it will.
If you accept that data like most things has its own character, then whatever is chosen as your backtesting data will simply result in unconscious curve fitting. We can then produce great results in that frame, but if you switch over to a database of a different character or time frame then the same system produces different results.
Robustness is a fashionable topic, the dictionary calls it "strength, straight foward not confused by subtleties." A tough call, to be sure. The devil is in the detail and the subtle differences in the detail is what affects the outcome. Experience has shown me what suits one data class or given time period may not suit another. Is that robust or is it that it is only robust in that class. If so then chose the class that will be the one used for trading.
* Mendenhall. Introduction to probability and statistics.
________________________________________
Doubts are healthy
I took statistically acceptable sample sizes* from a wide variety of data, including from different times, commodities, currencies,American and Australian stocks and randomly generated from the spreadsheet.
When I compared them in an unadulterated form they all exhibited specific characteristics, ie, class types, eg, futures, national stocks or random. These classes were correlated in their basic behaviour and how they responded to different statistical tests. There was variation in the way they responded over time, as much as how they differed from others in a similar class. In fact many of a similar class varied in a similar fashion over time. Barring random which performed much the same as the books say it will.
If you accept that data like most things has its own character, then whatever is chosen as your backtesting data will simply result in unconscious curve fitting. We can then produce great results in that frame, but if you switch over to a database of a different character or time frame then the same system produces different results.
Robustness is a fashionable topic, the dictionary calls it "strength, straight foward not confused by subtleties." A tough call, to be sure. The devil is in the detail and the subtle differences in the detail is what affects the outcome. Experience has shown me what suits one data class or given time period may not suit another. Is that robust or is it that it is only robust in that class. If so then chose the class that will be the one used for trading.
* Mendenhall. Introduction to probability and statistics.
________________________________________
Doubts are healthy