Benchmark Test Data

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Benchmark Test Data

Post by Forum Mgmnt » Wed Apr 23, 2003 6:24 pm

I'd like us to come up with a standard benchmark set of test data which we could use for the purposes of making comparisons between different systems/approaches. This data will also be used for the Trading System Contest.

Does anyone have any suggestions for what data would be a good choice (source, markets, years)? It would obviously have to be data that comes from a provider that doesn't prohibit redistribution of daily data or that offers the data for free.

Unfortunately, the data I have can't be redisributed.

damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian » Thu Apr 24, 2003 12:28 am

Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Thu Apr 24, 2003 2:51 am

Another thought(s).

For commodities EOD would CSI, say, allow release of a subset (20 commodities backadjusted from 1980-1999 only) along with acknowledgement of where it came from.

For stocks & indexes EOD everyone could be referred to yahoo.

For intraday there are a number of samples on the web but perhaps someone would permit a subset to be made available with acknowledgement?

bloom
Senior Member
Senior Member
Posts: 35
Joined: Thu Apr 17, 2003 12:45 am
Location: SARS infested HONG KONG..ahh

Post by bloom » Thu Apr 24, 2003 3:19 am

I like CSI and I think a lot of people use CSI.

By the way, Forum Mgmnt, Can Veritrader be used with intraday data too?

TradingCoach
Roundtable Knight
Roundtable Knight
Posts: 176
Joined: Thu Apr 17, 2003 9:52 am
Location: Sacramento, CA
Contact:

I use TS 2000i GlobalServer (open system)

Post by TradingCoach » Thu Apr 24, 2003 6:57 am

You can access and update 30 years worth of stock and commodity data
from the history bank it has C++ and VB API.
You can also get real time quotes into the GlobalServer via 4-5 vendors.
Just an idea....

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Tue Apr 29, 2003 3:42 am

I just visited Mark Johnson's site and he has a backadjusted data set for 20 commodities at:

http://www.mjohnson.com/rescap_lecture/lectop.htm

Dutchtrader
Roundtable Fellow
Roundtable Fellow
Posts: 58
Joined: Wed Apr 30, 2003 4:35 pm
Location: Netherlands

CSI as benchmark

Post by Dutchtrader » Tue May 06, 2003 6:06 pm

CSI proofed to be the best. So that must be the benchmark> most logical I think personally. It gives better insights what the differences are in testing results with different data providers.

Marc

wcb
Contributing Member
Contributing Member
Posts: 8
Joined: Wed Apr 16, 2003 10:30 pm

Post by wcb » Tue Jul 08, 2003 7:33 pm

I would like to see CSI data adopted for the standard data set. I think the 20 commodities listed on Mark Johnson's site would be a good place to start. Another good market group would be the 25 markets that make up the MLM and CMI. ( http://www.mtlucas.com/MLM_const.htm )

The next step would be to fully specify the CSI roll timing settings for each market. As shown in the attached image.
Attachments
CSI.jpg
CSI.jpg (76.13 KiB) Viewed 10953 times

damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian » Tue Jul 08, 2003 8:03 pm

The CSI back adjusting options annoy me. If you stipulate roll on OI, it will do so... and then jump between contracts depending on OI. Say you roll from June03 to Dec03 on OI and then come August the Oct03 contract is trading at higher OI. In this case the continous contract will stop using Dec03 data and start using Oct03 data on the OI basis. As a trend follower, I do not trade like this.

No back adjusting engine is of any use to me until it let me say "roll into the most distant contract that has the required liquidity".
Last edited by damian on Tue Jul 08, 2003 10:49 pm, edited 1 time in total.

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Tue Jul 08, 2003 10:28 pm

It used to do this but (i think last year) CSI changed to hold the later contract even if the OI in an earlier one popped up. I checked with Rudi and he confirmed that they had changed because most traders preferred this approach.

John

damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian » Tue Jul 08, 2003 10:51 pm

thanks john.

this goes to show that I should read that stuff they send me in the mail rather than complaining on an internet forum :wink:

steady_jake
Full Member
Full Member
Posts: 22
Joined: Fri Jun 27, 2003 8:05 pm
Location: Warsaw, Poland
Contact:

Post by steady_jake » Wed Jul 09, 2003 4:41 pm

damian wrote:Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?
Just throw in fatter tails and skewed distribution. And whatever you may say about prices in general. That would be a "statistical" benchmark :wink: .

Jakub

gfullmer
Full Member
Full Member
Posts: 18
Joined: Sun Jul 06, 2003 5:39 pm
Location: Phoenix, AZ
Contact:

Post by gfullmer » Sat Jul 12, 2003 10:30 am

steady_jake wrote:
damian wrote:Dare I say it.... don't throw too many stones at me.... but could we use a randomly generated times series?
Just throw in fatter tails and skewed distribution. And whatever you may say about prices in general. That would be a "statistical" benchmark :wink: .

Jakub
If you just randomized the selection of the block of time series - I mean they have to be contiguous, of course, isn't that a valid test?

Glen

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Sat Jul 12, 2003 8:30 pm

Glen,

It is a valid test ... but only of the series you have created. I'm not a statistician so someone correct me if I'm inaccurate.

Real world data has characteristics that a random series won't give you. The fat tails just means that where a normal distribution slopes nicely down to zero price data doesnt trail off so nicely - so you get more prices at the extreme than you would expect. Skew means that instead of being distributed evenly on both sides of the mean (average price) the data will be skewed to one side.

So when you test trading systems you need to use data with these characteristics or you won't see the extreme behaviours you can get in real markets.

John

DrHendricks
Roundtable Fellow
Roundtable Fellow
Posts: 81
Joined: Tue May 13, 2003 12:21 am

Post by DrHendricks » Sat Jul 12, 2003 10:26 pm

I agree with John. Moreover, how could meaningful random data be generated from what is an inherently unknowable distribution. Sure- it's "similar" to lognormal, stable Pareto,etc. but is "similar" generated data any better than the actual sample of historical data.

jimsta
Full Member
Full Member
Posts: 13
Joined: Sun Jun 15, 2003 2:45 am

Data

Post by jimsta » Wed Jul 23, 2003 6:31 am

Data is the heart of the whole issue for me.
I took statistically acceptable sample sizes* from a wide variety of data, including from different times, commodities, currencies,American and Australian stocks and randomly generated from the spreadsheet.
When I compared them in an unadulterated form they all exhibited specific characteristics, ie, class types, eg, futures, national stocks or random. These classes were correlated in their basic behaviour and how they responded to different statistical tests. There was variation in the way they responded over time, as much as how they differed from others in a similar class. In fact many of a similar class varied in a similar fashion over time. Barring random which performed much the same as the books say it will.
If you accept that data like most things has its own character, then whatever is chosen as your backtesting data will simply result in unconscious curve fitting. We can then produce great results in that frame, but if you switch over to a database of a different character or time frame then the same system produces different results.
Robustness is a fashionable topic, the dictionary calls it "strength, straight foward not confused by subtleties." A tough call, to be sure. The devil is in the detail and the subtle differences in the detail is what affects the outcome. Experience has shown me what suits one data class or given time period may not suit another. Is that robust or is it that it is only robust in that class. If so then chose the class that will be the one used for trading.

* Mendenhall. Introduction to probability and statistics.
________________________________________
Doubts are healthy

Post Reply