Metastock parsing ... in Java

Discussions about custom-built testing platforms written in C, C++, or Java.
Post Reply
verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Metastock parsing ... in Java

Post by verec » Thu Dec 18, 2003 4:41 pm

Since I needed this for my own trading system, I'm releasing the following in the public domain.

You need to know Java, as I didn't go the full route of packaging this into a directly executable code, and BTW I can't predict what actual use you'd have for such a code :)

The source:
http://www.jfb-city.co.uk/code/Metastock_Reader.zip

My test bed, actually executable:
http://www.jfb-city.co.uk/code/Metastock.jar

If you succeed in recompiling, or want to use Metastock.jar, then you can invoke the included main using:

Code: Select all

java -cp Metastock.jar com.mac.verec.datafeed.metastock.Main <path>
where <path> is where you have stored your Metastock format data.

For example, if your "MASTER" file path is : "~/Projects/Quotes/MASTER", pass "~/Project/Quotes/" for the path. YMMV according to your OS.

After a couple dozen lines of display, the program stops. You'll find, in the very directory you specified, as many I*.txt files as they are F*.dat files, Those "I" files are in CSV format directly usable by Excel.


Please, dont try to navigate the site elsewhere, you'd only get 404 ;-(

I hope this will be useful to someone ;-)

DISCLAIMER:

PLEASE, USE THIS CODE AT YOUR OWN RISK. I DON'T INTEND TO SUPPORT IT BEYOND MY OWN NEEDS.

THIS CODE IS 100% MY OWN WORK. I GOT HINTS FROM USING GOOGLE ON "METASTOCK DATA FORMAT", ALL FROM PUBLIC USENET NEWSGROUPS. I SIGNED NO NDA WHATSOEVER WITH EQUIIS/REUTERS REGARDING THIS CODE OR THIS FORMAT. ALL BUGS ARE MINE. ALL TROUBLES ARE YOURS.

8)
Last edited by verec on Sat Mar 20, 2004 6:57 pm, edited 1 time in total.

jagui
Contributor
Contributor
Posts: 1
Joined: Thu Mar 11, 2004 7:43 pm

Post by jagui » Thu Mar 11, 2004 7:47 pm

Thank You very much for the code... just what i was lookin for :P

I hope this will work for me...

Ted
Contributing Member
Contributing Member
Posts: 9
Joined: Fri Mar 19, 2004 2:39 pm
Location: Toulouse - France

Post by Ted » Fri Mar 19, 2004 3:10 pm

Well done, Verec! Thanks for sharing this piece of code.

I'd like to invite all Java developers to join this thread. Personally, I am developing using Java on a GNU/Linux box, and would love to get in loose contact with others regarding Java for trading purposes. While everyone might have different ideas and targets, I assume we share some basic problems, questions and solutions. What comes to mind is for example: historical data retrieval, storage and management, charting and graphs, portfolio definition etc.

If I am allowed to dream a bit, we'd come up with some common objects (quote, peergroup, portfolio etc.) and share what be build upon them.

Like, for example, I am currently facing quite a performance problem. I work on a universe of approx. 5500 stocks using EOD data. Although I am caching to the max and make use of threaded data retrieval, it's still a pain in the butt. I need to look into finding a suitable database and then the next to best database structure - I can't believe I am the first having this problem, and would thus love to learn from others.

You might have problems in different areas where I, or others, can step in. Why not join forces for the basics?

Best from France,
Ted

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Mon Mar 22, 2004 6:03 am

I'd rather think twice (at least) before using any kind of database, but if that's really (really!) what you want to do, I'd start reading The Prevaylence Java Database and especially the The Skeptic FAQ about the kind of performance claims this guy seems to be able to justify.

Version 1.0, the only one I spent some times playing with, was about a few hundred lines of heavily refactored code. A bit hard to wrap your brain around it, but probably worth the try ...

jdfagan
Contributing Member
Contributing Member
Posts: 5
Joined: Fri Nov 26, 2004 7:04 pm
Location: San Francisco, CA

Post by jdfagan » Tue Nov 30, 2004 2:06 am

I know something about Java and performance. :D

First of all, performance problems usually trace back to particular bottlenecks which may or may not be the result of your design. Measuring the proper metrics as your application is crucial for understanding what really is going on (are you CPU bound, I/O bound, thread bound, memory bound?). For example, how much RAM are you dedicating to your JVM. And which version JDK are you using - JDK 1.4 at least I hope. :o

If using a modern JVM like 1.4 or 1.5, then you should be taking advantage of as much RAM as you can afford on your main app server machine. And the database server should probably reside on a different server altogether (one that has very fast I/O). The app server should be dedicated to running the JVM with as much RAM as possible. JDK 1.4 and 1.5 with 64-bit version of the JVM can utilize huge RAM heaps and this can have a significant impact on performance (sometimes good, sometimes bad).

Well, let me know how's it going with your Java code.

JD

jdfagan
Contributing Member
Contributing Member
Posts: 5
Joined: Fri Nov 26, 2004 7:04 pm
Location: San Francisco, CA

Post by jdfagan » Tue Nov 30, 2004 2:25 am

Verec,

Just checked out Prevayler.org and I like the concept. I've never been a fan of the OO to RDBMS mismatch. I've tried other OO databases (ObjectStore) in the past and this one does have a nice appeal to it. :D

Have you used it successfully yourself? Any gotchas?

JD

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Tue Nov 30, 2004 4:52 pm

jdfagan wrote:Have you used it successfully yourself? Any gotchas?
As far as mechanical trading systems go, the answer is no. I don't need the database concept to store and retreive even large market datasets even more so for huge datasets, when I come to think of it :idea:

Regarding market data standards you may want to check the site that c.f. started at http://www.hr-xml.org.

Regarding Prevalence proper, I only toyed with the sample code and was deeply impressed by the simplicity and, how daring the concept simply was.

Sorry I don't have hard numbers to present.

chicagodev
Contributor
Contributor
Posts: 1
Joined: Mon Feb 19, 2007 10:47 am

Post by chicagodev » Mon Feb 19, 2007 10:59 am

While I realize this thread is rather old, but I was wondering if anyone had figured out how to parse the new metastock format with the XMASTER file. It appears that the MASTER only goes up to the 255 F*.dat and the rest are MWD files.

icecubetray
Contributor
Contributor
Posts: 1
Joined: Wed Oct 03, 2007 9:54 am

Post by icecubetray » Thu Oct 04, 2007 4:23 am

Hey guys/gals.

I'm working on an open source project in VB .NET. In the very beginning of my project about a year ago I used some of the java code above to get me started.

I have mapped out the metastock files quite extensively.

http://meta-all.sourceforge.net/

The main function of my application is a quote downloader that writes the data into metastock files.

But....It was written in such a way that it has a DLL that contains everything needed to read and write to metastock files.

I have just recently documented a good portion of the DLL API and put together a little bit of a SDK with some basic examples.

Its still a work in progress, but I thought some of the developers here might be interested in either the use of the application or API or maybe helping out with the project.

Cheers!

:)

Post Reply