Page 1 of 1
Metastock parsing ... in Java
Posted: Thu Dec 18, 2003 4:41 pm
Since I needed this for my own trading system, I'm releasing the following in the public domain.
You need to know Java, as I didn't go the full route of packaging this into a directly executable code, and BTW I can't predict what actual use you'd have for such a code
My test bed, actually executable:
If you succeed in recompiling, or want to use Metastock.jar, then you can invoke the included main using:
Code: Select all
java -cp Metastock.jar com.mac.verec.datafeed.metastock.Main <path>
where <path> is where you have stored your Metastock format data.
For example, if your "MASTER" file path is : "~/Projects/Quotes/MASTER", pass "~/Project/Quotes/" for the path. YMMV according to your OS.
After a couple dozen lines of display, the program stops. You'll find, in the very directory you specified, as many I*.txt files as they are F*.dat files, Those "I" files are in CSV format directly usable by Excel.
Please, dont try to navigate the site elsewhere, you'd only get 404 ;-(
I hope this will be useful to someone
PLEASE, USE THIS CODE AT YOUR OWN RISK. I DON'T INTEND TO SUPPORT IT BEYOND MY OWN NEEDS.
THIS CODE IS 100% MY OWN WORK. I GOT HINTS FROM USING GOOGLE ON "METASTOCK DATA FORMAT", ALL FROM PUBLIC USENET NEWSGROUPS. I SIGNED NO NDA WHATSOEVER WITH EQUIIS/REUTERS REGARDING THIS CODE OR THIS FORMAT. ALL BUGS ARE MINE. ALL TROUBLES ARE YOURS.
Posted: Thu Mar 11, 2004 7:47 pm
Thank You very much for the code... just what i was lookin for
I hope this will work for me...
Posted: Fri Mar 19, 2004 3:10 pm
Well done, Verec! Thanks for sharing this piece of code.
I'd like to invite all Java developers to join this thread. Personally, I am developing using Java on a GNU/Linux box, and would love to get in loose contact with others regarding Java for trading purposes. While everyone might have different ideas and targets, I assume we share some basic problems, questions and solutions. What comes to mind is for example: historical data retrieval, storage and management, charting and graphs, portfolio definition etc.
If I am allowed to dream a bit, we'd come up with some common objects (quote, peergroup, portfolio etc.) and share what be build upon them.
Like, for example, I am currently facing quite a performance problem. I work on a universe of approx. 5500 stocks using EOD data. Although I am caching to the max and make use of threaded data retrieval, it's still a pain in the butt. I need to look into finding a suitable database and then the next to best database structure - I can't believe I am the first having this problem, and would thus love to learn from others.
You might have problems in different areas where I, or others, can step in. Why not join forces for the basics?
Best from France,
Posted: Mon Mar 22, 2004 6:03 am
I'd rather think twice (at least) before using any kind of database, but if that's really (really!) what you want to do, I'd start reading The Prevaylence Java Database
and especially the The Skeptic FAQ
about the kind of performance claims this guy seems to be able to justify.
Version 1.0, the only one I spent some times playing with, was about a few hundred lines of heavily refactored code. A bit hard to wrap your brain around it, but probably worth the try ...
Posted: Tue Nov 30, 2004 2:06 am
I know something about Java and performance.
First of all, performance problems usually trace back to particular bottlenecks which may or may not be the result of your design. Measuring the proper metrics as your application is crucial for understanding what really is going on (are you CPU bound, I/O bound, thread bound, memory bound?). For example, how much RAM are you dedicating to your JVM. And which version JDK are you using - JDK 1.4 at least I hope.
If using a modern JVM like 1.4 or 1.5, then you should be taking advantage of as much RAM as you can afford on your main app server machine. And the database server should probably reside on a different server altogether (one that has very fast I/O). The app server should be dedicated to running the JVM with as much RAM as possible. JDK 1.4 and 1.5 with 64-bit version of the JVM can utilize huge RAM heaps and this can have a significant impact on performance (sometimes good, sometimes bad).
Well, let me know how's it going with your Java code.
Posted: Tue Nov 30, 2004 2:25 am
Just checked out Prevayler.org and I like the concept. I've never been a fan of the OO to RDBMS mismatch. I've tried other OO databases (ObjectStore) in the past and this one does have a nice appeal to it.
Have you used it successfully yourself? Any gotchas?
Posted: Tue Nov 30, 2004 4:52 pm
jdfagan wrote:Have you used it successfully yourself? Any gotchas?
As far as mechanical trading systems go, the answer is no. I don't need the database concept to store and retreive even large market datasets even more so for huge datasets, when I come to think of it
Regarding market data standards you may want to check the site that c.f. started
Regarding Prevalence proper, I only toyed with the sample code and was deeply impressed by the simplicity and, how daring the concept simply was.
Sorry I don't have hard numbers to present.
Posted: Mon Feb 19, 2007 10:59 am
While I realize this thread is rather old, but I was wondering if anyone had figured out how to parse the new metastock format with the XMASTER file. It appears that the MASTER only goes up to the 255 F*.dat and the rest are MWD files.
Posted: Thu Oct 04, 2007 4:23 am
I'm working on an open source project in VB .NET. In the very beginning of my project about a year ago I used some of the java code above to get me started.
I have mapped out the metastock files quite extensively.
The main function of my application is a quote downloader that writes the data into metastock files.
But....It was written in such a way that it has a DLL that contains everything needed to read and write to metastock files.
I have just recently documented a good portion of the DLL API and put together a little bit of a SDK with some basic examples.
Its still a work in progress, but I thought some of the developers here might be interested in either the use of the application or API or maybe helping out with the project.