Anybody using Quad Core Machine for testing

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Post by Roger Rines » Wed Jan 16, 2008 4:28 pm

I doubt there is any way to get around this issue, but it might be possible on a 64-bit OS, maybe.

Windows 2K, XP PRO (32-bit), and Vista (32-bit) will limit the amount program memory available to an application, even when you have 3-Gb of physical memory installed. Window XP PRO (64-bit) & Vista Ultimate (64-bit install) will access all the physical memory, but then if the installed application limits the size of its memory model you might be limited by the application.

A different approach might be to create a first pass screening process were you remove symbols from the testing symbol list for easy to screen rules like, volume is too low, not enough data, etc.

By generating a screening process that creates a testing list of symbols you should be able to reduce the size you are attempting to work with to something possible for the installed abilities of the computer. Another advantage a screened testing list offers is the ability to just test on symbols that have some hope of being included in a trading program, instead of trying to work with segregate symbols lumped into linear sequenced sized-chunks, and then manually attempt to make decisions after the computer stops running its test on each lump of data.

Tim Arnold
Site Admin
Site Admin
Posts: 8540
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Wed Jan 16, 2008 5:16 pm

If you use an x64 machine with 10 GB of memory you should be fine. We have a Trading Blox native x64 version for this purpose.

Rush
Roundtable Fellow
Roundtable Fellow
Posts: 50
Joined: Wed Dec 31, 1969 7:00 pm

database

Post by Rush » Thu Jan 17, 2008 4:56 pm

Roger and Tim thanks for your support.

Roger, even reducing the size of database via a pass screening, I think the main constrain remains the lenght of the database (40 years).

Also, is any way in TB to filter the portfolio instruments by a certain cretiria (e.g. avg 100 daily volume >200k) first in order to create a downsized new portfolio on which you can run the backtest? Should I use a Excel/VBA combo?

Tim could you specify your hardware details.

Would be interesting to know under a 32 bit - 3GB or x64 - 10GB framework which are maximum database sizes in term of number of stocks and number of years.

Portfolio selection is the king in stocks.

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Re: database

Post by Roger Rines » Thu Jan 17, 2008 6:07 pm

Hello Rush,
Rush wrote:Roger, even reducing the size of database via a pass screening, I think the main constrain remains the lenght of the database (40 years).
If you take your 40-years of date records, times 15,000 stocks and attempt to create a maximum short-term memory model, it might look like this:

Code: Select all

Each Stock Record of 6 fields, so it wouldn’t 
be unreasonable to say that each field will 
consume 8-bytes which needs:      8 * 6 = 48 bytes for each symbol record.  

With a symbol’s average year having about 
260 records you now need about:   48 * 260 = 12,480 bytes for each year.  

If all the files are truly 40-years long, which 
is very conservative for memory sizing, you 
now need :                                      40 * 12,480 = 499,200 for each symbol.

That is if you plan to keep it all in memory.  

In a population of 15,000 symbols the computer 
is going to need to have a large stomach to gulp 
that down, so plan on thinking like this:     15,000 * 499,200 = 7,488,000,000 bytes

When you get to this point, Tim’s recommendation 
of 10 GB indicates he has previous experience, or 
is prescient.  Either way 3 GB falls short. 
With the above said, there is a maybe in that pile of words because the OS can do some caching if the memory model of the OS and the application is capable of accessing more than 2 GB, the limit of 32-bit memory models (application and OS). However, disk caching a trade testing process will be about as much fun as watching the grass grow unless you’re a hay harvester.
Rush wrote:Also, is any way in TB to filter the portfolio instruments by a certain cretiria (e.g. avg 100 daily volume >200k) first in order to create a downsized new portfolio on which you can run the backtest? Should I use a Excel/VBA combo?
In simple terms, Yup!

Trading Blox (TB) provides all the tools you’ll need to generate a working portfolio list that you can use in the portfolio block section to load or ignore a symbol. So, if you find a filtering process you like and it can grind the candidate selection down to some reasonable amount of symbols, then the process you would create in TB would be to look at each symbol and test it for compliance to your acceptance rules. For the candidate symbols that pass your testing, TB can write that symbol to disk file as a viable candidate on a first pass, once a month or quarter process or as often as you want to be busy. This file can then be used as your portfolio list for signal testing.

With your accepted process detailed and reasonable size portfolio to chew through, you now have something your current installation can chomp on without gagging on error messages. This is where your trade signal testing process can take over as a second pass process and tell you what it thinks about what your trading logic.

As for using Excel & VBA, you could go that way, but that might be a little painful with something as slow as Excel. Still, if you like a third-party approach, which I do, then consider something like Visual Basic or my favorite, Power Basic. In this business I find my little utilities can make things like your discussing a lot faster and simpler.
Rush wrote:Would be interesting to know under a 32 bit - 3GB or x64 - 10GB framework which are maximum database sizes in term of number of stocks and number of years.
I don’t know about Tim’s hardware other than he has a Win XP Pro 64-bit installation he feeds using Trading Blox’s 64-bit memory model.

What I do know is that I have some work underway here where I’m looking at the problems it will take for me to move my Dual-Xeon workstation (64-bit CPUs) to Win XP PRO 64. So far I’ve ground to a halt trying to find a printer driver for my HP-4000 Laser Printer and my Microtech high-end scanner. I’ve not given up, but instead have purchased a high-end HP laptop that has Vista Ultimate installed for 64-bit operation to slowly work my way in that direction. As of today, I’ve found that ZoneAlarm’s firewall isn’t compatible as well as the other driver disappointments, but a lot of 32-bit programs seem to work fine in Vista Ultimate, if you don’t install them in the default “C:\Program Files\…â€

TEH
Full Member
Full Member
Posts: 11
Joined: Sun Jun 01, 2003 6:12 pm
Location: New York

Post by TEH » Fri Feb 01, 2008 12:18 am

Tim,

With where TB is now and will be in the near future, is it worth the extra money to buy a quad core machine in testing large baskets of stocks or is a dual core machine taking advantage of all the software's speed and functionality?

Thanks

Tim Arnold
Site Admin
Site Admin
Posts: 8540
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Fri Feb 01, 2008 5:59 am

Would be interesting to know under a 32 bit - 3GB or x64 - 10GB framework which are maximum database sizes in term of number of stocks and number of years.
Exact numbers depend on the type of system, how many indicators, how many trades, etc. But I run 6000 stocks for 10 years with just 2GB of memory. I have 6GB in my machine and don't have any issues with a full 7500 stocks testing 20 years.
With where TB is now and will be in the near future, is it worth the extra money to buy a quad core machine in testing large baskets of stocks or is a dual core machine taking advantage of all the software's speed and functionality?
Trading Blox will not use the Quad Core. The fastest machine we see in our speed benchmarks is the new Intel Core2 Duo machines such as the MacBook Pro and others.

Rush
Roundtable Fellow
Roundtable Fellow
Posts: 50
Joined: Wed Dec 31, 1969 7:00 pm

PreFilter Screener

Post by Rush » Sun Feb 03, 2008 7:26 pm

Roger,
thanks for the help and the suggestions.

I am wondering how can i achieve what you said:
... the process you would create in TB would be to look at each symbol and test it for compliance to your acceptance rules. For the candidate symbols that pass your testing, TB can write that symbol to disk file as a viable candidate on a first pass...

1) Does exist a function in TB which ignore doesnot load a symbol for a portfolio?
2) I have to filter for 15k stocks, should I have still memory problems in running the first pass filter?
3) Where and how do you store to disk file a viable symbol to be picked?

I have already tried some pre-filtering in the my database vendor but doesn't work properly.
Could you help me maybe with some pratical example?


Many thanks

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Re: PreFilter Screener

Post by Roger Rines » Mon Feb 04, 2008 6:27 pm

Hello Rush,
Thanks for the kind words.
Rush wrote:[snip]
I am wondering how can i achieve what you said:
... the process you would create in TB would be to look at each symbol and test it for compliance to your acceptance rules. For the candidate symbols that pass your testing, TB can write that symbol to disk file as a viable candidate on a first pass...
1) Does exist a function in TB which ignore doesnot load a symbol for a portfolio?
Yes. TB Uses a File_Set text file to determine which files to load. Take a few minutes and go to the File Set directory for STOCKS SETS:
"<your_drive_letter>:\Program Files\TradingBlox\Stock Sets"

".Set" files are simple text file that will open with NotePad.

Take a few minutes and be sure to read these sections in the Trading Blox Help File (Press F1 when TB appears on the screen).
  • Portfolio File Format
  • The Stock Dictionary
  • Data Folders and Options Preferences
[/b]
Rush wrote: 2) I have to filter for 15k stocks, should I have still memory problems in running the first pass filter?
I don't know because I use a home-made utility to do my filtering here and for creating my ".Set" file lists.
Rush wrote: 3) Where and how do you store to disk file a viable symbol to be picked?
TB has the ability to read a Symbol, and then add that Symbol name to a text file. In this case it would be named with a ".Set" suffix on the file name so that TB would know what you intend.
Rush wrote: I have already tried some pre-filtering in the my database vendor but doesn't work properly.
Could you help me maybe with some pratical example?
OK. Let me see what I can conjure up over the next few days. I'll create a Portfolio Manager that looks at the available symbols, test each symbol against some simple condition and if the symbol data passes the test, it will add that symbols name to a ".Set" file.

Caveat Emptor
One of things my Power Basic application does is to read the Signal Report so that it can keep track of open positions. It also has the ability to run through a set of subdirectories, count the files and then keep track of what it is doing. How to do some of this in TB will be new to me so I'll probably need to do some experimenting. Also, I work with a lot less files than you are talking about so when I find a file I want to use, it is easy to move a copy into my working Stock location.

None of this is impossible with how I approach it, but I might have to do some WISHing to get this working as a total TB solution. Whatever comes of this, it will make for some interesting coding that I'm sure will be of interest to you and others.

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Post by Roger Rines » Wed Feb 06, 2008 4:58 pm

Hello Rush,
I've uploaded a TB Blox that should help to get you started at this forum location: Stock File Set Builder
As for this topic, it would be best if it needs to be continued that the next posting be made in the Trading Blox Support section of the forum where we reference the Blox location using the topic Stock File Set Builder.

Post Reply