Historical Data Setup

Data Formats:

Trading Blox supports MetaStock and ASCII text formats.  Sample data is ASCII since it is humanly readable and easier to debug.  

 

Note:

MetaStock formatted files do not include the Unadjusted Close price, or the Delivery Month for Futures data.  They do not allow the use of extra fields to be added, their values imported and accessible to Trading Blox.

 

Comprehensive data setup tutorials:

CSI Data: http://www.tradingblox.com/tradingblox/howto-ua.htm

Worden Data: http://www.tradingblox.com/tradingblox/howto-worden.htm

 


Data File Names:

Metastock

With the Metastock format you don't need to deal with file names. If Trading Blox can read the Metastock Master file, it will locate all the files in the directory. For Futures, be sure that the symbol is correct in the Futures Dictionary.

 

ASCII Text

Futures and Forex:

Futures or Forex Dictionaries require file names assigned to each symbol match the Dictionary symbol, and that it is in the specified folder.

When the file name starts with the symbol letters, you can often find the file using the Futures Dictionary "Find Symbol" button.

 

Stocks:

Stock file specification: ^Top

CSI ASCII/EXCEL Setting:

File Name Example:

<symbol-letters>.<suffix>

Symbol 2015-10-25

AAPL.CSV or AAPL.TXT

<CSI#>.<suffix>

CSI-Number 2015-10-25

5902.TXT

<symbol-letters & CSI#>.<suffix>

Symbol & CSI-Num 2015-10-25

AAPL_5902.TXT

Suffix:

Both ".txt" or ".csv" suffix designators are allowed.

Notes: ^Top

Stock Dictionaries can be automatically updated using the main screen's Tool -> Build Stock Dictionary selection. When a Stock Dictionary is built, it will include all the stock symbol files in the selected "Stock" folder.  It will also include all the stock symbol files that are contained in all the folders that are in the selected "Stock" folder.  To select a stock folder destination that is different from where your stock files are being placed, the

 

When a Stock Dictionary is being built, it will first look to find that symbol, or CSI #, in the Unfair Advantage "sdbfacts.csv" database file.  During the building of the dictionary any symbols that have duplicates in the database file will not be included in the finished dictionary unless those symbols have a CSI # appended to its symbol, i.e. AAPL_5902.CSV.  

 

Symbol file names can have a ".CSV" or ".TXT" suffix identifier.

 

When defining the field columns in the CSI Unfair Advantage software, be sure to use an upper case 'V" for the volume column.  A Lower case "v" is intended to Futures files that are intended to use the contract's volume.  An upper case "V" for volume in a Futures' file selects the total volume value created by adding the volume values from all of the active contracts on a record date.

 


ASCII Data Files: ^Top

ASCII formatted files must be comma delimited. Dates must be in sequential order where the earliest date information is the first record, and the latest date  is the last record in the file.

 

Daily Data Columns & Price Adjustments:

All fields must be set up as follows. A header record is optional for daily data.

 

Instrument Type: ^Top

Daily Data Columns

Futures: ^Top

YYYYMMDD, Open, High, Low, Close, Volume, Open Interest, Delivery Month (YYYYMM), Unadjusted Close, Extra Data 1 ... 8

 

It is recommended that Futures contracts be created using the Back-Adjusted process.  Futures Back-Adjustments removes the price gap between expiration months on the date of when the expiring contract is added to the next forward contract.  

 

Roll calculations can use the Delivery Month information when that column is added after the Open Interest column.  Price fields of Open, High, Low and Close are the prices that get changed during a price adjustment process.  

 

When prices are back-adjusted, the date where prices are adjusted will be different than they were on that date when the prices were originally active in the exchange.  This means adjusted price values on previous dates after an adjustment will not create the same calculation result unless the price adjustment is removed when that date's adjustment price is applied to a calculation.  This reason is why we recommend the data file contain the UnAdjusted Close price column.  This price column will have the actual price for each date available and it can be used to discover the actual price for each of the four other price fields.

Forex: ^Top

YYYYMMDD, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8

 

Forex files don't use price adjustments.  All Forex records are assumed to be accurate transaction values for each record date.  In addition, Forex files should be updated daily so the current exchange rate information is available during testing.

Stocks: ^Top

YYYYMMDD, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8

 

Stock Splits:

Most stock files over time experience stock splits.  Each Stock split changes the price and the number of outstanding shares.  Stocks that have Dividends also affect the value of a share.   When a stock is split it will show a gap in price that isn't suitable to computer based calculations.  This failure happens because the change in number of shares that changes the price, isn't a change that was caused by an positive or adverse economic view of the company's stock.

 

In order for computer based calculations to be kept accurate, and provide transactions that  reflect the economic conditions of how a stock's is viewed, stock split calculation should adjust the prices in the file so the change in shares is reflected in the price.  This is important because if the change in price is not adjusted in the data, it is likely to cause testing to create an order transaction where it should not happen.  False order events like this can be avoided by back-adjusting the change in price on the date the change shares takes place.  

 

When prices are back-adjusted, the date where prices are adjusted will be different than they were on that date when the prices were originally active in the exchange.  This means adjusted price values on previous dates after an adjustment will not create the same calculation results unless the price adjustment is removed when that date's adjustment price is applied to a calculation.  Price adjustments can be removed in testing by including the unadjusted close price in the data file.  CSI data provides this price when the file creation settings in their software adds UnAdjustedClose price column to the data file (see the stock price fields recommended above).

 

When a Stocks or Futures file contains the data column UnAdjustedClose, that field value can be used to remove the price adjustment so the calculation result will be the same as it was on the date the exchange created that price record.  When it is available in the data, that price in a price adjusted data file can be used to calculate the split ratio and the split ratio direction.   By knowing the split ration the basis for adjusting the price becomes available and the transactions that happen during a period where the price had to be adjusted, will be based upon how it would have at that time when there was no price adjustment.

 

Dividends:

When a dividend is added to a stock it increases the price of the stock.  

Trading Blox can use CSI dividend information to build dividend files for all stocks that have dividends.  It builds dividend files that are visible in its Portfolio Editor's Instrument Symbol list section.

 

Trading Blox Stock File Search Structure Process:

Stocks are the only instruments that can have multiple dictionaries when it comes time to locate stock symbols.  This means that when Trading Blox starts, it looks for files using the path settings established in the Preferences Data and Folder section.  In each folder and each symbol it finds it will add it to a list of symbols that it will use to determine if it can be in a Stock Dictionary, or when a stock dictionary is not used, it will be displayed in the Portfolio Editor's Instrument Symbol list section.

 

In this next set of images, the path is set to D:\Data\Stocks.  That path has five sub-folders and a large number of symbol files.  All the files in all the folders, plus all the symbols in the Stock Set's portfolio files will be record so they can be used in a Stock Dictionary, when that is the process being created, or as a list of symbols in the Portfolio Editor's Instrument Symbol list section.

Example Stock Folder and Files Structure Example

Example Stock Folder and Files Structure Example

 

Stock Set Default Stock Portfolio Example

Stock Set Default Stock Portfolio Example

All the files and folders in all the places shown in the example above, which includes the symbols listed in the Stock Set Portfolio files, will be listed in the Instrument Symbol list section when a Stock Dictionary is not used.  When a dictionary is being assembled, the same search structure will be used to locate symbols.

 

If you do not have a Stock Dictionary (stockinfo.txt or stockinfo.csv file), it will look for a Metastock MASTER file with that name.   If it finds the MASTER file, it will assume Metastock data and use that format process to read the data.  If it does not find a MASTER file, it will assume ASCII Text files and look for all the .txt or .csv files in the specified folder, and all the folders contained with the top level Stock folder.   Once all the folders have been searched, it will then search all the Stock set portfolio files and collect the symbols in those files.

 

When there isn't a stock dictionary file, and the above symbol information is done its symbol search, Trading Blox will create a list of symbols from the file names and set files and display that list in the Stock option of the Portfolio Editor symbol listing.  

 

If a symbol is listed in a Stock file portfolio file, and there is no data file with the same name, that item in the list will be unable to load that symbol name information.  Trading Blox  flags portfolio symbols that don't have data files when it is first started and creates a warning message in the main screen's Log Window display area when the Log Window is enabled.

Portfolio Editor Symbols: ^Top

Description:

Stock Dictionary Available

Stock dictionaries control which symbols are listed in the Portfolio Editor's Instrument list section.  When new symbols are added, and the Stock Dictionary is rebuilt, the new dictionary file will include the newly added symbols.  When files are removed, and the dictionary is rebuilt, the deleted symbols will no longer be included in the Instrument Symbol list section.

 

In simple terms, when new symbols are added or removed from the computer, or the Stock folders list of folders and symbols, the dictionary should be rebuilt.

Stock Dictionary - Not Available

When a stock dictionary is not used, all the symbols in the structure of the Stock folder's list of folders and symbols will be displayed in the Instrument Symbol list section.

 

and are in your Stock Dictionary. The stock dividend data from CSI is encrypted by default. To use this feature you must contact CSI and make arrangements to have this data unencrypted for your subscription account. This is an additional cost service from them. Alternately you can build your own dividend files manually from another source for each stock and place in the Dividend Folder. The format is Date,Dividend and you can see example files in the sample data that ships with the product.

NOTE: ^Top

All fields after the Date, Open, High, Low, Close (DOHLC) are optional.  Files that only contains DOHLC are fine, but some features of Trading Blox will not be effective when the information, like "Delivery Month", or "UnAdjustedClose" are not in the file.

 

 

Adding Data Columns: ^Top

When a need to have more data columns for in a file is required, an additional 8-columns of information can be added to each date record after the file has been updated by your data service.

 

Trading Blox can provides access to additional data columns.  However, access to additional data columns might require additional commas ahead of the extra data so that the information expected by Trading Blox isn't in the wrong place.

 

Futures information will only access the extra column information after the first 9-columns of specified information is found.  Stock and Forex files will only access the extra column information after the first 7-columns of specified information is found.  If you are going to add extra columns to a Futures file, but don't include columns for Open Interest and Delivery Month, the file structure will will need to look like this:

YYYYMMDD, Open, High, Low, Close, Volume, Open Interest, , , Extra Data 1 ... 8

 

When this requirement is followed, Trading Blox will be able to access data in the extra columns because the Futures record found the first nine commas, and could then read the extra column information into the extra column properties for each date record in the file.

 

CSI Data Updating Service: ^Top

CSI's UnFairAdvantage Updating service will support all the data type requirements shown.  

 

When an UnFairAdvantage instrument portfolio of symbols is created for downloading each day, the following images of UnFairAdvantage's ASCII/Excel Fields will populate the data records with the above Trading Blox data requirements.  Less columns can be used when Extra-Data columns are not going to be added after the file updating has completed.

 

CSI's UnFairAdvantage Portfolio Data Letter Requirements: ^Top

Futures:

Forex:

Stocks:

Click to Enlarge, Click to Reduce

CSI's Futures data column letters.

Click to Enlarge, Click to Reduce

CSI's Forex data column letters. 

Click to Enlarge, Click to Reduce

CSI's Stock-Type data column letters. 

 


Intraday Data Columns:

For intraday data, a header must be used with at least the first two items "Date,Time", such as:

 

Date,Time,Open,High,Low,Close,Volume

 

Thereafter the format of the data is: ^Top

Futures:

YYYYMMDD, HHMM, Open, High, Low, Close, Volume, Open Interest, Delivery Month (YYYYMM), Unadjusted Close, Extra Data 1 ... 8

 

Stocks and Forex:

YYYYMMDD, HHMM, Open, High, Low, Close, Volume, Unadjusted Close, Extra Data 1 ... 8

 

Trading Blox accepts the following optional date formats: YYYYMMDD, YYMMDD, MM/DD/YYYY, MM/DD/YY.

Trading Blox accepts the following optional time formats: HHMM, HH:MM.

 

 


Data Dictionaries: ^Top

In order for Trading Blox to read your data, there needs to be a dictionary file for Futures and Forex data. This Dictionary file can be in the default Trading Blox Data/Dictionary folder, or it can be in the same folder as the data. These Dictionary files contains the market specific symbol information for each symbol.

 

If you don't have a Stock Dictionary in your stock data folder, Trading Blox will look for a Master file. If found, it will assume Metastock format, and if not found it will assume Text format. If text format, it will load all the files ending in ".txt" or ".CSV' into Trading Blox. When no Stock Dictionary is used the

following columns displayed in a Stock Portfolio display will not show

 

Stock files with a Dictionary:

Click to Enlarge, Click to Reduce

Stock Dictionary Data Information

 

Stock files without a Dictionary: ^Top

Click to Enlarge, Click to Reduce

Stock Data Information without a Dictionary.

 

If you don't have a Futures, or a Forex Dictionary available, Trading Blox will display data error information in Log Window showing no files could be found.  

 

When a Dictionary is listed, but one or more of the symbols listed in the selected portfolio are not listed in the Dictionary, an error record will be displayed in the main screen's Log Window area:  

Click to Enlarge, Click to Reduce

Log Window Menu Controls

Missing symbol information will show symbols selected in the portfolio, but not found in the dictionary.  Any portfolio selected symbols not included in a Dictionary, when a Dictionay is present, will be shown near the top of the portfolio list without any of the usual Dictionary information.  Symbols listed at the top that appear like what is shown next, will not be loaded and available during testing:

 

 

Data Dictionary File Names: ^Top

Stock:      StockInfo.txt

Futures: FuturesInfo.txt

Forex:     ForexInfo.txt.  

 

See Data and Folder Preferences to change data's location folders.

 


Interest Rates Files: ^Top

The rates files must be located in the Forex Rates folder, which is in the Data folder. Rate files are required for Forex cost of carry, and for the system wide base currency only, earning interest and paying margin. For example, if the base currency is USD, the USD_Rates.txt file will be used for earning interest on excess cash as well as paying margin on stocks on a negative cash balance. The Lending Rate is used when earning interest, and the Borrowing Rate is used for paying margin.

 

The file can be sparsely populated, and for any given test date, the last available date in the file will be used for the rates. The format for this file is as follows: Date, Lending Rate, Borrowing Rate. If only one rate is in the record, it is the Lending Rate, and the Borrowing Rate is the Lending Rate plus .5%. Note that the concepts of Borrowing and Lending are from the consumer's perspective.

 

All rate files use a name like "XXX_Rates.txt" or "XXX_Rates.csv". All rate files must be in the Forex Rates folder in the Data folder. The characters "XXX" to the left of "_Rates.txt" part in the example above is the ISO currency code required to find the correct rate file so the use of the information for calculation like computing cost of carry will be accurate. Be sure to include all required ISO currency codes in the ISOCurrencies.csv file location in the Dictionaries folder.

 

Example Forex Rate File:

20081008,0.015,0.0205

20081029,0.010,0.0155

20081031,0.0045,0.0155

20081216,0.0015,0.0125

 

Column Descriptions:

Date

Lending Rate

Borrowing Rate

20081008

0.015

0.0205

20081029

0.010

0.0155

20081031

0.0045

0.0155

20081216

0.0015

0.0125

 


Exchange Definition File: ^Top

The exchanges.csv file contains the abbreviation and description of all the exchanges that are used in Trading Blox. A warning will appear if the exchange is used by a  market, but missing from this file. To add an exchange to this file simply open the file in excel or notepad, add the exchange and description, and save. Restart Trading Blox.

 

Or you can press the Examine button next to the Exchange dropdown in the Futures Dictionary. The exchanges are the same for futures and stocks, so this will work for both. The exchange.csv file is located in the Dictionaries folder, as defined in preferences, default is Trading Blox/Data/Dictionaries.

 

This is a user maintained file.

 


Currency Definition File: ^Top

The ISOCurrency.csv file is located in the Data/Dictionaries folder and contains all the currencies that can be used by Trading Blox. The file has a version header, column header, and record format is comma delimited.

 

Version,7

Currency Code,Currency Description

ADF,Andorran Franc

ADP,Andorran Peseta

AED,United Arab Emirates Dirham

AFA,Afghanistan Afghani

ALL,Albanian Lek

 

This is a user maintained file.


Edit Time: 9/20/2017 07:56:26 AM


Topic ID#: 155

 

Created with Help & Manual 7 and styled with Premium Pack Version 2.80 © by EC Software