Trading markets that are electronic & pit

General discussions about futures.
Post Reply
ladidalimey
Roundtable Knight
Roundtable Knight
Posts: 197
Joined: Thu Sep 21, 2006 6:21 am
Location: London

Trading markets that are electronic & pit

Post by ladidalimey » Tue Oct 21, 2008 8:06 am

Hi,

I'm papertrading a strategy that trades futures markets that are trading both electronic and pit (e.g. Soybean Oil) during RTH.

I test using Pinnacle data which I assume is pit. I trade using Interactive Brokers and was hoping to trade electronic.

I have noticed that the electronic data is slightly more volatile than pit and can reach higher highs and lower lows during RTH. My problem is that I'm using a strategy that enters on a stop entry so it is possible that the electronic price could trigger, but the pit does not and so I get a mismatch of results.

Ideally I would like to test with daily bars consisting of electronic data for RTH only, but the providers I've looked at give daily bars for the whole elec. session.

Can anyone give me some advice for this?


Many thanks


L. :?

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Tue Oct 21, 2008 1:14 pm

Welcome to the swamp. Data is in flux at the moment. Think I recall CSI saying they planned a day session only electronic at some point. What I did was test all the options and decide which compromise worked best for my systems characteristics. No one size fits all answer. Skid overnight can be larger, but waiting till the next morning can result in even bigger 'skid' (gap). Some have suggested building daily bars from intraday data as one sees fit. This is cumbersome and error prone as I see it currently. There are few pits which still trade more volume than Globex. The meats for example. So the issue is becoming an easier or no choice decision.

Mats
Roundtable Knight
Roundtable Knight
Posts: 133
Joined: Thu Feb 28, 2008 2:59 am
Location: Stockholm

Another comment from the swamp..

Post by Mats » Wed Oct 22, 2008 4:12 am

As the former speaker said, welcome to the swamp. I use Interactive Brokers myself and you can set the order to fill/ or not to fill outside RTH.

I rarely use the fill outside RTH option as my experience is that i do not want the fills outside RTH. If the trend is strong i get the fill a day later or so anyway. Those sharp rise/declines rarely give my any winners.

However you must see that your strategy fits your system,I myself use a trend following strategy with low risk and this match my system best.

Good luck!

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Oct 22, 2008 8:17 am

You've identified one problem that can happen when you run your trading system software using pit data, but trade it in the electronic market: stop orders (and/or limit orders) can be filled in one but not filled in the other.

There is a second problem: trading system calculations (indicators and so on) sometimes use the high and the low. If the high or low prices are different between pit data and electronic market data, your system's calculations don't match the market data where it is being traded. The simplest example is the ATR or Average True Range calculation. If your stop is 2.0*ATR from entry, you need the correct value of ATR in order to place your stop. If ATR is too small (as it will be if you calculate ATR using pit data), your stop will be too close, and sometimes you'll be stopped out of a trade when you shouldn't be. This can even happen on a day when the pit data perfectly matches the electronic data, OHLC are all identical. How can this be? Because ATRs weren't equal, because prior days' highs and lows were different. Ugh.

Indicators that use high and low data include Stochastics, ADX, Parabolic, anything that employs True Range or ATR, etc.

What can you DO about it? There are several possibilities, but none are terribly appealing
  1. Decide to live with it (perhaps after lots of what-if testing). Accept that you will sometimes get different trades or different fills in your account, than in your software. Each time this happens in real-life trading, deal with it manually, on a case-by-case basis.
  2. Change your system so it doesn't use the high or low in any of its calculations and indicators. Don't use stop orders or limit orders. Now you and your system don't care about any prices except the open and/or the close.
  3. Find a data vendor who will supply you with electronic market data for the specific trading-times you actually trade. (Or, change the times you actually trade (via Interactive Brokers software setup?) to match the specific trading-times of your electronic market data).
  4. Become your own data vendor. Obtain 24 hour electronic session intraday data (5 minute bars?) and use it to build your own custom daily bars that match your own specific trading-times. You can do this for "data to trade with" (i.e. the present and the future), and you can also do it for "data to test with" (i.e. past history), for backtesting. Beware, historical intraday data isn't cheap, especially if you buy lots of it for lots of instruments. One piece of good news is that "data to trade with" need not be the super expensive realtime option. The far cheaper 20 minute delayed data (or even 60 minute delayed) is just fine, if all you want to do is construct daily bars after tonight's close, then generate signals before tomorrow's open.
Personally, I find option 3 to be the worst of the bunch. Who knows how long the world will have to wait, before such a vendor emerges? What is the opportunity cost of not trading while I wait? And how do I know that the first such vendor's infrastructure and data will be reliable, accurate, and worth risking my capital upon?

My own preference is to pursue a combination of the other options. Note that 1&2 are not incompatible if you trade more than one system. Note that 1&4 are not incompatible if you have a "time of transition" where you are doing a live trial (like papertrading except with real trades and real money). You might recoil in horror, feeling that only a stoic, a hair-shirt Minnesota libertarian, would prefer such a bothersome and annoying approach. To each their own.

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Oct 22, 2008 11:47 am

sluggo wrote:Become your own data vendor. Obtain 24 hour electronic session intraday data (5 minute bars?) and use it to build your own custom daily bars that match your own specific trading-times. You can do this for "data to trade with" (i.e. the present and the future), and you can also do it for "data to test with" (i.e. past history), for backtesting. Beware, historical intraday data isn't cheap, especially if you buy lots of it for lots of instruments. One piece of good news is that "data to trade with" need not be the super expensive realtime option. The far cheaper 20 minute delayed data (or even 60 minute delayed) is just fine, if all you want to do is construct daily bars after tonight's close, then generate signals before tomorrow's open.
Ive found such data , at least in the past, to be prone to spike errors intraday from a bad quote broadcast or other glitch (an offer transmitted with no trade, seen as the high). Monitoring for such errors on a broad portfolio on a daily basis seems a daunting task without staff. Even for a stoic north-country libertarian.

BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI » Wed Oct 22, 2008 4:03 pm

hi guys, I've also been thinking about it a lot lately. Since Cotton, Coffee and other markets gone electronic and trade 24/day there's no way but adapt I think. Most important is where markets trade, if its Cotton it'll have thin volume and no volatility during night hours of US, when american traders wake up Cotton market gets the action, same with currencies(local central banks) and other markets that depend locally. Also as you said that there are gaps in day sessions caused by night movement of the market, gaps are not good for backtesting, smoother data seems to be better. As I see it, any system will adapt better to electronic than day sessions these days, better fills(80% of the business is electronic today) and so forth. I didn't believe when Kiwi predicted that pits will disappear. They were right.

Mats
Roundtable Knight
Roundtable Knight
Posts: 133
Joined: Thu Feb 28, 2008 2:59 am
Location: Stockholm

The bad thing is...

Post by Mats » Thu Oct 23, 2008 4:24 am

As the previous comment shows that then the market gets electronic on a lot of spikes outside the retail trading hours (RTH) appears.

However even if the trading is 24 hour electronic the volume exists in the old RTH = old pit hours.

For my self i prefer to live the the that sometimes my simulations do not match my trading, use RTH (pit data) for my simulations, often there is no other data 20 years back in time.

I use interactive Brokers to set that i only trade in RTH hours. My belief is that those off hours spikes does not give me the trends. If there is a sustainable trend i get the fills later, ok i have to pay in with slippage.

Comments?

AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Thu Oct 23, 2008 4:33 am

One thing which I do find hateful with a product like MF Global's MTrade is trying to figure out the depth and liquidity of the market at the time of day I want to trade. This has been worsened by the advent of electronic markets.

With a market like Hogs, you are no longer given the option of putting in a MOO order secure in the knowledge that it will be executed when the pit opens and the liquidity is there. You have to deal electronic. So you have to wait, physically, until Chicago opens.

So you either have to call the dealing desk and ask the size of the bid and offer at various levels or wait until the relevant dealing centre wakes up for business or hazard a guess at volumes and depth by looking at the total volume for the session in MTrade.

Given the number of markets I trade paying exchange fees for each market is not an attractive option. Some exchange websites give live prices and volume of course but it is a bit hit and miss.

Perhaps it is simpler to dump the whole problem on the broker and let them do the dealing at the cost of higher ticket charges.

Any other options? And no, I would not want to use IB.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Thu Oct 23, 2008 9:11 am

Perhaps it might be possible to find, for each market, certain times of day "T1" and "T2", such that if you changed "Market On Open" orders into "Market at time T1" orders, and if you also changed "Market On Close" orders into "Market at time T2" orders, you would be approximately pleased. Just as some traders are approximately pleased to roll over from one contract to the next on a fixed calendar schedule.

You might be able to find these times T1 and T2, for each market, by studying 5-minute bars for the last N years (I might suggest N=3 but if you're the one doing the work, you're the one who gets to choose the horizon). Maybe there is a cluster of high volume trading at a certain time in the "morning". The center-of-mass of that cluster might be a good starting point in your search for time "T1", the moment when you will execute orders formerly called Market On Open. Maybe there is also a cluster of high volume trading at a different time, in the "afternoon". The center-of-mass of that second cluster might be a nice starting point in your search for "T2", the moment when you will execute orders formerly called Market On Close (or Stop Close Only, or Limit on Close, or ...).

Once you've got a T1 and a T2 that you can live with, you can use them to create your own personal customized "daily bars" data set, for historical testing and for live trading (order generation). These custom bars would be built from intraday data, with the "Open" at T1, the "Close" at T2, the "High" being the max price between times T1 and T2, the "Low" being the min price between T1 and T2. Perhaps you do this for all of past history. Or maybe you only do it for the most recent N years, and just use pit session data before that (saves money on historical intraday data).

Or maybe the whole idea is doomed. Maybe in the world of electronic trading, there is no reliable pattern of times-of-high-liquidity (unlike the Pit Open and Pit Close in days of old). The same research project, looking at N years of 5-minute bars and seeking an optimal T1 and T2, would also inform you if they don't exist (and therefore the idea is doomed).

If doomed then AFJ's option #5 (hand the problem to a human executing broker, then gladly pay extra for her hard-fought knowledge, experience, and discretion in executing your orders, at "the best time, today" in her opinion) might be a good approach. Option #6 is the same as option #5 except replace the word "human" by "software Algorithmic Execution package". It's not hard to find people who argue that options 5 and/or 6 pay for themselves many times over, thanks to improved fills.

However, in my experience, a human broker will gladly accept an account that submits 1 order per day, in 1 market, to buy 100 contracts on a Stop. But that same broker will decline an account that submits 100 orders per day, 1 order in each of 100 markets, each order to buy 1 contract on a Stop. Software doesn't seem to care.

ladidalimey
Roundtable Knight
Roundtable Knight
Posts: 197
Joined: Thu Sep 21, 2006 6:21 am
Location: London

Post by ladidalimey » Thu Oct 23, 2008 11:50 am

Just thought I'd add my thoughts so far. I've been playing around with Tradestation which I use for intra-day stuff.

If any of you have TS you might try this. I brought up the electronic data, but within Pit RTH i.e. day session. I then set the number of minutes in each bar to slightly longer than the whole day, say 500 mins. Then bingo, you get daily bars comprising electronic data with market hours. Its got ot be better than pit and misses out the overnight spikes.

Next task is to hive it off for input into TB!

L

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Thu Oct 23, 2008 3:13 pm

ladidalimey wrote:Just thought I'd add my thoughts so far. I've been playing around with Tradestation which I use for intra-day stuff.

If any of you have TS you might try this. I brought up the electronic data, but within Pit RTH i.e. day session. I then set the number of minutes in each bar to slightly longer than the whole day, say 500 mins. Then bingo, you get daily bars comprising electronic data with market hours. Its got ot be better than pit and misses out the overnight spikes.

Next task is to hive it off for input into TB!

L
Sorry to be thick, but I really don't understand how you did this. Can you explain it again perhaps? (p.s. I have TS)

RedRock
Roundtable Knight
Roundtable Knight
Posts: 941
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Thu Oct 23, 2008 3:57 pm

Ive seen that, in 'fee station'. You can build a custom data set from intraday i thought, including choice of contract stitching. Ive heard less than glowing reports of ts historical accuracy though. Perhaps patch it into CSI a shorter while ago and watch your files closely for errors each day.

I was doing a trial and just happened upon it so cant tell how its done.

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Thu Oct 23, 2008 4:27 pm

RR, I've done the custom symbol ploy - to no avail for the reason you cited - historic data is full of wildcards all over the place.

ladidalimey is talking about something different I thought?

ladidalimey
Roundtable Knight
Roundtable Knight
Posts: 197
Joined: Thu Sep 21, 2006 6:21 am
Location: London

Post by ladidalimey » Fri Oct 24, 2008 5:27 am

This is what I've done. Using Heating Oil as an example. First create a custom session of 9:00 - 14:30. Set the symbol to @HO, then set the session to the custom session. Then set the interval bar to 400 mins. The bars appear and you can print them off.

You only need to use this data where there is volume, probably back to about October 2006 in this case, then tag it on the end of the pit data.

L

Post Reply