Validity of system

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Validity of system

Post by Jwebster » Tue Dec 23, 2003 10:40 am

Hi all,

I'm messing around with a system and have been using $5,000 as starting capital (it's a spreadbetting based system therefore lower capital). While the system works with some markets at this starting capital, if I up the capital to 15,000 or 25,000 it really rockets off. Looking at trades the system has a tendency to make small losses and with only 5,000 the capital gets eaten up quickly so that when a good trade does come along there's not enough capital to size the positions off, so you don't get a chance to make the big profit that makes up for the losses and more.

The high capital seems to allow it to abosrb then losses and have enough firepower left to scale up on the good trades. The system would seem to be valid apart from the capital issue. Has anyone had similar experiences in their system design?? If I go for system that incurs less/lower losses so I can start with 5,000 will I be limiting the performance due to it most likely being inherently more conservative?? Any thoughts appreciated.

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Wed Dec 24, 2003 6:24 am

I'm wondering whether 5000 is enough, whatever your technique.

Since you are talking of spread-betting, well-known firm "A" starts accepting bets as low as 50p per point (most other houses start at £2/point).

Let's imagine you are using house A.

Take the DOW as an example (for Money Management, I'm not suggesting or not suggesting to trade or not trade the DOW -- who said I should be a lawyer? ;--)

The ATR, currently is at around 150. If you want to place your stops at 3 ATR, this means that your distance, or worst loss on that unit is 450 points. Multiply this by the lowest possible stake of 50p. You get a £225 exposure.

If you define your risk as 1% of your capital per unit traded, this means that your trading capital should be £22,500.00 Or nearly five times as much as what you are quoting.

To trade with your current capital means that such a trade would expose 5% of your 5000. Which is probably too high a risk if you want to be able to sustain rows of losses that WILL happen, no matter what technique you are using.

IMHO :roll:

BigBrad
Full Member
Full Member
Posts: 20
Joined: Tue Oct 28, 2003 8:39 pm

Post by BigBrad » Wed Dec 24, 2003 12:34 pm

I've seen something similar in my research on short term trading stock trading systems. The best that I can figure out for what I tested is that with a smaller starting capital the cost of trading in terms of slippage and commissions eats into the profits enormously, but with larger starting capital you can gain "economies of scale" because you can buy larger volume for the same cost.

I can buy 5000 shares of stock for the same commission as 100 shares.

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Post by Jwebster » Wed Dec 24, 2003 3:16 pm

Thanks guys. I'm just going to paper trade it for a while and see if the reality matches up with the testing. Hopefull it works out well :shock:

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Further question on system validity

Post by Jwebster » Mon Jan 12, 2004 11:43 am

Just wondering what people's thoughts are on the following scenario.

I have developed a system similar to PGO except that it uses difference between close and an adapatve moving average, plus some other tweaks (measurement of cycle period etc). To date I have focused on the Hang Seng index and over ten years of data (about the most data I have available), it does very well. I also tested each individual year starting at Jan. through to December based on starting capital of $100,000 and out of 10 years only three were losers (worst finish was around $55,000 at december, best was about $89,000). When I test the system on other markets (euro, swiss franc, ten year note so far) results have been pretty poor.

Anyway, given that system trades very well over both the whole ten years and each year individually on the Hang Seng, could this be a valid system for trading the Hang Seng? Given its good year by year performance (I call 70% hit rate good) the system would appear to be able to capture the moves/gyrations in this market. I guess I'm trying to work out what other tests I could do to verify this system has a reasonable chance. On the basic test of trying other markets with the same parameters it fails miserably, but on a year by year test with no re-optimisation the results on the Hang Seng are very good. Anyone got any thoughts?

verec
Roundtable Knight
Roundtable Knight
Posts: 162
Joined: Mon Jun 09, 2003 7:04 pm
Location: London, UK
Contact:

Post by verec » Mon Jan 12, 2004 11:59 am

The only thing that would make me think twice is that, as you say, your system only works on one market.

If you were to understand how specific the Hang Seng index is, and how your rules are adapted to that specific behavior, then using it for real would be much less questionnable. As it stands now, it seems that if you don't understand why your rules work on that system, irrespective of the fact that they do, then going live is something I would find too much risky for my taste.

Note, however, that if you had a system that worked across both time and markets, things would be totally different, even if the rules made no rational sense ...

The scary thought, really, is that your system is invalidated by all markets except the Hang Seng index ...

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Post by Jwebster » Mon Jan 12, 2004 12:20 pm

Thanks Verec.

As you say 'If you were to understand how specific the Hang Seng index is, and how your rules are adapted to that specific behavior, then using it for real would be much less questionnable'

I think that's it. I realize the Hang Seng has some does have some very specialised characterisitics and I believe that for the very reason the system works on the Hang Seng it can't work on other markets, simply because they don't exhibit that kind of behaviour. The other markets I have tested it on appear to be far less volatile than the Hang Seng. I wondering whether the system may be jsut suitable for high volatility markets. I need to find some suitable data and test it I guess!

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Mon Jan 12, 2004 6:48 pm

I would be exceptionally careful of drawing any conclusions based on tests on the hang seng. My reasons are:
- compared with dax, estx, es, ym etc hang seng is very thin with ugly spreads and the market maker behaviour is questionable. Your slippage is likely to be much greater.
- hang seng may well have behaved differently to other indexes in the past but might revert to normal index behaviour in future.

If I couldnt make something work on most indexes by adjusting for a few factors then I would be surprised if I was profitable in future. Having said that you might what to look at the HHI.HK the new chinese index which is stunningly trendy on an intraday timeframe.

John

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Post by Jwebster » Tue Jan 13, 2004 4:37 am

John,

Thanks for your comments. I am designing the system to work with spreadbetting. Do you believe that your comments on spreads and MM behaviour for the Hang Seng would still hold? I kind of guess they would as there would be some flow through to the SB prices.

UT
Contributor
Contributor
Posts: 1
Joined: Sun Dec 28, 2003 2:34 pm
Location: Chicago

Post by UT » Tue Jan 20, 2004 12:59 am

John,

You mentioned that the new chinese index is trendy intra-day. I assume you are refering to the H-shares futures, based on the Hang Seng China Enterprises Index (HSCEI), correct?

Do you know if there is anywhere on the internet that I can find intraday charts of this instrument?

Thanks,
UT

Kiwi
Roundtable Knight
Roundtable Knight
Posts: 513
Joined: Wed Apr 16, 2003 1:18 am
Location: Nowhere near

Post by Kiwi » Tue Jan 20, 2004 4:08 am

JWebster. Possibly not. The larger spreads on spreadbetting tend to nullify the issue that you face trading it directly --- excessive slippage/bad fills. Also u get a quote and choose whether to accept it which I find changes the mechanics somewhat.

UT. Yes you are correct. I am not aware of any free sources of real time data. You can get it free with an IB account. You can also get delayed data from:

http://www.futuresource.com/charts/char ... &b=bar&st=
John

masmit
Full Member
Full Member
Posts: 20
Joined: Sun Oct 12, 2003 7:28 pm
Location: London, England

Post by masmit » Thu Jan 22, 2004 2:09 am

Don't most of the spread betting companies web-sites give real time quotes of their prices. Since their prices are what you'd be trading, wouldn't they do?

Mark

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Update - for those who are interested

Post by Jwebster » Thu Jan 29, 2004 12:25 pm

As some of you may know I have been playing around with a system that among other things uses cyclical measurements to create an adaptive moving average. I have been focusing my research on the Hang Seng and over ten years of data the system has been doing very well. I also tested individual years without re-optimization and the results were really good. When i tested it on other markets however, the results were poor.

I was therefore stuck with trying to determine the validity of the system with it having failed one of the major tests normally used (ie it trades multiple markets well). Then I thought about how it works, given that it uses cyclical measurements (which I believe are specific to each market, ie each market has its own 'rhythm') the system parameters, which are set to the 'rhythm' of the underlying market should only work on that market. This may be flawed logic, but it seems to make sense to me.

Anyway, another test of system validity is to run it over more data, so I got Hang Seng data going back to 1969. Interestingly, the system does equally well over this new 34 years of data. If I widen the stops by just 0.25 'N' it really kicks butt (tighter stops seem to work better in the more volatile modern markets). I find this quite an interesting validation as the nature of the market (volatility) has changed dramatically from 1969 to 2004. So, given that the system picks good trades over this entire period gives me some cause for optimism. I'm still going over it trying to pick holes, but paper trading to date has yielded very good results. Anyone got any thoughts on this/am I just fooling myself for some obvious reason that I've so far missed?

Jason Czech
Roundtable Fellow
Roundtable Fellow
Posts: 68
Joined: Fri May 16, 2003 8:28 am
Location: Atlanta, GA

Post by Jason Czech » Thu Jan 29, 2004 1:39 pm

I guess the question is, do you believe there is a reasonable chance that the "rhythm" of HangSeng market could change in the future? And if so, how would you know when this has occurred so that you could get out before all of the damage was already done?

Personally, I would not use a system that performed badly in all but one (or a few) market(s). Particularly if by 'performs badly', you mean that it loses money in those markets.

Mark_et_Lizard
Roundtable Fellow
Roundtable Fellow
Posts: 62
Joined: Sun Apr 20, 2003 9:16 am

Post by Mark_et_Lizard » Fri Jan 30, 2004 10:32 am

Jwebster

How many trades does your system have over the 34 year test period? If it's more than a few thousand, say 5 to 6 thousand I wouldn't worry about it not working as well on other markets, although a complete failure on a similar index should be of concern.

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Post by Jwebster » Fri Jan 30, 2004 12:56 pm

Mark,

Its a reasonably long term system, I sought to catch major moves and try and get the moving average to adapt (tighten) to capture some of the choppy bits in the market (or conversely not get killed as much by ranging markets, ha!). So, over the 34 years it makes little over 700 trades, or close to two a month. I'm sure where this stands in the scheme of statistical significance, but it seems to show a proven ability to catch good moves year in year out. As for similar indices, if I make some changes to the parameters that feed off the measured period of the market cycle, it will produce very good results on the Nikkei (what I believe to be a similar index to the HS). However, once I make these changes it no longer works well on the HS. It was because the system performers differently based on the measured cycle period that got me thinking that maybe each market does have it's own 'rhythm'.

For example, I looked at the Nikkei and Hang Seng during 2003 and both had extended periods where they were trending strongly. I figured the period measurements for each market would be fairly similar given the similar behaviour of the indices. Well, they were actually fairly different. Different enough to make me think about the whole rhythm thing. I mean when you think about it, what determines prices? The participants in the market right? And even though this global age means funds are invested everywhere, each market still has its 'locals' who have their own make-up and way of reacting to market moves. I am tending to believe that because the participants and their reactions can vary so much from one market to another that maybe their is some foundation to each market having its own rhythm. This would certainly seem to be bourne out by the cycle period measurements I was looking.

Anyway, I guess the great thing about this trading game is that in the end no-one can really tell what's right and wrong (within reason!), just what you think has a reasonable chance for success. I keep paper trading for a while and see what happens.

edward kim
Roundtable Knight
Roundtable Knight
Posts: 344
Joined: Sun Apr 20, 2003 2:42 pm
Location: Silicon Valley / San Jose, CA USA
Contact:

Post by edward kim » Fri Jan 30, 2004 2:34 pm

Hi JWebster,

Which would you rather have:

A. 30 different optimized systems for 30 uncorrelated markets

B. 1 system that works for 30 uncorrelated markets

I use B and only B, and I put money into B now.

If you want to go the route of "A", then do it. But if your Hang Seng system doesn't work for a whole bunch of other uncorrelated markets, your system might have a huge problem down the line. By then, it might be too late for you :shock:

I used to have a system just like yours that only worked for Nasdaq and the S&P500. When I used it on any other market, it performed horribly. THEN I discovered what the inherent system problem was.

I know how many people here like the idea of paper trading and single market optimization. I do not like either, but that's the way I feel and I'm more comfortable going the other way.

Edward

Sebastian
Full Member
Full Member
Posts: 21
Joined: Mon Nov 24, 2003 3:26 pm

Post by Sebastian » Fri Jan 30, 2004 6:45 pm

For example, I looked at the Nikkei and Hang Seng during 2003 and both had extended periods where they were trending strongly. I figured the period measurements for each market would be fairly similar given the similar behaviour of the indices. Well, they were actually fairly different.
This sounds similar to the situation I'm running across in adapting a Turtle variant to trade the QQQ and SPY. The Nasdaq100 Index (also the Nasdaq) is simply a different animal from the SP500 Index. Even though they appear to make highs and lows at about the same points and mostly move in the same direction for about the same lengths of time, you just can't use all the same parameters on them.

Personally, I don't see anything terribly wrong in having a system that only works on one market, with this proviso: You should have some kind of credible theory as to WHY it works. Without this, for all you really know you've simply stumbled onto a statistical anomaly that will break down as soon as you put money into it. :)

Some system theorists might argue that successful testing of one system across many markets is the only way to be sure that system is truly valid. I'm not convinced of this idea (although I do agree that a system should be robust within the market you're trading). I've seen the performance of portfolios of futures, trading according to the rules of a single system: How many of those contracts are PERENNIAL losers that clearly DON'T work with that system, EXCEPT as part of a portfolio? How many of those contracts defy successful trading with ALMOST ANY system?

Find a niche, and then make yourself the smartest person in the world about it, that's what I say. :D



Luck,

Sebastian

MarkH
Roundtable Fellow
Roundtable Fellow
Posts: 53
Joined: Fri Dec 05, 2003 10:24 am
Location: Corpus Christi, Texas

Post by MarkH » Fri Jan 30, 2004 8:59 pm

When Chuck Lebeau started his web site, he advocated the idea of designing systems to be used on specific markets and sold simple, inexpensive systems for single markets. I don't know for sure, but I don't think most of the market specific systems have performed that well after release. I actually bought and traded a few on a small scale for a while a few years ago. As I recall, they all came with documentation that presented fairly sound logical concepts that formed the basis of the systems.

Unless you are talking about a market specific shorter term system that generates several hundred trades over the back test period, I would be very carefull about devoting too much capital to a market specific system.

However, I am willing to vary parameters of a simple, robust system on different markets, provided that there is a wide range of parameters that give profitable results. I do this now with part of my capital.

Mark

Jwebster
Roundtable Fellow
Roundtable Fellow
Posts: 52
Joined: Fri Jul 04, 2003 10:23 am
Location: London, UK

Post by Jwebster » Sat Jan 31, 2004 10:50 am

Thanks to all for your input. Like most of you I would like to trade a system that works over multiple markets, but I am yet to develop this. No problem, as it gives me something to work towards as I continue my research. Maybe I'm a little strange but I actually enjoy spending hours and hours staring at screen and trying out weird theories. :lol:

Post Reply