Danger of single market back test result

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
AlexT
Full Member
Full Member
Posts: 10
Joined: Fri Sep 03, 2004 9:32 pm
Location: Penang, Malaysia

Danger of single market back test result

Post by AlexT » Wed Sep 08, 2004 10:56 pm

Hi, I read in one of posting (sorry i forgotten which one is it) on the danger of back test result of a single market.

I am living in Malaysia where i only have access to 2 active future markets : Crude Palm Oil and Index Future.

i have develop a varition of Mid term simple breakout system that back tested result was very good (using TradeStation).

can some one point out the danger of following the system in real time?

i have traded the index future market using this system, the problem seems to be my stop order placed in advance can be seen by my broker.

if i do not place the stop order in advance and i place market order when it happen. it seems to work well.

MCT
Roundtable Knight
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm

Post by MCT » Thu Sep 09, 2004 1:33 am

Hi Alex

If you are interested in systematic trading, I suggest you study the following link and the pdf turtle manual thoroughly before risking a single cent viewforum.php?f=6

Don't worry about how much money you are going to make, instead worry how it is you make it.

Make sure concepts and processes are cristal clear, and that you are confident with your approach before departing with your hard earned $.

Theodros

TC
Roundtable Knight
Roundtable Knight
Posts: 101
Joined: Thu Jul 08, 2004 2:12 pm
Location: Toronto, Canada

Post by TC » Thu Sep 09, 2004 1:56 am

Alex

Without more information on how you developed your system and/or what data you used to optimise (if at all) your parameters it is not possible to comment on the validity of your back-tested results for future trading. However, if your system performed as well on the out-of-sample data as it backtested on the sample data then this should give you a measure of confidence.

For what it's worth, I can tell you what I do. I am extremely reluctant to develop any system on a single market. It is just too easy to curve fit when using only a single market.

To ensure robustness I prefer developing my systems using the maximum amount of data possible (currently 40+ US futures markets). If you currently use TradeStation then you should have access to all of the same US futures markets. When VT 2.0 is released, and I can import more data, I will increase this to ca 80 markets.

Unless my system performs well across a majority of all the markets tested I will not use it. For the small cost of acquiring additional data I would recommend buying yourself some extra peace of mind by backtesting your system on some other markets before committing real money. As these markets will all be out-of-sample the results will be a very good guide to what may happen in the future in whatever markets you trade.

Best of luck

Tom

AlexT
Full Member
Full Member
Posts: 10
Joined: Fri Sep 03, 2004 9:32 pm
Location: Penang, Malaysia

Post by AlexT » Thu Sep 09, 2004 3:26 am

yes, i have read turtle system by c.f., thanks to him alot .

i believe trend following can be profitable i have seen some similar system in books by George Pruitt and i have tested them.

on thinking about back testing more market is the best, can a market in a specific country has different charecteristic then let say US market as a general?

i am thinking Malaysia market is quite restricted in a sense, can it be different to US market, if can, then if i develope a system that work find in this market, will it work? (assuming very little obtimisation is used)

my system has only 1 variable and it work with other value, the system also work during financial crisis in 1997-1998.

can i have confident i will work in future?

so far i have paper trade it 6 months and with real money 3 months, so far so good.

Kvadrik
Full Member
Full Member
Posts: 15
Joined: Sat Jun 19, 2004 3:08 am
Location: USA - Israel
Contact:

Criteria for back testing

Post by Kvadrik » Thu Sep 30, 2004 3:30 pm

What criteria do you use for back testing? Why are you sure that the tested strategy will be profitable in the future period?

Murray Ruggiero
Senior Member
Senior Member
Posts: 41
Joined: Mon Oct 11, 2004 3:35 pm

Will a system make money in the future

Post by Murray Ruggiero » Fri Oct 15, 2004 12:03 am

One point I would like to make on this thread is that the premise of the system is the most important thing. What is the system based on; a good premise goes along way. You could have a system with a lot of trades but not a good premise and you’re in trouble. I once had a client who made a lot of money trading the NASDAQ in 1999 to late 2001, using a neural network. The problem was the signal where correlated very highly with the rate of change over the past three days. When the NASDAQ moved sideways he lost everything. The point is if he realized the premise was today minus three days ago, he would not have traded it because it did not make sense. Sometimes it takes a little analysis to figure out what a system is really doing. I have written several articles of interest in this thread. Check out my article 9/98 on report days trading bonds.

I also did an essay style article on system development in 5/98

Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann » Fri Oct 15, 2004 10:04 am

An example of a bad premise is the system that Murray Ruggerio sold in 1997, called "Taylor Bonds Pro". It included some universal market truths such as the following snippets, taken directly from the Taylor Bonds Pro code:

Code: Select all

If value1<.40 and Seq[1]=0 and Month(Date)<>9 and Month(Date)<>10 and Month(Date)<>12 and DayOfWeek(Date)<>4 then buy(.......

If value1>.40 and value1<.85 and DayOfWeek(Date)<>5 and DayOfWeek(Date)<>2 and Month(Date)<>4 then sell(.......
Those are just two of the eight compound-IF statements that are necessary to express the premise of Taylor Bonds Pro. You can see how it elegantly captures the true essence of market behavior, the universal laws that govern human trading.

I dug out the system just this morning and ran it again. The .ELS file is dated 22 April 1997. I ran it using $50 per contract for commissions and slippage, the exact figure that Murray recommended in his instruction sheet that shipped with the system. I ran it with the recommended stoploss ($1600), and then again with effectively no stoploss (stop = $25,000). The resulting equity curves for one-lot testing (Tradestation) are shown below.

You can see just how well Murray's premise worked in the marketplace. An arrow points to the date when the system was released to a grateful public and purchased by lucky traders like myself.
Attachments
murray.png
murray.png (19.75 KiB) Viewed 9854 times

Murray Ruggiero
Senior Member
Senior Member
Posts: 41
Joined: Mon Oct 11, 2004 3:35 pm

Post by Murray Ruggiero » Fri Oct 15, 2004 11:28 am

If you notice, the system was released in 1997, and did set a few equity peaks after release. I am not saying this was one of my best system in retrospect but sometimes mechanical system do not hold up because of various reasons. Another point I would like to make is this was seven years ago. In term of full time research in the markets that’s an incredible time to evolve. If you notice these articles were dated after the release of Taylor bond. Reviewing some of my earlier research both sold and not sold, I realized that some of my system worked well while other basically when flat after release so I did research to try to find our why, this was the premise research I talk about in these articles.

One problem with systems that have a rhythm type pattern changes to the underlying market are important. Maybe the rhythm changed because the government during the Clinton years balanced the budget, now we have an over stimulus due to low rates. The market moved to a different strange attractor so to speak and this is a problem with many systems.

I am not saying that everything I have ever developed works forever, but people with few ideas seldom have good ones. You don't know ahead of time if a system will work in the future. But for the past four or five years I have been developing systems on baskets of markets with large number of trades based on earlier lessons learned. The market are dynamic and trading mechanical system requires understand when a system is not performing as expect and when to stop trading it and to reevaluate. Market change for example the Monday day of week effect does not work like it use to and that was a really effect which was talked about for over a decade!.

I have had the courage to publish 100’s of system not all of them worked well, some have worked very well. I make comments in forum other than my TradersStudio thread because I want to help. I have learned a lot and am willing to share. This should not be an attack on me for sharing. You need to take my body of work, over 100 article, three books , hundreds of trading systems and judge that, just pick and choose, because no man or women could be held to the standard of always being perfect, only god.

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Post by Roger Rines » Fri Oct 15, 2004 11:42 am

Once you discover all the things you should and shouldn’t do in developing a system, spend the time paper-trading the signals to see what else you don’t understand about the system. Many systems that follow the best-practice rules don’t survive well when you’ve got to handle them within the constraints of real-time.

When we paper trade, we send our signals to our broker who paper-executes the signals and compiles the results at the end of the day on a spreadsheet they send back everyday. We have no chance to influence the results outside of what the signal ordered.

This approach gives us the chance to work out all the questions the broker might have about over-nights, or anything else that can be an issue. If your broker won’t do this for you, find a fellow trader who will and you’ll surprise yourself at how much you’ll both learn about the system.

Another advantage that comes from paper trading is that you’ll get to see if the system will perform on data that nobody has seen before. Many systems can follow the rules of development and look wonderful, but when it comes to reality, the model breaks down and the system’s performance never sees another new equity high.

shakyamuni
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Wed Jun 04, 2003 9:44 am
Location: Somewhere, Hyperspace

Post by shakyamuni » Fri Oct 15, 2004 10:07 pm

If you know how to design a robust trading system, then you can design one specifically to exploit a single market. If, on the other hand, you do not know how to design a robust trading system, then trading all the markets in the world won't be able to stop you from losing money.

Murray Ruggiero
Senior Member
Senior Member
Posts: 41
Joined: Mon Oct 11, 2004 3:35 pm

Number of trades

Post by Murray Ruggiero » Fri Oct 15, 2004 11:04 pm

The problem with building a system on a single market is do you have enough trades. That is one place which building them on multiple markets help.

Roger Rines
Roundtable Knight
Roundtable Knight
Posts: 1946
Joined: Wed Oct 06, 2004 10:52 am
Location: San Jose, CA

Re: Number of trades

Post by Roger Rines » Sat Oct 16, 2004 12:14 pm

Murray Ruggiero wrote:The problem with building a system on a single market is do you have enough trades. That is one place which building them on multiple markets help.
This warning should be pasted onto the screen of all development monitors because it is one of the primary problems that plague all system development.

Post Reply