How Do You Test?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Turtle40
Roundtable Knight
Roundtable Knight
Posts: 201
Joined: Wed Oct 19, 2005 1:53 pm
Location: Guernsey, Channel Islands

How Do You Test?

Post by Turtle40 »

An open question:

How do you test? I have spent quite a lot of time thinking about the testing process, and the more testing I do the less I think I know. Using TB is very easy, we can now test our edges, draw fabulous Monte Carlo graphs and so on, but with so many infinite combinations and parameters, is it possible to test incorrectly? Probably.
As far as I am aware there is no "definitive" guide on how to design a trading system. Testing and designing in isolation lacks input from others, so how do different traders complete a testing cycle?

I don't want to pry into people's detailed trading systems, but am curious about how others deal with the process of building a system from scratch.

What makes it acceptable?

I think we all realise "robustness" is good. Too much curve fitting is a bad thing, money management is vital, but going a little deeper what else is important.
From my experience thus far, I tend to start with the markets I wish to trade. Set a %/trade that I am happy with then divide the data into three 10 year periods. I usually test with older data, then check different time frames, and other markets to look for consistency. I like the increment test for robustness.

Originally I went for CAGR as the main measure of success of a system, nowdays MAR is more important. Limiting drawdown is perhaps more important than shooting for the highest CAGR. (D/Ds above 30% are too high for me).
What about profit per trade and expectation? Should they feature quite highly? Do you rank CAGR, MAR, DD, Expectation, P/Trade in any kind of order. Are there ranges of results that have to be achieved?

The list of questions goes on and on. I would like to hear how others go about the process.

Thanks,

-Turtle40
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

these are some good questions... I test in Wealth Lab so its a bit different from TB, the concept remains the same though. What I look for in short term systems: Winning Percentage above 60%, Profit Factor above 1.5 , draw down less than 30%, CAGR above 30%. In Long term systems: Profit Factor above 2 minimum, Win Percentage above 30%, Cagr above 30% DD less than 30%. I test market orders only.
Turtle40
Roundtable Knight
Roundtable Knight
Posts: 201
Joined: Wed Oct 19, 2005 1:53 pm
Location: Guernsey, Channel Islands

Post by Turtle40 »

Guys thanks for the replies thus far.

Ted I will re-read the links you provide. The main thing that interests me is the "process of process", in other words how others go about bringing all the various pieces together.
Nussgipfel
Roundtable Knight
Roundtable Knight
Posts: 115
Joined: Sun Nov 20, 2005 12:02 pm
Location: Westport

Post by Nussgipfel »

Turtle40,

I am also playing around with trying to come up with a long only system for stocks that is acceptable to me.

I thought that taking the possibly worst time in stocks in most recent history which in my thinking would probably be from 1/1/2000 to 12/31/2002 and then coming up with a way to avoid being long will help when stocks are starting to move up again. Well, it worked on being out alright but when stocks started to move up the performance was underwhelming....so back to the drawing board for me.....again!

I am coming to the conclusion that you can not have it both ways and trying to be out is coming at the price so for me I still have to find the pain/gain ratio which is acceptable. My system should be simple and easy to understand and build. The frustration seems that some of the ideas that seem to work on a set of up-trending stocks just are not holding up when tested against a basket such as the S & P 100. I had a system that gave me the impression of working great on the NASDAQ 100 only to see it being useless when running it against the S & P 100 or S & P 400.
painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

comparisons are difficult

Post by painless »

I look for low deviation from optimum and as few parameters as possible. I do not understand the basis by which % figures are quoted on this forum. You say MAR, for example, but what is the initial capital? D/D above 30%?
Is there some standard that is used?
I trade futures and do all my testing based on a starting capital of
(DD + margin) * 2
I abandonded compound returns when testing since they seemed a bit unreal to me.
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

painless whats your average holding period?
painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless »

Well that would be about 3 weeks to 2 months. Depends on the market but I like to mix up the trading periods. If I go for the shorter time frame then it just means smaller losses and more of them but the drawdown is about the same. I like being hundreds of points in profit coz I don't have to stress about it or look at the screen. I tried a daytrade method but I don't have the personality for that. At the moment I am only trading 2 markets and I will add more as my account grows and it takes me about 3 months to analyse a market and then decide if I want to trade it. Guess I am a snail in that respect but until I have a computer with a lot more grunt it just takes time.
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

I guess its something I also have to work on (holding longer without a need to look at the screen every day)
painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless »

Oh now I see. Blox is a software package. I came upon this site via a google link. It seemed to have traders like me rather than pip traders at forex sites and the reference to turtle traders is a good thing I think.
So does blox have a standard that you use to determine starting capital or somebody enlighten me please.
painless
Roundtable Fellow
Roundtable Fellow
Posts: 59
Joined: Tue Nov 14, 2006 12:52 pm

Post by painless »

Hi Turtle40,
I guess I should try to answer your post, the polite thing to do.
I think that the markets have a band of noise and sometimes it breaks out of this band and goes somewhere else to make noise. I am a trend follower type so I don't want to trade the noise.
It doesn't take much to capture the non-noise but the noisy part takes back all the wins. So I study ways to not give it back (don't we all?). So I look at a monthly chart of whatever market I am dreaming about :) and try to determine the size of the noise and the magnitude of the moves. So move minus two noisy bands is the amount I could make on each significant move. If after all that it looks in the realm of worthwhile then I look at a weekly chart and think about it again.
I have to find something that suits me and my tiny budget. I have not got the money management down pat yet but I don't have the luxury of choosing how many to trade I am happy to just be in the market and trading with my winnings at the moment. But I have started on the money management and like so many of you guys, who obviuosly know what you are talking about, it is a very significant part of what we do.
One of the problems with testing is you might have a really great system amongst all those numbers but because of three bad trades it gets obscured but if you had a protection stop or something similar then it might have been the answer to your prayers. But testing with that stop included may skew the results of your tool.
The size of the data sample is another problem and I would suggest out of sample tests which it appears you are onto.
All the best
Blair
Stephen
Senior Member
Senior Member
Posts: 41
Joined: Tue Jul 18, 2006 12:17 am
Location: Jakarta, Indonesia

Post by Stephen »

Back to Turtle40’s 'process of process' question in the October 24th post in this thread.

My frustration has been the lack of a comprehensive roadmap on how to approach system development, one that guides a novice through the complex and potentially overwhelming array of software features available to change the behavior and results of a mechanical trading system. I realize that this is the novice’s dilemma and, in one sense, everyone has to work this out for themselves. So, this is me working it out.

The lack of a roadmap results in one wasteful activity that I call “parameter fiddleâ€
Last edited by Stephen on Fri Dec 01, 2006 2:13 am, edited 1 time in total.
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

Stephen wrote:Back to Turtle40’s 'process of process' question in the October 24th post in this thread.

trading system. I realize that this is the novice’s dilemma and, in one sense, everyone has to work this out for themselves. So, this is me working it out.

Steve
You seem to be on a clear and scientific path towards your goals. There are many books on the theory of proper system development. But I believe you already spoke the core truth... "Everyone has to work it out for themselves." (Or, as Sluggo suggests in another post, hire a professional manager to do the professional work for you.)

You have heard some say instead of providing an answer, instead speak of clues which can be tested to find the "answer" for ones own understanding. I'm suggesting that the process of design and how one tests, fits into the same process. If there were one specific correct answer, then nobody would be working at McDonalds as everyone would have found the path to their dreams. (not to say owning a McDonalds isnt a worthy dream..)

Keep up the good work :!:
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

RedRock, you talk a lot about working at McDonalds, do you have any affiliation with it? :wink:
RedRock
Roundtable Knight
Roundtable Knight
Posts: 944
Joined: Fri Jan 30, 2004 3:54 pm
Location: Arizona

Post by RedRock »

BARLI wrote:RedRock, you talk a lot about working at McDonalds, do you have any affiliation with it? :wink:
It was used as a metaphor, barli.
BARLI
Roundtable Knight
Roundtable Knight
Posts: 650
Joined: Sat Jan 17, 2004 6:01 pm
Location: USA

Post by BARLI »

I was kidding :wink:
Stephen
Senior Member
Senior Member
Posts: 41
Joined: Tue Jul 18, 2006 12:17 am
Location: Jakarta, Indonesia

Post by Stephen »

Thank you, RedRock. As I find my own road, it helps to know that I'm not driving off a cliff! - in terms of methodology.

I have trimmed some detail out of my post above to allow others that follow more room to explore how they would make it happen.

Steve
danZman
Roundtable Knight
Roundtable Knight
Posts: 117
Joined: Thu Mar 10, 2005 10:07 am
Location: Newport Beach,CA

Post by danZman »

For stock trading, I want a very robust system. You can't simply choose stocks in the S&P 500 or NASDAQ 100 for backtesting, since many are new to those indices (and their performance got them in there).

I have a list of 3200+ stocks for backtesting using TB. I do a lot of filtering such as $ volume and average volatility. From there, I rank them over a 6 month period using a gain/pain ratio instead of the common performance over 6 months. I find it's better to have smooth performance from A to B instead of volatile moves (just like a portfolio).

What's kind of interesting with short-term stock systems is that profit taking after a certain ATR seems to be the best system for a good MAR. I've done well with some trend following methods, but nothing close to going for those quick base hits.

One big caveat is that after about 10 years of data, there are fewer trades. The cause is fairly simple - my quote vendor doesn't supply me with old stocks that have gone defunct. I need the data to get a better feel for anything prior to 1996 or so. This is VERY important if you're creating a short-selling system (ever notice that the NASDAQ a/d is consistently falling).

One other thing for stock trading systems: correlation. Most stocks move as a group. Therefore, it's good to come up with not just an edge in trading individual stocks, but an edge for timing the general market. I've found a good edge for IT timing, and I'm in the process of using it's signals as a filter for all my stock trading systems. Luckily TB is very flexible. By far, it's much easier to design futures trading systems.

My idea for system trading is to have a bunch of non-correlated systems over various markets that have excellent MAR and are not in the market that much (but I don't want too small of a sample...I want to see at least 1000 trades by each system). Basically, I want a big edge at the sacrifice of CAGR. However, when all the systems are online together, CAGR becomes very large.

Hope that helps.

D
zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Post by zentrader »

...perhaps it's not the best idea to develop a trading system only testing with "historical" data.

Markets change.

One answer is to test the system also with synthetic data, which simulates different market conditions.

The good old monte carlo simulation method is her again an important part to generate such data (data simulation or data scrambling).

Only my two cents...

bye,
zentrader
mmm
Contributor
Contributor
Posts: 3
Joined: Mon Aug 31, 2009 6:19 am

Post by mmm »

nice post Stephen. Since this post, any new updates on your generic process? Thanks.

Stephen wrote:Back to Turtle40’s 'process of process' question in the October 24th post in this thread.

So, based on my first few months experience in system development, and after much browsing through this site, here is a take on the generic process as I see it (for now):
Post Reply