Testing Results Question

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Testing Results Question

Post by HWG » Mon Mar 12, 2012 12:50 pm

I've been testing a pretty simple system over about 30 years data in 65 different worldwide futures. I've been using in sample and out of sample data to test for robustness, etc.

My question is that my results I feel are decent (for me) but if you look at the profit contribution all of the winners and losers are within 1R. In addition, my win/loss rate is nearly 50% which is a red flag for me as I would expect something more like 30/70 losses to wins or similar. So basically I am getting profitable results but not the big huge winners (10-15R).

It seems that if I'm happy with the results then it shouldn't matter but I have to say I'm a bit surprised there weren't bigger wins.

To be clear this is a long only system with no profit target.

Would like to hear some thoughts on this and maybe some ideas how I'm missing these bigger moves or whatnot.

'Preciate it.
Attachments
EquityCurve.png
EquityCurve.png (35.94 KiB) Viewed 6010 times
ProfitContribution.png
ProfitContribution.png (13.7 KiB) Viewed 6008 times
Win_LossStats.png
Win_LossStats.png (24.98 KiB) Viewed 6008 times

Robwynge
Senior Member
Senior Member
Posts: 42
Joined: Wed May 25, 2011 10:33 pm

Post by Robwynge » Mon Mar 12, 2012 1:37 pm

Well, how are you instructing the system to exit trades? And when you look at the trades tab, is the system exiting based on how you expected it to?

rossb34
Full Member
Full Member
Posts: 12
Joined: Fri Dec 31, 2010 7:04 pm
Location: Middleton, WI

Post by rossb34 » Mon Mar 12, 2012 1:44 pm

It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Mon Mar 12, 2012 1:48 pm

Robwynge wrote:Well, how are you instructing the system to exit trades? And when you look at the trades tab, is the system exiting based on how you expected it to?
I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Mon Mar 12, 2012 1:50 pm

rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
Right? That's why I posted this question. It's not a pre-supplied system but one I picked up through the Blox Marketplace and combined with some pre-supplied blox.

My entry/exits look ok but just doesn't make sense why everything is so clustered around the mean so to speak.

Robwynge
Senior Member
Senior Member
Posts: 42
Joined: Wed May 25, 2011 10:33 pm

Post by Robwynge » Mon Mar 12, 2012 2:09 pm

HWG wrote:I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.
So then I am confused about why you are confused. :-)

If it's doing what you want by following the signals of your indicator faithfully, is your indicator such that it only has small winners? If not, and your indicator sometimes triggers exits after small moves, and other times triggers exits after large moves, then clearly something must be wrong with your code. Maybe soemthing is wrong with the risk manager forcing you out of trades when your risk gets "too large"? Just throwing something out there...

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Mon Mar 12, 2012 3:49 pm

Robwynge wrote:
HWG wrote:I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.
So then I am confused about why you are confused. :-)

If it's doing what you want by following the signals of your indicator faithfully, is your indicator such that it only has small winners? If not, and your indicator sometimes triggers exits after small moves, and other times triggers exits after large moves, then clearly something must be wrong with your code. Maybe soemthing is wrong with the risk manager forcing you out of trades when your risk gets "too large"? Just throwing something out there...
Right, its not that its not working properly its just not what I expected so trying to figure out why. The more interesting part is what rossb34 said where the expectation is 0.0 and Average Risk Percent is N/A. That doesn't seem right to me. I will go through checking again. Cheers.

Aaron01
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Thu Jun 02, 2011 1:21 pm

Post by Aaron01 » Mon Mar 12, 2012 4:46 pm

rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
With an average holding period of ~198 days and an average trade return of ~1.75%, I would venture there's a bug in the code, or he's betting so small he can't fully capitlize on the moves.

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Mon Mar 12, 2012 4:51 pm

Aaron01 wrote:
rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
With an average holding period of ~198 days and an average trade return of ~1.75%, I would venture there's a bug in the code, or he's betting so small he can't fully capitlize on the moves.
I'm using starting test equity of 1mm and 2% risk per trade so seems big enough to make a difference. I did go back and look to see what percentage wise some of my biggest winners and losers were. Had some in the 60-70% range and a bunch in the 30s. Biggest losers were in the 5s and a few in the 6-8% range so seems as if some of those would be more than 1R or maybe that 1R is not registering properly.

This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Mon Mar 12, 2012 6:53 pm

At least one of the presupplied systems that ships with Trading Blox software, exhibits the same behavior. See figure below.

If anybody is really interested, I suppose they can try all eleven presupplied systems, one by one, to find out which of them behave this way.
Attachments
Presupplied_with_Blox.png
Presupplied_with_Blox.png (23.99 KiB) Viewed 5900 times

drm7
Roundtable Fellow
Roundtable Fellow
Posts: 96
Joined: Sun Apr 20, 2003 9:02 pm
Location: Richmond, VA

Post by drm7 » Tue Mar 13, 2012 9:37 am

This might be a bit tedious, but you should use a charting program to bring up a chart with the necessary indicators and go through 25 to 50 trades BY HAND, to see entry/exit/risk, etc. This should help you figure out if the system matches the output.

LeviF
Roundtable Knight
Roundtable Knight
Posts: 1428
Joined: Mon Dec 22, 2003 12:24 pm
Location: Des Moines, IA
Contact:

Post by LeviF » Tue Mar 13, 2012 9:46 am

I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Tue Mar 13, 2012 10:05 am

LeviF wrote:I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.
Correct, no stops. System exits when the indicator changes a certain way.

So if that's the case then it can't really calculate the R-multiples as far as profit contribution right? That makes sense as I said before I have some winning trades in the 30-70% range and losing trades in the 5-8% range so this would appear to be further out on the curve then what its showing.

Chuck B
Roundtable Knight
Roundtable Knight
Posts: 469
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B » Wed Mar 14, 2012 8:15 am

If the exit (stop) is undefined when (before) an entry signal is triggered, then you have no means to define the risk on the trade or size the position based on that risk.

babelproofreader
Roundtable Knight
Roundtable Knight
Posts: 138
Joined: Wed Nov 10, 2004 4:36 pm

Post by babelproofreader » Wed Mar 14, 2012 9:37 am

I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.
If the exit (stop) is undefined when (before) an entry signal is triggered, then you have no means to define the risk on the trade or size the position based on that risk.
I'm afraid I disagree with these two statements; it might not be possible for TB to calculate R-multiples, expectancy, average risk etc. without a hard coded stop, but these calculations can be done if you have access to the history of the losses. These losses will form a distribution and thus measurements can be taken:
  • the average loss can be taken as 1R
    the average loss plus 1/2 standard deviations can be taken as 1R
    if the loss distribution is highly non-normal, bootstrap it and take the bootstrap average, with or without +1/2 bootstrap standard deviations as 1R
    take a suitably high nth percentile value as 1R
    some other "robust" measure of the distribution can be taken as 1R
the point being that having the history of losses enables one to come up with a meaningful number for 1R which can be plugged in to the relevant formulas and expectancy etc. can be calculated and the question of positioning sizing addressed.

Chuck B
Roundtable Knight
Roundtable Knight
Posts: 469
Joined: Thu Apr 17, 2003 6:34 am

Post by Chuck B » Wed Mar 14, 2012 10:51 am

My point about having no means to define the risk at entry was in regard to this specific case where the OP is trying to evaluate a system and seeing strange results. No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Thu Mar 15, 2012 9:40 am

HWG wrote:This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)
Chuck B wrote:No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

I am pleased to present some good news: Trading Blox has a feature which makes this possible. In fact, easy. For those with access to the "Customers Only" section of the website, see (THIS thread) . HWG can command Trading Blox software to use his variable "estimated_risk" in its risk calculations, including the R-multiple calculations, expectancy calculations, etc. Without placing stop orders in the market. It's exactly what you want, and mirabile dictu, it's exactly what MS-DOS "Trading Recipes" software offered in 1997 with its keyword "SYSTEM" (later replaced by "MANAGER[n]").

HWG
Roundtable Fellow
Roundtable Fellow
Posts: 84
Joined: Fri Jun 04, 2010 10:21 am
Location: Omaha, NE
Contact:

Post by HWG » Thu Mar 15, 2012 11:32 am

sluggo wrote:
HWG wrote:This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)
Chuck B wrote:No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

I am pleased to present some good news: Trading Blox has a feature which makes this possible. In fact, easy. For those with access to the "Customers Only" section of the website, see (THIS thread) . HWG can command Trading Blox software to use his variable "estimated_risk" in its risk calculations, including the R-multiple calculations, expectancy calculations, etc. Without placing stop orders in the market. It's exactly what you want, and mirabile dictu, it's exactly what MS-DOS "Trading Recipes" software offered in 1997 with its keyword "SYSTEM" (later replaced by "MANAGER[n]").
Hmmm..thanks Sluggo. The Entry Day Replacement is a % correct? I changed it to -1% however didn't see any change as far as calculating the risk and R-Multiples. Will keep poking around unless I'm going in the wrong direction.

Post Reply