Page 1 of 1

Testing Results Question

Posted: Mon Mar 12, 2012 12:50 pm
by HWG
I've been testing a pretty simple system over about 30 years data in 65 different worldwide futures. I've been using in sample and out of sample data to test for robustness, etc.

My question is that my results I feel are decent (for me) but if you look at the profit contribution all of the winners and losers are within 1R. In addition, my win/loss rate is nearly 50% which is a red flag for me as I would expect something more like 30/70 losses to wins or similar. So basically I am getting profitable results but not the big huge winners (10-15R).

It seems that if I'm happy with the results then it shouldn't matter but I have to say I'm a bit surprised there weren't bigger wins.

To be clear this is a long only system with no profit target.

Would like to hear some thoughts on this and maybe some ideas how I'm missing these bigger moves or whatnot.

'Preciate it.

Posted: Mon Mar 12, 2012 1:37 pm
by Robwynge
Well, how are you instructing the system to exit trades? And when you look at the trades tab, is the system exiting based on how you expected it to?

Posted: Mon Mar 12, 2012 1:44 pm
by rossb34
It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.

Posted: Mon Mar 12, 2012 1:48 pm
by HWG
Robwynge wrote:Well, how are you instructing the system to exit trades? And when you look at the trades tab, is the system exiting based on how you expected it to?
I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.

Posted: Mon Mar 12, 2012 1:50 pm
by HWG
rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
Right? That's why I posted this question. It's not a pre-supplied system but one I picked up through the Blox Marketplace and combined with some pre-supplied blox.

My entry/exits look ok but just doesn't make sense why everything is so clustered around the mean so to speak.

Posted: Mon Mar 12, 2012 2:09 pm
by Robwynge
HWG wrote:I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.
So then I am confused about why you are confused. :-)

If it's doing what you want by following the signals of your indicator faithfully, is your indicator such that it only has small winners? If not, and your indicator sometimes triggers exits after small moves, and other times triggers exits after large moves, then clearly something must be wrong with your code. Maybe soemthing is wrong with the risk manager forcing you out of trades when your risk gets "too large"? Just throwing something out there...

Posted: Mon Mar 12, 2012 3:49 pm
by HWG
Robwynge wrote:
HWG wrote:I have and indicator that when it reaches a threshold it goes long and then exits when it reverses to another level.

Double checked and yes it is entering and exiting how I'm instructing.
So then I am confused about why you are confused. :-)

If it's doing what you want by following the signals of your indicator faithfully, is your indicator such that it only has small winners? If not, and your indicator sometimes triggers exits after small moves, and other times triggers exits after large moves, then clearly something must be wrong with your code. Maybe soemthing is wrong with the risk manager forcing you out of trades when your risk gets "too large"? Just throwing something out there...
Right, its not that its not working properly its just not what I expected so trying to figure out why. The more interesting part is what rossb34 said where the expectation is 0.0 and Average Risk Percent is N/A. That doesn't seem right to me. I will go through checking again. Cheers.

Posted: Mon Mar 12, 2012 4:46 pm
by Aaron01
rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
With an average holding period of ~198 days and an average trade return of ~1.75%, I would venture there's a bug in the code, or he's betting so small he can't fully capitlize on the moves.

Posted: Mon Mar 12, 2012 4:51 pm
by HWG
Aaron01 wrote:
rossb34 wrote:It seems odd to me that all winners are less than 1R for a system with no profit target. Are you using a pre-supplied system or a custom trading system? Another thing that jumps out is the Expectation is 0.0 and Average Risk Percent is N/A. There could be a bug in your code, data, or both.
With an average holding period of ~198 days and an average trade return of ~1.75%, I would venture there's a bug in the code, or he's betting so small he can't fully capitlize on the moves.
I'm using starting test equity of 1mm and 2% risk per trade so seems big enough to make a difference. I did go back and look to see what percentage wise some of my biggest winners and losers were. Had some in the 60-70% range and a bunch in the 30s. Biggest losers were in the 5s and a few in the 6-8% range so seems as if some of those would be more than 1R or maybe that 1R is not registering properly.

This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)

Posted: Mon Mar 12, 2012 6:53 pm
by sluggo
At least one of the presupplied systems that ships with Trading Blox software, exhibits the same behavior. See figure below.

If anybody is really interested, I suppose they can try all eleven presupplied systems, one by one, to find out which of them behave this way.

Posted: Tue Mar 13, 2012 9:37 am
by drm7
This might be a bit tedious, but you should use a charting program to bring up a chart with the necessary indicators and go through 25 to 50 trades BY HAND, to see entry/exit/risk, etc. This should help you figure out if the system matches the output.

Posted: Tue Mar 13, 2012 9:46 am
by LeviF
I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.

Posted: Tue Mar 13, 2012 10:05 am
by HWG
LeviF wrote:I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.
Correct, no stops. System exits when the indicator changes a certain way.

So if that's the case then it can't really calculate the R-multiples as far as profit contribution right? That makes sense as I said before I have some winning trades in the 30-70% range and losing trades in the 5-8% range so this would appear to be further out on the curve then what its showing.

Posted: Wed Mar 14, 2012 8:15 am
by Chuck B
If the exit (stop) is undefined when (before) an entry signal is triggered, then you have no means to define the risk on the trade or size the position based on that risk.

Posted: Wed Mar 14, 2012 9:37 am
by babelproofreader
I assume you are not using (or properly using) stops. You cant calculate R-multiples, expectancy, avg risk, etc without stops.
If the exit (stop) is undefined when (before) an entry signal is triggered, then you have no means to define the risk on the trade or size the position based on that risk.
I'm afraid I disagree with these two statements; it might not be possible for TB to calculate R-multiples, expectancy, average risk etc. without a hard coded stop, but these calculations can be done if you have access to the history of the losses. These losses will form a distribution and thus measurements can be taken:
  • the average loss can be taken as 1R
    the average loss plus 1/2 standard deviations can be taken as 1R
    if the loss distribution is highly non-normal, bootstrap it and take the bootstrap average, with or without +1/2 bootstrap standard deviations as 1R
    take a suitably high nth percentile value as 1R
    some other "robust" measure of the distribution can be taken as 1R
the point being that having the history of losses enables one to come up with a meaningful number for 1R which can be plugged in to the relevant formulas and expectancy etc. can be calculated and the question of positioning sizing addressed.

Posted: Wed Mar 14, 2012 10:51 am
by Chuck B
My point about having no means to define the risk at entry was in regard to this specific case where the OP is trying to evaluate a system and seeing strange results. No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

Posted: Thu Mar 15, 2012 9:40 am
by sluggo
HWG wrote:This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)
Chuck B wrote:No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

I am pleased to present some good news: Trading Blox has a feature which makes this possible. In fact, easy. For those with access to the "Customers Only" section of the website, see (THIS thread) . HWG can command Trading Blox software to use his variable "estimated_risk" in its risk calculations, including the R-multiple calculations, expectancy calculations, etc. Without placing stop orders in the market. It's exactly what you want, and mirabile dictu, it's exactly what MS-DOS "Trading Recipes" software offered in 1997 with its keyword "SYSTEM" (later replaced by "MANAGER[n]").

Posted: Thu Mar 15, 2012 11:32 am
by HWG
sluggo wrote:
HWG wrote:This is what I'm using for risk:

estimated_risk = 3.0 * MAX(myATR, (5.0 * instrument.minimumTick))

ncars = riskPerTrade * system.totalEquity / (estimated_risk * instrument.BigPointValue)

order.setQuantity(ncars)
Chuck B wrote:No doubt you can come up with many different possibilities to "create" an initial exit to apply to each entry as you point out. My point was that he/she needs *something* defined as an exit for each occurrence of an entry.

I am pleased to present some good news: Trading Blox has a feature which makes this possible. In fact, easy. For those with access to the "Customers Only" section of the website, see (THIS thread) . HWG can command Trading Blox software to use his variable "estimated_risk" in its risk calculations, including the R-multiple calculations, expectancy calculations, etc. Without placing stop orders in the market. It's exactly what you want, and mirabile dictu, it's exactly what MS-DOS "Trading Recipes" software offered in 1997 with its keyword "SYSTEM" (later replaced by "MANAGER[n]").
Hmmm..thanks Sluggo. The Entry Day Replacement is a % correct? I changed it to -1% however didn't see any change as far as calculating the risk and R-Multiples. Will keep poking around unless I'm going in the wrong direction.