Identifying Efficiency in a 3D Matrix

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
Sean Travers
Contributing Member
Contributing Member
Posts: 8
Joined: Tue Aug 05, 2003 3:36 pm
Location: United States

Identifying Efficiency in a 3D Matrix

Post by Sean Travers »

Using tradestation's standard set of optimization parameters as building blocks (net profit, gross profit, gross loss, # of trades, roa, profit factor, % profitable, max DD), what is the best measure of linearity in an equity curve that also factors in the number of trades to evaluate a strategy's highest areas of efficiency?

Using a multi-dimensional matrix of strategy inputs and optimization outputs, I want to find loci of the most linear curves produced with the fewest trades possible.

Would this be ROA / # of trades, as an ROA per trade?

Or (Net Profit / max drawdown) / # of trades? Or is there a better measure?

Thanks very much for your help
Last edited by Sean Travers on Tue Nov 04, 2003 12:39 am, edited 1 time in total.
Bondtrader
Roundtable Knight
Roundtable Knight
Posts: 101
Joined: Mon Apr 28, 2003 11:07 am
Location: Zimbabwe

Post by Bondtrader »

Tradestation is a "red belt" program.

You want to obtain "brown belt" results. Tradestation is not capable of this.

The highest confidence path (also the biggest investment) is to roll your own software. Among the writers here on this postnews system it is called "Creating A Testing Harness". You can get exactly what you want, exactly, without invoking approximations or workarounds or dirty side-effects. And when you are all done you will have earned a "black belt" and entered a rare & elite group indeed.
Sean Travers
Contributing Member
Contributing Member
Posts: 8
Joined: Tue Aug 05, 2003 3:36 pm
Location: United States

Post by Sean Travers »

Bondtrader,

Thanks very much for your thoughts on my question. I'm gratified to learn that ROA/# trades, custom metrics or efficiency matrices in general are considered brown belt methods in your opinion, I take this to mean I'm on the right track, and I'm in full agreement that Tradestation is extremely limited when it comes to finer details of system evaluation.

I believe the best solution going forward will be a custom harness as you suggest. Time to hit the books and get ready for a few all nighters! my c++ is a little rusty :shock:

Thanks,
Sean
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

Sean,

You have hit on a subject dear to my heart, and with the concurrence of the list I feel we could explore this further - what indeed is a usful measure of system of strategy efficiency?

In addition to your suggestions, would members care to comment on (and add to) the following brief list:
  • RoA (Return on Account, where 'Account' = [(MDD * 2) + (margin * 4)])
    Mathematical Expectancy and Expectancy Score [Expectancy * Opportunity]
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Performance Measures

Post by ksberg »

Hi Ross! :-)

I'll throw a couple things into the kettle, both from Ralph Vince "Portfolio Management Formulas".

On page 45 of Ralphs starts with "Let's begin by determing what is a good measure of performance ...". He then derives a Pessimistic Return Ratio (PRR) as follows:

Code: Select all

PRR = (((W-(W^0.5)))/T)*AW) / (((L + (L^0.5)))/T)*AL)

W = number of winning trades
L = number of losing trades
T = total number of trader (winning + losing)
AW = average win
AL = average loss
In short, PRR is the profit factor weighed down inversely by the number of trades. I haven't seen PRR appear many places, but it seems reasonable enough.

The other I'll offer up is the geometric mean. I prefer this over expectation or expectancy, simply because multiplying by number of trading opportunities gives a more realistic result. Try it for yourself. Derive the expectancy (R-multiple type) or expectation (ME) and multiple times the number of trades you see in a month, times the number of months that were traded. How close is the result to actual? ... not very.

If we replaced expectancy with geometric mean, this problem disappears, because geometric mean is EXACTLY derived from the product of intermediate returns (Ralph Vince, page 65).

So, one measure might be

Code: Select all

KB Metric = (geometric mean) * (#trades / month)
I use month here, but any consistent time period would suffice. This would give us the potential expected geometric growth for a given system over a period of time. Normalizing the time measure, we can effectively compare growth characteristics of any two systems.

I think this would be a starting point for also evaluating how a system deviates from its geometric growth. For instance, we could now measure standard deviation from the projected geometric growth curve. If we wanted to get fancy, I suppose we could assume individual returns will not be guassian, and substitute another probability distribution function.

From here, given that two systems have an equal 'KB Metric', it makes sense to choose the one with the smoothest equity curve ... or least deviation from the projected geometric.

...

Truth be told, I use mostly use a combination of MAR, CAGR, and dive in from there. But, I'm always looking for better measures.

Cheers,

Kevin
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

Hi Kevin!

Maybe we might also consider the Rod Sharp/Bo Thuman Pessimistic Return on Margin (PRoM)?
Assume that you one std dev more losers and one std dev more winners that you actually did. Then calculate the total profit using the system's average win and average loss. If, using that assumption, you still have a positive rate of return, chances are you have a pretty good system.


Does that sound like it might make a worthy addition to our list?

Best,
Roscoe
Last edited by Roscoe on Tue Feb 17, 2004 3:01 am, edited 1 time in total.
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

System Performance

Post by ksberg »

Ted, thank you much for organizing the links ... very insightful discussions.

I have a question for moderator: with all these different threads, where is the best place to continue the discussion?

Thanks,

Kevin
Ted Annemann
Roundtable Knight
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona

Post by Ted Annemann »

I happen to think that "By What Measure? How Do You Know if a System is Good?" is an excellent forum for the presentation of data on the topic of "why X is my favourite way of evaluating trading system performance." It is found in Trader's Roundable Forum Index -> Testing and Simulation -> By What Measure

teda
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

I agree Ted, I started that thread to discuss this topic.

- Forum Mgmnt
Roscoe
Roundtable Knight
Roundtable Knight
Posts: 250
Joined: Sat Jan 24, 2004 2:06 am
Location: Houston TX

Post by Roscoe »

Mr Moderator: would it be possible to move this entire set of posts to the "By What Measure? How Do You Know if a System is Good?" topic as referenced above by Ted and c.f.?

In that manner we can continue the discussion without doubling up or missing something. Thanks in advance!
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Unfortunately, the forum software doesn't let you move posts to another topic. You can only move posts to a new topic. :(
Post Reply