Robust Optimization

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
ksberg
Roundtable Knight
Roundtable Knight
Posts: 208
Joined: Fri Jan 23, 2004 1:39 am
Location: San Diego

Post by ksberg »

Forum Mgmnt wrote:Ah, now you get to the real point of all of this, "Automatic Optimization" where what you are looking for is NOT the peak of the test, but the best Robust Peak.

If we have a mechanism for coming up with automated robustness checking and some numeric robustness measure suited to individual tastes, then we can do automatic optimization in stages like:

1) Auto-optimize the Parameters to find some candidate peaks using genetic algorithms or advancing granularity iteration.

2) Run Automatic Robustness Checks on those peaks

3) Use this information for another genetic optimization pass where we use the result of the Automatic Robustness Check as the goodness measure rather than the raw input.
I like the idea of an automatic robustness check. The suggestions for user specified epsilon or variance could work, as could two-pass variance calibration. It seems that the former would be simplier as a first step.

Ok, I'm going to rain on my own parade a bit. There might be a kink in how distributions are collected and analyzed. It always pays to ask: distribution of what? What's of interest, as c.f. mentions, is not the best peak, but best robust peak. If we're using distributions for parameter sensitivity, maybe it makes sense to look at how the robust peak varies over time, not the peak itself (which is likely to jump around). I don't have proof, but a local peak, even multiple local peaks may not equate in aggregate to the desired robust peak. On the other hand, what we would hope is that a distribution of robust peaks is a tight cluster (i.e. that we really found something optimal and stable).

I'd imagine the robust peak algorithm might use a goodness/fitness function in combination with image edge-detection techniques, or something similar.

Cheers,

Kevin
Last edited by ksberg on Wed Sep 22, 2004 7:42 pm, edited 1 time in total.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Asterix wrote:Wouldn't the Auto Optimization just return the same parameters as you've identified with your Automatic Robustness Check? I'm wondering if you would be in a circular iteration at this point.
No, the first pass is optimizing for the goodness function only.

The second genetic optimization pass is optimizing for the best value after the Robustness Check, not the single-test goodness function but the scramble everything in lots of different ways and tell me what I can expect in real life value.

The reason to do it in separate stages is that pass three will take much more time per test than pass one (perhaps 100s or 1,000s more time per iteration), so you want to have some idea what the raw parameter space looks like to narrow down the search areas for pass three. This way you will be able to do the whole test in some reasonable timeframe, like a few hours perhaps. In total, we might have to perform tens or hundreds of thousands of tests to get a good answer.

Even at unhead of speeds like 2 tests per second for a full-portfolio 25 year test, it will take 15 hours to run 100,000 complete tests. So we want to make sure we are smart about how those tests are run.
Asterix also wrote:Forum Mgmnt, if you can implement anything close to what you suggested in your last post, Veritrader will be head and shoulders above any other system trading development toolkit.
We can and will do all of this. At the end of the day, we can dream about how wonderful this or that might be, but what really matters is what we can deliver.

It's not actually a very had problem from where we sit today. Perhaps two or three weeks work on my part with a lot of good feedback from our Beta Testers along the way. That's because we've anticipated features like this and have incorporated the required support into our design. For example, the way we handle parameters internally is very flexible. It took much more work to get things working initially with this level of flexibility; but the extra work building a strong foundation makes it possible to add features like the above with much less work now.

The bad news is that we don't have the full automatic optimization scheduled for 2.0; but we will do this for the following release which is scheduled for 3 to 5 months after 2.0 ships. So by mid-2005 you will all be able to see this in action.

The Robustness Check is something we do have penciled in for 2.0, so perhaps we're following Kevin's suggestion of a good first step.

- Forum Mgmnt
Asterix
Senior Member
Senior Member
Posts: 44
Joined: Mon Apr 05, 2004 11:16 pm
Location: San Diego

Post by Asterix »

c.f. wrote:
Even at unhead of speeds like 2 tests per second for a full-portfolio 25 year test, it will take 15 hours to run 100,000 complete tests. So we want to make sure we are smart about how those tests are run.
One additional suggestion here. All instruments (futures, stocks, etc) trade in finite increments (ticks) so they don't have truly continuous prices.

When you start changing input parameters by miniscule amounts, sometimes there is no real effect on the overall results because you can't trade at the exact prices calculated by the simulations. So an input value of 2.0 might produce exactly the same results as a value of 2.001 depending on how that particular parameter is being used.

Consequently, the math precision of the optimization and robustness checking should be limited so you aren't re-running calculations on values that won't cause any changes in your results.
Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt »

Earlier, in a post that I neglected to reply to, TK wrote:How is "Parameter Sensitivity" different from this method and what new concepts does it introduce or what other questions does it help answer that Incremental Stepping doesn't? Or do the two interact and complement each other? I'm a bit confused, sorry.
Parameter sensitivity is a measure of how sensitive the results are to the variation in a given parameter. It is an indicator of degree of effect.

Stepping lets you examine a specific set of tests of a parameter across a range. This also shows you the degree of effect caused by a parameter change, so they are related concepts.

However the intent of the "Parameter Sensitivity" measure is for comparing the relative degree of effect among various parameters. So one might say if a system in relatively more sensitive to changes in parameters A and B than parameters C and D, this means that changes in Parameter A and B cause the most effect on the outcome while changes in parameters C and D have relatively less effect.

- Forum Mgmnt
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

For those who've studied calculus, "parameter sensitivity" = "partial derivative".
shakyamuni
Roundtable Knight
Roundtable Knight
Posts: 113
Joined: Wed Jun 04, 2003 9:44 am
Location: Somewhere, Hyperspace

Post by shakyamuni »

Sluggo,

Please elaborate on the procedure that you propose. I don't understand what you mean.

Thanks
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

c.f.'s definition of "parameter sensitivity" perfectly coincides with the mathematical definition of partial derivatives as used in calculus.

If you wish to compute the parameter sensitivities yourself, it'd be natural to perform ordinary garden variety numerical differentiation. Assuming the grid of results is sufficiently fine-grained, you'll get quite workable estimates of the partial derivatives. And, if you happen to be severely masochistic, you can dig out the holy bible of Abramowitz and Stegun http://www.amazon.com/exec/obidos/tg/de ... ingblox-20 to code up a program that'll calculate their N-point approximation formulae (N>3) for additional precision.

However, only do this if you really enjoy calculus.
Post Reply