Optimization - Independently testing parameters

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
bluefightingcat
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Fri Nov 20, 2015 3:09 pm

Optimization - Independently testing parameters

Post by bluefightingcat »

In my current test I have 5 different parameters. I have been wondering whether it is wiser to test each parameter independently and compare it to a base system or whether to carry out a Brute Force backtest with all parameters stepped. Of course the brute force test would take a long time to complete.

I read a book that said that you should start with a baseline system and then test each change (e.g. adding a hard stop or adding a trailing stop) independently. Then when all the different aspects have been independently tested compared to the base system. The best "bloxs" so to speak for the system should be combined.

However my concern is that different aspects of the system might not necessarily be completely independent and that changing one parameter setting in one place will cause a change in the ideal setting for another parameter somewhere else?

How do you optimize your systems?
Tim Arnold
Site Admin
Site Admin
Posts: 9015
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Re: Optimization - Independently testing parameters

Post by Tim Arnold »

The first rule of optimization is, don't.

But seriously, assuming the goal is a robust system, with robust defined as high probability of future performance similar to past performance, be very careful to 'keep it simple' and reduce the number of parameters. Fully understand each and why they are included. For each parameter the risk of robust breakdown is greatly increased.

Test with a large set of markets over a large number of years.

Vary in two parameter pairs and use the surface map to determine if they are high influence parameters and/or related. If they are related then you should step them together. If they are not high influence then eliminate.

Use large steps to start. No need to get super granular on the first pass of parameter investigation and general goodness concepts.

Never pick the 'best' historical parameter set -- be sure to pick a plateau on the surface map that makes sense.

Be careful with slippage, commission, interest and other administrative assumptions. It's easy to throw out ideas based on invalid slippage, stop distance, entry day retracement assumptions.
Post Reply