Adaptive systems vs. curve fitting

[skpatel73]

Hi all,

I'm a newbie, so please take it easy on me on my first post.

I've read a lot about how curve-fitting a system is bad, and I can definitely understand that for a static system whose rules don't change, curve-fitting is a disaster.

However, if you design a system that adapts over time by adjusting a few key parameters, in theory you can adjust to changing market conditions and perform a lot better.

In essence, you are dynamically curve-fitting, and on paper you can come up with some great looking systems.

Is this basically just as bad as curve-fitting in a static system whose parameters don't change? Or is this even curve-fitting at all? Or does it approach curve-fitting when the system can adjust itself too quickly, but it is not so much with a longer time constant?

This is the kind of thing that I can test and test, but it seems to go against conventional wisdom. Any advice?

Thanks!!
Samir

[sluggo]

They do this a lot over at the Wealth-Lab website. Hunt around for the key words "self tuning". Some of the algorithms are very sophisticated.

[Roscoe]

Excellent point. In many cases curve fitting is taken to mean something like “making things better after the factâ€

[tobbe]

Is this basically just as bad as curve-fitting in a static system whose parameters don't change?

IMHO, it’s worse.

An adaptive system has more degrees of freedom than a non-adaptive system and is by design more susceptible to curve fitting than a corresponding non-adaptive (static) system. Both types of systems need initial optimization but it is harder or impossible to find the robustness of the set of parameters that are allowed to vary in the adaptive system. Even if you back test to put constraints on how the dynamic parameters are allowed to change, the uncertainty on system performance has increased in comparison. Trying to assess the robustness of a system is essential (to me, at least) and is hard enough without the added complexity of dynamic adaptation.

There are other ways of adaptation to market changes that might be more robust though. For example if you have two systems that are negatively correlated it might make sense to try to identify the conditions that can be used to decide when to take trades from either system. And some systems are inherently adaptive (Bollinger etc).

There are many good discussions on these topics at this forum. If you haven’t already, try the search function and look for “walk forwardâ€

[James A]

I've read a lot about how curve-fitting a system is bad, and I can definitely understand that for a static system whose rules don't change, curve-fitting is a disaster.

However, if you design a system that adapts over time by adjusting a few key parameters, in theory you can adjust to changing market conditions and perform a lot better.

In essence, you are dynamically curve-fitting, and on paper you can come up with some great looking systems.

Is this basically just as bad as curve-fitting in a static system whose parameters don't change? Or is this even curve-fitting at all? Or does it approach curve-fitting when the system can adjust itself too quickly, but it is not so much with a longer time constant?

This is the kind of thing that I can test and test, but it seems to go against conventional wisdom. Any advice?

Thanks!!
Samir

Here's my take on it, from another newbie.

Curve-fitting is where you assume the data should take a particular form, like linear regression assumes the data should fit a straight line.

AFAICT adaptive systems may or may not be curve-fitting. For digital signal processing, the adaptive systems seem to be primarily curve-fitting, because the adaptive output is typically compared to an "ideal" output and adjustments are made accordingly. But adaptive systems in trading tend to be more free-form. For example, the Kaufman Adaptive Moving Average doesn't fit the data to a form; it just adjusts the degree of smoothing based on whether or not the data is trending.

So I guess the answer is "it depends."

[Bernd]

[tobbe]

I am also testing for robustness: I show the GA only some thousands stocks and test the performance of the resulting system on several sets of other thousands stocks.

Out of sample data easily becomes in sample data. As soon as a set of parameters have been discarded as a result of the system not performing well on out of sample data, the out of sample data has become in sample data since it's now part of the optimization process . In such a case testing on the out of sample data doesn't really say much.

To me, testing for robustness is trying to assess how sensitive the system is to changes in parameter values as well as how sensitive the system is to minor/major changes in the prerequisites (the environment that the system is thought to work in). This is what I find so hard to do with an adaptive system.

Do you use genetic optimization to find new functions (behaviour) on the fly or do you use it to optimize a given set of parameters for some fixed functionality? I think there's a big difference in complexity (as well as degrees of freedom).

Anyway, I'm not implying that adaptive systems don't work. It's just they're so hard to test.

This is an interesting topic and I hope you'll post updates on your results.

cheers,
tobbe

[Bernd]

[freeman]

I believe this is curve fitting.Funny enough this system has similar results on what ever i test it on(all liquid,stocks and fx demo data.It is a boli counter trend.Commisions is included but no slippage,as soon as i hit with 8% slippage it false on its backside.
CAGR: 368%
MAR : 8.17
max draw: 45%
on 10 yrs data,8590 trades
Can anyone explain this?
Hey nice to dream thou

[freeman]

Ill just make it a bit clearer on my question.
Once i realized that i did not account for slippage i purposely curved fitted it for my usd portfolio,(just for fun),after i did this i then ran it over everthing else i had,to my suprised it work on everything,(until i added slippage).i quess my question is,how can this system be so curved fitted for usd only and still have excellent results on everything else?Or maybe it is a robust system but trades to aggresively for the slippage.
Any thoughts and opinions appreciated.

[BARLI]

freeman, its not curve fitting, your code is peeking somewhere. post the code of the system and I am sure that Tim or c.f. will be glad to help you understand where you peek ...

[Forum Mgmnt]

It shouldn't really be possible to "peek" as in use postdictive data with TBB.

However, I've often seen this problem with very small stops and the "Entry Day Retracement (%)" parameter set to 0%.

If you change the value of the Global Parameter "Entry Day Retracement (%)" to 100%, how does this affect your results?

- Forum Mgmnt

[BARLI]

Forum Mgmnt, it doesn't depend on data though, I believe that he in his trading system he's entering trades on yesterday or today, or there's also an option when he uses stops/limits that predict the move and thus get him out on a profit only. I've done such mistakes myself. Another mistake I made that as peaking is that I was enetering trades on "bar+2" while placing a stop on "bar+1" it was peeking as well.freeman, try not using stop/limit orders, and enter all of your trades on tomorrow's open/close only, see how it effects your results. both of your entries and exits are supposed to be on bar+1

[Forum Mgmnt]

Barli,

I've made these sorts of mistakes myself, however, the type of peeking you refer to is not possible in TradingBlox because unlike with products like WealthLab, with TradingBlox you can only see the data at any point in the simulation that you would actually be able to see in real life. We won't let you make postdictive errors.

For example, when you enter the orders for the day, you can only see yesterday's prices. There is no ability to index bars after yesterday's bar.

TradingBlox let's you index bars backwards from today. So close[0] is yesterday's close or the last close visible as of the morning before the open; close[1] is the day before. Each day the meaning of close[0] changes to reflect another day in the future. So you can index backwards from the last day but not forward into the unknown future.

The problem with tight stops I previously mentioned can exist in TradingBlox if you set the "Entry Day Retracement (%)" parameter to 0% or something low like 5% to 10%.

The idea is that if that parameter is set to 0% and you enter on a stop order then TB will only assume your subsequent protection stop was hit if the market closes through that price. So 0% is being very optimistic. 100% is pessimistic, it assumes that if the low of the day is below your protection stop then you get a same day stopout for a loss. This means that on a day where the market climbs through your entry stop and keeps going up, TB will assume you were stopped out if the low was below your stop. The parameter setting we provide let's you be as optimistic or pessimistic in your testing as you want.

I've discussed this before when referring to the Turtle whipsaw exit which cannot really be tested reliably with EOD data. We just can't know what happens between the open and the close. Did the market rally to the high, sink to the low, rally again and close on the highs? Did it slowly climb steadily? Did it hit the highs three times during the day with two dips to the lows in between? We can't know so we let you decide how to handle it, optimistically, pessimistically, or somewhere in between.

If you use large enough stops this is not a problem; however, with small stops, say less than 0.8 ATR this problem starts to create potential inaccuracies. Stop entries with tiny protection stops will exhibit the behavior freeman sees above, huge MAR, high returns, etc.

In short, leaving "Entry Day Retracement (%)" set to 0% with small stops will give inaccurate results.

- Forum Mgmnt

[BARLI]

100% agree with you...I just tried different peaking scenarios. What are other peaking situations besides"Entry Day Retracement (%)" can arise in TBB simulations?

[freeman]

c.f. you are spot on,entry day retracement was set at 0%,the stop was set at 1.2 ATR,looking over the trades i see this much more clearly.
Barli raises a good question,is there others we should be aware of ?

[BARLI]

freeman how did your results change after you changed the parameters?

[freeman]

Anything more than 10% on slippage and entry day retrace and numbers pyramid backwards.Chart looks the same but backwards.

[BARLI]

how annualized gain changed?