Walk Forward Analysis

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
bluefightingcat
Roundtable Knight
Roundtable Knight
Posts: 427
Joined: Fri Nov 20, 2015 3:09 pm

Walk Forward Analysis

Post by bluefightingcat »

I've been doing some reading about walk forward analysis. I think I have understood the principal behind it but I thought I would come here to get confirmation on whether I have understood everything properly.

1. The principal behind walk forward analysis is to see whether a system can be more profitable if the system is regularly re-optimized compared to just being optimized once. Is this correct?

2. Is it better to have a robust system that has been tested over a long test window (say 20 years) with lots of instruments and trades and then validated on OOS window (say 5 years) or would it be better to have a system that is regularly re-optimized at a regular time period or once the equity curve starts going south using a smaller data set that is more similar to the current market conditions?

3. In walk-forward analysis, the test period is divided into segments. The first segment is tested and then validated in OOS data. Then we move on to the next segment, until all segmens have been tested. Couldn't I just skip all the segments in the beginning and then just test and validate the last segment and use the parameters from that one? I'm don't understand the link between the different segments.
Tim Arnold
Site Admin
Site Admin
Posts: 9015
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Re: Walk Forward Analysis

Post by Tim Arnold »

1. I'm sure there are different thoughts on that. You can test a regular re optimization strategy with the walk forward feature.
2. You could test both and see how the historical results look. Lots of variables and opinions involved in the correlation between historical results and future results. Much depends on how the system was conceived and how the optimization process is conducted.
3. This is a historical test of the walk forward process over time. If you had optimized every x days. Going forward from today you can just use the last set of parameters.
Bob Pardo
Contributor
Contributor
Posts: 3
Joined: Sun Nov 04, 2018 6:48 am

Re: Walk Forward Analysis

Post by Bob Pardo »

1. The primary value of walk-forward analysis (WFA) is to prove robustness. The secondary benefit of WFA is to improve performance by adapting to changing market conditions.

2. In the scenario you have described, the strategy sounds robust (however, the devil's in the details, so that is somewhat of an unknown as described.) However, I suspect it would prove robust in WFA. Depending on the trading pace (long-term versus short-term) WFA may or may not offer improved performance. Another unspoken benefit of WFA is to turn the strategy off if it fails in the most current IS testing window.

3. This entirely begs the question. If you did not do a WFA then what emperically proven window sizes would you use? However, when using WFA, the process is to update the windows discovered by WFA periodically as defined by the IS and OS window sizes.

However, in conclusion, I would NEVER use a strategy that did not pass our WFA robustness testing. I have been doing this and using exclusively WFA for our robustness testing for a long time. The results speak for themselves.

This is all described in full detail in my book The Evaluation and Optimization of Trading Strategies.

Bob Pardo
bobsyd
Roundtable Knight
Roundtable Knight
Posts: 405
Joined: Wed Nov 05, 2008 2:49 pm

Re: Walk Forward Analysis

Post by bobsyd »

Hello Bob

Read your book years ago and have been a big proponent of WFA ever since as you could see by searching walk forward and my name.

Interested in your comment re “Another unspoken benefit of WFA is to turn the strategy off if it fails in the most current IS testing window.”

TB has a great facility to string OS test results together, but it is cumbersome and time consuming to determine walk forward efficiency (ratio of OS/IS profit, eg CAGR). However, the few times I have done it there was not an overall benefit to going flat if an IS test lost money. That is, overall, the times that a losing IS made money OS more than compensated for the times a losing IS lost money. However, profitable IS tests are also often followed by unprofitable IS tests.

For context, my trading is mostly automated short term trading of a forex portfolio using intraday data. I have tested many IS/OS window sizes over the years, but one of the ones I have recently found performs well is to use a month for both IS and OS. I generally use over 6 years of data, so I get around 80 IS/OS tests, with around 7,000 OS trades (i.e. 70-100 in each IS and OS test). A typical good result would be that 50 out of 80 OS tests make money.

Any thoughts on this? For a long time I stuck with more “usual” window size choices, eg 12 months OS/3 months IS, i.e. a 25% IS/OS ratio.

Bobsyd
Bob Pardo
Contributor
Contributor
Posts: 3
Joined: Sun Nov 04, 2018 6:48 am

Re: Walk Forward Analysis

Post by Bob Pardo »

Hi Bobysyd,

Glad that you have found benefit in and use WFA.

Another benefit of WFA is to identify optimal IS and OS window sizes. If the ones that you use are markedly superior to longer more "traditional" windows there is nothing wrong with using them as you have been doing.

I must say, that with all the testing we have done over the years, we never did a 1x1. Just never occurred to me. Compliments on keeping an open mind and paying attention to what the data tells you.

Did you evaluation of the benefit or lack thereof of turning off the OS after a losing IS period cover a lot of different window sizes? Was it on your 1x1? And, are you doing it based on a calendar month (June IS and July OS, etc.) and not just 22 trading days rolling? If yes, since there are behavioral patterns based upon the calendar, that may be why it is working well.

The fact that you are getting a lot of trades in each window and a lot of WFA tests certainly contributes to its effectiveness.

With very fast strategies as yours, a strategy which has highly variable values and very volatile markets with a lot of regime changes, fast windows will always perform better.

I personally feel that there is not any where near as much difference currency to currency as there is in different markets, like OL, S and NQ for example.

Have you tried your strategy and 1x1 in markets other than FX?
Post Reply