Does TBB support walk-forward tests?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
TK
Roundtable Knight
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Does TBB support walk-forward tests?

Post by TK » Tue Nov 10, 2009 1:59 pm

Does/Will TBB support fully automated walk-forward tests?

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Tue Nov 10, 2009 3:05 pm

Have you tried this?
Attachments
ug.png
ug.png (26.41 KiB) Viewed 6389 times

Tim Arnold
Site Admin
Site Admin
Posts: 8540
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Tue Nov 10, 2009 3:15 pm

Trading Blox supports Start Date Stepping, which is a great way to check the robustness of your system over many different start dates.

Trading Blox does not support the conventional definition of Walk Forward Testing because the out of sample data becomes in sample far too quickly, so the benefit of this approach can be overstated leading to over confidence.

bobsyd
Roundtable Knight
Roundtable Knight
Posts: 376
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd » Tue Nov 10, 2009 3:33 pm

The only way I've been able to do it is to:

1. Set up Print Output to print dates, parameter values and objective function (eg MAR) you want to optimize

2. Do optimization run using start date stepping

3. Sort Print Output by date then by objective function in Excel

4. Do walk forwards based on best objective function result for each date range


VERY time consuming. If anyone has come with anything quicker it would be great to hear about it.

It would be wonderful to have it automated in TB. Doesn't seem like it would be that hard. For some reason I don't understand, there doesn't seem to have been a great demand for it from users based on searching past posts. On the one hand, Pardo (proponent of Walk Forward testing) and his books are often mentioned in the forum, but the general gist of the conversations leads to "it isn't the best way to go" conclusions which don't convince me. Sure there are potential issues (especially for LTTF in regards to maybe not enough trades), but I don't see a better way to go for short term/swing trading system development.
Last edited by bobsyd on Tue Nov 10, 2009 3:57 pm, edited 1 time in total.

bobsyd
Roundtable Knight
Roundtable Knight
Posts: 376
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd » Tue Nov 10, 2009 3:38 pm

Tim, didn't see your post before I made mine. Don't understand the issue with

"the out of sample data becomes in sample far too quickly, so the benefit of this approach can be overstated"

Could you please expand on this?

Tim Arnold
Site Admin
Site Admin
Posts: 8540
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Tue Nov 10, 2009 5:17 pm

You have a binary option when it comes to out of sample data. When the data is used for more than parameter test on the same system, it becomes part of the curve fit conundrum. The only way to remain pure is to throw away the system design and start from scratch, the moment the out of sample data shows an unacceptable system performance.

There is nothing wrong with dynamic parameters -- that is a design consideration that is quite valid. If you compute a channel based on ATR, or allocate system equity percent based on performance, or even change system parameters based on performance or market types such as consolidating or expanding, then you have a dynamically adjusting system. You can test what you trade and trade what you test.

The other problem is assuming that the 'optimal' parameter set is easily identifiable from a back test. Often if you look at the surface map, the most 'robust' parameter set is not the 'best' parameter set at all.

TK
Roundtable Knight
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Post by TK » Tue Nov 10, 2009 5:51 pm

I am with bobsyd on this one.

I consider walk-forward tests the closest thing to real-life trading and the final check on the validity of my backtests. If done properly, walk-forward testing is a much better indicator of a real-time performance than back testing. The latter tends to be curve-fitted to some degree no matter how prudent a back-tester you are.

About two years ago, I became increasingly curious about the differences in the 2000s performance of JWH and Bill Dunn on one hand, and the "London Camp" (AHL / Winton Capital / Aspect Capital) on the other. While all of them are trend-followers and mechanical system traders, it was clear to me that there must be some important differences in the way they approach the systematic trading game. I started digging the Web, and after reading dozens and dozens of press articles, industry reports, interviews, studies and disclosure documents – all public domain anecdotal evidence just waiting online for you to be found and discovered – I have found out the following:

(1) Markets DO CHANGE. Markets are non-stationary. While many people like comparing markets to poker, pointing out many similarities, they often overlook a major difference. Poker rules don’t change. Market rules do. If you really have to compare markets to poker, imagine a poker game in which the number of Aces in the deck changes from one deal to another. There may be four Aces in the deck in deal #1, five Aces in deal #2, and no Ace in deal #3. You never know the exact number, just a fuzzy range. Now, good luck with calculating the odds of your hand in such a game! On the other hand, market are not random, either. They change but slowly enough for you to take advantage of their non-random component.

(2) The "London Camp" has been smart enough to account for market non-stationarity for a long time now. David Harding of Winton Capital admitted (in Financial Times) to tuning their systems once a month. Anthony Todd of Aspect Capital repeatedly stressed the evolutionary nature of markets and the need to adapt or die. AHL in their fund manager profile document revealed they use out-of-sample testing in their system development process.

(3) While the old-school system traders frown upon the idea of re-optimization or using different system parameters for different markets, funds such as AHL make money doing just that. They do reoptimize their systems, and they do use different parameter values for different sectors. And they tell you about it in their online corporate documents.

Now, if you accept that markets are indeed non-stationary, then periodic reoptimization becomes a must. To me, the best approach is to use walk-forward out-of-sample testing since you can emulate (almost exactly) the backest-to-real_life_trading process.

As for my personal trading, I’m a private trader using two simple short-term / swing systems (of Donchian and volatility breakout type) on the CSI_Data Market #953. After a lot of struggle in 2006 and 2007, I bought the Pardo book about two years ago. After reading it, I decided to include walk-forward testing in my system development and boy, it made a change! I made 100%+ in 2008, and I am up more than 60% this year. I consider walk-forward testing and the ideas from Pardo’s book instrumental in my success.

I attach the AHL Manager Profile document. It is an excellent read and offers great ideas for trend-following mechanical system developers.
Attachments
ahl_manager_profile(2).pdf
(178.72 KiB) Downloaded 841 times

bobsyd
Roundtable Knight
Roundtable Knight
Posts: 376
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd » Tue Nov 10, 2009 6:21 pm

Great post TK - of course I'm biased though! Was writing the below, when I saw your post - think it will still add to what I think is a very important discussion.

Re: "binary option" - Yes, but as I understand it (largely from Pardo's EOTS) there are many aspects of curve fitting, all optimization has some element of it, but one shouldn't throw out the baby with the bath water.

Overfitting, as Pardo describes it, is what one should try to avoid. At the end of the day it is the results that matter. If you do, say, 12 walkforwards and 9 of them are profitable, is that random chance? What if it's 100 walkforwards over different markets and 75 of them are profitable?

One of the chicken and egg difficulties at the moment is that it is so computer and person time intensive to do the research that I suspect not many people posting to this forum have done it in a systematic way. Would be great to hear from anyone who has done at least some true walkforward analysis (i.e. multiple sequential walkforwards). My limited experience so far is promising, but hard to do enough to get anywhere close to a conclusion.

Re: "surface maps" This isn't unique to walkforward testing. Looking at 2 dimensional maps when you have, say, 4 or more adjustable parameters gives only limited information about how to find the broad high utopian hills of robustness as opposed to the dangerous daggers of spikes. Even if you only have 2 significant parameters, it would still be useful to have an inbuilt customizable algorithm rather than just looking and making judgments based on looking at a map.

plan_trader
Contributing Member
Contributing Member
Posts: 7
Joined: Fri Oct 30, 2009 11:51 pm
Location: Queensland, Australia
Contact:

Post by plan_trader » Fri Nov 13, 2009 2:52 am

TK,

Do you have a quick summary of the performance of the managers you mentioned. Failing that, where would you find a summary. I went to a couple of the website and it is a bit of a drama to register.

Thanks

TK
Roundtable Knight
Roundtable Knight
Posts: 167
Joined: Tue Apr 15, 2003 5:45 pm

Performance Tables

Post by TK » Fri Nov 13, 2009 3:23 am

Look at the tables below.

Please also note that John W Henry first and flagship program was Original Investment. I think it was closed in mid-2005 after 23 years in operation. At the time it was terminated, over these 20+ years, it had returned less than "Buy and Hold S&P500" over the same period. Its statistics and performance are now conveniently removed from JWH website as well as from CTA tracking websites such as iasg.com or autumngold.com.

As for Dunn, if you look up the WMA program at iasg.com (link), you will find out that its last equity high was in February 2003, so despite the nice recovery last year, the program has been in a drawdown for 6.5 years now.
Attachments
ahl_diversified.png
AHL Diversified
ahl_diversified.png (80.11 KiB) Viewed 6170 times
winton_capital.png
Winton Capital
winton_capital.png (4.39 KiB) Viewed 6168 times
JWH_financial&metals.png
John W Henry - Financial and Metals
JWH_financial&metals.png (4.5 KiB) Viewed 6168 times
Dunn_WMA.png
Dunn Capital - WMA
Dunn_WMA.png (4.4 KiB) Viewed 6168 times

bobsyd
Roundtable Knight
Roundtable Knight
Posts: 376
Joined: Wed Nov 05, 2008 2:49 pm

Post by bobsyd » Sun Nov 15, 2009 5:16 pm

See Feature Requests - Automated Walkforward Testing

Post Reply