Requirements before buying backtester software
Posted: Thu Jul 14, 2005 9:21 am
Some new systems testing programs are being offered for sale to the public, including but not limited to: Mechanica, Traders Studio, Trading Blocks Builder, MechTrade. These programs offer the capability to perform historical back-testing of mechanical trading systems, including advanced features like dynamic money management, simultaneously trading several systems, Monte Carlo equity curve analysis, and so forth.
They are attacking the market that is presently served by well-established incumbents, including Trading Recipes, Wealth-Lab, and (to a certain degree) Tradestation. It is natural for a trader to wonder, when would be the best time to buy one of these newcomers and try it out? Remember that this involves a substantial investment, including both money and a more precious commodity: time. Setting up a new software package, getting the historical price data arranged to its liking, feeding in some test systems, and analyzing the results takes time.
Some people love to be first. "Early adopters," they are called. They'll buy revision 1 of a new program, they'll put up with its quirks, bugs, absence of features (which are always promised to be included in a soon-forthcoming next release), mysterious crashes, occasional lack of repeatability, clumsy user interfaces, spotty documentation, and limited user base. Often these customers also cheerfully serve as unpaid workers in the software vendor's "Quality Control" effort. They find bugs, inconsistencies, wrong answers, and other problems with the software, report them to the developer, and optimistically hope for a quick resolution. What is the reward for this eager-beaver free labor? The joy of knowing you're using the newest (and hopefully best) software around.
Other people are late- or extremely-late adopters. They hate the idea of using software that still has, or still might have, bugs that produce wrong answers. Hate it, hate it, hate it! They'd much rather stick with what they know is stable and reliable, and let some other poor schmuck deal with the problems of early-rev software. Perhaps the most notable example of this approach is Larry Williams, who still uses "System Writer Plus" (Omega Research's first product, that preceded Tradestation). He even subtracts 28 years from the dates of historical prices he feeds into SWP, because it is not a Y2K-compliant program! Laugh if you wish but Larry is very successful with this approach and sees no reason to change.
For the vast majority of potential users, it's difficult to decide exactly when to invest money and time into a new software offering. Too early and you're only wasting your time, chasing down bugs and reporting errors (rather than testing trading systems and obtaining results that you trust completely). Too late and you've missed the opportunity to exploit the advantages of the program, improve your systems (or find some new ones), and increase your investment Annual Rate of Return.
In an email exchange with Murray Ruggiero, he (inadvertently!) suggested an excellent way to decide when to buy and use one of these software packages. I asked him to send me reprints of magazine reviews of his product. Murray replied (paraphrasing): It's not in good enough shape to submit for review. Stunning! He doesn't mind taking people's money for the product, which he excuses by saying he charges less for the early edition, but he'd be embarrassed for it to be reviewed!
He has provided an objective, easily verified method for deciding whether a software package is ready for serious use: Has it been reviewed yet in a mainstream magazine? If so then potential users can feel confident that the obvious bugs have been removed, the "QA suite of tests" has been created and run (and it passes), feedback from customers and real users has been incorporated into the product, the documentation is in a decent state, customer support is in place, and to sum it all up in one phrase: the software is moderately stable.
And that is my modest proposal: don't buy software testing packages before you see them reviewed in mainstream magazines. To do so is to take a huge risk: you may be using software that produces wrong answers, to make important investment decisions! Let someone else take that risk. Let them experience the horror of finding that an indicator is calculated wrong (this happened in Wealth-Lab release 1, with the EMA. Really!). Let them discover problems with the calculation of slippage, or with unintended post-dictive behavior. Let them suffer through early releases that don't quite have all the calculations and indicators and preconfigured systems, that magazine editors (and you) require.
If you're thinking about buying software sooner than that, I recommend you ask the vendor for a copy of the bug list. If they say no, and give an excuse, be very skeptical. Why is it a secret what's wrong with the program? The early-adopter eager beavers already know; why shouldn't YOU? I recommend similar skepticism towards user testimonials. These are frothy praise from frothy eager beavers, the very people most inclined to disregard problems and forgive errors. Instead, be stingy with your time (and your money). Don't make trading decisions based on the output of software that isn't good enough for Stocks and Commodities to see just yet.
My opinions.
They are attacking the market that is presently served by well-established incumbents, including Trading Recipes, Wealth-Lab, and (to a certain degree) Tradestation. It is natural for a trader to wonder, when would be the best time to buy one of these newcomers and try it out? Remember that this involves a substantial investment, including both money and a more precious commodity: time. Setting up a new software package, getting the historical price data arranged to its liking, feeding in some test systems, and analyzing the results takes time.
Some people love to be first. "Early adopters," they are called. They'll buy revision 1 of a new program, they'll put up with its quirks, bugs, absence of features (which are always promised to be included in a soon-forthcoming next release), mysterious crashes, occasional lack of repeatability, clumsy user interfaces, spotty documentation, and limited user base. Often these customers also cheerfully serve as unpaid workers in the software vendor's "Quality Control" effort. They find bugs, inconsistencies, wrong answers, and other problems with the software, report them to the developer, and optimistically hope for a quick resolution. What is the reward for this eager-beaver free labor? The joy of knowing you're using the newest (and hopefully best) software around.
Other people are late- or extremely-late adopters. They hate the idea of using software that still has, or still might have, bugs that produce wrong answers. Hate it, hate it, hate it! They'd much rather stick with what they know is stable and reliable, and let some other poor schmuck deal with the problems of early-rev software. Perhaps the most notable example of this approach is Larry Williams, who still uses "System Writer Plus" (Omega Research's first product, that preceded Tradestation). He even subtracts 28 years from the dates of historical prices he feeds into SWP, because it is not a Y2K-compliant program! Laugh if you wish but Larry is very successful with this approach and sees no reason to change.
For the vast majority of potential users, it's difficult to decide exactly when to invest money and time into a new software offering. Too early and you're only wasting your time, chasing down bugs and reporting errors (rather than testing trading systems and obtaining results that you trust completely). Too late and you've missed the opportunity to exploit the advantages of the program, improve your systems (or find some new ones), and increase your investment Annual Rate of Return.
In an email exchange with Murray Ruggiero, he (inadvertently!) suggested an excellent way to decide when to buy and use one of these software packages. I asked him to send me reprints of magazine reviews of his product. Murray replied (paraphrasing): It's not in good enough shape to submit for review. Stunning! He doesn't mind taking people's money for the product, which he excuses by saying he charges less for the early edition, but he'd be embarrassed for it to be reviewed!
He has provided an objective, easily verified method for deciding whether a software package is ready for serious use: Has it been reviewed yet in a mainstream magazine? If so then potential users can feel confident that the obvious bugs have been removed, the "QA suite of tests" has been created and run (and it passes), feedback from customers and real users has been incorporated into the product, the documentation is in a decent state, customer support is in place, and to sum it all up in one phrase: the software is moderately stable.
And that is my modest proposal: don't buy software testing packages before you see them reviewed in mainstream magazines. To do so is to take a huge risk: you may be using software that produces wrong answers, to make important investment decisions! Let someone else take that risk. Let them experience the horror of finding that an indicator is calculated wrong (this happened in Wealth-Lab release 1, with the EMA. Really!). Let them discover problems with the calculation of slippage, or with unintended post-dictive behavior. Let them suffer through early releases that don't quite have all the calculations and indicators and preconfigured systems, that magazine editors (and you) require.
If you're thinking about buying software sooner than that, I recommend you ask the vendor for a copy of the bug list. If they say no, and give an excuse, be very skeptical. Why is it a secret what's wrong with the program? The early-adopter eager beavers already know; why shouldn't YOU? I recommend similar skepticism towards user testimonials. These are frothy praise from frothy eager beavers, the very people most inclined to disregard problems and forgive errors. Instead, be stingy with your time (and your money). Don't make trading decisions based on the output of software that isn't good enough for Stocks and Commodities to see just yet.
My opinions.