Requirements before buying backtester software

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Post Reply
stancramer
Roundtable Fellow
Roundtable Fellow
Posts: 60
Joined: Sat Feb 07, 2004 10:34 am
Location: Washington DC

Requirements before buying backtester software

Post by stancramer »

Some new systems testing programs are being offered for sale to the public, including but not limited to: Mechanica, Traders Studio, Trading Blocks Builder, MechTrade. These programs offer the capability to perform historical back-testing of mechanical trading systems, including advanced features like dynamic money management, simultaneously trading several systems, Monte Carlo equity curve analysis, and so forth.

They are attacking the market that is presently served by well-established incumbents, including Trading Recipes, Wealth-Lab, and (to a certain degree) Tradestation. It is natural for a trader to wonder, when would be the best time to buy one of these newcomers and try it out? Remember that this involves a substantial investment, including both money and a more precious commodity: time. Setting up a new software package, getting the historical price data arranged to its liking, feeding in some test systems, and analyzing the results takes time.

Some people love to be first. "Early adopters," they are called. They'll buy revision 1 of a new program, they'll put up with its quirks, bugs, absence of features (which are always promised to be included in a soon-forthcoming next release), mysterious crashes, occasional lack of repeatability, clumsy user interfaces, spotty documentation, and limited user base. Often these customers also cheerfully serve as unpaid workers in the software vendor's "Quality Control" effort. They find bugs, inconsistencies, wrong answers, and other problems with the software, report them to the developer, and optimistically hope for a quick resolution. What is the reward for this eager-beaver free labor? The joy of knowing you're using the newest (and hopefully best) software around.

Other people are late- or extremely-late adopters. They hate the idea of using software that still has, or still might have, bugs that produce wrong answers. Hate it, hate it, hate it! They'd much rather stick with what they know is stable and reliable, and let some other poor schmuck deal with the problems of early-rev software. Perhaps the most notable example of this approach is Larry Williams, who still uses "System Writer Plus" (Omega Research's first product, that preceded Tradestation). He even subtracts 28 years from the dates of historical prices he feeds into SWP, because it is not a Y2K-compliant program! Laugh if you wish but Larry is very successful with this approach and sees no reason to change.

For the vast majority of potential users, it's difficult to decide exactly when to invest money and time into a new software offering. Too early and you're only wasting your time, chasing down bugs and reporting errors (rather than testing trading systems and obtaining results that you trust completely). Too late and you've missed the opportunity to exploit the advantages of the program, improve your systems (or find some new ones), and increase your investment Annual Rate of Return.

In an email exchange with Murray Ruggiero, he (inadvertently!) suggested an excellent way to decide when to buy and use one of these software packages. I asked him to send me reprints of magazine reviews of his product. Murray replied (paraphrasing): It's not in good enough shape to submit for review. Stunning! He doesn't mind taking people's money for the product, which he excuses by saying he charges less for the early edition, but he'd be embarrassed for it to be reviewed!

He has provided an objective, easily verified method for deciding whether a software package is ready for serious use: Has it been reviewed yet in a mainstream magazine? If so then potential users can feel confident that the obvious bugs have been removed, the "QA suite of tests" has been created and run (and it passes), feedback from customers and real users has been incorporated into the product, the documentation is in a decent state, customer support is in place, and to sum it all up in one phrase: the software is moderately stable.

And that is my modest proposal: don't buy software testing packages before you see them reviewed in mainstream magazines. To do so is to take a huge risk: you may be using software that produces wrong answers, to make important investment decisions! Let someone else take that risk. Let them experience the horror of finding that an indicator is calculated wrong (this happened in Wealth-Lab release 1, with the EMA. Really!). Let them discover problems with the calculation of slippage, or with unintended post-dictive behavior. Let them suffer through early releases that don't quite have all the calculations and indicators and preconfigured systems, that magazine editors (and you) require.

If you're thinking about buying software sooner than that, I recommend you ask the vendor for a copy of the bug list. If they say no, and give an excuse, be very skeptical. Why is it a secret what's wrong with the program? The early-adopter eager beavers already know; why shouldn't YOU? I recommend similar skepticism towards user testimonials. These are frothy praise from frothy eager beavers, the very people most inclined to disregard problems and forgive errors. Instead, be stingy with your time (and your money). Don't make trading decisions based on the output of software that isn't good enough for Stocks and Commodities to see just yet.

My opinions.
TrendMonkey
Roundtable Knight
Roundtable Knight
Posts: 154
Joined: Fri Apr 22, 2005 9:14 pm
Location: Vancouver, Canada

Post by TrendMonkey »

That's good advice, I hope someone from TBB comments, because my mental finger has been hovering over the Buy button for a while now, but I can't quite bring myself to click...

Incidentally, having gone through many vendors' web sites in the last couple weeks, without mentioning names, I'd like to comment on what a huge turn-off it is to me when I see unprofessional web sites... eg. typos, amateur fonts and colors, broken links, etc. I'd say this is more common than not, and as far as I'm concerned, if they can't be bothered to produce a decent web site, I'm not going to trust them to sell me complicated software.

TBB and TS's sites are nicely done, though I was totally turned off by two separate customer service experiences with TS, so I won't be dealing with them anytime soon.
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

Some people love to be first. "Early adopters," they are called. They'll buy revision 1 of a new program, they'll put up with its quirks, bugs, absence of features (which are always promised to be included in a soon-forthcoming next release), mysterious crashes, occasional lack of repeatability, clumsy user interfaces, spotty documentation, and limited user base. Often these customers also cheerfully serve as unpaid workers in the software vendor's "Quality Control" effort. They find bugs, inconsistencies, wrong answers, and other problems with the software, report them to the developer, and optimistically hope for a quick resolution. What is the reward for this eager-beaver free labor? The joy of knowing you're using the newest (and hopefully best) software around.
That sounds like me and TeeBeeBee.

It is a risk and I have a stop-loss and an alternate product lined up should my TBB stop get hit.
Tim Arnold
Site Admin
Site Admin
Posts: 9015
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold »

OK, I'll bite :)

I like your description of early adopters, bleeding edge types vs. more cautious wait and see types. Corporations are like this as well. Some buy the latest and greatest, while some wait for most of the Fortune 500 to implement before they buy.

In my experience, the reason you would be an early adopter is because the software is part of your Core Competency. If the software is an accounting package, and you are a home entertainment company, then you probably don't want to be on the bleeding edge. On the other hand, if the software will help you program the latest features into your wireless home entertainment product, then you will probably jump. I would think that if you are a trader, using the latest and greatest would be a significant advantage. Helping shape the direction and functionality of a product that is core to your business is a sustainable competitive advantage vs. your competition.

The other thing that I find, being an early adopter myself, is that I learn a tremendous amount about myself and my trading strategy through the discovery involved in implementing a new trading platform. If you use the same platform and features for enough time, you wear a path through it. In this business, Larry notwithstanding, I think it's good to shake things up and try a new approach, not only for the profits but also for the educational value.

At Trading Blox, we strive for a clean professional website, a typo free and comprehensive manual, and a rock solid easy to use yet powerful product. As TBB has been in General Availability for just about 2 weeks, we have still some areas of improvement, but are fast approaching our goals.

Trading Blox Builder is ready to be reviewed, and reviews will be published as the magazines' schedules permit. There is already some early feedback from Futures Truth Magazine on this forum.

The Trading Blox Support forum is a dynamic exchange of issues and resolutions. We are open about the issues as reported, and encourage customers to share these with each other in a customer forum so we can all make informed decisions and discuss priority, impact, and resolution possibilities. We are a customer driven organization, and those that find value in influencing the direction and function are welcome to join with us in this endeavor.

Happy to answer any other questions that may arise from this excellent topic.

Tim
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

stancramer wrote: I recommend you ask the vendor for a copy of the bug list. If they say no, and give an excuse, be very skeptical. Why is it a secret what's wrong with the program?
Tim wrote:Happy to answer any other questions that may arise from this excellent topic.
Tim, here is my question: May I have a copy of the bug list, please?
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

I quite agree with Tim. Intimacy with your testing platform is a very valuable asset as is an established working relationship with those that develop the product. Both can be can be obtained by being involved from the beginning, seeing a little of the intestines behind the scenes and growing moment by moment with the product. As I said, this also comes at a risk.

I have benefited by my involvement from the start more so than the developers have benefited. It is a better deal for me than them and I am glad I got involved.
Tim wrote: At Trading Blox, we strive for a clean professional website,
Who came up with the cookie cutter styled header graphics? Attractive yet astute lady engaged in business with her laptop outdoors with a cup of coffee, the suggestive one looking over her shoulder, the content husband and wife trading team in white shirts....the space filler image of some CD's in coloured cases. Really, who's work is this? ;)
Old European
Roundtable Knight
Roundtable Knight
Posts: 144
Joined: Mon Apr 19, 2004 3:04 pm
Location: Old Europe

Post by Old European »

I think stancramer's message is pretty clear. And he definitively has a point.

Although I'm convinced that the Trading Blox team is very genuine in its effort to come up with a state-of-the-art product, I can't help having the feeling that the release of TBB version one has been a bit premature. Minimal stability is lacking. I say this after having worked happily with VeriTrader for a year and having played with Trading Blox for a few weeks.

I think most people interested in these great (future) products would be much better off buying VeriTrader 1.6.4 first and upgrading to Trading Blox at a later stage.

Cheers,

Old European
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

In this thread and others I can’t help but detect feelings of perhaps resentment, bad blood, a gung-ho ‘take no prisoners’ attitude. Product developers aren’t our enemy. If you don’t like it then don’t buy it. There seems to be self promoted sheriffs of product quality that rather than just not buy, they pick away at the low hanging fruit. TBB is not yet perfect and quite a few people seem very fast to seemingly ‘get even’ or settle a score on that account. No one can beat the markets, but by gum many will taking easy pot shots at the next best thing: beating down market related product developers. I see a self fulfilling prophecy, I see people heading down the dreary path of getting exactly what they want. Keep this up and TBB will fail and you will all smugly claim ‘I told you so’. It is as though half this board applied for jobs with TBB Inc and, on rejection, have become bitter and twisted (that was tongue in cheek). These rather strong comments of mine are not solely directed at the (frequently intelligent, consistently respected and obviously experienced) participants in this thread, more so I am speaking to an atmosphere of hostility that is becoming more and more prevalent. I am tentative regarding a few TBB glitches and I have bombarded Tim with a biblical flood of emails hitherto. Yet I decided to buy the product and if it turns out to be a losing decision then that is my loss (note that the bold font is an intentional application of the popular method bringing focus to your decision within trading forums such as this). Do you all speak so poorly about Dec corn when it spends 3 weeks moving against your position? I doubt it. Perhaps that is why Trading Blox is such a tempting target for contempt?

Given my historically prolific internet babbling (which has been well document by some mean forum participants), I am perhaps the last person the TBB team want on their side... but beggars can’t be choosers.

Maybe TB would get less attention if MC or DrVT provided a bit more juicy material like the good old days.

Thus spake Frothy Optimism.

(as an afterthought, I myself concede that I have spoken with frustration in the past and been a touch too blunt in my expression regarding TBB. The famous debugger is a point in case).
TrendMonkey
Roundtable Knight
Roundtable Knight
Posts: 154
Joined: Fri Apr 22, 2005 9:14 pm
Location: Vancouver, Canada

Post by TrendMonkey »

I'm not going to besmirch TBB in any way shape or form but, well, Corn doesn't have a web site promising that every ear will be sweet and perfect and that Corn 2.0 will have husks made of pure gold.

By which I mean to say, personally, I think some hostility towards systems vendors in general might be deserved, though I personally feel the TBB guys are a notable exception.
sluggo
Roundtable Knight
Roundtable Knight
Posts: 2987
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo »

I noticed an amusing parallel, hope you enjoy it too:
damian wrote:I have benefited by my involvement from the start more so than the developers have benefited.
Winston Churchill wrote:I have taken more out of alcohol than alcohol has taken out of me.

http://www.quotedb.com/quotes/2449
damian
Roundtable Knight
Roundtable Knight
Posts: 815
Joined: Tue Apr 15, 2003 8:43 pm
Location: dusseldorf

Post by damian »

...and no one ever beat WC when it counted.

:wink:
Post Reply