Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
redbullpeter
Roundtable Fellow
Posts: 82
Joined: Mon Apr 28, 2003 9:12 am
Location: London, UK

Hi All,

As a part of testing my systems, I ran a system that traded randomly, buying/selling at market open and closing at the end of the day. The markets are stock index futures.

I noticed a consistent bias. Percentage win was 2.5% above loss and the bias was greater for long trades.

I don't know what to make of it. What conclusions can we draw if any?

Thanks

Peter

Mark Johnson
Roundtable Knight
Posts: 122
Joined: Thu Apr 17, 2003 9:49 am
I conclude that you don't yet have a good handle on the rate of convergence of your Monte Carlo approximation. The exercise of calculating an error estimate will be illuminating. Percentage wins minus percentage losses was 2.5% plus or minus what?

I ran a test even simpler than yours, as it randomly traded a randomly generated stream of "prices". I found that the rate of convergence was very slow. After 5000 trades there was still a significant error component:

Code: Select all

``````50 repetitions of (100 trades per experiment)
Best  winning percentage  63.0000
Worst winning percentage  37.0000
Avg   winning percentage  51.4800
Best  win/loss ratio    2.1498
Worst win/loss ratio    0.6218
Avg   win/loss ratio    1.0636``````
Running 50,000 random trades reduced the error to about 1/2 percent, and running five million random trades reduced the error to around 0.01 percent. If you want to repeat my experiment, help yourself:

Code: Select all

``````/* randomly go long/short and add up the random profits */
#define REPS  (50)
main()
{
int	i, j, w ;
double	x, y;
double	sumw, suml;
double	best_sumw, worst_sumw ;
double	best_suml, worst_suml ;
int	wins_best, wins_worst ;
int	nwins_all ;
double	buxw_all, buxl_all ;

extern double unirand() ;
extern double gaussrand() ;

nwins_all = 0;
buxw_all = buxl_all = 0.0 ;
best_sumw = -1e10 ;
worst_sumw = 1e10 ;
wins_best = 0 ;
wins_worst = 0 ;
for(i=0; i<REPS; i++)
{
w = 0;
sumw = 0.0 ;
suml = 0.0 ;
/* randomly choose trade direction */
/* long if x>=0.5, short otherwise */
x = unirand() ;
/* open to close amount is a random number */
y = gaussrand(0.0, 1.0) ;
if(x < 0.5) y = -1.0 * y ;
if(y > 0.0) { w++; sumw += y ; buxw_all += y ; }
if(y < 0.0) { suml += y ; buxl_all += y ; }
}
if(sumw > best_sumw) { best_sumw = sumw ; best_suml = suml ; wins_best = w ; }
if(sumw < worst_sumw) { worst_sumw = sumw; worst_suml = suml ; wins_worst = w ; }
nwins_all += w ;
}
printf("Best  win/loss ratio  %8.4f\n", fabs(best_sumw / best_suml)) ;
printf("Worst win/loss ratio  %8.4f\n", fabs(worst_sumw / worst_suml)) ;
printf("Avg   win/loss ratio  %8.4f\n", fabs(buxw_all / buxl_all)) ;
}``````

Mark Johnson
Roundtable Knight
Posts: 122
Joined: Thu Apr 17, 2003 9:49 am
One way to visualize the rate of convergence of a Monte Carlo simulation
Attachments
.png format
monte20.png (21.2 KiB) Viewed 10549 times

MCT
Roundtable Knight
Posts: 102
Joined: Fri May 16, 2003 7:27 pm
Hi peter,

I'm of the opinion, attempting to squeeze significance out of such tests is out of place.

redbullpeter
Roundtable Fellow
Posts: 82
Joined: Mon Apr 28, 2003 9:12 am
Location: London, UK
Wow, thanks for the explanation.

I increased the iterations from 10,000 to 500,000 and the bias narrowed to insignificance.

Live and learn.

However, is there a condition where a statistically significant bias could exist asides from errors and deficiencies in the testing?

Peter

Chris Murphy
Roundtable Fellow
Posts: 84
Joined: Thu May 29, 2003 12:11 pm
Location: Eugene, OR
Contact:
Mark,

As always, you are a much appreciated member of this board. Great post showing how it's just statistical noise. Good post.

Chris

jimsta
Full Member
Posts: 13
Joined: Sun Jun 15, 2003 2:45 am
Interesting posts.
Random entry is one thing, but random exit is another.
Any random exit will automatically revert to zero.
A statistical hypothesis is a claim that a result can be obtained from a sample test. The level of significance is determined by the rejection region of the bell curve. The tail of the curve is generally about 2.5%. That is why all these media polls, etc, call their margin of error at about 5%. If inferences from results are to be drawn then it would pay to stick inside the confidence intervals. Any results that come close to the tails should be tested again with more data.

crazybuddha
Full Member
Posts: 10
Joined: Thu Sep 18, 2003 1:18 pm
Location: boston

Okay. I'm sort of hijacking this thread, but I couldn't bear to add any clutter to the forum.

I am interested in the notion of a system that trades randomly and how to verify the randomness of it. I'm not talking about trading on randomly generated prices or going over the same sequence repeatedly. Rather, I mean starting at the beginning of a price series and (if necessary) jumping to the next contract.

Here is an example using an old coffee contract (included in the attachment).

Code: Select all

``````#   Example of usage:  perl base.pl <kch90.txt

\$scale = 375;		# contract size
\$acct = 100000;
\$ndays = 0;		# days left

while(<STDIN>) {           # put line into \$_
chop;                    # remove newline character from EOL

\$ndays++;

(\$date[\$ndays],\$time[\$ndays],\$price[\$ndays]) = split(/\s/);
}

\$entry = int( rand(\$ndays) );

do {			# at least one trade
\$range = \$ndays - \$entry;
\$exit = \$entry + int( rand(\$range) );
\$remainder = \$ndays - \$exit; # remaining range for next entry

\$result = \$price[\$exit] - \$price[\$entry];

if( int(2*rand()) % 2 ){  # flip a coin
\$result = -\$result;	# if one, we'll short
}

\$result *= \$scale;
\$acct += \$result;

printf("account: %8.2f\n", \$acct);

\$entry = \$exit + int( rand(\$remainder) );

} while( \$remainder > 10 ); # don't use last 10 days

``````
I was reading in Christina Ray's "The Bond Market" (p400) about a trading manager mistaking luck for skill, and it got me thinking about how to optimize a test for randomness.

For this bit of code, the price series is every tick of the March 1990 Coffee contract. The initial entry price is chosen at random. The exit is random and long/short is random. Then more trades are made as the series permits. No evaluation is made. The code is given merely as a starting point.

I welcome any suggestions or comments. Eric
Attachments
base.tar.gz

Mark Johnson
Roundtable Knight
Posts: 122
Joined: Thu Apr 17, 2003 9:49 am
I modified your code to run through the Coffee data multiple times:

Code: Select all

``````#
#  Usage:  perl base.pl <kch90.txt
#
\$iterations = 3000000;
\$scale = 375;
\$nticks = 0;

while(<STDIN>) {
chop;
(\$date[\$nticks],\$time[\$nticks],\$price[\$nticks]) = split(/\s/);
\$nticks++;
}

\$total_profits = 0.0;
for(\$j=1; \$j<=\$iterations; \$j++) {
\$acct = 0.0;
\$entry = int( rand(\$nticks) );
do {
\$range = abs( \$nticks - \$entry );
\$exit = \$entry + int( rand(\$range) );
\$remainder = \$nticks - \$exit;
\$result = \$price[\$exit] - \$price[\$entry];
if( int(2*rand()) % 2 ) { \$result = -\$result; }
\$result *= \$scale;
\$acct += \$result;
\$entry = \$exit + int( rand(\$remainder) );
} while( \$remainder > 10 ); # don't use last 10 days

\$total_profits += \$acct ;
\$avg_profit = \$total_profits / \$total_trades ;
if(1 == (\$j%5000)) { printf "%9d    %16.6f\n", \$j, \$avg_profit ; }
}``````
This code performs a "test" which consists of three millions runs through the Coffee data. The average profit per trade is accumulated. I performed this test sixteen times and plotted the results. Notice that one test, in Forest Green, shows an average profit per trade of \$1.50. For three million trades, that's pretty good money. However, another test in Bright Yellow, shows an average loss per trade of \$2.00. Six million reasons to feel sad.

When you look at the average profit per trade, it converges (very slowly!) to zero, as expected.
Attachments
.png gives small file size!
grapha.png (22.31 KiB) Viewed 10134 times

crazybuddha
Full Member
Posts: 10
Joined: Thu Sep 18, 2003 1:18 pm
Location: boston

random results redux

Although I dropped the ball on this topic, I can't get it out of my head. I mentioned the quote from The Bond Market about a desk manager needing to be able to distinguish skill from results which may have been achieved purely by chance. Recently I read Fooled By Randomness (subtitled The Role of Chance in Life and the Markets). The basic idea is that human brains are wired for seeing/creating patterns but not for discerning statistical significance. As a result, we "misinterpret" success, ignoring survivorship bias and sheer dumb luck combined with risk taking.

I have not been successful in coming up with a way to evaluate trading performance relative to that which may have been achieved by chance. This is complicated by the number of trials (e.g. trades) needed to establish long-term results. I was recently working on some gambling simulations and I was surprised at how many trials were needed to approach the theoretical expectation. Could it be that traders mostly deal with the short-term -- that is, an insufficient number of trades to establish anything other than random results??? Anyone who has developed Monte Carlo simulations can appreciate this problem.

Has anyone given this matter much (original) thought???

Rubber side down!
Eric

Ted Annemann
Roundtable Knight
Posts: 118
Joined: Tue Apr 15, 2003 7:44 pm
Location: Arizona
Some people wish to prove (in the strictest academic, mathematical sense) to an audience of skeptics, that such-and-such method unquestionably gives results better than random chance, or gives a Sharpe Ratio better than buy-and-hold, or gives a risk/reward point outside the Markowitz Efficient Frontier. Sunny Harris, Ph.D., gave several interviews in which she stated that as her goal.

Other people want to see overwhelming anecdotal evidence that such-and-such method has performed well in the past, for many years after its release into the public domain, across a wide spectrum of market conditions, political climates, oil embargoes, wars, famines, and pestilences. Arnold Bernhard (founder of Value Line) and Richard Thaler (leader of the Behavioral Finance school of economists) are among them.

Another group of people attempt to show evidence that such-and-such method is robust, meaning that it gives pleasing results across different portfolios, and/or over long periods of time, and/or with sizeable perturbations of its parameters (moving average lengths, etc). After reading the manifesto of OriginalTurtles.org, I would conclude that c.f. is one of this group.

Another group of people just want to see that some authoritative person or organization (or magazine) has tested a method and endorsed it. Possibly a sizeable portion of the vendor-system-buying public falls into this camp.

First know your audience, then select your goals, then select your tools for accomplishing those goals. If Monte Carlo is the right tool for the task, randomise away. If it isn't, then don't.

crazybuddha
Full Member
Posts: 10
Joined: Thu Sep 18, 2003 1:18 pm
Location: boston
I mentioned Monte Carlo only as an example of needing a large number of trials to arrive at an expected result.

My actual issue is the one addressed by Fooled By Randomness: just because someone has performed well doesn't mean their methods are any better than what chance would produce. They may have been lucky long enough to seem gifted or skilled. In another context, Lester Thurow discusses this in Building Wealth. In his examples, he talks about being in the right place at the right time with enough brains to recognize what you've stumbled onto (e.g. Bill Gates and IBM deal). I would tend to place the legends of trading in this group. The story seems to be: get lucky, leverage your good fortune, diversify your pursuits.

As a securities broker, I've heard lots of anecdotal evidence that supports the idea that successful people are by-and-large lucky. Most of my prospects (but of course not my clients!) got creamed over the last few years. Some of them fortuitously overweighted bonds and REITs at the right time -- mostly due to lifestyle changes. People who jumped on the stock bandwagon at the end of the nineties either came out flat or lost a ton. Some made a killing in IPOs. Some got beaten up when their start-up company options turned to dust, yet they owed taxes. Lastly, there are people who ignorantly concentrated in one stock or other way back when and made a lot of money.

And of course, anyone who trades with any frequency feels the sting of the house take (spread, commissions, fees, and taxes), so theirs is a somewhat Sisyphean task. If you try to buy low and sell high, as a discretionary trader, it's even worse. Oh, and if you are under-capitalized for the variations in your securtiy of choice, forget it. It becomes very difficult to recover from drawdowns over the long-term.

Doesn't it behoove a trader to find ways to see if she is a victim of self-delusion?? I wince every time I look at the Ibbotson stock history chart in my office and think of the poor souls who use it to justify their buy-and-hold strategies.

Eric

crazybuddha
Full Member
Posts: 10
Joined: Thu Sep 18, 2003 1:18 pm
Location: boston
I changed this so that the decision to enter/exit a trade is made randomly at every tick, rather than at random points along the life of the contract. This makes for a lot more trades but seemed to seemed to me to represent "random" trading better. I lowered Mark's iterations (and printing intervals) so it wouldn't take so long to run. It would have to be bumped up again to watch the convergence as he did.

Code: Select all

``````#
#  Usage:  perl base.pl <kch90.txt
#
\$iterations = 10000;
\$scale = 375;
\$nticks = 0;

while(<STDIN>) {
chop;
(\$date[\$nticks],\$time[\$nticks],\$price[\$nticks]) = split(/\s/);
\$nticks++;
}

\$total_profits = 0.0;

for(\$j=1; \$j<=\$iterations; \$j++) {
\$acct = 0.0;
\$open = 0;

for(\$i=1; \$i<=\$nticks; \$i++ ) {
if( rand >= 0.5 ) {  # flip a coin to see whether to trade
if( \$open == 1) {
\$exit = \$i;
\$open = 0;
# diff is pos if long, neg if short
\$result = \$price[\$exit] - \$price[\$entry];
# flip a coin to see if long or short
if( rand >= 0.5 ) { \$result = -\$result; }
\$result *= \$scale; # multiply by contract value
\$acct += \$result;  # trade is done, record result
#printf("account: %8.2f\n", \$acct);
} else {
\$entry = \$i;
\$open = 1;
}
}
}
\$total_profits += \$acct ;
\$avg_profit = \$total_profits / \$total_trades ;
if(1 == (\$j%10)) {
printf "%9d    %16.6f\n", \$j, \$avg_profit ;
}
}

``````