In your experience as a system trader... 80/40 or 40/40 ?

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
AFJ Garner
Roundtable Knight
Roundtable Knight
Posts: 2040
Joined: Fri Apr 25, 2003 3:33 pm
Location: London
Contact:

Post by AFJ Garner » Wed Feb 28, 2007 1:47 pm

You must predict the future, and hope that you have predicted well.
Precisely my feeling. The same applies accross the investment universe. It is little good relying solely on statistical analysis. It is futile to claim that mechanical systems trading requires no human judgement.

You have to judge now what you think most likely to work tomorrow and for the next few months if not years. Charts like those of the equity curves above show very clearly the trend in favour of wider stops and longer term systems in recent years.

But it is now down to judgement as to where such trends go from here. I have placed my bets and will see what happens. But I have diversified accross other investments and styles and am aiming to do so further.

RedRock
Roundtable Knight
Roundtable Knight
Posts: 939
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Feb 28, 2007 1:55 pm

Most excellent points Sluggo!

I have taken a defensive posture of late. And yes at the risk of missing out on a windfall. Rather than a synthetic data set I have sought a system and combination of parameters which performs consistently, mmmm has performed consistently, on not only futures but also stocks forex and most any other data series I could throw at it. I'm confident that its as likely to continue in this manor as anything my current knowledge and 'power' level allows. Its my best guess. Prediction. If this pans out Ill look to ratchet up risk or diversify with a time frame spread. Only time will tell.

Perhaps the synthetic series could be built using random inputs within a range of plausible and not so likely, not user specified per se. It could just be another tool. Not the magic formula for spinning thread to gold.

Like the MC sims, Could it be a useful tool? Could it be another piece of rope on which to hang oneself? Either Ill guess.

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Post by sluggo » Wed Feb 28, 2007 3:06 pm

I think a big problem with the very idea of synthetic data, is that we don't even know what we want. Because our systems trade multiple different markets simultaneously, we certainly need different markets' synthetic price streams to be correlated with each other, since real markets are correlated with each other (as Tuesday 27 Feb 2007 vividly demonstrated!). Neither the Barnes method nor the Chande method of synthetic data generation, mentioned earlier in the thread, makes any attempt to capture or to create correlation between markets. Assuredly this is because these methods were developed by people of the "Single System, Single Market, Single Contract Position Size" mind-set, in the olden days before Blox et al.

But I think we don't have any idea how much correlation we want, nor how we want it to vary with time. Neither do we know among which markets we want that correlation to occur. Historically, correlations haved changed with time, and then at price shock events, everything suddenly becomes correlated. It seems to me to be asking a lot, to require synthetic data generators to "realistically" simulate the shifting correlations among markets, that have been observed in the past, and furthermore to "realistically" forecast and simulate how this may work in the future.

And speaking of price shocks (Silver+Gold mania of 1979-1980, Stock Index crash of 1987, Petroleum madness in Kuwait War of 1990, Natural gas mega-spike during Shock and Awe bombing in 2003, etc, etc), do you or don't you want price shocks to appear in your synthetic data? If so, how often? More often than they occurred in the past? Less often? How big do you want the shocks to be? Bigger than in the past? Smaller? How many previously-correlated markets do you want to participate in the shock? How many previously-uncorrelated markets do you want to participate in the shock? In Bahston they say, "Its enough to make you wanna bahf."

zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Synthetic data...

Post by zentrader » Wed Feb 28, 2007 3:49 pm

@all,

simply said:
I think most of you think to complicate concerning this topics.

Synthetic data (when generated in a"intelligent" way...) expands the area of testing.

If a system is successful not only with its regular historical data, but also with such generated synthetic data you can assume, that such a system is better prepared for changing markets.

Simple enough.

Clear, that this is not a rule, that such a system survives at all thinkable and possible market conditions but it's a step further (and for many traders, who are astonished about the differences between their backtests and real trading a big step further !!!).

Only my two cents.

bye,
zentrader
Last edited by zentrader on Wed Feb 28, 2007 3:53 pm, edited 1 time in total.

Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt » Wed Feb 28, 2007 3:52 pm

I'm of the opinion that synthetic data is basically useless. For it to be useful it would have to be more likely to be representative of the future than actual historical trading data. I don't see any rationale for why this might be the case, nor have I ever seen anyone propose one.

The past is all we have. I wish there were crystal balls for this sort of thing but there aren't.

What does it mean when our systems perform well on synthetic data? That they perform well on synthetic data, nothing more.

Since synthetic data is less likely to represent the future than actual data, testing using synthetic data is less likely to give results which will indicative of what we will encounter when trading in the future. Since we have enough problems with historical data not representing the future, I don't see any reason to use synthetic data if it is even less predictive.

- Forum Mgmnt

zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

Syntethic data...

Post by zentrader » Wed Feb 28, 2007 4:07 pm

Forum Mgmnt,

Perhaps we have a different definition of synthetic data.

The kind of synthetic data, which I use and generate has the following generation process:

1. basis are the original data sets (historical data) and their characteristics
2. the time series of this historical data sets are then changed using random components
3. optional, additional customizing of the generation is possible

This kind of synthetic data generation produces data sets which can exist in the future , but don't have to.

But if a system also works with such data generating similar system test report results, the danger of over optimization is banned and the chance, that such a system survives under different market conditions rises...

Imho, data generation ("data scrambling") is one of most important methods to contribute to a validation of a trading system and it's very usefull!

bye,
zentrader

Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt » Wed Feb 28, 2007 4:30 pm

zentrader,

Sorry, I must disagree.

If you take past data and then determine that a certain set of characteristics are sufficient to define that data, you can then proceed to generate new data using those characteristics, I don't doubt that this can be done and that most people be impressed with each level of sophistication you add to your model and definition of price movement, with how closely it begins to resemble actual price data.

I have another perspective. I believe the model will be more reflective of your beliefs about how the market works than reality, and to the extent your beliefs differ from reality you will end up having worse data than real data.

Besides, there is no lack of data available for us to test with. You can test thousands of market years of futures data alone. With stocks we have hundreds of thousands of years of price data. I don't see how creating more data that is based on a particular set of assumptions about price movement will help.

- Forum Mgmnt

nickmar
Roundtable Knight
Roundtable Knight
Posts: 191
Joined: Tue Oct 26, 2004 12:38 pm
Contact:

Post by nickmar » Wed Feb 28, 2007 4:40 pm

How about the use of randomized portfolios instead of synthetic data as a means of testing system robustness?

About a year ago, I wrote a Portfolio Manager blox which generates random portfolios based on specific sector constraints (see attached). The sector names must be in a specific format and need to be included in the "Group" field of every market in the Futures Dictionary - here's a list of the sector names currently used by the Blox: Currencies, Energies, Grains, Long_Rates, Short_Rates, Meats, Metals, Softs, Stock_Indices. You should be able to easily make changes to the Blox in order to adapt it to the sector names that you use but feel free to contact me if you need some guidance.

The "Simulations" parameter is used to specify the number of random runs to generate. For example, in order to generate 1000 runs, step from 1 to 1000 in increments of 1.

The "Markets Per Sector" parameter specifies how many members of each sector should be included in each random portfolio.
I have also included parameters that allow you to exclude specific sectors - simply change the parameter value of the sector you wish to exclude from "No" to "Yes".

Feel free to modify and rip apart the code - you may want to remove the sector constraints altogether.


2011-06-15 Updated block to work with new version of Trading Blox.
Attachments
Random Portfolio PM.tbx
Random Portfolio
(9.87 KiB) Downloaded 209 times

LeapFrog
Roundtable Knight
Roundtable Knight
Posts: 695
Joined: Mon May 17, 2004 4:18 pm
Location: Boston, MA

Post by LeapFrog » Wed Feb 28, 2007 6:49 pm

Nickmar - truly incredible - thank you so much for sharing this - I could never have programmed this.

Once I jiggered all my group names (using global replace in Notepad) to match exactly your strings, this blox ran beautifully.

This is a very useful and powerful tool. If a system can produce good results after being put through this meat grinder, it is robust indeed!

Jake Carriker
Site Admin
Site Admin
Posts: 1493
Joined: Fri Sep 12, 2003 10:32 am
Location: Austin, Texas

Post by Jake Carriker » Wed Feb 28, 2007 10:35 pm

Great Work Nickmar!

Here is a slightly altered version that allows the user to choose exactly how many markets from each sector to include. For instance, you might want to create a portfolio that always has 4 stock indices, but always has 2 Long Rates. It might be useful for creating random portfolios that are purposefully balanced (or unbalanced!) in specific ways.

Enjoy,
Jake


2011-06-15: Updated block for use with new Trading Blox version.
Attachments
PM_Random_Portfolio.tbx
Alternate Random Portfolio Manager
(9.49 KiB) Downloaded 225 times

RedRock
Roundtable Knight
Roundtable Knight
Posts: 939
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Wed Feb 28, 2007 10:54 pm

LeapFrog wrote:Nickmar - truly incredible -!
Jake Carriker wrote:Great Work Nickmar!
wow :!:

zentrader
Full Member
Full Member
Posts: 18
Joined: Wed Feb 09, 2005 10:01 am
Location: Germany
Contact:

MCS...

Post by zentrader » Thu Mar 01, 2007 1:54 am

Forum Mgmnt,

it seems we have a different view of the things.

With characteristics of a price data set I mean only that what I see: the OHLC bars and their relation to their predecessors. It' a "Zen" view of the things... :-)

If I use this given state and randomize the time series there is no influence of me in my role as developer, trader etc.

Optional I offer the possibility to customize future scenarios (expetation of a up/down market, lower/higher volatility) or let the developer/trader decide whether he wants to simulate single bars or block of bars etc. etc.

Imho this is a valid method to get nearly unlimited test data with unlimited different scenarios. And that you can't achieve with past data, independent how much past data you have.

But everyone is free, to select the methods and tools of his choice.

bye,
zentrader

nickmar
Roundtable Knight
Roundtable Knight
Posts: 191
Joined: Tue Oct 26, 2004 12:38 pm
Contact:

Post by nickmar » Thu Mar 01, 2007 9:01 am

Excellent modification Jake - makes sense to allow for unequal sector weightings (e.g. less exposure to the meats). It may also be worthwhile exploring modest random sector underweightings/overweightings.

TradingBlox Builder makes these types of tools easy to implement. Kudos to c.f. and Tim.

Forum Mgmnt
Roundtable Knight
Roundtable Knight
Posts: 1842
Joined: Tue Apr 15, 2003 11:02 am
Contact:

Post by Forum Mgmnt » Thu Mar 01, 2007 10:00 am

nickmar,

Excellent idea. Since individual markets are often in different states, trending or not, volatile or not, etc. scrambling the portfolio gives you a better sense of the range of potential outcomes which might occur.

This too is a form of Monte-Carlo simulation and a good addition to the robustness checking Arsenal.



zentrader,

The reason I don't like synthetic data is the same reason I don't like Monte Carlo simulation of trades, it hides important relationships between the days that will tend to make the synthetic data kinder to your equity than the markets will be when one considers multi-market trading.

sluggo covered most of the issues in some detail above. First when things go bad, they tend to go bad all at once, and for a few days or months at a time, how do you preserve this with synthetic data? How does synthetic data account for the fact that on the day after the crash of 1987, I had my Eurodollar, Gold, Silver, Bonds, TBills and Sugar positions all gap against me in a massive way?

As sluggo mentioned, this might not matter if you are testing using single-market-at-a-time tools like TradeStation but I certainly don't recommend doing that either.

So I prefer ideas like nickmars and like what we do with Monte Carlo simulation in Trading Blox, we randomize while preserving the underylying information rooted in the human interactions in the markets.



On a related note, nickmar's suggestion gave me an interesting idea for extending Trading Blox's Monte Carlo analysis. In essence his portfolio manager lets you determine what would happen if you picked a different set of the available market data.

This got me thinking about what would happen if you did the same thing over time instead of markets. If you picked say only two or three years of data out of the many that were available. This is somewhat like what we do with Monte Carlo analysis but with one important difference: the simulation timeframe would be less than the underlying test from which the simulation data is drawn.

Currently we scramble chunks of the equity curve over the same length as the underlying test. So if you test for 20 years we scramble those 20 years of data. This gives you some idea how the 20 years could have been different.

This process does not actually answer the question that is on most people's minds when the decide to run a Monte Carlo test. I suspect most people are concerned with what the next few years might bring, or perhaps the first year of actual trading.

I suspect many people would like to use the 20 years of data as a pool from which to draw a test on a much shorter period, say 1 or 2 years instead of the entire 20 years. There will be much more variability in the results of a two-year extraction from a 20 year simulation. This increased variability is reflective of the actual increased variability one is likely to encounter when considering only one or two years of trading.

An improvement to Trading Blox's Monte Carlo simulation system like this should help traders get a more realistic sense of the possibilities for a given year of trading.



Finally, since I too have done many of these random tests using parameters whose sole purpose was to control the number of iterations in the test, it struck me that these are just another Monte Carlo test and it would be nice if we allowed you to specify that the test was a Monte Carlo test so that you could get the same output graphs, i.e. the distribution of max drawdowns, MAR, CAGR%, etc. that we get when we turn on the Monte Carlo reports.

I'm putting this idea into the suggestion box.

Thanks again everyone for an interesting dicussion, and nickmar in particular for stimulating my thinking here.

- Forum Mgmnt

Jake Carriker
Site Admin
Site Admin
Posts: 1493
Joined: Fri Sep 12, 2003 10:32 am
Location: Austin, Texas

Post by Jake Carriker » Thu Mar 01, 2007 11:12 am

Michael Clarke is a CTA with a long and respectable track record. I find his comments (excerpted from the blurb about his programs on the IASG website) about model development interesting. I find especially interesting his thoughts on out of sample testing, which I have highlighted in bold below.
CCM's philosophy when developing a model is to test it using a large pool of commodity interests, approximately 105, with data as far back as 1945. In order for the model to be accepted into CCM's portfolio of available models, it must trade all 105 markets using the same rules and parameters and the results should indicate excellent performance characteristics for the vast majority (at least 90%) of the markets and for the group of markets as a whole. Also no model is accepted unless it shows stability of performance during tests involved with shifting parameters and altering rules. Much effort has been expended in developing tools to assist in this effort to assure robustness of the models. I consider the software we have developed in this area to be one of our edges in the markets. With regards to out-of-sample testing CCM no longer performs this test. I noticed that if the length of the out-of-sample period was longer that the average length of time that the particular model might be expected to have a mediocre or poor performance, then the effective parameters and rules were very similar to the test sample.
Jake

Tim Arnold
Site Admin
Site Admin
Posts: 8501
Joined: Tue Apr 06, 2004 1:41 pm
Location: Boston, MA
Contact:

Post by Tim Arnold » Thu Mar 01, 2007 1:11 pm

In the case that you might have less markets in a sector than specified, it would be prudent to add a failsafe trigger to avoid an infinite while loop:

Code: Select all

' Set a fail safe counter in case NumMarkets is greater than the available markets in that sector.
failSafe = 0
WHILE ( y < NumMarkets AND failSafe < 1000 )
	failSafe = failSafe + 1
	Rnd = Random( system.totalInstruments )
	PortfolioInstrument.LoadPortfolioInstrument(Rnd)

danZman
Roundtable Knight
Roundtable Knight
Posts: 117
Joined: Thu Mar 10, 2005 10:07 am
Location: Newport Beach,CA

Post by danZman » Mon Mar 05, 2007 6:06 pm

sluggo wrote:I'm glad to hear that you believe Monte Carlo analysis cannot be used to forecast system performance in the future.

Although System 1 had a nice smooth equity curve during 1973-1994 (21 years!) and although it outperformed System 2 during that period, the markets changed. System 2 outperformed System 1 quite dramatically in 1994-2007(Q1), despite System 1's vastly nicer-looking Monte Carlo results for 1973-1994.

System 1 follows intermediate-term trends, and makes good profits when intermediate-term trends are smooth. These type of trends were plentiful before 1994 and not so plentiful afterwards. System 2 follows long-term trends, which were present the entire time from 1973 to 2007(Q1), but not quite as profitable as smooth intermediate-term trends. The full equity curves, from 1973 to 2007(Q1) are contained in the original .zip archive, see picture below. Their password is "Roundtable". I've copied those equity curve pictures here, and added a marker at 1994, showing the date where the MC simulations were run.

Obviously no software could predict that the nature of trends was going to change in the future. No software could predict that the type of trends which profit System 2 were going to keep occuring in the future, while the type of trends which profit System 1 were going to stop occurring in the future. Keep in mind that Systems 1 and 2 traded the identical same portfolio of markets for the entire period.

System 1 was not a "bad system"; it was merely a machine that profited handsomely when the markets behaved a certain way. The markets behaved that certain way for 21 years, and then they stopped. No analysis of the data of those 21 years, could have predicted the change. Not even Monte Carlo analysis.

And now we have 34 (!!) years worth of profits for System 2. Now that we see what happened from 1994 to 2007, System 2 looks like an excellent machine that profits quite nicely when the markets behave a (different) certain way. Especially in comparison to System 1 !! But: can we predict whether or not the markets will continue to behave that way in the future? No. Can Monte Carlo analysis predict whether or not System 2 will continue to be profitable in the future? No. Can we use any of this data to predict whether the markets will revert to their old behavior, such that System 1 starts to be profitable again? (e.g.(this post) )? No.

Too bad.
This is a great example of a changing market structure. The question then becomes....when do you abandon a system? When do you go back to using a system you threw out? This is something I've been having trouble answering. I like to trade a suite of systems, if one doesn't make a new equity peak for a certain amount of time (say double what testing has shown), I sure don't want to trade it anymore...or at least allocate less to it. Anyone have suggestions?

D

tovetski
Senior Member
Senior Member
Posts: 46
Joined: Fri Jan 13, 2006 10:49 am
Location: Montréal, Québec, Canada

How about creating synthetic price series of the markets

Post by tovetski » Fri Mar 23, 2007 2:05 pm

Hi everyone,

I was wondering if anyone tried to create synthetic price series for the markets they trade based on the historical distribution of daily returns. I guess one could do so in a similar way to Monte Carlo simulations in Trading Blox where daily returns of historical simulations are shuffled and realigned so as to construct theoretical equity curves. The interesting part of doing so at the instrument level would be to preserve to historical volatily distribution of each market. It obviously does not solve the problem that the volality of each market will change from it's historical distribution, but I think it could serve as a reasonable "out-of-sample" testing environment. I haven't thought about volume and O.I. much however. Since simulations over many years can lead one to trade ridiculously high # of contracts on certain markets, we'd have to have a way to project volume and O.I. levels into the future so as to apply liquidity constraints.

Looking forward to some feedback on this topic.

Good day

sluggo
Roundtable Knight
Roundtable Knight
Posts: 2986
Joined: Fri Jun 11, 2004 2:50 pm

Re: How about creating synthetic price series of the markets

Post by sluggo » Fri Mar 23, 2007 2:33 pm

Pierre_Thauvette wrote:... create synthetic price series for the markets they trade based on the historical distribution of daily returns.
Don't forget correlation.

Many people trade a portfolio of instruments and hold several positions simultaneously. These traders are interested in the movements of the individual markets AND in the correlation of movements between markets; as for example when all the world's stock indices spiked down together, in perfect correlation, on 27 Feb 2007. The Nikkei, the CAC, the Canadian S&P 60, the DAX, the Nasdaq, the EuroSTOXX, the SPI-200, and dozens more: all into the toilet, all at once.

Perhaps you wish to create synthetic price series for the markets you trade based on the historical JOINT distribution of daily returns?

RedRock
Roundtable Knight
Roundtable Knight
Posts: 939
Joined: Fri Jan 30, 2004 3:54 pm
Location: Chicago

Post by RedRock » Fri Mar 23, 2007 2:57 pm

27 Feb 2007 Global Market Unity Day. I think it's taking brotherly love WAY TO FAR.

As c.f. mentions in his book, and others have alluded to. After a period of trend-following harmony, markets tend to auto correlate in a most unfortunate way against those trends. All at once. Systems which had an especially lucky run, may well face added pressures and sub par performance for some time to come.

Anyways.. Seems like a difficult, (though probably not impossible) task to simulate in a synthetic series. And even then, the future will probably be a bit different than our hopes, as it has a way of being...

Post Reply