Synthetic Contracts

[Roger Rines]

Most markets haven't traded long enough to provide enough trades that a developer can use as a statistical basis for evaluating the robustness of a trading approach. Just as important, as the market data gets older, it stops being representative of current conditions.

To get around some of this, I'm considering using synthetic contracts created from a random selection of bar records using the market data that has been available over the last 15 to 20 years. While this approach will probably lose the rhythm inherent in the psychology of the market, it should retain the volatility and expand upon the range and patterns the trading system will encounter.

I aim to end up with a large sample of data contracts to test against to understand how the trading approach will perform on unseen data. I'm also hoping to get a broader range of results that can be used to test trade sizing and asset allocation.

Before I bang out the software to create synthetic contracts, I would like to hear from others who might have some insights on where these contracts might fail to produce my intended result.

[si]

You may benefit from reading Benoit Mandelbrot's book "On (Mis)behaviour of Markets". He talks about his two-step generation of synthetic data, and tries to precisely define the term "market-like".

[Roger Rines]

Thanks, Si.

I've just ordered a used copy from Amazon in the UK. Amazon in the US doesn't even know about the book.

Before your post, I didn't know anyone had seriously considered this idea. Nearly all of the traders I've spoken with hadn't been about trying to expand the testing range of their data.

Can you share you're impression of what Mandelbrot said about the validity of creating synthetic contracts?

Thanks for responding,

[si]

The link to amazon is here: hardcover


I read the book some months back. Here is my broad understanding of the main theses of the book as my notes show from then:

1. A plot of frequency of price changes shows too many points at the ends, and not enough in the middle. Such fat tails follow a power law.

2. Price changes show dependence, short-term as well as long-term. This dependence damps to 0 very very slowly, perhaps taking decades.

3. Volatility clusters together in an unpredictable manner. The market seems to be often in slow motion or in fast forward mode.

I think most LTTFers agree with all three. The second sentence of each point above is his contribution. Indeed he suggests mathematical ways of looking at each. He models price in two steps. He describes "trading time" as a function of "real time" and then "price" as a function of "trading time". Each function is a fractal. The two combine to produce a model of price in real time.

There are two parameters for the price vs. trading time:
1. "H" for dependence. Flips of a fair coin are memoryless. (H=0.5) Typical markets show higher H, about 0.7 or so.
2. "alpha" is the exponent in a power law equation describing the fat tails.

The description of trading time vs. real time is his model of accounting for volatility clusters. The description is short and a bit vague.

The author's style is often self-important and melodramatic: he never tires in calling the (orthodox economics) emperor naked. He also dumbs everything down from math to prose. There is a large bibliography at the end with references to technical publications. Those may have more details but I suspect that this work is a) very new and b) not exactly publishable in an orthodox economics or finance journals.

Regards.

[si]

I’ve just ordered a used copy from Amazon in the UK. Amazon in the US doesn’t even know about the book.

I think that is due to the spelling -- "behaviour" in UK versus "behavior" in USA.

[Roger Rines]

Thanks for such a great outline Si. From what I’ve read of his work previously, I found myself agreeing with you on some of the points about the writing style. Here is hoping it doesn’t get in the way of his ideas.

In thinking about the points he makes about the data being market like, I wonder how complicated his idea will be to implement. If he gives enough of a vision I should be able to emulate in code fairly well, but if he is spotty I might take myself across a broken bridge. I’ll see what hits me when it arrives. There is also the goal of wanting to push my systems into strange market behaviour so I can see how they respond to unseen conditions. Maybe both approaches will turn out to be important.

Your Amazon link would have saved me some money. I think I was so anxious to find someone else talking about synthetic contracts that I shot at the first target to appear on the screen. Thanks for finding it.

In trying to understand your point for item #1 above, I looked at the Close to Close difference of the Swiss Franc for the last 23-years and found these results:

Code: Select all

Mean                       0.001343728
Standard Error	          0.006794911
Median                     0
Mode                       0
Standard Deviation         0.513364409
Sample Variance	         0.263543016
Kurtosis                   2.682901744
Skewness                   0.094091584
Range                      6.75
Minimum                   -3.41
Maximum                    3.34
Sum                        7.67
Count                      5708
Confidence Level(95.0%)	 0.013320604

From how the numbers play out the Skewness seems fairly good, but both tails are very long. From a Pareto distribution perspective, this simple single example shows that 95.6% of the changes fall within 2 standard deviations indicating that the daily price changes are more constrained than I had earlier realized. The visual is a very tall single mode bell with a flat mouth.

Can you explain your first point a little more, I’m not sure I’m getting it yet.

Thanks again.

[Roger Rines]

Hello Walker,

Thanks for the two leads. I'll spend time studying both approaches.

Chande's approach places emphasis upon the relationship between the Open, High, Low, and Close of each individual bar.

Prior to this thread, I hadn't gotten past what you mention about Chande's approach.

Barnes presents three different approaches which vary from emphasis on individual bars, to emphasis on (overlapping) 2-bar sequences, to emphasis on the intermediate term swings.

This sounds intriguing and complicated. It will be informative to see what Robert has to say about creating new data.

It is good to hear others have written on this before now. I’ve not seen any forum threads here or elsewhere talk about this issue before, so I wasn’t sure if I was missing the obvious.

Thanks again for the leads.

[si]

In thinking about the points he makes about the data being market like, I wonder how complicated his idea will be to implement.

The book will give you some ideas to explore and point you in the direction that Mandelbrot wishes to persuade others to pursue: multi-fractal modelling. I do not have the book handy to look up the specifics right now, so you have to wait till you get your book. However, here is a quick flow of the algorithm, from memory:

0. First you arbitrarily set a start price and a length for your series.

1. Determine: The frequencies of price differences from a power law curve . This distribution empirically seems to approximate market behavior, i.e., if you take a real time series, and plot frequency vs the absolute change in price, the curve you get has the form of such a power law eqn. This is his mathematical model of the fat tails. [I think you are on the right track with Pareto.]

2. Determine: The effect the size of a price bar has on the size of the next. (Note: The correlation is on the size of the bars, i.e. prices maybe uncorrelated but volatility is.) A coin toss, for instance, has no memory, whereas real markets have memory which damps with time. He has a model for this memory and its damping from hydrology. The damping coefficient varies with individual markets. [Look up Hurst coefficient.]

3. Combine the start price, the distribution of bars and the memory to obtain price as a function of "trading" time. Visually this looks a lot like market data, except it is much less volatile than real market data.

4. Define a fractal mapping of "trading time" in terms of "real time". The fractal ensures that you get volatility clusters in every time frame.

5. Combine the results of 3 and 4, Price->Tradingtime->Realtime, to get price series in terms of real time.

Note: This process will let you create synthetic data by changing the choices for alpha, H and the fractal for time. But it is not clear how to use it to model existing markets. In particular, step 4 above is not unique, so if you already have a price series then the procedure above cannot be reversed to get a unique pair of alpha and H.

Happy reading!

[si]

Most markets haven’t traded long enough to provide enough trades that a developer can use as a statistical basis for evaluating the robustness of a trading approach. Just as important, as the market data gets older it stops being representative of current conditions.

What is your basis for these assumptions at the starting point? There is anecdotal evidence to the contrary -- people have looked up grain prices for 100+ years and seen some consistency in behaviour.

[Roger Rines]

Hello Si,

My comment is based upon my focus on the currency and financial markets. In the currency markets I don’t see data going back before the Gold standard was dropped in the early 70s, and even then the earlier data doesn’t seem like a good representation of what is happening now. My last sentence is based upon how much less the trades deliver now versus at the beginning of the data. For example, the number of trading days in a signal is much shorter now than it is on the earlier data. In looking at what is happening with the reduce trade length, it seems those markets are getting more efficient, and intervention seems to happen more now than it did in the past.

Whether any of the above is anything other than anecdotal, the total number of trades generated by my long-term system isn’t much more than 30 to 50 trades for each of these markets. This last fact is the primary motivation for wanting to expand the testing scope of the trading model so that it can see more data streams. I’m hoping from the expansion to synthetic data that I’ll be able to really shake out the models methods and my sizing approach.

With a synthetic data generation tool there is no reason why I can get the trade sample up into the many hundred area to see if the model is consistent on unseen data. In reading the Chande reference given by Warren above, and just starting on the Barnes reference, it seems both of these writers use this approach in their method development. In Chande’s book he provides a simple example of how he used the synthetic data to evaluate a trading model on the Swiss Franc. It was also interesting to see how the trade stats showed believable results even though not all generated a profit.

Would you be interested in playing with the software that comes out of this exercise?

[bolter]

Hi Roger,

I'll put my hand up. If you're creating some software to generate synthetic data I'd love to have the opporunity to "play" with it.

Cheers

[Roger Rines]

I've been surprised at how well this project is working and will post some results here later.

Okay on the testing. Send me an email and I'll send you the next release. I'm hoping to release the next update sometime on Monday, but I won't have much information in the HELP file by then, but you can always ask questions if anything is confusing.

[Roger Rines]

If you've asked for a copy of the synthetic data building software you should have received an email instructing where it can be downloaded. If you didn't get an email, either we didn't get the request or our download instructions were lost in the mail. In either case just send another message and we'll send the information right away.

[yoyo2000]

Rines,could you please give me a copy of the newest version of your synthetic data generator?
My email address is yoyo2000yoyo2000@hotmail.com.

And i found a post which discussed 'test date generator',address is here:
http://www.wealth-lab.com/cgi-bin/Wealt ... 85&start=0

Hope it's useful for you.

[Roger Rines]

yoyo2000,
You should be getting an email telling you where you can download the current release of the software. If you don't get it, let me know and we can see how else we might make access possible.

As for the Wealth-Lab discussion, I've been talking directly with some of the discussion leaders about the their approach and our software. Currently, they have our software and they have also been sending back data files. We are both running test, and if they prove successful, the understanding should help us expand the capabilities of the software's data generation.

[yoyo2000]

Ok,I'v downloaded it,and will study it as soon as posible.
Thanks.