## Conf. Interval around a system change - UPDATE + Blox

Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.
Magnus
Senior Member
Posts: 35
Joined: Mon May 10, 2010 10:18 am
Location: UK

### Conf. Interval around a system change - UPDATE + Blox

This is an update based on input I received since my first post on the topic in June 2020.

The below and attached Excel Sheet describe a framework that can be used to statistically assess if a change to a system results in a significant improvement across a large number of different portfolios.

The work was done mainly in relation to stocks but can be used for other instruments as well. I share it here as it might be of interest to some and there are a few useful online tools that can be used for other similar problems as well.

I have posted a Portfolio Manager Blox (in "Blox MarketPlace") that can be used to run simulations with multiple randomly selected (generated) sets of stocks (or futures) from a larger portfolio.

Main Question:
How can we statistically assess that a change to a system results in a significant improvement and can /should be applied to any portfolio of stocks?

Example:
Stock trading system, Simulation over 21 years. Portfolio of 1000+ S&P stocks including delisted.
We want to assess if a change to the system (from “Before” with no rebalancing of positions to “After” with an ongoing rebalancing) results in a significant improvement to the MAR and Sharpe (or any other output). What we change to the system doesn't really matter, this is about the methodology used.

Initial simulation:
Includes the entire portfolio (1000+ stocks) over 21 years and results in a significant improvement in MAR from "Before" (0.35) to "After" (0.57) the change. The MAR figures might not be very “impressive” but this is an example only.

Main Issue:
Maybe a small number of stocks drive the improvement? How to statistically check if the change to our strategy results in an improvement across a large set of different portfolios?

Methodology used:
1. Generate a number (30-50) of random portfolios.
2. For each random portfolio, run tests "Before" and "After" the system change.
3. Calculate the difference in the selected metrics (e.g. MAR, Sharpe…) from the simulation results before and after the change.
4. Test the Sample data for Normality (Online tools can be used). The statistical calculations (e.g. Confidence interval) work on the assumption that the data is normal.
--> IF NOT Normal, "Boot Strapping" can be applied.
5. Calculate the Margin of Error and Confidence Interval for the difference in Means to assess if the change to the system results in a significant improvement (online tools or in Excel).
6. If desired, calculate the required sample Size to reduce the margin of error.
Attachments
Applied Statistics - AlgoTrading - v2.xlsx