Portfolio Design

Advanced concepts in stock investment portfolio design.  Fundamentals, technical analysis and many other related topics are discussed.

How Not to Optimize a Simulation

Today I would like to demonstrate one of the most basic of mistakes that can be made during optimization of a trading system. For background information please refer to Time The Stock Market Using Credit Spreads.

Stock analysis tools provided by Portfolio123.


The system for this demonstration is a simple market timer based on the BofA Merrill Lynch US Corporate BBB Option-Adjusted Spread index. I'm not going to bore the reader with the intimate details of how to set up the simulation and optimizer study as I will be keeping this post short. In summary, the simulation uses the Buy rule:

Eval(Close(0 ,##CORPBBBOAS) < (SMA(100, 0, ##CORPBBBOAS) + SMA(100, 100, ##CORPBBBOAS)) / 2, Ticker("RSP"), FALSE) 

The Buy rule tests the closing price of the index against the 200 day Simple Moving Average (SMA) of the index.  I have split the moving average into two parts due to limitations imposed by Portfolio123 on the SMA parameters i.e. maximum period of 500, maximum offset of 500. The Buy rule is the starting point, and the Optimizer will successively increase the length of the moving average.

Simulation Buy rule

The permutations for the Optimizer study are set up as shown below, with the index price compared against the 200-day SMA, 400-day SMA, 600-day SMA, 800-day SMA, and 1000-day SMA. The last moving average is approximately four years, the longest that can be practically tested.

Optimizer permutations

The reason for testing for such a long moving average is that there is no "normal" spread that one can assume, as there is for the TED Spread (0.5%).  So the long average represents a nominal spread for recent history that is used as "normal".

The results of the Optimizer study are shown below. (Click to enlarge.)

Optimizer backtest results

Note that the return goes up with each permutation, and the drawdown goes down.  Naturally, I was drawn to the fifth permutation, the one with the highest return and close to the lowest drawdown.

This is where I stopped when I originally designed this market timer. However, that was an error, because the results were partially attributable to limitations in the simulation that I didn't investigate.

Below is a Portfolio123 multi-chart of the BofA Merrill Lynch US Corporate BBB Option-Adjusted Spread, from its start date, December 1996, or lets say January first 1997, as the chart appears to have a resolution of one month.

BofA Merrill Lynch US Corporate BBB Option-Adjusted Spread index

Lets have a closer look at the permutations, the first and last permutation actually.

Backtest results for permutation #1

Backtest results for permutation #5

As you can see from the first permutation, the drawdown from 1999-2003 is almost equal to the benchmark drawdown. In the fifth permutation, the data up until the start of 2001 is not valid, as the Buy rule uses a four year moving average (1997 + 4 = 2001). Thus at least part of the decreased drawdown and higher return is a result of limitations in the simulation.

While I did make a judgement error in where I started the simulation, and used the flawed results to decide on the best parameters for the timer, I believe that there is enough left of the simulation post 2001 to conclude that the long SMA is still better than using a short SMA.