Pre-Launch Versus Post-Launch Stock Model Performance

Added on by Steve Auger.


There has been lots of discussion at Portfolio123 regarding prediction of stock model performance out of sample (post-launch) based on factors extracted from past data. I thought that I would take the time to add my own analysis.

Most of the analytical work has involved something called Gain Stock per Day (GSD). Some of the problems I have noticed with the previous work is the focus on (pre-occupation with) GSD and the prediction (annual excess return).

Some of the problems with the analysis include inconsistent pre-launch and post-launch time periods and use of look ahead data (GSD includes post-launch period).

I expect my analysis to occur over many weeks/months. I won't be making any great discoveries today but I am going to kick off the process.

Model Selection Criteria

I decided to start by analyzing third party R2G stock models with at least 180 days of post-launch performance. In addition, I decided to only look at models that hold 20 stocks initially. I expect less volatility and hence more consistency with 20 stock models than those with fewer stocks. I also chose to ignore models that hold more than 20 stocks.

Pre- and Post-Launch Time Periods

My first step was to download the daily equity data for eleven models that meet my criteria into a spreadsheet. I then separated the data into two time periods. The first time period was from April 6, 2004 to Dec 31, 2012. This is the pre-launch time period that all of the 11 models have backtest performance data for.

The second time period is Aug 7, 2013 to Feb 7, 2014. This is the last six months of post-launch. All 11 models have been "live" for this time period.

Input(s) Versus Outputs(s)

I don't particularly like predicting annual excess returns as this output doesn't tell the whole story regarding drawdown, alpha, beta, etc. I have decided to initially look at excess Sortino Ratio. By "excess" I mean compared to a benchmark as opposed to risk-free return.

Since I will be looking at post-launch excess Sortino Ratio, I felt it would be logical to also look at pre-launch excess Sortino Ratio as a possible predicter.

As a first attempt, I chose the SPDR S&P 500 ETF trust as the benchmark, although many of the models are actually smallcap or microcap models.

The results are provided on the chart below.


Just to be certain about the choice of benchmark, I decided to create the same chart except with the Russell 2000 ETF used as the benchmark.  The results are similar except that the points are shifted to the left on the graph.  This suggests that it was more difficult to outperform the Russell 2000 than it was the S&P 500 Pre-Launch, but Post-Launch they were about as equally difficult.


From these charts it can be seen that two models are performing better post-launch than pre-launch: Keating's med turnover and Quantonomics smallcap gems.  There are two models that are underperforming:  Sherman's with low max drawdown and Thank Buffet.

Above-Average S&P500 is performing in line with the benchmark, indicating that there has been no advantage to trading this model in the last 6 months.  The same holds true for Marco's Merger&Acquisition model.  And here is where it becomes evident that excess Sortino Ratio is not sufficient.  The M&A model is very low beta... it is not expected to outperform the benchmark, but it is expected to have lower drawdown, and that is why looking at excess returns or any kind of excess performance does not tell the whole story.

The Problem With Conditional Nodes

Added on by Steve Auger.

While writing my last post I discovered a major issue with how Portfolio123 processes NAs in conjunction with Conditional Nodes within ranking systems. At least I view the issues as major, perhaps I was expecting too much.

The problem is that the Ranking System algorithm processes conditional nodes in a seemingly irrational fashion.  as an example, I created this very simple ranking system with one conditional node RSI(14)>50.  I used the stock factor SiRatio (Short Interest Ratio) for the TRUE condition. 

For the FALSE condition, I used the formula (NA.).  The idea is to rank all stocks with RSI(14)<=50 at the bottom and the remainder at the top, ranked by SiRatio.


So what does actually happen?  Well have a look below...

NA Processing.jpg

The rankings for the stocks meeting the FALSE condition are placed at the top (rank=90).  The rankings for the stocks satisfying the TRUE condition are placed below those stocks meeting the FALSE condition.  For me this is counter-intuitive and it appears there is no way to get them placed at the bottom of the universe.  I've tried using the statement FALSE instead of NA, changing the node to Binary, etc.  This is a major problem and should be fixed by P123!