Criticism with old BPM metric

For the most part, Daniel Myers of Basketball-Reference created a good predictor of RAPM through a box-score regression on Jeremias Engelmann’s 2001-2014 RAPM data set. This metric accomplishes taking RAPM out of a black box and using standard box-score metrics to predict a player’s impact on a per 100 possession level. However, based on my statistics background, I always thought there were some issues with the metric’s creation.

First, from the description given by Daniel Myers, there is not much information on how the training and test set were designated. If the metric was modeled and tested on the whole data, there could be some clear overfitting because the model would basically not be tested with the idea of uncertainty. In essence, when the data is fitted on new data (like 2020 data), the metric may not have the same uncertainty as the previous data that was tested on.

Second, I thought the handling of the team adjustment on BPM was outside the scope of statistics and weirdly handled. Instead of using a variable to control for this effect, Daniel Myers uses a team adjustment that factors in that teams perform 20% worse when the team is ahead and 20% better when the team is behind. With the combination of not handling the variable in the regression itself and the use of not a fully fleshed idea, I found there was some weird results on the fitted values for players.

Changes to the BPM model

For the new BPM model, I used Jeremias Engelmann’s 1997-2014 dataset. Within this dataset, I used a similar procedure as Daniel Myers to limit the dataset to a certain set of players based on a weighted possessions threshold of 250. With this dataset, I made sure to include 1st order terms for interaction terms (not included in Daniel Myers’ regression), on-off court +/-, and overall +/-. The last two metrics for +/- were included to basically capture the team effect to fit the data better to RAPM. The designation of training/test set always seems to be a hot debate for me because, with a smaller data set, the coefficients can change with any changes to either data set when creating these regressions. So, I found using a pseudo-bootstrapping technique to be useful by measuring the uncertainty in coefficients, error term, and overall correlation to the original RAPM metric. Based on the 1000 simulations, BPM has a centered error of +1.05/-1.05 and a centered correlation of 0.77.

Using the 1000 simulations, I took the mean of the coefficients and made sure the variables coefficient signs make sense to intuition.

Fitting the model to new data

With the BPM metric, I was able to create a WAR (Wins Above Replacement) by subtracting the replacement level BPM (BPM level for players on NBA 10-day contracts) from BPM, then multiplying with the percentage of total minutes played (MP / team MP), then multiplying with the ratio of games played (Team GP / 82). Then, with the idea of using average score differential, as stated by Daniel Myers, I multiplied this with the previous number to get a WAR metric. As shown below, the image shows the distribution of WAR throughout the years.

Top Ten BPM performers during the 2019 NBA season