ESPN's Receivers Tracking Metrics — Confusing Mess or Valuable Tool?

ESPN released their receiver tracking metrics based on chip data. But the results were pretty confusing. Is that helpful or hurtful?

Seven weeks into the season Seth Walder, a sports analytics writer at ESPN, announced the release of ESPN’s receiver tracking metrics, using the chip data from every player on the field — as well as in the ball — to determine a receiver’s quality, route distribution, catch capability and more.

Walder, who is not the author of the model, has been put into a position to defend it and has seen the direct feedback from fans and writers about the model on social platforms.

One problem.

What is Tyreek Hill doing down there

ESPN’s receiver tracking has been a genuinely great addition to the analytics conversation and something I have used before (in a different iteration, provided then by FiveThirtyEight) to provide background knowledge when discussing receivers or directly when providing analysis on what distinguishes one receiver from another.

Their explanation of Open Score, provided both in article form and in a very useful video, gives credence to their approach and opens up a number of fantastic and potentially dynamic conversations about NFL receivers.

So, What’s Wrong?

But some of the overall scores are truly mind-boggling. While new analytical models should always have some counterintuitive or unexpected results — what’s the point of a model telling you everything you already know? — this doesn’t quite pass the sniff test.

It’s not just that Hill is ranked 16th instead of near the top — it’s that players like Kendrick Bourne, Nico Collins and Tank Dell are as high as they are. It also seems to be high on D.J. Moore, Adam Thielen and Brandon Aiyuk.

Not all of these are failures of the model. They could be failures in my perception. After all, Collins, Moore and Aiyuk rank high in Pro Football Focus’ rankings.

One might think that Justin Jefferson is absent because of his injury. Not so — he just ranks 27th.

What are we doing here

Jefferson and Hill are top-five receivers by almost everyone’s estimation. Other highly respected receivers like Stefon Diggs, Cooper Kupp and Davante Adams rank low. If this were an issue of a model being outside of consensus on just one or two players, that would be one thing. There are players all over the spectrum it seems to diverge from public perception.

These, to me, represent significant weaknesses in the model — weak enough that it would be difficult to use in any serious football analysis.

At least in any analysis meant to break down the game. It’s very useful in breaking down how models are created, what we can learn from them and how to read them.

It should be noted that for a model that does not directly incorporate yards or outside rankings (e.g. PFF grades or player rankings of any sort), it does a remarkable job building from the ground up to approximate the general feeling of the public.

But there’s a difference between being abstractly impressive given the inputs and being ready for publish. This purports to be an all-purpose receiver ranking that happens to put the yards-per-game leader at 16th and a receiver who started off the season averaging over 135 yards a game at 27th.

It is difficult for a multiple-time yardage leader to exceed his previous pace while being largely an average player.

Walder has done quite a bit in the world of public-facing football analytics and his contribution to our understanding of metrics like pass block and run block win rate (or its opposites, pass rush and run stop win rates) or the advantage pre-snap motion provides offenses is extremely useful.

His charts mapping double-team rates for defensive linemen against their win rates provides significant texture to the conversation around those positions. On top of that, he adopts a fairly humble approach to criticisms of the model.

All of that is great, though it would have been good to look at the model and reason whether it was more likely that the YAC leader was below average at generating yards after the catch or that the model was wrong.

The Confidence of Assumed Knowledge