Why Are The Consensus Boards Different?

There are a few different consensus boards out there. I looked at four of them, including the one at Wide Left, and found that they have different player rankings, sometimes drastically so. Why?

I’ve been producing consensus boards since 2014 and have been publishing them since 2015. At the time, there weren’t many – if any – that one could easily find in draft media. Now, there are several. But they don’t agree, even though their goals and data sources are theoretically the same.

Some of the disagreements are fairly small and easy to explain – Marvin Harrison Jr. and Caleb Williams are virtually tied in Wide Left’s most recent consensus board, now with 80 included analyst rankings. The smallest of tiebreakers gave Williams the nod. A slightly different dataset might give Harrison first-place billing instead of Williams.

But that’s not the only set of differences.

Wide Left’s Consensus Big Board ranks J.J. McCarthy 22nd overall while Jack Lichtenstein’s Consensus Board ranks him 28th. NFL Mock Draft Database’s Consensus Board ranks McCarthy seventh. That’s a significant spread and the largest difference in value between any set of boards of the four consensus boards Wide Left looked at for this project.

The boards included were Wide Left’s, the NFL Mock Draft Database’s, Jack Lichtenstein’s and the Athletic’s. There were some other extreme differences near the bottom of the boards – Frank Crum is ranked 236th by the Mock Draft Database but 363rd by Lichtenstein– but those are a lot less interesting than disagreements in the top 100.

This is particularly important because the Athletic only ranks the top 100 – a practice that has a lot of solid backing – making it difficult to create a consistent understanding of the differences that exist outside of the top 300.

Another quarterback, Drake Maye, has created some pretty significant disagreement as well. He ranks sixth in the Lichtenstein board and second in the Mock Draft Database.

At the other end, Joe Alt ranks eighth in the Database’s board while Lichtenstein’s board ranks him fourth. In between the two, Wide Left and the Athletic ranked him fifth and sixth, respectively. Below is a table of every player ranked in the top 100 by any of the four sets of consensus boards.

The NFL Mock Draft Database is the most unique of the three boards, but there are differences between Wide Left’s consensus calculation and the ones produced by Lichtenstein and the Athletic. A good example, after Maye, would be Edgerrin Cooper. Wide Left’s consensus ranks him 37th while Lich’s consensus ranks him 53rd.

Answering why this is the case reveals the fundamental nature of data analysis and how it will be impacted by choices at every level: how to gather data, which data to gather and how to treat gathered data. All of these choices will depend on the fundamental goal of the analyst or the intuition of the researcher attempting to solve a problem.

How Do You Average?

Normally, one might be able to chalk this up to the datasets. Lichtenstein keeps a tightly focused set of included boards, generally about 16, while Wide Left’s database now includes 80. While there shouldn’t be a huge difference for most players as a product of this different sample size – 16 is effectively a sample of 80 in this case – some players, just through a product of statistical quirk, will produce larger differences in the data.

In Cooper’s case, I’m not so sure this is the entire reason. It is most of the reason, but the other contributing factors also point to a difference in methodology. One element in play is the fact that Cooper just happens to be ranked fairly low by two of the boards used by Lichtenstein in ways that don’t reflect the variance of the larger group of analysts.