Grading the NFL Draft Big Boards, 2016-2021: Who Had the Best, Who Had the Worst?

One of the projects I’ve become most well known for in the NFL draft community has been the Consensus NFL Draft Big Board, which pulls together dozens of player rankings across the industry, using industry professionals and newcomers to the game, to produce a composite rank of the draft-eligible players for the NFL draft.

This has been something I’ve done since 2014, and the project has generated a lot of interesting insights into coverage of the NFL draft. When the 2024 board is released, we’ll go over some of them, including the most polarizing players in the draft and what draft boards can tell us about who teams might “secretly” have higher or lower than the media.

I’ve done that work every year and it’s been great, but something I’ve had access to but unable to do is to evaluate media boards. Previous stops have been hesitant about me evaluating draft boards that include analysts they also employ and may also have been wary of alienating future employees. So I hadn’t looked at the draft board success of other analysts for years.

Now that I no longer have that governor on me, I can take a look at analysts across the industry to see how they’ve done stacking up players in the NFL draft. Who’s been best? Who needs to go back to the drawing board? And how does the consensus board do in predicting outcomes when compared to individual boards or the NFL draft itself?

Before anything else, we have to have a measure of outcomes. While it’s trivially easy to determine that Ja’Marr Chase was a better pick than Zach Wilson and about as easy to say the same about Trevor Lawrence, it becomes pretty difficult to figure out if, for example, we should give the Bears more credit for missing on Justin Fields than for the 49ers missing on Trey Lance.

And how do we incorporate the success of Mac Jones’ rookie year against the rest of his career? How does that compare to the relatively consistent but somewhat underwhelming career of Zaven Collins?

Ultimately, I relied on a few tools and brought them together until they fit my intuitive expectations. Combining Pro Football Focus’ Wins Above Replacement statistic with snap count and modifying all values for positional standards (many times an unsuccessful QB will play more snaps than a successful defensive tackle) was a good start.

I had to recalibrate the results to match the general positional biases found in big boards; Patrick Ricard is by far the most successful fullback to come from the 2016-2021 draft classes but is not more valuable than Amon-Ra St. Brown. I also had to do this to make sure the top of the value chart wasn’t all quarterbacks -- ranking Micah Parsons above Trevor Lawrence probably should not be punished.

That still led to QB bias, but it was pretty reasonable. Most draft boards rank quarterbacks pretty high and though it would be a bit much to argue that Baker Mayfield has produced more value than Justin Jefferson, it’s at least a start.

The top non-quarterbacks were, in order, Micah Parsons, Chris Jones, Justin Jefferson, Fred Warner, Tristan Wirfs, Penei Sewell and Myles Garrett. Everyone is going to have a different method to rank players – just look at the comments on every quarterback ranking post – but this gives us a baseline to work off of.

With a slightly more precise rubric in hand, we can begin to grade big boards. Using a value chart similar to the Jimmy Johnson trade chart when tracking players, we can use a variance calculation – that is, take the difference in value from a big board and subtract it from the value of the player outcome ranking and squaring it – to see where boards stack up.

The NFL draft itself, when treated like a big board (indeed, it is a partially blind revealed preference ranking), ranked number one or number two in all of these exercises. That’s not a surprise. In addition to having unique information like injury or character, the NFL controls outcomes.

That doesn’t just mean that they pick-and-choose, allowing higher picks more chances and more opportunity – they do – but also that they get to avoid many of the pitfalls that an outside analyst cannot avoid.

For example, if an NFL team runs a very classic 3-4 system reminiscent of the 1980s Bill Parcells teams, then they’ll have a lot of use for a beefy nose tackle who can two-gap a center and take on double teams at the point of attack without offering much pass rush. They might have more use for an undersized edge rusher, moving him to “outside linebacker” and giving him more space to convert speed to power on his pass rush.

An analyst can only rank a player based on collective strengths and weaknesses. Some may choose to project best possible fit, but many see a one-dimensional player and move him down the board against a player who can be used in every system in a variety of roles whose best outcomes are only slightly behind that first player.

This isn’t necessarily the case of the NFL “knowing better,” though it sometimes is. It’s also the case of different incentives structures involved with ranking players writ large and selecting players for a specific fit.

When constructing these boards, I decided to avoid the 2022 and 2023 big boards when evaluating player outcomes because the careers of the players in question hadn’t taken off yet and only a few of them have flamed out effectively enough to be called “busts.” I’ve also excluded Jeff Gladney, Max Tuerk and Dwayne Haskins from the outcome scores because of their recent deaths.

There is something valuable about going back to previous results and seeing who underperformed but it can be easy to overfocus on “bad” boards. While I will be looking at the “worst” boards, I don’t think this is a significant knock on these analysts, many of whom improved pretty significantly over the course of the exercise or have contributed in big ways to the NFL draft media experience, including two who have pioneered different types of engagement and draft analysis, like mock draft simulators and collaborative or computer-driven mocks.

One of the boards that ranked last came with an interesting story. It’s down below, but I’ll quote myself here.

Something to keep in mind with the “worst” boards – they took significant risks in this exercise by diving deeper into the draft. The top boards tend to stop at 100 players while the bottom boards watched over a thousand players, providing insight into Division III and unheralded prospects no one else could provide.

On top of that, these boards operate as an excellent learning exercise. In addition to finding out why an analyst may have ranked a player a certain way, we learn more about the way they think about football and that can change our understanding of the sport. And getting things wrong is part of that process, too.

Let’s look at the best – and worst – big boards from every year. At the end, we’ll look at the final results for anybody who has submitted at least four boards over the six years of data we have and point out any interesting quirks that have emerged as a product of the whole sample process rather than just looking at year-to-year data.

The 2016 Big Board Awards will be available to everyone, but everything after that is behind the paywall.

This one’s long, so if you want to jump around:

The first of many years where the NFL draft itself outpaced the analysts, 2016 was a year when a number of people in the draft community were at least close. The best board of 2016 belongs to a website that no longer exists, unfortunately. He used to own the NFLDraftGeek account on twitter, but that is currently run by a different person who doesn’t maintain the website.

NFLDraftGeek, run by a software administrator in Michigan, had the single best board of 2016. Some of his best called shots:

The top-ranked board willing to rank more than 100 players was The Huddle Report’s value board, which was built by Robby Esch. Players outside of the top 100 that Esch got closest on were running back Jonathan Williams (ranked 134th, picked 156th, consensus ranked 112th, outcome ranked 136th), defensive back Jalen Mills (ranked 114th, picked 233rd, consensus ranked 71st, outcome ranked 129th) and linebacker Jaylon Smith (ranked 71st, picked 34th, consensus ranked 31st, outcome ranked 51st).

This was a good year for short boards, though it wasn’t always a big disadvantage to have more players ranked. The top five boards belonged to:

The board that least matched NFL outcomes belonged to Real Football Network, a now-defunct website that was once spearheaded by former CBS analyst Pat Kirwan. The biggest mistakes for the very expansive board – over 1,000 players were ranked – were the exclusion of wide receiver Tyreek Hill and defensive tackle David Onyemata. Other unranked players that ended up spending a reasonable amount of time in the NFL included receiver Robbie Chosen, cornerback Brandon Williams and linebacker Kamu Grugier-Hill.

Of players they ranked, ranking Isaac Seumalo 1,108th and Will Fuller 1,013th both represented significant mistakes. Placing Chris Jones 134th and Ezekiel Elliott first overall hurt as well. Ranking Jalen Ramsey 28th dinged their grade pretty significantly, too.

That’s not to say they didn’t have hits. They ranked quarterback Dak Prescott higher than almost anyone else at 67th, ranked cornerback Vernon Hargreaves low (25th) and were right to value cornerback Kendall Fuller at 12th overall.

The five worst boards belonged to:

If we eliminate every player ranked outside of a board’s top 100, we might get a fairer view without punishing experts for doing more work. By evaluating the outcomes of each of the player in every board’s top 100, we end up with the following lists.

Already in 2017, we begin to see trends emerge. Some of the analysts that had done well in 2016 show up again in 2017. The NFL still outpaced everyone, but the difference between the NFL and the top board in predicting player outcomes was astoundingly close.

Before he became the Raiders general manager, Mike Mayock was one of the most valued analysts in the industry, and it’s not hard to see why. In addition to having encyclopedic on-call knowledge of virtually every prospect in the draft, he turned out to be pretty good at projecting them.

His biggest hit might have been on Mitch Trubisky, who he ranked 30th overall. The NFL selected Trubisky at the 2nd overall pick and the consensus ranked Trubisky 21st. That, along with other best-ofs:

The second-best board was also one of the more expansive boards, belonging to Pro Football Focus. The bias towards shorter boards is still present but it’s not that severe in 2017. The top five boards from that year:

Though there was less of a bias towards larger boards, we still saw bigger boards punished – the five worst boards all ranked over 300 players. This meant that if they didn’t rank a player that did well, they were punished more severely.

SportsSkeeda’s Tony Pauline fared the worst as a result of this, having ranked 975 players. Pauline didn’t have moments like Real Football Network where unranked players finished as top-50 players in the draft, but some back-of-the-draft players slipped through his fingers. Missing out on Marquez Williams (consensus ranked 481st, outcome ranked 114th) and Alex Armah (consensus ranked 633rd, outcome ranked 112th) isn’t that bad.

What does hurt him, however, are highly-ranked prospects who fizzled out.

The five worst boards belonged to:

If we eliminate every player ranked outside of a board’s top 100, we might get a fairer view without punishing experts for doing more work. By evaluating the outcomes of each of the player in every board’s top 100, we end up with the following lists.

In 2018, the NFL draft community did better than ever, with the smallest average error in player rankings thus far. The NFL still barely beat out the top boards in the market, but more of them got close than did in 2017 or 2016.

It is likely the case that College Football Metrics had boards before 2018, but it was difficult for me to get a hold of them back then, in part because many of the boards acquired for the project every year require funds or connections. Nevertheless, I was able to acquire their board in 2018 and it was a big hit, getting closer than anyone else to predicting NFL outcomes.

R.C. Fisher runs the site, but he did have Xavier Cromartie put together his big boards. College Football Metrics has become FF Metrics but both Fisher and Cromartie are still there. His biggest hit from the draft comes from wildly veering from consensus, ranking quarterback Josh Allen number one overall. What a shot. Let’s take a look at that and others from the draft.

The best boards other than Cromartie’s were familiar names, though they did include another expansive board, this time from Dane Brugler now of the Athletic. Top five:

Like I mentioned above, this was a particularly tight race with the NFL in determining outcomes. The “worst” boards here mean a lot less than they did in 2017 or 2016. That said, we do get a repeat from Pauline.

The worst example might have been the decision not to rank guard Dalton Risner, who was selected 41st overall and ranked 31st by consensus. His outcome score was 50th. He also didn’t rank Zach Sieler, which is not as egregious. Nevertheless, the 238th-pick in the draft was ranked 333rd by consensus and finished with a 32nd rank in outcome score.

The other bottom-five boards in an otherwise very good year for draft analysts:

If we eliminate every player ranked outside of a board’s top 100, we might get a fairer view without punishing experts for doing more work. By evaluating the outcomes of each of the player in every board’s top 100, we end up with the following lists.

In 2019, we actually had an analyst beat the NFL – a big turnaround from his 2017 showing, where he earned a bottom-five finish.