Grading the NFL Draft Big Boards, 2016-2021: Who Had the Best, Who Had the Worst?
I have a database of big boards from draft analysts going back years. Time to take a look at how they've done.
One of the projects I’ve become most well known for in the NFL draft community has been the Consensus NFL Draft Big Board, which pulls together dozens of player rankings across the industry, using industry professionals and newcomers to the game, to produce a composite rank of the draft-eligible players for the NFL draft.
This has been something I’ve done since 2014, and the project has generated a lot of interesting insights into coverage of the NFL draft. When the 2024 board is released, we’ll go over some of them, including the most polarizing players in the draft and what draft boards can tell us about who teams might “secretly” have higher or lower than the media.
I’ve done that work every year and it’s been great, but something I’ve had access to but unable to do is to evaluate media boards. Previous stops have been hesitant about me evaluating draft boards that include analysts they also employ and may also have been wary of alienating future employees. So I hadn’t looked at the draft board success of other analysts for years.
Now that I no longer have that governor on me, I can take a look at analysts across the industry to see how they’ve done stacking up players in the NFL draft. Who’s been best? Who needs to go back to the drawing board? And how does the consensus board do in predicting outcomes when compared to individual boards or the NFL draft itself?
How Do We Rank the Boards?
Before anything else, we have to have a measure of outcomes. While it’s trivially easy to determine that Ja’Marr Chase was a better pick than Zach Wilson and about as easy to say the same about Trevor Lawrence, it becomes pretty difficult to figure out if, for example, we should give the Bears more credit for missing on Justin Fields than for the 49ers missing on Trey Lance.
And how do we incorporate the success of Mac Jones’ rookie year against the rest of his career? How does that compare to the relatively consistent but somewhat underwhelming career of Zaven Collins?
Ultimately, I relied on a few tools and brought them together until they fit my intuitive expectations. Combining Pro Football Focus’ Wins Above Replacement statistic with snap count and modifying all values for positional standards (many times an unsuccessful QB will play more snaps than a successful defensive tackle) was a good start.
I had to recalibrate the results to match the general positional biases found in big boards; Patrick Ricard is by far the most successful fullback to come from the 2016-2021 draft classes but is not more valuable than Amon-Ra St. Brown. I also had to do this to make sure the top of the value chart wasn’t all quarterbacks -- ranking Micah Parsons above Trevor Lawrence probably should not be punished.
That still led to QB bias, but it was pretty reasonable. Most draft boards rank quarterbacks pretty high and though it would be a bit much to argue that Baker Mayfield has produced more value than Justin Jefferson, it’s at least a start.
The top non-quarterbacks were, in order, Micah Parsons, Chris Jones, Justin Jefferson, Fred Warner, Tristan Wirfs, Penei Sewell and Myles Garrett. Everyone is going to have a different method to rank players – just look at the comments on every quarterback ranking post – but this gives us a baseline to work off of.
With a slightly more precise rubric in hand, we can begin to grade big boards. Using a value chart similar to the Jimmy Johnson trade chart when tracking players, we can use a variance calculation – that is, take the difference in value from a big board and subtract it from the value of the player outcome ranking and squaring it – to see where boards stack up.
The NFL Wins
The NFL draft itself, when treated like a big board (indeed, it is a partially blind revealed preference ranking), ranked number one or number two in all of these exercises. That’s not a surprise. In addition to having unique information like injury or character, the NFL controls outcomes.
That doesn’t just mean that they pick-and-choose, allowing higher picks more chances and more opportunity – they do – but also that they get to avoid many of the pitfalls that an outside analyst cannot avoid.
For example, if an NFL team runs a very classic 3-4 system reminiscent of the 1980s Bill Parcells teams, then they’ll have a lot of use for a beefy nose tackle who can two-gap a center and take on double teams at the point of attack without offering much pass rush. They might have more use for an undersized edge rusher, moving him to “outside linebacker” and giving him more space to convert speed to power on his pass rush.
An analyst can only rank a player based on collective strengths and weaknesses. Some may choose to project best possible fit, but many see a one-dimensional player and move him down the board against a player who can be used in every system in a variety of roles whose best outcomes are only slightly behind that first player.
This isn’t necessarily the case of the NFL “knowing better,” though it sometimes is. It’s also the case of different incentives structures involved with ranking players writ large and selecting players for a specific fit.
When constructing these boards, I decided to avoid the 2022 and 2023 big boards when evaluating player outcomes because the careers of the players in question hadn’t taken off yet and only a few of them have flamed out effectively enough to be called “busts.” I’ve also excluded Jeff Gladney, Max Tuerk and Dwayne Haskins from the outcome scores because of their recent deaths.
Holding Analysts “Accountable”
There is something valuable about going back to previous results and seeing who underperformed but it can be easy to overfocus on “bad” boards. While I will be looking at the “worst” boards, I don’t think this is a significant knock on these analysts, many of whom improved pretty significantly over the course of the exercise or have contributed in big ways to the NFL draft media experience, including two who have pioneered different types of engagement and draft analysis, like mock draft simulators and collaborative or computer-driven mocks.
One of the boards that ranked last came with an interesting story. It’s down below, but I’ll quote myself here.
It was exhaustively researched, with information on player measurements that even included player age – hard to find at the time – and ran hundreds of prospects deep. _____ watched every one of them, often writing extensive scouting reports on most of them.
His rankings never accorded with the opinions of the community at large. He took quite a bit of heat for it and didn’t earn much in the way of donations after offering the database for free, without advertisement. I’m not sure if that heat is the reason he took a brief break from scouting or if it was a re-evaluation of his process, but I say all of this to reiterate that having a bad big board does not mean providing no value to the online draft community.
Indeed, _______ clearly provided immense value without much in return and certainly didn’t produce a poor board by being lazy or being rigid in his thought process. _______ has consistently shown flexibility in his approach and even has a 35,000-word thesis on the subject as it relates to roster construction.
It’s important to hold the NFL draft community “accountable,” by keeping a running record, but it definitely doesn’t mean that the inability to predict necessarily means something about hard work or intelligence. And it certainly doesn’t mean focusing ire on the “weirdest” boards.
Something to keep in mind with the “worst” boards – they took significant risks in this exercise by diving deeper into the draft. The top boards tend to stop at 100 players while the bottom boards watched over a thousand players, providing insight into Division III and unheralded prospects no one else could provide.
On top of that, these boards operate as an excellent learning exercise. In addition to finding out why an analyst may have ranked a player a certain way, we learn more about the way they think about football and that can change our understanding of the sport. And getting things wrong is part of that process, too.
Let’s look at the best – and worst – big boards from every year. At the end, we’ll look at the final results for anybody who has submitted at least four boards over the six years of data we have and point out any interesting quirks that have emerged as a product of the whole sample process rather than just looking at year-to-year data.
The 2016 Big Board Awards will be available to everyone, but everything after that is behind the paywall.
This one’s long, so if you want to jump around:
2016 Big Board Awards
The Best Big Board of 2016
The first of many years where the NFL draft itself outpaced the analysts, 2016 was a year when a number of people in the draft community were at least close. The best board of 2016 belongs to a website that no longer exists, unfortunately. He used to own the NFLDraftGeek account on twitter, but that is currently run by a different person who doesn’t maintain the website.
NFLDraftGeek, run by a software administrator in Michigan, had the single best board of 2016. Some of his best called shots:
Tyler Boyd, who he ranked 33rd. The NFL picked Boyd 55th and the consensus ranked him 51st. His outcome ranking was 31st.
Maliek Collins, who he ranked 54th. The NFL picked Collins 67th and the consensus ranked him 83rd. His outcome ranking was 52nd.
Sterling Shepard, who he ranked 60th. The NFL picked Shepard 40th and the consensus ranked him 43rd. His outcome ranking was 56th.
Sheldon Rankins, who he ranked 46th. The NFL picked Rankins 12th and the consensus ranked him 13th. His outcome ranking was 42nd.
The top-ranked board willing to rank more than 100 players was The Huddle Report’s value board, which was built by Robby Esch. Players outside of the top 100 that Esch got closest on were running back Jonathan Williams (ranked 134th, picked 156th, consensus ranked 112th, outcome ranked 136th), defensive back Jalen Mills (ranked 114th, picked 233rd, consensus ranked 71st, outcome ranked 129th) and linebacker Jaylon Smith (ranked 71st, picked 34th, consensus ranked 31st, outcome ranked 51st).
This was a good year for short boards, though it wasn’t always a big disadvantage to have more players ranked. The top five boards belonged to:
NFLDraftGeek
Mike Mayock, formerly ESPN
Gil Brandt, formerly NFL.com
The Worst Big Board of 2016
The board that least matched NFL outcomes belonged to Real Football Network, a now-defunct website that was once spearheaded by former CBS analyst Pat Kirwan. The biggest mistakes for the very expansive board – over 1,000 players were ranked – were the exclusion of wide receiver Tyreek Hill and defensive tackle David Onyemata. Other unranked players that ended up spending a reasonable amount of time in the NFL included receiver Robbie Chosen, cornerback Brandon Williams and linebacker Kamu Grugier-Hill.
Of players they ranked, ranking Isaac Seumalo 1,108th and Will Fuller 1,013th both represented significant mistakes. Placing Chris Jones 134th and Ezekiel Elliott first overall hurt as well. Ranking Jalen Ramsey 28th dinged their grade pretty significantly, too.
That’s not to say they didn’t have hits. They ranked quarterback Dak Prescott higher than almost anyone else at 67th, ranked cornerback Vernon Hargreaves low (25th) and were right to value cornerback Kendall Fuller at 12th overall.
The five worst boards belonged to:
Real Football Network
Tony Pauline, SportsKeeda
Kyle Crabbs, Locked On
Steve Shoup, FanSpeak
DraftTek
What About the Top 100?
If we eliminate every player ranked outside of a board’s top 100, we might get a fairer view without punishing experts for doing more work. By evaluating the outcomes of each of the player in every board’s top 100, we end up with the following lists.
The Five Best Top 100s of 2016
NFLDraftGeek
Pro Football Focus
DraftBoardGuru
Corey Chavous, formerly DraftNasty
The Five Worst Top 100s of 2016
Kyle Crabbs, Locked On
Jon Dove, formerly With The First Pick
Tony Pauline, SportsKeeda
Oscar Hazell, formerly SB Nation
DraftTek
Keep reading with a 7-day free trial
Subscribe to Wide Left to keep reading this post and get 7 days of free access to the full post archives.