The recent attention surrounding an elite performance by Chicago Bears’ quarterback Caleb Williams being graded as rather average by the organization Pro Football Focus has attracted a great deal of well-earned criticism, but it would ultimately be meaningless if it were a simple one-time mistake. It could be as insignificant as an interception thrown in the final stages of a game that never even enters the stat line because of a penalty. However, it could also be indicative of an organization that has fundamental
flaws in how it approaches and presents its work.
Grading Quarterbacks
Let’s get the obvious one out of the way. Imagine, for a moment, that you (the person reading this) are driving a perfectly lovely 2014 Honda Accord LX. Now, imagine that next to you is Michael Schumacher magically restored to his athletic peak in a custom-built Ferrari. You both have the same goal–make it across town as efficiently as possible without crashing.
Do you think that you will make the same decisions? Do you think that you should make the same decisions? Even if you are a moderately talented driver, there are things Schumacher will be able to do easily that are not even options for you.
Pro Football Focus, however, would grade your decisions by the same scale. A lot of quarterbacks have a Honda Accord for an arm, but Caleb Williams has a Ferrari, and they still insist on evaluating the throws he makes by the scale applied to other arms. There are moves he can make, decisions he can access, that simply are not reasonable considerations even for the majority of professional athletes. Josh Allen is another quarterback who fits into this same category, in that he has options to him at the position that are not available to a vast majority of players, and hence the system fails.
However, I mentioned Michael Schumacher as the driver of this Ferrari, and that’s also relevant to the discussion. Because just as you or I could not drive the Ferrari nearly as well as a professional racecar driver, the average moderately-trained analyst cannot break down defensive tendencies and situations nearly as well as a professional quarterback being guided by Ben Johnson over a week’s worth of film review. Yet, again, PFF believes that they know better how to read defensive tendencies than the quarterback in the situation performing the review.
Of course, someone moderately knowledgeable about football should spot a fundamental problem in my driving analogy, and it’s this–most of the time, the other cars on the road will not actively be trying to hinder your (or Schumacher’s) progress. I know it might seem otherwise (cough, Eisenhower Expressway, cough). Teams are actually engaged in a match against each other that is as much mental as it is physical, and aggressive play from the quarterback on a handful of throws can force a defense to react even if those individual throws for that individual play are not as effective. If I’m playing chess, I will gladly sacrifice a pawn to take a rook. If you are playing football, you might gladly take a deep shot where only your receiver can get it even if it doesn’t get a first down if it influences the defense to play the next series differently. The very act of grading a quarterback on a per-play basis ignores the degree to which football works on developing and exploiting tendencies across 60 minutes.
Simply put, for all their talk about process, the process Pro Football Focus uses is fundamentally flawed.
Drafting
Next, however, it’s important to point out that Pro Football Focus fails in other areas. Twice, recently, I have evaluated which “big boards” do the best job of evaluating offensive linemen and edge defenders.
With offensive linemen, I compared PFF to Daniel Jeremiah, Jacob Infante, Mock Draft Database’s Consensus Board, and Brandon Thorn. The study had more than a hundred offensive line prospects over three years of the draft. While I originally included PFF only for the sake of a different perspective, I found that Pro Football Focus was the least accurate of any public board I studied, and that only a third of their top prospects were selected near where they were drafted. They frequently disagreed with other boards and with the NFL itself in which players should be valued.
Additionally, “when reviewed by a third-party evaluator once they entered the NFL, the prospects uniquely favored by PFF played worse than the players uniquely devalued by them.” In other words, PFF’s draft board preferences for offensive linemen not only did not resemble the players that NFL teams actually wanted to draft, but they also did not resemble the actual performance of those players once they got on the field. As I concluded at the time, “it’s worth asking if their grades are so inaccurate when checked at other times, how valuable can their in-season grades be?”
When I pivoted to edge defenders, I also found a disconnect between which players were favored (or disfavored) by Pro Football Focus and which ones were actually productive on the field. This was a more limited study that compared singular measures of performance (such as combine metrics) and also ranking by board evaluators to see if there was any trait that aligned favorably with the performance of a new edge defender in terms of actually generating disruption, measured by pressures and also by disruptive plays (sacks, tackles for a loss, and defended passes). Of all individual public boards, PFF did the worst in terms of aligning their favored players with total pressures generated by edge defenders in their first two years. As I concluded in that article regarding draft prospects, if PFF is an outlier to other systems, it is likely PFF that is in error.
Again, it is fine that they disagree in evaluation. However, if they consistently produce player evaluations that do not reflect the actual performance of the players, then what exactly is the value of their product?
It’s the Process
On a fundamental level, the product that Pro Football Focus puts out is nothing more and nothing less than the subjective evaluation of whether or not a given player tends to act the way that PFF itself thinks they should have acted. It’s an aesthetic score based on a set of normalized evaluators who are asked: “Does the performance look like we think it should look?” For those old enough to remember the media broadcasting of red carpet shows from the Joan Rivers era, PFF tells subscribers whether or not a given player matched the arbitrary fashion rules that they themselves generated.
In and of itself, there is nothing wrong with that. As an entertainment product, it’s fine. However, the problem comes from the way that PFF presents itself. By using a number scale (complete with decimal points) and presenting their work as “analytics”, implying a claim to objectivity, they risk engaging in pseudoscience. From Oxford Reference:
One of the key differences between science and pseudoscience is that a scientific statement or theory is stated in such as a way as to be falsifiable. In contrast, pseudoscientific statements are usually not falsifiable using objective experimental or observational evidence. Pseudoscience provides no room for challenge and tends to dismiss contradictory evidence or to selectively decide what evidence to accept.
The scores they give are not derived from objective measures but instead from vibes. Their methodology is proprietary, too, so it is not possible to have a third party check those vibes. Pro Football Focus seems to want the reputational benefits of claiming that their work is above reproach because it is “fact,” while not actually opening themselves up to the rigor of science by testing to see if others, using the same data and methodology, produce the same results.
All-time great J.J. Watt frequently explained the flaws in their system, highlighting how the success of their grading system is from marketing, not from any value added by the product. That product is now ubiquitous and is unlikely to go anywhere. However, hopefully as new measures come out—like those provided by Pro Football Network and NextGen Stats—conversations can continue to be had, and more people will be willing to hold all grading systems to a higher standard.
