QB analysts are better than ever, but the draft is a different beast
CORONADO, Calif. J.T. O’Sullivan spread out some of his old playbooks on the counter. Cincinnati, Minnesota, New Orleans. He played quarterback for 11 NFL teams across nine seasons, and a few years into retirement he started a YouTube channel called “the QB School” on the idea that he could provide high-level analysis of X’sand-O’s because he “played for more NFL offenses than any other QB in NFL history.”
Over the past six years, the channel has amassed 335,000 subscribers during a boom in longform quarterback analysis. Increasingly, former signalcallers are going outside of legacy media to teach football fans how to assess the most important position in America’s favorite sport.
But even though he’s an expert in multiple systems, and even though he has been an instructor at a camp for elite high-schoolers, he insists he cannot predict how college quarterbacks will perform in the NFL. And he’s certainly
not alone.
For a century, teams, researchers and others have poured an incalculable amount of time and money into projecting quarterbacks, and yet no one knows for sure which prospect will bust and which will go to the Hall of Fame. Many analysts call the process a “crapshoot.”
“It would be hard to point to the results and say we’ve made progress, as sad as that is,” NFL Network draft analyst Daniel Jeremiah said.
“I don’t think we’re [getting] closer,” O’Sullivan said, laughing, on a recent morning in his home, where he had spent weeks in his library-turned-recording-studio analyzing the top prospects. “Selfishly, I’m probably further away than ever. The longer I do this, the less confident [I am in projections].”
When the NFL draft starts Thursday night, a handful of general manag
ers will tie their futures to 20-something prospects who have never taken an NFL snap, betting big on the hope they actually do know more than everyone else. If they pick the right guy, their franchises will gain a significant edge in a league with widespread parity; if they pick wrong, it could cost them their jobs and reputations.
“The hardest thing to do in sports, I think,” said Washington Commanders GM Adam Peters, “is evaluate and develop a quarterback.”
Analysts such as O’Sullivan, who does not consult for any NFL teams, face a similar problem but far lower stakes.
“If I was going to get fired if I didn’t make the right pick, [concerns about quarterback scouting] would make me a hell of a lot more uncomfortable than me being a YouTuber giving my rankings,” he said.
- - -The root problem here - devoting resources to research without making progress - is not unique to quarterback scouting. Derek Thompson, writer for the Atlantic, noted in a recent podcast that, despite billions of dollars spent to study pancreatic cancer, the mortality rate “hasn’t budged in 60 years.”
In recent years, though, former quarterbacks have made huge strides in educating the public about quarterback play. O’Sullivan and his peers, including Kurt Warner, Chase Daniel and Dan Orlovsky, have increasingly stepped out of the confines of the TV studio and broadcast booth to deliver exhaustive analyses of coaches’ film, known as “All-22,” on social media. They have amassed hundreds of thousands of followers, mostly on X or YouTube, fans with seemingly
insatiable appetites for detailed breakdowns of quarterback play.
In the videos, they scrutinize the process of each snap. Did the offensive coordinator call a good play? Did the quarterback recognize the blitz and adjust the protection? Was the interception actually the fault of the receiver, who ran a bad route?
In one breakdown, O’Sullivan pointed out a backup Chiefs tight end keyed a critical play by setting a pick for his teammate without hitting the defender, which would have drawn a flag. The play wouldn’t show up in the box score but helped Kansas City win the Super Bowl.
“Wow,” a commenter wrote. “I would’ve never even noticed that.”
Surprisingly often, analysts caveat their takes with reminders that they don’t know what the quarterback is being coached to do.
“[Good analysis is] not about whether you played or not,” said Robert Griffin III, who hosts the “RG3 and The Ones” podcast. “[You] have to be careful not to assume what the read was or where the ball should have gone” because “no two systems in the NFL are exactly the same” and every coach teaches concepts differently.
Griffin added: “Using experience to say, ‘This is what I was taught’ is fair game, but to say, without a shadow of a doubt, that this is what the QB should have done isn’t analysis; it’s just an uninformed opinion. Be aware of that, and your journey to find the ‘why’ of QB play will avoid the potholes of groupthink and angry fan analysis.”
Last season, subscribers clamored for O’Sullivan to study San Francisco’s Brock Purdy and Chicago’s Justin Fields. They wanted to know how much credit Purdy deserved for the 49ers’ success and how
much to blame Fields for Chicago’s disappointing season. Fields was personal; O’Sullivan had ranked him No. 1 in the 2021 class ahead of top pick Trevor Lawrence - because he loved Fields’s growth, athleticism and high ceiling.
Over the first two years, Fields was uneven, and O’Sullivan regularly went back and forth with Bears fans. He processed Fields’s third season in real time on the channel, with breakdowns that regularly ran more than an hour. Ultimately, he concluded Fields and Purdy were somewhat responsible for their offenses’ performances but had contrasting situations: The 49ers maximized Purdy and the Bears hindered Fields.
For anyone paying attention, 49ers and Bears fans fueled the growth of O’Sullivan’s channel and presented a model for growth in the booming market: Expertise and engagement.
Chase Daniel - a journeyman quarterback over 14 NFL seasons - has a similar story to O’Sullivan, and after he retired in 2023 he started podcasting and going in studio for NFL Network. But he noticed that, for massive national games, network broadcasters had to define even basic terms such as “cover-two.” Daniel realized a large audience on social media wanted to go deeper, so he started posting quarterback breakdowns. On Oct. 1, his YouTube channel had 4,900 subscribers; six months later, it had 48,000.
“When I was playing, I didn’t see a lot of [deep analysis] out there because no one really thought people would like it,” Daniel said. But he believes fans are becoming smarter and more curious, maybe in part because of the rise in sports gambling, and saw an opportunity to improve their understanding.
“QB analysis was there,
like on the deep, dark web of Twitter, but it was people doing it that had no idea about it. It was your-mom’sbasement people,” he said. “[QB analysis has] gotten a lot more serious, especially this past year.”
The market will probably continue to grow. Analysts can receive ad money from YouTube - the healthiest economy on the entire internet - and build direct relationships with viewers. O’Sullivan said he has about 4,000 paying subscribers on Patreon, where memberships start at $5 per month.
But during the draft process, NFL expertise becomes less applicable. The college game is not the same - the hash marks are wider, some rules differ - and it’s impossible to project performance.
For analysts, the draft presents a paradox: Fan interest goes up as the quality of their analysis - and the availability of game film goes down. And they know it. That’s why, during a recent episode, O’Sullivan looked visibly uncomfortable.
“Today,” O’Sullivan said into the camera, “one of the most anticipated videos of the year. One of my least favorite videos of the year to make. You guessed it: The Quarterback School 2024 draft quarterback rankings.”
Repeatedly, O’Sullivan reminded viewers that his rankings were not projections or a mock draft. He evaluated prospects exclusively on the quality of their game film because their future performance would be determined by so many factors, including coaching, health, organizational stability, scheme fit, supporting cast and a good bit of luck.
Like every analyst, O’Sullivan knew these qualifiers would be lost on some fans who, not long from now, would screenshot his errors and attempt to discredit his work. He grinned, squirmed
and put up the graphic:
1. Caleb Williams
2. Jayden Daniels
3. J.J. McCarthy
4. Drake Maye
5. Michael Penix Jr.
6. Bo Nix
A few days later, O’Sullivan still felt okay about his ranking given its limited perspective.
“I can live with that take,” he said. “I can’t live with just, like, making s—- up about, ‘Oh, I watched every single throw of every single person, and this is going to be the best organizational fit in this team culture and this scheme.’ … I just kind of, like, shiver when I hear people say s—- like that.”
General managers have the same challenge, except with their jobs on the line. They too saw Maye’s mechanical flaws, which caused him to miss short throws, but they must guess whether, given the right situation, he could reach his superstar ceiling.
“It’s a lot easier to evaluate a defensive end than it is a quarterback,” said Peters, the Commanders’ GM. “There’s so many other things that go into a quarterback than just what you see on the tape.”
Given the challenge, the league is shifting its approach on how it handles the uncertainty surrounding quarterbacks.
“There’s more attention being paid now to the [NFL] environment than the actual quarterback,” Jeremiah said, adding: “Different [playing] styles, that’s not something to get carried away with. To me, it’s more a focus of: Do we have the right play caller, the right offensive line to protect him, and do we have some guys he can get the ball to? That seems to be what’s going to lead to these guys being a success or not, more so than even just how good the player at the quarterback position might be.”
Kevin Cole, a football data analyst who writes the
Unexpected Points newsletter, believes teams have been “marginally better” at quarterback evaluation in recent years because they’re more willing to take shots on players from smaller schools and with more limited résumés, and quicker to admit whiffs.
“Teams are doing the right thing in moving off picks earlier,” he wrote in an email. “That’s part a recognition of how few QBs improve after really poor starts to their careers (not rookie year, but being bad in years 2 and 3), and part changes in the rookie wage scale from 2011. You’re writing off a much smaller percentage of the salary cap now when you move on from a top pick than you were for those drafted before 2011.”
O’Sullivan, for his part, isn’t trying to improve the evaluation process. In fact, he said, he has “let go of the idea that there’s a significantly better way to do it.” His goal is to deepen fans’ understanding of why quarterbacks succeed or fail, and this year there very well could be another Fields.
Though he misses competing on the field, and the life of a YouTuber can get a little lonely, O’Sullivan said he’s not interested in becoming a coach or GM. He lives in a nice house on an island across the bay from San Diego with his wife and three sons. He never has to travel, go to work meetings or worry about getting fired. He loves watching, talking and teaching football.
So, the other day, he pulled out some of his old playbooks and found a few examples of “Hank,” a short-passing concept he hates. He fired up the cameras that would beam his face onto YouTube, that would help fans get just a little smarter about a field that’s still filled with unknowns, and said: “Hi everybody, and welcome to the QB School.”