Advanced Analytics - Success or Failure?

[skip]

I am curious if anyone has done enough of an "analysis" on the success or failure rate of "advanced analytics". While this may sound silly, I think it is a very appropriate question. Hopefully we all realize that statistics can be presented in different ways to support or disprove a position. Does it really matter to me that the offensive tackle that was just signed as a free agent by team X graded out as the 3rd worst in the league last season on passing downs and what this means as the potential impact on the passing game? While this may be an interesting study to some statistics guy behind closed doors, it really doesn't matter to me in the scheme of things.

An example... Ladarius Green a couple of seasons back graded out as the best TE in the game on something like "fantasy points per route run". This gave a considerable boost to his value but has ultimately been pretty meaningless. While he is now (presumably) in a better opportunity, I think owners are still holding out hope based on that same analysis.

So the basic question is this...are analysts over complicating things? Does that extra level of analysis really provide greater insight or does it end up leaving us chasing false hope? Would more traditional, simpler analysis just be better?

[dlf_jules]

Seems like you're mostly talking about Pro Football Focus. The trouble there is that their grades are largely film-driven. They watch every play and assign grades on a play-by-play basis.

The fantasy points per route run stuff is a different thing (though also Pro Football Focus). Danny Tuccito has done a lot of work on that -- well, on closely related questions.

There are many "flavor" statistics out there, but only a few predictive ones.

[dlf_jules]

So the basic question is this...are analysts over complicating things? Does that extra level of analysis really provide greater insight or does it end up leaving us chasing false hope? Would more traditional, simpler analysis just be better?

Just realized I didn't really answer your question. Yes, you can get greater insight from more complicated statistics. Maybe the best proof I can give you is this: the guys winning in DFS -- the ones making a living off it -- they're all running complicated computer algorithms to generate lineups. I did fairly well in DFS (actually very well), but I don't have the programming or statistical chops to make it a full time thing. Plus, I think you almost have to play MLB and NBA to go full time.

Of course, we can also be led astray by bad stats, whether they be simple or fancy.

One more thing: drafters are generally getting smarter. The guys on Living the Stream were talking this week about how Blake Bortles would have been way overvalued three years ago, but now people (somewhat) understand regression to the mean. So you don't have to read the stats to get the knowledge. It's part of the conventional wisdom. But those who know how to filter out the good from the bad still have a small advantage.

[Phaded]

So the basic question is this...are analysts over complicating things? Does that extra level of analysis really provide greater insight or does it end up leaving us chasing false hope? Would more traditional, simpler analysis just be better?

I think all of all these different number crunches are interesting - but personally I take them with a grain of salt and not gospel. They make for interesting reads and are generally food for thought. However, I don't live by them. There is no perfect formula out there. There will always be outliers and exceptions to any formula someone can come up with. It's not a perfect science, nor should it be. I will say that some people place too much value in some of these analytics. They can help you find a diamond in the rough but they can also give you unrealistic hope for a player that may never hit the expected level from those analytics.

At the end of the day - I still think the number one tool for analysis should be what you see on tape. How the player performs in-game when you watch him. I know it is still my number one tool. That said - there are also plenty of other resources that literally tell you what is on tape or break down the tape for you as well.

[maxhyde]

So the basic question is this...are analysts over complicating things? Does that extra level of analysis really provide greater insight or does it end up leaving us chasing false hope? Would more traditional, simpler analysis just be better?

I think all of all these different number crunches are interesting - but personally I take them with a grain of salt and not gospel. They make for interesting reads and are generally food for thought. However, I don't live by them. There is no perfect formula out there. There will always be outliers and exceptions to any formula someone can come up with. It's not a perfect science, nor should it be. I will say that some people place too much value in some of these analytics. They can help you find a diamond in the rough but they can also give you unrealistic hope for a player that may never hit the expected level from those analytics.

At the end of the day - I still think the number one tool for analysis should be what you see on tape. How the player performs in-game when you watch him. I know it is still my number one tool. That said - there are also plenty of other resources that literally tell you what is on tape or break down the tape for you as well.

...but don't throw the baby out with the bathwater.
Meaning your formula isn't garbage because it doesn't account for 100%...very few things in life and even fewer in gambling are 100% accurate

[OhCruelestRanter]

Good analytics are good precisely because they can tell you how often they succeed or fail. See football outsiders for an example of good analytics. See KC Joyner for an example of bad analytics.

[ninotoreS]

Well, I've arrived at the opinion that 'NEP' (Net Expected Points) isn't very reliable for projecting future success. It's numberfire's 'signature metric'. Many times now I've seen the metric used to justify pessimism or optimism with several players, and the prediction ended up faceplanting. I think it's fine for quatifying a deeper appreciation for a given player's productivity in the present, but used as a forecasting tool it'll burn you.

I think deep metrics are fine, but too often they're produced without any awareness of the tape, without actually seeing the football that was played. And when they're geared around 'point scoring' I think they're especially faulty and overlooking too much of the mitigating context that goes on with 11 v 11 football, coaching/playcalling factors, etc. There's so many moving parts in pro football that can artificially drive down or inflate a given player's performance through factors other than their own merit or lack thereof. So, you've got to watch the tape. There is no substitue for tape study, but many 'advanced analytics' authors don't want to believe that.

[OhCruelestRanter]

Well, I've arrived at the opinion that 'NEP' (Net Expected Points) isn't very reliable for projecting future success. It's numberfire's 'signature metric'. Many times now I've seen the metric used to justify pessimism or optimism with several players, and the prediction ended up faceplanting. I think it's fine for quatifying a deeper appreciation for a given player's productivity in the present, but used as a forecasting tool it'll burn you.

I think deep metrics are fine, but too often they're produced without any awareness of the tape, without actually seeing the football that was played. And when they're geared around 'point scoring' I think they're especially faulty and overlooking too much of the mitigating context that goes on with 11 v 11 football, coaching/playcalling factors, etc. There's so many moving parts in pro football that can artificially drive down or inflate a given player's performance through factors other than their own merit or lack thereof. So, you've got to watch the tape. There is no substitue for tape study, but many 'advanced analytics' authors don't want to believe that.

I'm comfortable saying the people who are actually really good at this rely more heavily on analytics than on film study.

[Tsunami]

The sample sizes in the NFL are all too small to get reliable statistics like this.

[ninotoreS]

I'm comfortable saying the people who are actually really good at this rely more heavily on analytics than on film study.

I'm comfotable saying the people who are actually really good at this study film every bit as much as they pay attention to analytics.

Waldman, Harmon, Cosell? All noted film rats. Not to mention nearly every employed NFL scout.


The reality with people who depend on analytics over tape is that they want a shortcut to making their conclusions and predictions. Studying tape takes more time and effort than looking at numbers. And an All-22 subscription, college conference network subscription, etc. It's time-consuming and more expensive, so some people want to believe there's an alternative. Well, that's a delusion. No. Substitute. For. Tape study.

[dlf_jules]

The reality with people who depend on analytics over tape is that they want a shortcut to making their conclusions and predictions. Studying tape takes more time and effort than looking at numbers. And an All-22 subscription, college conference network subscription, etc. It's time-consuming and more expensive, so some people want to believe there's an alternative. Well, that's a delusion. No. Substitute. For. Tape study.

You're right, I do want shortcuts. To that point, there's a great substitute for me studying tape: other people studying tape.

Waldman "beats" the draft about 44% of the time. I'm not going to be that good. It's not worth the time investment, and I don't love it that much. Plus, why bother? Just go with draft position and you'll be right more often. As for metrics, Jon Moore's Phenom Index also "beats" the draft about 44% of the time. Again, no advantage.

But when Waldman and the Phenom Index agree that the draft has it wrong, they're right 59% of the time. That's a serious advantage. So, yeah, there's no substitute for film study. But analytics still have value.

[gkritikos]

Advanced Analytics (or just analytics in general) is a set of tools no different than film. Both are trying to understand what happens and project what will happen. To Brian's point on Waldman and Moore, it's better to take in more information than less and it will ultimately get you closer to 100% accuracy than either on its own.

I'm a numbers guy and understand them well. That said, I know the value of film and rely on the expertise of others to fill those knowledge gaps for me. My opinion is that anyone who ignores one side of it in an effort to prove the other right (or because they only know one side) is doing themselves a disservice. Why use a hammer for everything when an entire toolbox is just sitting there?

[OhCruelestRanter]

The idea isn't that film study in general isn't important. It's that you, the fantasy football player, cannot add anything of demonstrable value to your team by watching film. Between the millions of dollars committed by NFL teams, plus the public opinions of guys like Waldman and Harmon, there is nothing that you can do by staring at All-22 film that will help your team beyond just looking at the draft and reading what those guys say.

In the contrary, as Jules has demonstrated, you can use analytics to find ways to out-perform the NFL draft.

If you like watching film, good for you. I'm not trying to talk you out of doing something you enjoy. I just hope you realize that you're doing it purely for your enjoyment, because if you think you're helping your team, you're deluded and wasting your time.

[TomBobAnderson]

In that case I think most people watching tape and/or crunching numbers are wasting their time. Cruelest Ranter, do you really believe that NFL teams also don't Ivy league grads crunching numbers? Most fantasy football players won't be able to out analyze or out film watch NFL front offices.

[ccj]

I feel like this conversation is a little all over the place. Using draft position as a guide isn't pure analytics because what you're relying on is the evaluation of the NFL and that evaluation is going to be a blend of film, analytics, intangibles, team need, etc. I think when we as a fantasy community use it, it's a fair predictor, but if you use it exclusively, you're not an analytics guy.

As mentioned above ProFootballFocus grades are based on someone watching the film so if you decide to try and blend those grades into your statistical projections... that's film.

All that being said, film study is useless (not to say it isn't fun) to the fantasy community unless you need to address a stat that isn't being published. If you're trying to find the success rate of a DB defending the slot vs the X etc. You might watch film to address that, but what you're doing is gathering statistics (of course, defining success/fail could be film study depending on how you're doing it). I liken fantasy to poker a lot because that's my gambling background. Unskilled poker players will always tell you how they can't play online because they need to be at the table to read the other players... it's not true, they just lose faster online because the hands come faster. Betting patterns are a MUCH better predictor of your opponent's hand strength than any mannerisms you're hoping to glean.

DFS has put this argument to bed. That's an analytics driven game and where the most money stands to be won and lost, so the highest level of our endeavor. Just like online poker was first hit by hand tracking software which allowed you to log thousands of hands your opponent's play (and was later utilized by bots). It's not about catching a tell, it's about running the numbers.

Now, don't conflate my statement with running an NFL team. Running an NFL team requires certain types of players, requires team chemistry, and film study can make a huge difference in those evaluations. Running an NFL fantasy team requires numbers, watching the games is superfluous.