Success Profiler Study
Posted: Wed Jan 11, 2017 8:54 am
Last year about this time I came across a post from Dynasty DeLorean about his RB report. I was curious about what he was doing with his work and sent him a PM to try to see if he would share his initial process because I wanted to check it out and see what some of my take away's would be. He was more than gracious, and I started this process using with his initial format.
He was looking for a combination that only sure fire studs would have and I went took my study a different route. Instead of looking for a top tier only, I wanted to see if there were certain characteristics that successful RB's had, with the theory being that if you found a player with more traits that lead to successful RB's then you could better determine which RB's had the ability to succeed in the NFL and which did not. With that basis, we went very different routes with calculating the numbers - to use his description his "model is more to predict studs while (my) model is more to predict which RB's will become a starter in some form or fashion at some point in their career". While I disagree slightly with that statement, it does show some of our difference in process. I also ended up expanding my study to both WR and TE as well. I do not know him and we have not worked on this together, but I want to make sure to give him credit for helping me start this study. While I am doing this long term (want to re-evalute the scoring setup after 3 years to see what needs to be tweaked) I am posting it this year because the scoring ended up holding shockingly consistent when I entered the results from this NFL season (varying no more than 3%). I welcome any feedback and criticism about how it is done, because my goal is not be "be right", but to hopefully put all the measurable information in a usable context.
To provide more context for my study and the scores I will post, I want to share some of the process. Initially I used the measurables that DD had used, but kept hearing about this number that mattered or that number that matter and kept trying to find what actually did matter. As I compiled everything it just lead to a jumbled mess, so I had to figure out what actually was potentially predictive and what was not. So I came up with the idea of using a "hit rate", meaning there was a benchmark that when a player got to in their career (a single season benchmark, and if they got it at any point in their career they always had it) it was counted as a "hit" and they were considered a "hit" in the NFL.
I did not want my benchmarks to chase fantasy scores that fluctuate year to year, but rather see who was actually beneficial in the NFL for their team, because the thought was that would more likely indicate future usage (and hopefully future success) for that player and I was looking for just that, future indications of success. Again for the sake of transparency here are the benchmarks I settled on: 1,000 total yards for RB, 600 yards and 45 receptions for WR, and 400 yards for TE. This led to roughly 30 RB's per year (with no false peaks due to high TD rates), 80-90 WR per year (no TD for the same reason), and 20 TE's per year. These ranges were purposeful as they roughly represent the "starting" NFL skill position (1 starting RB per team, 2 outside WR plus a slot for most teams, and just under a starting TE per team as not every team uses their TE in the passing game). If a player put up startable levels of production, that was enough for them to be considered to be included in the players that we wanted to isolate successful attributes from.
After entering all the measurables that were said to be important, and going back to 2005 with the numbers of all players drafted 6th round or earlier or currently in the NFL (as of last year), I compiled a "hit" rate for all three positions. They came in for all players charted at 26.7% for RB's, 28.4% for WR's, and 30.3% for TE's. From there I had something to measure all these marks that claimed to be predictive against. The measurables I charted were the following: height, weight, 10 yard split, 40 yard dash, vertical, broad jump, 3 cone, 20 yard short shuttle, final year start age, speed score, height adjusted speed score, burst score, agility score, BMI, college dominator, and round drafted. I then compared them to 11 previous years of successful hits data.
Some claimed to be important measurable numbers didn't stand up, but that seemed to be overlooked did. As expected round drafted was easily the highest indicator of success. I analyzed each metric verses the average hit rate, and selected the ones that showed a marked and consistent progression and/or regression from the average for each position. I gave every 6% increase or decrease a value of 0.5 points. That number was used because it represented the roughly equal steps to reach a max of +3 or -3 for the most influential of all metrics used except draft position (not every mark fell on that 0.5 scale, so some did fall on other increments).
I want to touch on draft position separately because that is one thing I have read people say "nothing is better than it so why use anything else". First this isn't intended to replace anyone's individual way of doing things or to try to be better than the NFL at drafting (I am not payed enough to have to live up to that). The only intent here is to try to find the skill position players that have the most positive attributes that lead to success and to put some of those measurable numbers in perspective (what does a 3 cone time actually mean for this player).
That said I was pleasantly surprised at how well the success profiler scoring ended up holding up to draft position. I made the choice to account draft position at half value and capping it at 3 up or down like every other metric for a couple reasons - 1) because draft position already includes valuations on all the other metrics included but adds in scouting so I did not want to double up on any metrics by overvaluing draft position while still including the professional scouting 2) because at full value draft position would have mitigated all other metrics and made the rest of the study pointless. In the end when I compared the total players drafted at a position each round to the same sample size based on the success profiler scoring here is how they measured up based on hit rates (Draft position/SP score hit rates):
RB 1st round - 96.2%/80.8%, 2nd round - 40.0%/60.0%, 3rd round - 35.7%/28.6%, 4th round - 24.2%/24.2%, 5th round - 14.2%/23.1%, 6th round - 11.4%/11.4%, 7th round - 12.5%/16.7%, undrafted - 10.1%/7.6%
WR 1st round - 71.7%/76.1%, 2nd round - 54.9%/60.8%, 3rd round - 37.3%/32.2%, 4th round - 21.6%/23.5%, 5th round - 19.5%/24.4%, 6th round - 8.0%/0.0%, 7th round - 17.4%/21.7%, undrafted - 13.2%/10.7%
TE 1st round - 100.0%/87.5%, 2nd round - 63.2%/63.2%, 3rd round - 46.2%/50.0%, 4th round - 37.5%/37.5%, 5th round - 18.5%/37.0%, 6th round - 37.5%/18.8%, 7th round - 0.0%/7.1%, undrafted - 9.4%/3.8%
Simply being higher or lower than the draft position hit rate is neither good nor bad. Ideally I would want higher top end hit rates and lower bottom end hit rates to try to create bigger gaps from players making decisions easier, but this is simply to show how it lines up with draft position. What I learned is that 1st round RB's will get enough usage to count as HIT's even if they aren't good long term (I'm looking at you Donald Brown, Felix Jones, and Cedric Benson), and that if a TE gets drafted in the 1st, throw all other scouting away and draft him (not a large sample size, but 100.0% hit rate is impressive).
I also will include a pre-draft position hit rate to find both sleepers and players that may be over drafted. We will get into those more in future posts.
To be continued...
He was looking for a combination that only sure fire studs would have and I went took my study a different route. Instead of looking for a top tier only, I wanted to see if there were certain characteristics that successful RB's had, with the theory being that if you found a player with more traits that lead to successful RB's then you could better determine which RB's had the ability to succeed in the NFL and which did not. With that basis, we went very different routes with calculating the numbers - to use his description his "model is more to predict studs while (my) model is more to predict which RB's will become a starter in some form or fashion at some point in their career". While I disagree slightly with that statement, it does show some of our difference in process. I also ended up expanding my study to both WR and TE as well. I do not know him and we have not worked on this together, but I want to make sure to give him credit for helping me start this study. While I am doing this long term (want to re-evalute the scoring setup after 3 years to see what needs to be tweaked) I am posting it this year because the scoring ended up holding shockingly consistent when I entered the results from this NFL season (varying no more than 3%). I welcome any feedback and criticism about how it is done, because my goal is not be "be right", but to hopefully put all the measurable information in a usable context.
To provide more context for my study and the scores I will post, I want to share some of the process. Initially I used the measurables that DD had used, but kept hearing about this number that mattered or that number that matter and kept trying to find what actually did matter. As I compiled everything it just lead to a jumbled mess, so I had to figure out what actually was potentially predictive and what was not. So I came up with the idea of using a "hit rate", meaning there was a benchmark that when a player got to in their career (a single season benchmark, and if they got it at any point in their career they always had it) it was counted as a "hit" and they were considered a "hit" in the NFL.
I did not want my benchmarks to chase fantasy scores that fluctuate year to year, but rather see who was actually beneficial in the NFL for their team, because the thought was that would more likely indicate future usage (and hopefully future success) for that player and I was looking for just that, future indications of success. Again for the sake of transparency here are the benchmarks I settled on: 1,000 total yards for RB, 600 yards and 45 receptions for WR, and 400 yards for TE. This led to roughly 30 RB's per year (with no false peaks due to high TD rates), 80-90 WR per year (no TD for the same reason), and 20 TE's per year. These ranges were purposeful as they roughly represent the "starting" NFL skill position (1 starting RB per team, 2 outside WR plus a slot for most teams, and just under a starting TE per team as not every team uses their TE in the passing game). If a player put up startable levels of production, that was enough for them to be considered to be included in the players that we wanted to isolate successful attributes from.
After entering all the measurables that were said to be important, and going back to 2005 with the numbers of all players drafted 6th round or earlier or currently in the NFL (as of last year), I compiled a "hit" rate for all three positions. They came in for all players charted at 26.7% for RB's, 28.4% for WR's, and 30.3% for TE's. From there I had something to measure all these marks that claimed to be predictive against. The measurables I charted were the following: height, weight, 10 yard split, 40 yard dash, vertical, broad jump, 3 cone, 20 yard short shuttle, final year start age, speed score, height adjusted speed score, burst score, agility score, BMI, college dominator, and round drafted. I then compared them to 11 previous years of successful hits data.
Some claimed to be important measurable numbers didn't stand up, but that seemed to be overlooked did. As expected round drafted was easily the highest indicator of success. I analyzed each metric verses the average hit rate, and selected the ones that showed a marked and consistent progression and/or regression from the average for each position. I gave every 6% increase or decrease a value of 0.5 points. That number was used because it represented the roughly equal steps to reach a max of +3 or -3 for the most influential of all metrics used except draft position (not every mark fell on that 0.5 scale, so some did fall on other increments).
I want to touch on draft position separately because that is one thing I have read people say "nothing is better than it so why use anything else". First this isn't intended to replace anyone's individual way of doing things or to try to be better than the NFL at drafting (I am not payed enough to have to live up to that). The only intent here is to try to find the skill position players that have the most positive attributes that lead to success and to put some of those measurable numbers in perspective (what does a 3 cone time actually mean for this player).
That said I was pleasantly surprised at how well the success profiler scoring ended up holding up to draft position. I made the choice to account draft position at half value and capping it at 3 up or down like every other metric for a couple reasons - 1) because draft position already includes valuations on all the other metrics included but adds in scouting so I did not want to double up on any metrics by overvaluing draft position while still including the professional scouting 2) because at full value draft position would have mitigated all other metrics and made the rest of the study pointless. In the end when I compared the total players drafted at a position each round to the same sample size based on the success profiler scoring here is how they measured up based on hit rates (Draft position/SP score hit rates):
RB 1st round - 96.2%/80.8%, 2nd round - 40.0%/60.0%, 3rd round - 35.7%/28.6%, 4th round - 24.2%/24.2%, 5th round - 14.2%/23.1%, 6th round - 11.4%/11.4%, 7th round - 12.5%/16.7%, undrafted - 10.1%/7.6%
WR 1st round - 71.7%/76.1%, 2nd round - 54.9%/60.8%, 3rd round - 37.3%/32.2%, 4th round - 21.6%/23.5%, 5th round - 19.5%/24.4%, 6th round - 8.0%/0.0%, 7th round - 17.4%/21.7%, undrafted - 13.2%/10.7%
TE 1st round - 100.0%/87.5%, 2nd round - 63.2%/63.2%, 3rd round - 46.2%/50.0%, 4th round - 37.5%/37.5%, 5th round - 18.5%/37.0%, 6th round - 37.5%/18.8%, 7th round - 0.0%/7.1%, undrafted - 9.4%/3.8%
Simply being higher or lower than the draft position hit rate is neither good nor bad. Ideally I would want higher top end hit rates and lower bottom end hit rates to try to create bigger gaps from players making decisions easier, but this is simply to show how it lines up with draft position. What I learned is that 1st round RB's will get enough usage to count as HIT's even if they aren't good long term (I'm looking at you Donald Brown, Felix Jones, and Cedric Benson), and that if a TE gets drafted in the 1st, throw all other scouting away and draft him (not a large sample size, but 100.0% hit rate is impressive).
I also will include a pre-draft position hit rate to find both sleepers and players that may be over drafted. We will get into those more in future posts.
To be continued...