Legit rankings without preseason ranking consideration

[Great Dayne]

I love me some Massey ratings.

College Basketball Ranking Composite

It is a comprehensive rating system as it includes majority of the major ranking systems including human and computer polls.

 

[TrollyMcTroller]

College Basketball Ranking Composite

It is a comprehensive rating system as it includes majority of the major ranking systems including human and computer polls.

Which isn't as great of an idea as it seems, even in theory, and in practice they've actually done a worse job.

Arbitrarily lumping a bunch of different stuff together and taking the average really doesn't tell you much. Most of those systems are all measuring different things. For instance if you wanted to take an average of the coach's poll and the AP that kinda makes sense or if you wanted to do it with the RPI and Sagarin ELO_CHESS rating, they are sorta measuring the same thing so it makes sense to put them together.

But lumping everything together makes no more sense that trying to find out what the best movie of the year was by averaging viewer scores, popcorn sales, ticket sales, critic ratings, budget, and profit.

Some of those things may have some bearing on how good the movie was, but just lumping all of that stuff together, doesn't really tell you anything meaningful.

And that's not the worst of it.

Beyond the flawed premise in the first place, their methodology is flawed too.

For starters they are weighting things without even realizing it. Sagarin computes his final rating based on his PREDICTOR and ELO_CHESS scores. Those two directly influence the final rating, yet the composite above includes all three. So Sagarin's calculations are getting way more weight than KenPom's since his numbers are doubled up.

The other problem is that they're using the ranking, not the raw data. Take the human polls for instance (but this could apply to the computer rankings just as much) Say the #1 team in the AP got 1500 votes. #2 got 1450 and #3 got 1200. According to the AP voters, #1 and #2 are very close, but there is a significant drop off to #3. But when you just use the rankings, the gap between 1 and 2 is the same as 2 and 3. It misrepresents the data. It's a lot easier to calculate that way, but the data is much less meaningful (assuming it makes sense to lump it together in the first place, which it doesn't)

The proper way to do it is to take the raw data, and normalize it so that the highest value and lowest are equal. then take the average (or arithmetic mean, or whatever is appropriate) and get your rankings from that.

A few years ago just as a proof of concept I did that very thing daily and posted a composite of KenPom and Sagarin daily, with the disclaimer, that they two didn't belong together (since they were measuring fundamentally different things) but it was actually taking weighted and normalized versions of Sagarin's ELO_CHESS and PREDICTOR and KenPom's Offensive and Defensive efficiency numbers. It was fun to set up the spreadsheet to make it all happen, but at the end of the day, the numbers were meaningless. But they did spark a lot of conversation.

 

[Monte Carlo Wildcat]

Which isn't as great of an idea as it seems, even in theory, and in practice they've actually done a worse job.

Arbitrarily lumping a bunch of different stuff together and taking the average really doesn't tell you much. Most of those systems are all measuring different things. For instance if you wanted to take an average of the coach's poll and the AP that kinda makes sense or if you wanted to do it with the RPI and Sagarin ELO_CHESS rating, they are sorta measuring the same thing so it makes sense to put them together.

But lumping everything together makes no more sense that trying to find out what the best movie of the year was by averaging viewer scores, popcorn sales, ticket sales, critic ratings, budget, and profit.

Some of those things may have some bearing on how good the movie was, but just lumping all of that stuff together, doesn't really tell you anything meaningful.

And that's not the worst of it.

Beyond the flawed premise in the first place, their methodology is flawed too.

For starters they are weighting things without even realizing it. Sagarin computes his final rating based on his PREDICTOR and ELO_CHESS scores. Those two directly influence the final rating, yet the composite above includes all three. So Sagarin's calculations are getting way more weight than KenPom's since his numbers are doubled up.

The other problem is that they're using the ranking, not the raw data. Take the human polls for instance (but this could apply to the computer rankings just as much) Say the #1 team in the AP got 1500 votes. #2 got 1450 and #3 got 1200. According to the AP voters, #1 and #2 are very close, but there is a significant drop off to #3. But when you just use the rankings, the gap between 1 and 2 is the same as 2 and 3. It misrepresents the data. It's a lot easier to calculate that way, but the data is much less meaningful (assuming it makes sense to lump it together in the first place, which it doesn't)

The proper way to do it is to take the raw data, and normalize it so that the highest value and lowest are equal. then take the average (or arithmetic mean, or whatever is appropriate) and get your rankings from that.

A few years ago just as a proof of concept I did that very thing daily and posted a composite of KenPom and Sagarin daily, with the disclaimer, that they two didn't belong together (since they were measuring fundamentally different things) but it was actually taking weighted and normalized versions of Sagarin's ELO_CHESS and PREDICTOR and KenPom's Offensive and Defensive efficiency numbers. It was fun to set up the spreadsheet to make it all happen, but at the end of the day, the numbers were meaningless. But they did spark a lot of conversation.

See I've never been one to look at all these math sites as a source but recently I've started to. It sounds like you heavily favor sagarin over all other sites. IMO, they're ll fairly similar. I guess it's simply a matter of preference but again, I'm not an expert on college mathematics.

 

[TrollyMcTroller]

See I've never been one to look at all these math sites as a source but recently I've started to. It sounds like you heavily favor sagarin over all other sites. IMO, they're ll fairly similar. I guess it's simply a matter of preference but again, I'm not an expert on college mathematics.

I don't live or die by the numbers. I still think at the end of the day the human polls still do a better job overall. They take into account intangible things, like injuries, travel, time zones, start times, and other things. (For instance AZ traveling across two times zones, starting early and playing in a very hostile UM arena isn't the same thing as SU traveling to NYC and playing against St Johns, even though most will almost certainly be weighted simply as "road" games.)

I like Sagarin's stuff, and I'm most familiar with it, but I also like the BPI a lot. But I'm not married to either one. The biggest point I'm trying to make though, is that it is important to know what you're looking at when checking out the computer rankings, and for the most part, I think most people don't. Most fans know there is an RPI and have a vague idea what it is, but only a fraction of them seem to know what exactly it is and what it is intended to measure. Similar idea to Sagarin, KenPom, Massey, etc. They get thrown around a lot but most people that talk about them don't really seem to know what they are.

 

[Great Dayne]

I don't live or die by the numbers. I still think at the end of the day the human polls still do a better job overall. They take into account intangible things, like injuries, travel, time zones, start times, and other things. (For instance AZ traveling across two times zones, starting early and playing in a very hostile UM arena isn't the same thing as SU traveling to NYC and playing against St Johns, even though most will almost certainly be weighted simply as "road" games.)

I like Sagarin's stuff, and I'm most familiar with it, but I also like the BPI a lot. But I'm not married to either one. The biggest point I'm trying to make though, is that it is important to know what you're looking at when checking out the computer rankings, and for the most part, I think most people don't. Most fans know there is an RPI and have a vague idea what it is, but only a fraction of them seem to know what exactly it is and what it is intended to measure. Similar idea to Sagarin, KenPom, Massey, etc. They get thrown around a lot but most people that talk about them don't really seem to know what they are.

Correct, humans can also take into the account of anecdotal evidence that a team should've won (UW vs ASU football game example). Also, the example of UW's schedule with the most games played by any other team in the nation in a small amount of time For the duration of the season the human polls ranked UW ahead of ASU as though they beat them and the computers were the contrary.

The BPI does factor in injuries or games missed by players unlike most other computer ranking systems.
In the end BPI picked a higher percentage of tournament games over the other ranking systems at nearly 75% from 2007-2011. Glad to see the Badgers at #1 as they should be .

Also, thank you for all of your insight on the various ranking systems as most people (myself included) won't take the time to weed through the details even though I enjoy statistics.

 

[Great Dayne]

Which isn't as great of an idea as it seems, even in theory, and in practice they've actually done a worse job.

Arbitrarily lumping a bunch of different stuff together and taking the average really doesn't tell you much. Most of those systems are all measuring different things. For instance if you wanted to take an average of the coach's poll and the AP that kinda makes sense or if you wanted to do it with the RPI and Sagarin ELO_CHESS rating, they are sorta measuring the same thing so it makes sense to put them together.

But lumping everything together makes no more sense that trying to find out what the best movie of the year was by averaging viewer scores, popcorn sales, ticket sales, critic ratings, budget, and profit.

Some of those things may have some bearing on how good the movie was, but just lumping all of that stuff together, doesn't really tell you anything meaningful.

And that's not the worst of it.

Beyond the flawed premise in the first place, their methodology is flawed too.

For starters they are weighting things without even realizing it. Sagarin computes his final rating based on his PREDICTOR and ELO_CHESS scores. Those two directly influence the final rating, yet the composite above includes all three. So Sagarin's calculations are getting way more weight than KenPom's since his numbers are doubled up.

The other problem is that they're using the ranking, not the raw data. Take the human polls for instance (but this could apply to the computer rankings just as much) Say the #1 team in the AP got 1500 votes. #2 got 1450 and #3 got 1200. According to the AP voters, #1 and #2 are very close, but there is a significant drop off to #3. But when you just use the rankings, the gap between 1 and 2 is the same as 2 and 3. It misrepresents the data. It's a lot easier to calculate that way, but the data is much less meaningful (assuming it makes sense to lump it together in the first place, which it doesn't)

The proper way to do it is to take the raw data, and normalize it so that the highest value and lowest are equal. then take the average (or arithmetic mean, or whatever is appropriate) and get your rankings from that.

A few years ago just as a proof of concept I did that very thing daily and posted a composite of KenPom and Sagarin daily, with the disclaimer, that they two didn't belong together (since they were measuring fundamentally different things) but it was actually taking weighted and normalized versions of Sagarin's ELO_CHESS and PREDICTOR and KenPom's Offensive and Defensive efficiency numbers. It was fun to set up the spreadsheet to make it all happen, but at the end of the day, the numbers were meaningless. But they did spark a lot of conversation.

Perhaps taking the most important parts of each system and combining them into one and figure out a scheme that accurately weighs in each factor from MOV, injuries, time travel, bad calls late in game, home/away/neutral, efficiency, SOS, games vs top 25, 50, 100, total wins, etc. BPI is the closest system to this but lacks the human factor of time travel, should've won/lost, etc. I believe these factors can be consolidated but determining the weight of each factor would be the issue. Should MOV have a higher weight than time travel? Should the factor of getting robbed by the refs in the game weigh more than SOS? If anything, it's probably case by case basis and would be to time consuming even for people who crunch the numbers for a living.

 

[Monte Carlo Wildcat]

I don't live or die by the numbers. I still think at the end of the day the human polls still do a better job overall. They take into account intangible things, like injuries, travel, time zones, start times, and other things. (For instance AZ traveling across two times zones, starting early and playing in a very hostile UM arena isn't the same thing as SU traveling to NYC and playing against St Johns, even though most will almost certainly be weighted simply as "road" games.)

I like Sagarin's stuff, and I'm most familiar with it, but I also like the BPI a lot. But I'm not married to either one. The biggest point I'm trying to make though, is that it is important to know what you're looking at when checking out the computer rankings, and for the most part, I think most people don't. Most fans know there is an RPI and have a vague idea what it is, but only a fraction of them seem to know what exactly it is and what it is intended to measure. Similar idea to Sagarin, KenPom, Massey, etc. They get thrown around a lot but most people that talk about them don't really seem to know what they are.

Much agreed Trolly. I'll be honest, I'm one of those people who doesnt fully understand the logic and reasoning behind the math equations without having to look it up. I've always relyed on the intangibles as you mentioned. I agree that BPI does a great job. I referred back to BPI often last year and it seemed to be fairly accurate, for the most part.

Recently, I've been looking at Massey quite a bit because I believe some of his stuff is pretty accurate. Thus far, Massey has been pretty good at determining spreads from what I've seen. Once again, I don't go too much into the math but I like to look at the big stuff like power rankings. Would you say that sagarin does the best job at determining overall power rankings? If so what other aspects of sargarin do you like?

 

[TrollyMcTroller]

Recently, I've been looking at Massey quite a bit because I believe some of his stuff is pretty accurate. Thus far, Massey has been pretty good at determining spreads from what I've seen. Once again, I don't go too much into the math but I like to look at the big stuff like power rankings. Would you say that sagarin does the best job at determining overall power rankings? If so what other aspects of sargarin do you like?

I've heard a lot of good stuff about Massey, but I just haven't done much research to find out how they arrive at their numbers. I'm sure it's reasonably sound, but I just don't know about it enough to make an informed comment.

To give you a little background btw, I'm a poker player, and I'm all about the process, not the results. Poker is a great example of making all the right decisions, but not necessarily seeing all of the right outcomes, and vice versa. I see the computer rankings in a similar fashion... I like the methodology of Sagarin the most (based on what I know about it) even if the results don't always line up with my expectations. I think a lot of times, KenPom yields results that are more in line with what people would expect, but I just don't agree with the methodology. (at least for using it as power rankings) KenPom is just measuring offensive and defensive efficiency, and I think it does a great job of that, and I think that those efficiency ratings can be a good indicator of team performance, but I don't think I think the relationship is backwards from what the rankings (when used as power rankings) implies... specifically that efficiency makes for a good team. I think there is a lot of overlap, and I think that most really good teams are efficient with the ball, but I think teams are efficient because they are good, not that they are good because they are efficient.

In other words, if you're wanting to track lightening strikes, counting thunder claps may be a good indicator, but I'd rather just count the lightening strikes specifically, instead of using the indirect measurement, which is what KenPom is doing.

I haven't had the time the last few seasons to actually track any data, but a few years ago, I used Sagarin for college football wagering and made a small fortune. I would throw out bets where I knew something not stat based that would affect the spread/outcome (hurt players, rivalry games, Dave Wannstedt was coaching) but other than that is was purely based on Sagarin numbers. I don't have my spreadsheet in front of me, (that computer is unfortunately in a heap in the corner at the moment) but my success for the season was in the 70% range.

Hopefully before next season starts, I'll be able to get my stat machine going again and have some time to start betting again.

 

[Monte Carlo Wildcat]

I like your thinking bro. I'll start looking at sagarin and his methodology a little harder with the understanding that there is no one true poll/equation when evaluating teams and players. I like to base a lot of my decisions based on odds. I am firm believer in minimizing risk and increasing my chances of success. Just this weekend I was in Vegas and played the don't pass line on craps haha. Not too many people were happy with me. If I was efficient at counting cards then I would have played more blackjack but I've never tried that. I just want the best odds possible. It sounds like you play your 'best odds' as well. Correct? I assume you have to being a poker player and all. And it sounds like ken Pom relies on correlations opposed to actual direct data. I think all these sites can provide good info; all depends on wha you're looking for.

 

[TrollyMcTroller]

Yeah, I try to minimize risk as much as possible when betting, but most people consider me to be a "wide open" poker player, but that's a different game entirely since you're playing against other players, and not the house. I usually don't even play any table games in Vegas unless the whole crew wants to go hit a table for a while then I'll tag along, but it's a rarity.

As far as the stat sites, I like 'em all, it's just a matter of figuring out what they're trying to tell you. I could make a site that ranks teams based on # of timeouts used, and offensive rebound efficiency, and some chucklehead would inevitably point at it as proof that his team was the best. But all of those ratings are good tools, but like any tool it's knowing how to use it properly.

FWIW, I think there is a pretty strong correlation between off and def efficiency and overall team quality, which is why KenPom usually looks pretty reasonable if you do use it as a straight power ranking, but it can be a slippery slope if you're not paying attention.

I once read an article where some genius with a research grant noticed a strong correlation between children with behavior problems and children who get spanked, and the conclusion they came up with was spanking caused behavior problems. No it wasn't on The Onion, and No, I'm not making that up. Moral of the story is geniuses with research grants don't always know the difference between correlation and causation.

 

[rmilia1]

A site I really enjoy is teamrankings.com. They have full on projections, probability matrix, bracketology based on likely outcome etc etc. Its a really cool site if you havent been to it yet you should check it out. They dont have "rankings" per se but you can extraploate based off of their projections. Have conference projections, individual team projections, tourney probabilities. The whole shebang.

 

[TrollyMcTroller]

rmilia1, thanks for the heads up. I'll give that site a look.

And your post reminded me of something that I need to address.. (though this isn't aimed at you or your post)

Bracketology...

Time for my yearly bracketology rant. To the 900 sites out there dedicated to putting up brackets in December, and the inevitable forum posts that coincide... do me a favor. Either PROJECT something or STFU. Nobody cares what would happen in some shitty hypothetical 68 team torunament in December (or January, or February, or even in mid March) Any stupid shitheel can figure out who is ahead in conference standings and then plug the remaining slots with the top available RPI teams. I'm not even really impressed if you go to the trouble of mimicking the NCAA bracketing guidelines for your own December imaginary tournament and don't allow teams to play on their home court, and avoid match ups from earlier in the season, and follow the S-Curve, and all of that fun stuff, because no matter how realistic you make it, it still isn't real. Fucking quit it. There's bound to be some stupid fantasy league that you can join now that your shitty NFL fantasy team didn't make the playoffs for the 4th straight year, so go occupy yourself with that inane bullshit and you can commiserate with others who like to play pretend.

Now, on the other hand, if you're going to the trouble to do some actual goddamn analysis, and prognostication, then more power to you. For those that take the trouble to look at schedules and see what matchups are coming and looking at likely outcomes and how that will effect a team's RPI and their seeding, and whatnot, then I applaud you and your effort.

But to all of the people inventing tournaments so they can look smart without ever actually doing any real analysis at all... knock it the fuck off you posers.

Thank you.
/end rant.

 

[rmilia1]

rmilia1, thanks for the heads up. I'll give that site a look.

And your post reminded me of something that I need to address.. (though this isn't aimed at you or your post)

Bracketology...

Time for my yearly bracketology rant. To the 900 sites out there dedicated to putting up brackets in December, and the inevitable forum posts that coincide... do me a favor. Either PROJECT something or STFU. Nobody cares what would happen in some shitty hypothetical 68 team torunament in December (or January, or February, or even in mid March) Any stupid shitheel can figure out who is ahead in conference standings and then plug the remaining slots with the top available RPI teams. I'm not even really impressed if you go to the trouble of mimicking the NCAA bracketing guidelines for your own December imaginary tournament and don't allow teams to play on their home court, and avoid match ups from earlier in the season, and follow the S-Curve, and all of that fun stuff, because no matter how realistic you make it, it still isn't real. Fucking quit it. There's bound to be some stupid fantasy league that you can join now that your shitty NFL fantasy team didn't make the playoffs for the 4th straight year, so go occupy yourself with that inane bullshit and you can commiserate with others who like to play pretend.

Now, on the other hand, if you're going to the trouble to do some actual goddamn analysis, and prognostication, then more power to you. For those that take the trouble to look at schedules and see what matchups are coming and looking at likely outcomes and how that will effect a team's RPI and their seeding, and whatnot, then I applaud you and your effort.

But to all of the people inventing tournaments so they can look smart without ever actually doing any real analysis at all... knock it the fuck off you posers.

Thank you.
/end rant.

Word Trolly, thats why I like the teamrankings site. They use projections for their bracketology

 

[Monte Carlo Wildcat]

A site I really enjoy is teamrankings.com. They have full on projections, probability matrix, bracketology based on likely outcome etc etc. Its a really cool site if you havent been to it yet you should check it out. They dont have "rankings" per se but you can extraploate based off of their projections. Have conference projections, individual team projections, tourney probabilities. The whole shebang.

Teamrankings is a great site. I refer to that website to see how players and teams are doing nationally. It's a great tool to compare team match ups and see where their strengths and weaknesses.

 

[Monte Carlo Wildcat]

Have you guys ever checked out hoop-math? This is another site I use for reference. I find it very useful in scouting a team and their players. They break down a players offense and allow you to see where a players offense is most likely to come from. I've used it several times and it's been fairly accurate in determining where an opposing players offense will come from, i.e. An outside shooter versus at or around the rim. Anyways, if you haven't checked it out you should; you may find some useful stuff on there.

http://hoop-math.com

 

[TrollyMcTroller]

Thanks for the link MCW, I'll give it a look too.

 

[H2S]

great thread. good stuff, guise.

 

[Great Dayne]

Bracketology - NCAA College Basketball Brackets and Predictions - ESPN

Well Bracketology for Joe (68/68 last season) has Wisconsin and OSU as 1 seeds right now. I'll use this from now on but open for discussion to determine who the #1 overall seed should be. I believe it's correct for now.

 

[CatsTopPac]

Bracketology - NCAA College Basketball Brackets and Predictions - ESPN

Well Bracketology for Joe (68/68 last season) has Wisconsin and OSU as 1 seeds right now. I'll use this from now on but open for discussion to determine who the #1 overall seed should be. I believe it's correct for now.

The only reason why Joey Brackets is ever even close is because he only counts the last one he does right before Selection Sunday. Other than that, he's just picking his brackets during the season and changing them as it goes along, like everyone else.

The BPI is tailored to be the closest to the specifications that the committee uses when deciding on Selection Sunday, and it takes more accurate statistics into account for their index.

BPI -The College Basketball Power Index explained - ESPN

 

[john01992]

The only reason why Joey Brackets is ever even close is because he only counts the last one he does right before Selection Sunday. Other than that, he's just picking his brackets during the season and changing them as it goes along, like everyone else.

The BPI is tailored to be the closest to the specifications that the committee uses when deciding on Selection Sunday, and it takes more accurate statistics into account for their index.

BPI -The College Basketball Power Index explained - ESPN

i dont trust any espn developed ranking system