Tuesday, November 5, 2013

Measure for Mismeasure

There is also, in any sector, the issue of what the data really mean. One of my favorite (sad) stories concerns consumer credit ratings in Russia. A retail banker once told me that in making credit decisions in Russia, his analysts had faced a shortage of data and had assumed that a criminal record would be one easy way to weed out the bad eggs.
CommentsView/Create comment on this paragraphBut when the bank ran the numbers, they discovered that although about half of the people applying for low-end, high-interest-rate credit had a police record, this did not seem to reflect on payment outcomes. I suspect the reason was that the people with no police record were the middle 50%. At the bottom were 25% who had a police record for some kind of bad behavior. At the top were 25% who had a police record for exceptionally good behavior, like refusing to pay bribes or to overlook corruption.
CommentsView/Create comment on this paragraphThe moral of this story is that numbers do not tell us everything. Do we end up focusing on price because we can measure it, to the exclusion of other things that matter? This phenomenon has rendered the airline business largely unprofitable – and customers increasingly unhappy. Quality used to be included in the price of a ticket; now, with surcharges for seat selection, meals, and the like, customers are charged for it separately.
CommentsView/Create comment on this paragraphIn short, the paradox of choice (as elegantly described in an eponymous book by Barry Schwartz) is that too many choices confuse us and raise the possibility of regret; how can we be sure that we are getting just what we want, and that something else would not be better?

Read more at http://www.project-syndicate.org/commentary/on-making-sense-of-ratings-and-rankings-by-esther-dyson#4kQxsKQmRaxJiImt.99
There is also, in any sector, the issue of what the data really mean. One of my favorite (sad) stories concerns consumer credit ratings in Russia. A retail banker once told me that in making credit decisions in Russia, his analysts had faced a shortage of data and had assumed that a criminal record would be one easy way to weed out the bad eggs.

But when the bank ran the numbers, they discovered that although about half of the people applying for low-end, high-interest-rate credit had a police record, this did not seem to reflect on payment outcomes. I suspect the reason was that the people with no police record were the middle 50%. At the bottom were 25% who had a police record for some kind of bad behavior. At the top were 25% who had a police record for exceptionally good behavior, like refusing to pay bribes or to overlook corruption.


The moral of this story is that numbers do not tell us everything. Do we end up focusing on price because we can measure it, to the exclusion of other things that matter? This phenomenon has rendered the airline business largely unprofitable – and customers increasingly unhappy. Quality used to be included in the price of a ticket; now, with surcharges for seat selection, meals, and the like, customers are charged for it separately.


In short, the paradox of choice (as elegantly described in an eponymous book by Barry Schwartz) is that too many choices confuse us and raise the possibility of regret; how can we be sure that we are getting just what we want, and that something else would not be better?

There is also, in any sector, the issue of what the data really mean. One of my favorite (sad) stories concerns consumer credit ratings in Russia. A retail banker once told me that in making credit decisions in Russia, his analysts had faced a shortage of data and had assumed that a criminal record would be one easy way to weed out the bad eggs.
CommentsView/Create comment on this paragraphBut when the bank ran the numbers, they discovered that although about half of the people applying for low-end, high-interest-rate credit had a police record, this did not seem to reflect on payment outcomes. I suspect the reason was that the people with no police record were the middle 50%. At the bottom were 25% who had a police record for some kind of bad behavior. At the top were 25% who had a police record for exceptionally good behavior, like refusing to pay bribes or to overlook corruption.
CommentsView/Create comment on this paragraphThe moral of this story is that numbers do not tell us everything. Do we end up focusing on price because we can measure it, to the exclusion of other things that matter? This phenomenon has rendered the airline business largely unprofitable – and customers increasingly unhappy. Quality used to be included in the price of a ticket; now, with surcharges for seat selection, meals, and the like, customers are charged for it separately.
CommentsView/Create comment on this paragraphIn short, the paradox of choice (as elegantly described in an eponymous book by Barry Schwartz) is that too many choices confuse us and raise the possibility of regret; how can we be sure that we are getting just what we want, and that something else would not be better?

Read more at http://www.project-syndicate.org/commentary/on-making-sense-of-ratings-and-rankings-by-esther-dyson#4kQxsKQmRaxJiImt.99
There is also, in any sector, the issue of what the data really mean. One of my favorite (sad) stories concerns consumer credit ratings in Russia. A retail banker once told me that in making credit decisions in Russia, his analysts had faced a shortage of data and had assumed that a criminal record would be one easy way to weed out the bad eggs.
CommentsView/Create comment on this paragraphBut when the bank ran the numbers, they discovered that although about half of the people applying for low-end, high-interest-rate credit had a police record, this did not seem to reflect on payment outcomes. I suspect the reason was that the people with no police record were the middle 50%. At the bottom were 25% who had a police record for some kind of bad behavior. At the top were 25% who had a police record for exceptionally good behavior, like refusing to pay bribes or to overlook corruption.
CommentsView/Create comment on this paragraphThe moral of this story is that numbers do not tell us everything. Do we end up focusing on price because we can measure it, to the exclusion of other things that matter? This phenomenon has rendered the airline business largely unprofitable – and customers increasingly unhappy. Quality used to be included in the price of a ticket; now, with surcharges for seat selection, meals, and the like, customers are charged for it separately.
CommentsView/Create comment on this paragraphIn short, the paradox of choice (as elegantly described in an eponymous book by Barry Schwartz) is that too many choices confuse us and raise the possibility of regret; how can we be sure that we are getting just what we want, and that something else would not be better?

Read more at http://www.project-syndicate.org/commentary/on-making-sense-of-ratings-and-rankings-by-esther-dyson#4kQxsKQmRaxJiImt.99
There is also, in any sector, the issue of what the data really mean. One of my favorite (sad) stories concerns consumer credit ratings in Russia. A retail banker once told me that in making credit decisions in Russia, his analysts had faced a shortage of data and had assumed that a criminal record would be one easy way to weed out the bad eggs.
CommentsView/Create comment on this paragraphBut when the bank ran the numbers, they discovered that although about half of the people applying for low-end, high-interest-rate credit had a police record, this did not seem to reflect on payment outcomes. I suspect the reason was that the people with no police record were the middle 50%. At the bottom were 25% who had a police record for some kind of bad behavior. At the top were 25% who had a police record for exceptionally good behavior, like refusing to pay bribes or to overlook corruption.
CommentsView/Create comment on this paragraphThe moral of this story is that numbers do not tell us everything. Do we end up focusing on price because we can measure it, to the exclusion of other things that matter? This phenomenon has rendered the airline business largely unprofitable – and customers increasingly unhappy. Quality used to be included in the price of a ticket; now, with surcharges for seat selection, meals, and the like, customers are charged for it separately.
CommentsView/Create comment on this paragraphIn short, the paradox of choice (as elegantly described in an eponymous book by Barry Schwartz) is that too many choices confuse us and raise the possibility of regret; how can we be sure that we are getting just what we want, and that something else would not be better?

Read more at http://www.project-syndicate.org/commentary/on-making-sense-of-ratings-and-rankings-by-esther-dyson#4kQxsKQmRaxJiImt.99

- Esther Dyson

No comments: