email this
print this
reprint or license thisSAVE AND SHARE
 |
|
|
|
|
|
|
 |
Book Chat: How stats freaks are changing everything
Ian Ayres discusses 'Super Crunchers,' which shows the very calculated way society is changing.
Peter Mucha (Inquirer): Welcome, Ian and visitors. Since people may be unfamiliar with your book, let's start this off as an interview. I found Super Crunchers to be a fascinating book, right up there with Freakonomics and Money Ball for people who the truths that can be in numbers. Take that, Mark Twain. Or whoever said: "There are three types of lies - lies, damn lies, and statistics."
Ian Ayres: Thanks. It really is amazing how many different fields are now relying on statistical prediction -- sports, business, education, finance, medicine... even government.
Peter Mucha: And well they probably should. As you point out, people are actually pretty bad at predicting. You have a test, asking people to answer questions with as wide a range as they want -- a range that expresses 90 percent confidence. I have to say I flunked!
Ian Ayres: Yes, humans tend to be damnably overconfident. We have trouble about thinking about the degree of precision in our predictions. But one of the coolest things about statistics is that the technique that produces a prediction simultaneously tells you how precise the prediction is.
Peter Mucha: Not let's not make eyes glaze over -- yet -- with talk of standard deviations. Your book isn't some jargon-heavy tome. It's full of examples we all can relate to. From the guy who predicted good years for wine based on weather to doctors being overconfident. Man, I'll never able to watch House again without thinking, "You idiot! Do a search! Ask a diagnostic program!" You give a great example of a young doctor who shocked colleagues by daring to Google for a diagnosis. And she got it right.
Ian Ayres: One of the most important changes in medicine in the last decade is that physicians have started to do patient specific research. In retrospect, it is kind of weird that doctors' offices never had libraries (lawyers' offices always do). A dirty little secret is that physicians almost never did research on individual patients. Now with the Internet and with an explosion of statistical studies they can and do. And yes something as crude as Google can sometimes help. But increasingly their are software backed by substantial databases that can help in both diagnosis and treatment decisions.
Peter Mucha: It almost seems like new maladies are regularly being discovered (or invented), and of course new treatments and studies abound. No way a doctor can keep up ... How close are we to having programs that are more reliable even the best physicians in diagnosing? You say that programs already predict Supreme Court cases more accurately than legal experts, without even knowing the law.
Ian Ayres: There will probably always be extraordinary doctors who will excel (well maybe not always). But most of us can't know whether our doctor is extraordinary or not. Within 2 to 5 years, we're going to see a breakthrough in improved diagnosis. From a statistical perspective, most of us have died in vain -- because the data from our deaths and/or hospitalization has been lost. But within 2 years, we'll have real time epidemiology where doctors will be able to see what happened to the last five patients who presented with your symptoms.
Peter Mucha: That's amazing. I'm thinking it's this kind of number crunching, aided by bigger and bigger databases, that helped determine that medical error is actually one of the leading causes of death in this country. That's not knock doctors and nurses ... they have enormously complicated jobs.
Ian Ayres: The bigger point is humans have particular trouble predicting as complexity increases. We have trouble putting the right weights on different causal factors. We think we need experts particularly when their are subtle underlying factors to suss out -- but perversely this is just the situation in which statistical models tend to do better. That's why a statistical model with just 6 factors did better than a bunch of legal experts at predicting how justice on the supreme court would vote.
Peter Mucha: There's also a certain arrogance, though, in which experts like to believe they're better, have some kind of better judgment or intuition. And sometimes they do, as Malcolm Gladwell's Blink explores. But other times it's an illusion, as with the those wine experts who refused to accept that weather was so predictive for the best years for Bordeaux.
Ian Ayres: Sometimes it's arrogance; but sometimes it's self-interest. In field after field, traditional experts who base their predictions on intuition and experience are losing out to Super Crunchers. It's not pleasant to lose your discretion. Loan officers at local banks used to have some real input into whether loans would issue, now they're closer to being glorified secretaries who enter the info and wait for the computer algorithm to kick out the answer. The same thing is happening at radio stations where the program manager is losing discretion. It's happening in baseball, where traditional scouts are losing out. It's even happening in education where scripted lesson plans tend to do a better job at teaching reading than giving classroom teacher's discretion to innovate.
Peter Mucha: How much promise is there for taking on major social problems, like poverty, drugs, crime and finally finding effective solutions?
Ian Ayres: A lot. Governments (both state government in U.S. and internationally) have started to do hundreds of randomized test to find out what social policies work and which don't. Mexico ran a very simple test about giving poor women money and food supplements if their kids go to school and if the family members made regular visits to the doctor. They at random chose 200 villages to give the subisides and at random chose 200 villages where the contingent subsidies would not be given. The results were clear and powerful. The subsidies not only led kids to stay in school, but within the space of just a couple of years the average kid with the subsidy was a centimeter taller. That's a huge indicator of improved health. The program is now being used in about 30 other countries (and may come soon to U.S.).
Peter Mucha: Often there's plenty of will and money -- if they're directed wisely. You give an example of an airline that found it was wasting marketing efforts on customers who actually lost them money. I'll bet plenty of government programs have been like that ... money down the drain .... or they even made problems worse.
Ian Ayres: Yep. When decision makers make bad predictions, serious money can be lost. BTW, this can also happen when Super Crunchers make mistakes. This past summer has shown that some the Quant (quantitative) hedge funds are susceptible to error. What we need is not to rely blindly on Super Crunching and secondly, we should have multiple modes of super crunching ... Firms should have statistical audits and multiple statisticians attach data from a multiplicity of approaches.
Ian Ayres: FYI, your readers might like to do some Super Crunching themselves. If they go to http://islandia.law.yale.edu/ayres/predictionTools.htm they can find about 30 prediction tools that will allow them to predict how long they'll live or their kid's height or their due date... all kinds of things including how long your marriage might last... just plug in some information and the tool with make a statistical prediction.
Peter Mucha: There are dangers there, you're careful to point out out. Before we wrap up, I wanted to also ask about sports. You write about how Super Crunching was able to detect point shaving in basketball. But what about strategy? Should baseball teams go back to having four starters pitch more innnings? Should football teams go for 2 all the time, especially since the practice would improve odds of succeeding not only then but in scoring touchdowns? Any thoughts along those lines?
Ian Ayres: I haven't crunched the numbers on those questions. But David Roemer has crunched lot of numbers about whether NFL teams should go for it on 4th downs. He has a decision rule which is a simple function of where the ball is and Roemer finds that teams are better off going for it 4th sometimes even when it is 4th and 5 yards.
Peter Mucha: Here's another interesting link you mentioned in the book. Lulu.com has program that predicts which titles have the best chance of being a best-seller. It's fun and interesting. Link: http://www.lulu.com/titlescorer/. Thanks, Ian. Good luck with the book. Hope your sales numbers are good!
Ian Ayres: Thanks. I love it when the questions are as interesting and informed as yours.
Jen: BTW, it was Disraeli who coined the "three types of lies" quote.
Peter Mucha: After I said "Twain," I Googled it and found out nobody's sure. Twain cited Disraeli, so it wasn't Twain. But others may have said it first.
Ian Ayres: Yes. And while there is much truth in the saying, I think it is less true with regard to randomized trials. You give half the people the subsidy and half you don't and then you just compare the two averages. You don't have to be a statistician to understand it. It's not "trust me" statistics. Anyway, thanks again for letting me do this.
Peter Mucha: Well, I do wonder if anyone's ever done a study of what percentage of studies turn out not to be true. Thanks again. I enjoyed it.
Ian Ayres: Thanks. It really is amazing how many different fields are now relying on statistical prediction -- sports, business, education, finance, medicine... even government.
Peter Mucha: And well they probably should. As you point out, people are actually pretty bad at predicting. You have a test, asking people to answer questions with as wide a range as they want -- a range that expresses 90 percent confidence. I have to say I flunked!
Ian Ayres: Yes, humans tend to be damnably overconfident. We have trouble about thinking about the degree of precision in our predictions. But one of the coolest things about statistics is that the technique that produces a prediction simultaneously tells you how precise the prediction is.
Peter Mucha: Not let's not make eyes glaze over -- yet -- with talk of standard deviations. Your book isn't some jargon-heavy tome. It's full of examples we all can relate to. From the guy who predicted good years for wine based on weather to doctors being overconfident. Man, I'll never able to watch House again without thinking, "You idiot! Do a search! Ask a diagnostic program!" You give a great example of a young doctor who shocked colleagues by daring to Google for a diagnosis. And she got it right.
Ian Ayres: One of the most important changes in medicine in the last decade is that physicians have started to do patient specific research. In retrospect, it is kind of weird that doctors' offices never had libraries (lawyers' offices always do). A dirty little secret is that physicians almost never did research on individual patients. Now with the Internet and with an explosion of statistical studies they can and do. And yes something as crude as Google can sometimes help. But increasingly their are software backed by substantial databases that can help in both diagnosis and treatment decisions.
Peter Mucha: It almost seems like new maladies are regularly being discovered (or invented), and of course new treatments and studies abound. No way a doctor can keep up ... How close are we to having programs that are more reliable even the best physicians in diagnosing? You say that programs already predict Supreme Court cases more accurately than legal experts, without even knowing the law.
Ian Ayres: There will probably always be extraordinary doctors who will excel (well maybe not always). But most of us can't know whether our doctor is extraordinary or not. Within 2 to 5 years, we're going to see a breakthrough in improved diagnosis. From a statistical perspective, most of us have died in vain -- because the data from our deaths and/or hospitalization has been lost. But within 2 years, we'll have real time epidemiology where doctors will be able to see what happened to the last five patients who presented with your symptoms.
Peter Mucha: That's amazing. I'm thinking it's this kind of number crunching, aided by bigger and bigger databases, that helped determine that medical error is actually one of the leading causes of death in this country. That's not knock doctors and nurses ... they have enormously complicated jobs.
Ian Ayres: The bigger point is humans have particular trouble predicting as complexity increases. We have trouble putting the right weights on different causal factors. We think we need experts particularly when their are subtle underlying factors to suss out -- but perversely this is just the situation in which statistical models tend to do better. That's why a statistical model with just 6 factors did better than a bunch of legal experts at predicting how justice on the supreme court would vote.
Peter Mucha: There's also a certain arrogance, though, in which experts like to believe they're better, have some kind of better judgment or intuition. And sometimes they do, as Malcolm Gladwell's Blink explores. But other times it's an illusion, as with the those wine experts who refused to accept that weather was so predictive for the best years for Bordeaux.
Ian Ayres: Sometimes it's arrogance; but sometimes it's self-interest. In field after field, traditional experts who base their predictions on intuition and experience are losing out to Super Crunchers. It's not pleasant to lose your discretion. Loan officers at local banks used to have some real input into whether loans would issue, now they're closer to being glorified secretaries who enter the info and wait for the computer algorithm to kick out the answer. The same thing is happening at radio stations where the program manager is losing discretion. It's happening in baseball, where traditional scouts are losing out. It's even happening in education where scripted lesson plans tend to do a better job at teaching reading than giving classroom teacher's discretion to innovate.
Peter Mucha: How much promise is there for taking on major social problems, like poverty, drugs, crime and finally finding effective solutions?
Ian Ayres: A lot. Governments (both state government in U.S. and internationally) have started to do hundreds of randomized test to find out what social policies work and which don't. Mexico ran a very simple test about giving poor women money and food supplements if their kids go to school and if the family members made regular visits to the doctor. They at random chose 200 villages to give the subisides and at random chose 200 villages where the contingent subsidies would not be given. The results were clear and powerful. The subsidies not only led kids to stay in school, but within the space of just a couple of years the average kid with the subsidy was a centimeter taller. That's a huge indicator of improved health. The program is now being used in about 30 other countries (and may come soon to U.S.).
Peter Mucha: Often there's plenty of will and money -- if they're directed wisely. You give an example of an airline that found it was wasting marketing efforts on customers who actually lost them money. I'll bet plenty of government programs have been like that ... money down the drain .... or they even made problems worse.
Ian Ayres: Yep. When decision makers make bad predictions, serious money can be lost. BTW, this can also happen when Super Crunchers make mistakes. This past summer has shown that some the Quant (quantitative) hedge funds are susceptible to error. What we need is not to rely blindly on Super Crunching and secondly, we should have multiple modes of super crunching ... Firms should have statistical audits and multiple statisticians attach data from a multiplicity of approaches.
Ian Ayres: FYI, your readers might like to do some Super Crunching themselves. If they go to http://islandia.law.yale.edu/ayres/predictionTools.htm they can find about 30 prediction tools that will allow them to predict how long they'll live or their kid's height or their due date... all kinds of things including how long your marriage might last... just plug in some information and the tool with make a statistical prediction.
Peter Mucha: There are dangers there, you're careful to point out out. Before we wrap up, I wanted to also ask about sports. You write about how Super Crunching was able to detect point shaving in basketball. But what about strategy? Should baseball teams go back to having four starters pitch more innnings? Should football teams go for 2 all the time, especially since the practice would improve odds of succeeding not only then but in scoring touchdowns? Any thoughts along those lines?
Ian Ayres: I haven't crunched the numbers on those questions. But David Roemer has crunched lot of numbers about whether NFL teams should go for it on 4th downs. He has a decision rule which is a simple function of where the ball is and Roemer finds that teams are better off going for it 4th sometimes even when it is 4th and 5 yards.
Peter Mucha: Here's another interesting link you mentioned in the book. Lulu.com has program that predicts which titles have the best chance of being a best-seller. It's fun and interesting. Link: http://www.lulu.com/titlescorer/. Thanks, Ian. Good luck with the book. Hope your sales numbers are good!
Ian Ayres: Thanks. I love it when the questions are as interesting and informed as yours.
Jen: BTW, it was Disraeli who coined the "three types of lies" quote.
Peter Mucha: After I said "Twain," I Googled it and found out nobody's sure. Twain cited Disraeli, so it wasn't Twain. But others may have said it first.
Ian Ayres: Yes. And while there is much truth in the saying, I think it is less true with regard to randomized trials. You give half the people the subsidy and half you don't and then you just compare the two averages. You don't have to be a statistician to understand it. It's not "trust me" statistics. Anyway, thanks again for letting me do this.
Peter Mucha: Well, I do wonder if anyone's ever done a study of what percentage of studies turn out not to be true. Thanks again. I enjoyed it.