Thursday, October 22, 2015

Can We Build Better Machine Learning Models Based on Philip Tetlock's Good Judgement Project?

Thanks to the Good Judgment Project, the forecasting tournament has been solidified as one of the best modes of prediction, says Matheny, the Iarpa director. "Tetlock’s research really did help to inform how we should run forecasting tournaments," he says. "Not just that we should run them, but that there’s best practices in how to run them, and especially in picking questions that are neither too easy nor too hard."

Though Iarpa’s funding has ended, the team has formed a for-profit entity, Good Judgment Incorporated, which is recruiting members for a public tournament to begin later this fall. Corporate, nonprofit, government, and media clients can sponsor forecasting "challenges" on the public site, and the company will offer custom forecasts and training. It is also studying the potential of machine-human hybrids — like having IBM’s Deep Blue collaborate with Garry Kasparov in chess — that could prove more accurate than either one alone.

Now that we know some limitations, and strengths, of forecasters, Tetlock wants to focus on asking the right questions. He hopes to create what Kahneman has called "adversarial collaboration tournaments" — for instance, bringing together two politically opposed groups to discuss the Iran nuclear deal. One group thinks it’s great, one group thinks it’s terrible, and each must generate 10 questions that everyone will answer.

The idea is that each side will generate questions with answers that favor their position, and that, with everyone forced to consider all questions, a greater level of understanding will emerge. Maybe, in time, this will become the new norm for punditry, public debate, and policymaking.

The ultimate goals? Intellectual honesty. Better predictions. And, says Tetlock, "I hope we can avoid mistakes of the Iraq-war magnitude."


- More Here


No comments: