A couple years ago, Google wrote a blog post about how their internal prediction markets were working. It was an inspiring picture and one that got many people excited about using prediction markets. Now that we've been hosting prediction markets for several years, we have quite a bit of our own data. Looking at well over 3 million predictions and thousands of questions across thousands of sites, we can easily say Google's impressive results weren't a fluke.
The most popular type of question users create determines the probability of an event happening: Will sales of X product beat the forecasted estimate? What is the probability of Y risk occurring? Will the project meet milestone Z on time?
Despite the media's tendency to judge prediction markets based on the results of a single question, you can't just look at a single outcome of one prediction market question to determine how "accurate" your site is.
When Mrs. Burnette told us in 4th grade that flipping a coin had .50 probability of coming up heads, she didn't stop there. She made us measure it to prove that this was in fact true by measuring the relative frequency of heads coming up. For homework we had to flip the coin over and over and write down the outcome. Flipping nickels at home we got: heads tails heads heads tails heads tails tails tails tails heads heads tails. Using our very new skills with fractions we could then measure the probability by computing 6 heads/13 trials = 0.46, pretty close to 0.5.
Mrs. Burnette appeared to know what she was talking about.
So just like flipping a coin, if Inkling told you something has a 15% probability of coming true, you can't just look at one outcome (i.e. one coin flip). You need to look at multiple scenarios where Inkling said something would happen 15% of the time. If those things actually come true, 15% of the time, Inkling is doing well.
Below is a graph we plotted matching final prediction values to probabilities. We counted the number of questions that predicted an event would occur 5% of the time, and saw how many of those occurred: almost 5% of them. We counted the number of questions that predicted an event would occur 15% of the time, and sure enough 15% of them ended up occurring. And so on. Until we got the graph below:
The green line is what we'd look like if we were perfect: things predicted to happen 15% of the time happen 15% of the time, things predicted to happen 65% of the time happen 65% of the time, etc. Inkling is the black line hugging pretty close to perfect.
Another type of prediction market is one that predicts the numerical outcome of something: What will the population of New York City be in 2013? How many utility patent applications will be filed in the US in 2014?
In this case we'd like to see a plot of what is the value we predicted with what actually happened. We plotted hundreds of these questions in the graph below.
The green line is perfect again. We'd be perfect if Inkling said you’d sell 100 units of something, and you sold 100 units. If Inkling said you'd sell 1000 units, you sold 1000 units. The red line is a line of best fit through the data. Not too shabby.
There are significant misconceptions about what the results of a prediction market actually mean, especially in the media. Hopefully these graphs reinforce what a prediction market is revealing as this is the first step in using the new information as input to decision making.
