Why uncertainty is important to make better decisions.

January 22, 2022

From the archive. I wrote this in 2022 for an earlier version of this site. It is kept as it was then, apart from the layout and a few links.

Summary

As a data professional you will interact with business stakeholders on a regular basis. Usually they have little time and simply want a brief summary of your analysis. Consequently, important information is sometimes left out.

In this post I will demonstrate why incomplete analysis is a bad idea, and how Bayesian thinking helps to make better informed decisions.

Why learn Bayesian Inference?

As a former database marketer, I evaluated hundreds of marketing campaigns. A common setup of these campaigns is:

90% of the audience receives an email with a discount coupon (target)
10% of the audience receives the email without the coupon (control)

The control group is kept small to prevent turnover loss in case the coupon works.

Business stakeholders are interested how much better the target group performed in comparison to the control group. Classical hypothesis tests cannot answer this question, but Bayesian Inference can, as explained in the first section of this VWO white paper.

Although a useful technique, it can be intimidating to learn since most books and blogs rely heavily on theory and math equations. Therefore I will use a practical example with minimal theory, and focus on the thought process behind Bayesian Inference. Once you get the idea, it will be easier to understand the theory and math.

The setting.

Let’s imagine you are asked to analyze the following marketing campaign.

	n	Converted	Conversion rate
Target	900	360	40%
Control	100	30	30%

Since most business stakeholders prefer brevity and simplicity, it makes sense to communicate a 10% uplift without any additional information. But what about the following campaign?

	n	Converted	Conversion rate
Target	90	36	40%
Control	10	3	30%

Would you communicate the results in the same way?

If you do, your business stakeholders will perceive both campaigns as equally successful. They will also expect similar future performance when these campaigns are repeated. This doesn’t feel right.

Since the result of the first campaign is based on more data, we should be more confident in that result. But how much more confident in comparison to the second campaign? What if we had even more data? And how are we going to communicate the level of confidence to our stakeholders? To answer these questions we need a way to formally describe (un)certainty, based on the data.

Look inside the coin.

First, we need to change our perspective. Instead of thinking about n as the number of recipients, let’s see them as the number of coin tosses. And we will think about Converted as the number of heads. This means the number of heads is directly influenced by the bias of the coin. For example, a fair coin has a bias of 50%, which returns 50 heads in 100 tosses, on average.

From the book Bayesian Methods for Hackers we learn that the bias of a coin can be seen as the true underlying conversion rate, and cannot be interpreted directly.

The true frequency can be interpreted as the probability of an event occurring, and this does not necessarily equal the observed frequency. For example, the true frequency of rolling a 1 on a six-sided die is 1/6, but if we roll the die six times we may not see a 1 show up at all (the observed frequency!). (…)

Unfortunately, noise and complexities hide the true frequency from us and we must infer it from observed data.

Let’s apply the learnings from this book for the example of 100 coin tosses and 30 observed heads. Instead of taking the observed conversion rate of 30% for granted, we will go beyond the standard analysis and try to find the true conversion rate.

What we need to do is figure out is if it is possible for other conversion rates to have caused the same observed result. For example, is it possible that a conversion rate of 29% produces 30 heads in 100 tosses? What about a conversion rate of 25%? Or 20%?

We can answer these questions via simulation.

Simulation.

In Statistics, we use probability distributions to describe real world processes. Each distribution has certain characteristics which make them useful for specific situations. The data from a coin toss (or a marketing campaign, or any experiment with a yes/no outcome and multiple trials) can be modeled with the Binomial distribution, which has 2 parameters:

n: number of independent experiments
p: probability of success

In the first simulation we will simulate the marketing campaign with 100 tosses and 30 heads for different values of the probability of success (p).

from collections import defaultdict
from itertools import product
from scipy.stats import bernoulli

import numpy as np

np.random.seed(123)

simulation_dictionary = defaultdict(list)
conversion_rates = np.arange(0.1, 0.51, 0.01)
n_tosses = 100
target = 30
n_simulations = 500

for i in range(n_simulations):
    for conversion_rate in conversion_rates:
        number_of_conversions = np.sum(bernoulli.rvs(p=conversion_rate, size=n_tosses))
        if number_of_conversions == target:
            simulation_dictionary[conversion_rate].append(number_of_conversions)

{round(key, 2): len(value) / n_simulations for key, value in sorted(simulation_dictionary.items(), key=lambda x: x[0])}

There is a lot going on in this code snippet so let’s break it down step by step.

Create an array with conversion rates that could have resulted in 30 heads in 100 tosses ([0.1, 0.11, .., 0.49, 0.5])
Simulate the experiment 500 times for each conversion rate
For each round, calculate and sum the number of heads
If the sum equals 30, store it together with the conversion rate

The last line of code transforms and sorts the list of target values to a likelihood, which is the number of times a conversion rate hit the target, divided by the number of simulations (500).

In case this explanation goes too fast, in my experience it helps to run the code line for line in a Jupyter notebook. The simulation dictionary looks as follows.

{
  0.18: 0.002,
  0.2: 0.006,
  0.21: 0.008,
  0.22: 0.022,
  0.23: 0.026,
  0.24: 0.036,
  0.25: 0.05,
  0.26: 0.05,
  0.27: 0.062,
  0.28: 0.09,
  0.29: 0.082,
  0.3: 0.1,
  0.31: 0.128,
  0.32: 0.074,
  0.33: 0.068,
  0.34: 0.052,
  0.35: 0.054,
  0.36: 0.05,
  0.37: 0.024,
  0.38: 0.012,
  0.39: 0.012,
  0.4: 0.014,
  0.41: 0.014,
  0.42: 0.002,
  0.43: 0.002,
  0.46: 0.002,
  0.47: 0.002
}

There are 2 interesting things we can learn from this dictionary.

The observed conversion rate of 30% does not have the highest likelihood
Although we checked for conversion rates between 10% and 50%, not every conversion rate got a hit (<18%, >47%, and some conversion rates in between)

In the next simulation we will learn why the conversion rate of 30% did not have the highest likelihood, and examine the relationship with n tosses. Before moving on let’s visually examine the simulation dictionary to better understand it.

A colored bar plot of the conversion rates on the x-axis and the likelihood on the y-axis.

More data.

What would happen if we had 10 times more tosses and heads, n=1000, p=300? The conversion rate would be the same, but what about the uncertainty? When we run the simulation again with these numbers we get the following dictionary and visualization.

{
  0.27: 0.002,
  0.28: 0.008,
  0.29: 0.016,
  0.3:  0.022,
  0.31: 0.016,
  0.32: 0.01,
  0.33: 0.004
}

A colored bar plot of the conversion rates on the x-axis and the likelihood on the y-axis.

There are fewer possible underlying conversion rates, and the likelihood of a conversion rate of 30% has increased. If you divide the likelihood of a conversion rate of 30% by the sum of all the likelihoods of the result dictionary, the probability increases from 9.6% to 28.2%. Hence, there is less uncertainty with more data and you can be more confident that the observed conversion rate (30%) is close to the true unobserved conversion rate.

Back to business.

In the introduction I mentioned that leaving important information out of an analysis is a bad idea. Uncertainty is that important information. By including it into your analysis and explaining the thought process behind it, your business stakeholders will be able to make better decisions.

A great way to start is to include distributions in your analysis instead of using point estimates. They contain more information, and tell the complete story. Your business stakeholders might be hesitant at first, but when you clearly explain the benefit for them, I am confident they will embrace these insights quickly.

Formal Bayesian Inference.

As promised I will not go into technical mathematical details and theory. However, I do want to point out that we covered some serious ground in this post.

The bernoulli.rvs() function helped us calculating the likelihood, which is the first important part of the right-hand side of Bayes’ Theorem. Next, by defining a range of possible conversion rates before doing any calculations, we actually defined a prior. Based on our experience we decided to leave out any conversion rate <10% and >50%.

We are only missing the denominator but in practice this part is often bypassed due to the complexity of the calculations. Instead, by combining the likelihood and prior in a very smart way you get a result that is said to be proportional to the posterior, which is the left-hand side of the formula.