The Confusion Matrix and ROC

Table of Contents

1. Business Use Case
2. Taking the Confusion Out of the Confusion Matrix
   • Terminology in Context
   • Measuring the Values
     • Example 1
3. Receiver Operating Characteristic (ROC)

Business Use Case

A company - Topo’s Tacos sells DIY taco kits on a monthly-based continuity program. Subscribers pay $20.00/month to receive taco kits complete with taco shells, packaged Mexican cheese, and a variety of meats or veggies. Topo’s Tacos has been cash flow positive since fiscal year 2021 through the first 2 months of 2022. However, in recent months, the company’s CFO has become quite concerned with the downward-trending trajectory of Topo’s Tacos’ revenue. He thinks it has something to do with customer retention. So, he tasks his sharpest analysts to build a binary classification model for predicting churn. A random sample of n=200 observations is drawn.

The analysts quickly begin to implement scientific decision making by setting-up the hypothesis testing framework as follows.

\[\begin{eqnarray} H_0 &=& \text{The customers will not churn = 0} \\ H_1 &=& \text{The customers will churn = 1} \end{eqnarray}\]

Taking the Confusion out of The Confusion Matrix

The confusion matrix may sound confusing by name alone, but it is simply a table of predicted vs. actual values as part of the holistic assessment of classification models.

Sweeping through the confusion matrix below in a clockwise fashion produces the following results.

When we have an observation that is predicted as a “NO” and the actual observation is also a “NO,” we have what is called a True Negative (TN):

 

n = 200	PREDICTED NO	PREDICTED YES
ACTUAL NO	TN
ACTUAL YES

Next, when we have an observation that is predicted as a “YES” and the actual observation is a “NO,” we have what is called a False Positive (FP). Think of it this way: we predicted a “YES,” but got it wrong.
 

n = 200	PREDICTED NO	PREDICTED YES
ACTUAL NO	TN	FP
ACTUAL YES		TP

A Predicted “YES” that yields an actual “YES,” produces a True Positive (TP). No explanation necessary on this front.

Conversely, a Predicted “NO” that yields an actual “YES,” produces a False Negative (FN). False expectations yet again…

And now for a quick recap of what each value on the confusion matrix represents.

Terminology in Context

• True Negative (TN): a prediction for no churn (“NO”) was made. No churn (“NO”) was actually observed.
  
• False Positice (FP): a prediction for churn (“YES”) was made. No churn (“NO”) was actually observed.
  
• True Positive (TP): a prediction for churn (“YES”) was made. Churn (“YES”) was actually observed.
  
• False Negative (FN): a prediction for no churn (“NO”) was made. Churn (“YES”) was actually observed.

Measuring the Values

Remember the sample size of n=200 observations from the Business Use Case way up top? Let’s take a smaller sample size of n=30 to illustrate the mechanics of calculating confusion matrix values based on different thresholds.

Receiver Operating Characteristic (ROC)

On a high level, the ROC plots the relationship between the True Positive Rate (TPR) on the y-axis vs. the False Positive Rate (FPR) on the x-axis. Thus far, we have only calculated the TPR and FPR over one single threshold of 0.4. To make this a sound and proper curve, additional (FPR, TPR) coordinates for more thresholds would have to be calculated and plotted, respectively.