Undefined Metrics and Zero Denominators

Why Zero Denominators Need Careful Reporting

A classification metric is a ratio of counts from a confusion matrix. If its denominator is zero, the ratio is undefined: there is no observed group over which to calculate the rate. This is different from a metric that equals zero, which means the denominator is positive and the numerator is zero. Confusing these cases can make a classifier appear to have measured performance that the data do not support.

This matters in small validation samples, rare-event settings, and any sample where a classifier assigns every case to one class. A responsible report names the metric, states why it is undefined, and gives the relevant counts rather than quietly substituting zero. The approach here is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Confusion-Matrix Counts and Metric Denominators

Suppose a binary classifier labels cases as positive or negative, and the observed outcome is also positive or negative. A true positive (TP) is an observed positive case predicted positive; a false negative (FN) is an observed positive predicted negative. A false positive (FP) is an observed negative predicted positive; a true negative (TN) is an observed negative predicted negative.

	Observed positive	Observed negative
Predicted positive	TP	FP
Predicted negative	FN	TN

The four common rates use different groups as their denominators. Precision and negative predictive value condition on the classifier's prediction. Sensitivity and specificity condition on the observed outcome.

Definition: For a binary classification confusion matrix, $$ \text{Precision}=\frac{TP}{TP+FP},\qquad \text{Sensitivity}=\frac{TP}{TP+FN}, $$ $$ \text{Specificity}=\frac{TN}{TN+FP},\qquad \text{Negative predictive value (NPV)}=\frac{TN}{TN+FN}. $$ A metric is undefined when its own denominator is zero. A zero numerator with a positive denominator gives a defined metric equal to zero.

The denominator tells you exactly which cases contribute to each rate:

Precision: all predicted-positive cases, $TP+FP$.
Sensitivity: all observed-positive cases, $TP+FN$.
Specificity: all observed-negative cases, $TN+FP$.
NPV: all predicted-negative cases, $TN+FN$.

Therefore, precision is undefined when there are no predicted-positive cases; sensitivity is undefined when there are no observed-positive cases; specificity is undefined when there are no observed-negative cases; and NPV is undefined when there are no predicted-negative cases. These conditions concern the sample being evaluated. A zero denominator does not establish how the classifier would perform in a different sample.

Why a Zero Denominator Means “Undefined”

Each rate can be read as a conditional proportion in the observed sample. For example, among the cases predicted positive, precision is the proportion that are actually positive. The group size is $TP+FP$, and the number of actual positives in that group is $TP$. Thus the rate is $TP/(TP+FP)$, provided the group exists.

More formally, when $TP+FP>0$, the empirical conditional proportion is the number of cases satisfying both “predicted positive” and “observed positive” divided by the number satisfying “predicted positive.” If there are no predicted-positive cases, both counts are zero and there is no observed conditional proportion to calculate. This is not evidence that precision is zero: zero precision would require at least one predicted-positive case and no true positives.

The same reasoning applies to the other metrics. For sensitivity, the reference group is observed-positive cases; for specificity it is observed-negative cases; and for NPV it is predicted-negative cases. The denominator identifies the group whose proportion is being summarized.

Worked Example: No Cases Predicted Positive

A hypothetical screening classifier is evaluated on 240 applications. Sixteen applicants have the observed outcome of interest, and the classifier predicts every applicant negative. The resulting counts are $TP=0$, $FN=16$, $FP=0$, and $TN=224$. Calculate all four metrics and identify any undefined result.

Method. First check each denominator, then calculate the ratio only when that denominator is positive. The common mistake to avoid is assigning precision a value of zero just because there are no true positives.

Precision: $TP+FP=0+0=0$. There are no predicted-positive applications, so precision is undefined; $0/0$ is not zero.

Sensitivity: $TP+FN=0+16=16$, so sensitivity is $0/16=0$. Of the 16 observed-positive applicants, none were identified.

Specificity: $TN+FP=224+0=224$, so specificity is $224/224=1$. Every observed-negative applicant was predicted negative.

NPV: $TN+FN=224+16=240$, so NPV is $224/240=0.9333$, to four decimal places.

Check. The observed-positive count is $TP+FN=16$; the observed-negative count is $TN+FP=224$; and $16+224=240$, the sample size. Precision is the only undefined metric. The classifier's perfect specificity does not compensate for its zero sensitivity; these measures summarize different observed groups.

Worked Cases: Separate the Four Denominators

A reliable calculation starts with the group named by the metric, not with the numerator alone. The next examples show why checking all four denominators separately matters.

Worked Example: No Observed Positive Cases

In a hypothetical audit of 180 transactions, none has the observed event. The classifier predicts nine transactions positive and 171 negative. Thus $TP=0$, $FN=0$, $FP=9$, and $TN=171$. Calculate all four metrics.

Method. Use observed class counts for sensitivity and specificity, and predicted class counts for precision and NPV. The common mistake to avoid is treating sensitivity as zero because its numerator is zero without checking its denominator.

Sensitivity: $TP+FN=0+0=0$. There are no observed-positive transactions, so sensitivity is undefined.

Specificity: $TN+FP=171+9=180$, so specificity is $171/180=0.9500$.

Precision: $TP+FP=0+9=9$, so precision is $0/9=0$. There are predicted positives, and none is an observed positive.

NPV: $TN+FN=171+0=171$, so NPV is $171/171=1$.

Check. The predicted-positive group has nine transactions and the predicted-negative group has 171; together they total 180. Sensitivity alone is undefined because the validation sample contains no observed positives. Precision, specificity, and NPV are defined.

Worked Example: No Predicted-Negative Cases

A hypothetical triage rule is tested on 40 records. It predicts 40 records positive; 32 are observed positive and eight are observed negative. The counts are $TP=32$, $FP=8$, $FN=0$, and $TN=0$. Calculate all four metrics.

Method. Evaluate each denominator independently, including specificity even though there are no true negatives. The common mistake to avoid is declaring both specificity and NPV undefined merely because $TN=0$.

Precision: $TP+FP=32+8=40$, so precision is $32/40=0.8000$.

Sensitivity: $TP+FN=32+0=32$, so sensitivity is $32/32=1$.

Specificity: $TN+FP=0+8=8$, so specificity is $0/8=0$. The denominator is positive: there are eight observed-negative records, all predicted positive.

NPV: $TN+FN=0+0=0$. There are no predicted-negative records, so NPV is undefined.

Check. There are 32 observed-positive and eight observed-negative records, for 40 total. Specificity is defined as zero, while NPV is undefined. A zero numerator does not by itself make a metric undefined; its denominator must also be checked.

Worked Example: Two Metrics Undefined in One Sample

A hypothetical validation sample contains 14 observed-positive claims and no observed-negative claims. A rule predicts every claim negative. Thus $TP=0$, $FN=14$, $FP=0$, and $TN=0$. Find the four metrics.

Method. Identify each denominator before dividing. The common mistake to avoid is supposing that every metric can be calculated whenever the total sample size is positive.

Precision: $TP+FP=0+0=0$, so precision is undefined because there are no predicted-positive claims.

Sensitivity: $TP+FN=0+14=14$, so sensitivity is $0/14=0$.

Specificity: $TN+FP=0+0=0$, so specificity is undefined because there are no observed-negative claims.

NPV: $TN+FN=0+14=14$, so NPV is $0/14=0$. All 14 predicted-negative claims are observed positive.

Check. The sample contains $14+0=14$ records. Precision and specificity are undefined for different reasons: there are no predicted positives and no observed negatives, respectively. Sensitivity and NPV are both defined and equal zero.

Reporting Undefined Metrics Responsibly

An undefined metric is a statement about the available validation sample and the relevant group, not a score of poor performance and not a guarantee about future cases. Report the underlying counts so readers can see what was and was not observed. For example: “Precision is undefined because the classifier made zero positive predictions ($TP=0$, $FP=0$).”

Do not confuse an undefined metric with a metric estimated from a small but nonzero denominator. If there is one predicted-positive case and it is a true positive, precision is $1/1=1$, so it is defined, although the evidence is limited. A zero denominator means the rate cannot be computed from this sample; a small denominator means it can be computed but may be sensitive to individual cases.

Software or reporting conventions may display an undefined value as “NA,” “NaN,” or another designated symbol. Some workflows may choose a numerical replacement for a specific downstream purpose, but that replacement is a separate convention, not the mathematical value of the ratio. State the convention and retain the original counts; do not silently convert an undefined result to zero.

Write the four counts.
Record TP, FP, FN, and TN from the evaluation sample, keeping predicted class and observed class distinct.

Match metric to denominator.
Use $TP+FP$ for precision, $TP+FN$ for sensitivity, $TN+FP$ for specificity, and $TN+FN$ for NPV.

Check before dividing.
If the denominator is zero, report the metric as undefined and state which group is absent. If it is positive, calculate the ratio, including when the numerator is zero.

Verify and communicate.
Check that $TP+FP+FN+TN$ equals the sample size and report the counts alongside any undefined metric.

Key takeaway: Zero numerator and zero denominator are not interchangeable. A zero metric requires a positive denominator; a zero denominator makes that metric undefined. Always check the denominator specific to the metric and explain the missing group in context.

Check Your Understanding

For each hypothetical confusion matrix, determine which metrics are undefined and calculate the defined metrics. Give rates to four decimal places where needed.

A review sample has $TP=11$, $FP=0$, $FN=4$, and $TN=85$. Calculate precision, sensitivity, specificity, and NPV. (Answer: Precision $=1$; sensitivity $=11/15=0.7333$; specificity $=1$; NPV $=85/89=0.9551$.)
A validation sample has $TP=0$, $FP=6$, $FN=0$, and $TN=94$. Which metric is undefined, and what are the other three values? (Answer: Sensitivity is undefined; precision $=0/6=0$; specificity $=94/100=0.9400$; NPV $=94/94=1$.)
A rule produces $TP=15$, $FP=0$, $FN=5$, and $TN=0$. Identify any undefined metric and calculate the rest. (Answer: Specificity is undefined; precision $=15/15=1$; sensitivity $=15/20=0.7500$; NPV $=0/5=0$.)
A classifier predicts every record negative in a sample with $TP=0$, $FP=0$, $FN=18$, and $TN=72$. Calculate all four metrics and explain the undefined result. (Answer: Precision is undefined because $TP+FP=0$; sensitivity $=0/18=0$; specificity $=72/72=1$; NPV $=72/90=0.8000$.)
A data extract contains no evaluated records, so $TP=FP=FN=TN=0$. What should be reported for the four metrics? (Answer: All four metrics are undefined because each denominator is zero; report that the extract contains no evaluated records rather than reporting four zero rates.)

What You'll Learn