Testing an Advertised Success Rate

When an Advertised Rate Is Put to the Test

A service may advertise that a certain percentage of its transactions succeed. A sample of transactions can provide evidence about whether the service’s actual success rate is higher than that advertised benchmark. The direction of the claim matters: when the question is whether the rate exceeds the benchmark, unusually large sample success rates count as evidence against the null hypothesis.

This tutorial develops the upper-tail one-proportion z test for that question. The outcome must be binary, such as success or failure, and the data must come from a suitable sample or process. The test uses the advertised rate as the value specified by the null hypothesis, then measures how far the sample proportion lies above it relative to the variation expected under that null.

Set Up the Test Before Calculating

First define what counts as a success and identify the population of outcomes to which the claim applies. For example, “success” might mean that a randomly selected service request is completed within the advertised time. The population proportion \(p\) is then the proportion of all eligible requests meeting that definition. A different population or a different definition of success describes a different claim.

The hypotheses for the question “Does the true rate exceed the advertised rate?” are \(H_0:p=p_0\) and \(H_a:p>p_0\). The null uses equality to specify the reference model for the test; the alternative represents the direction of evidence being investigated. The alternative is not chosen after seeing whether the sample proportion happens to be high or low.

Let \(n\) be the number of sampled outcomes and \(x\) the number classified as successes. The sample proportion is \(\hat{p}=x/n\). Under the null model, the expected success count is \(np_0\), and the expected failure count is \(n(1-p_0)\). These expected counts, not the observed counts, are used to check the normal approximation condition for this z test.

The conditions address different concerns. A random sample helps make the observed outcomes informative about the target population. Independence means that one sampled outcome does not determine or strongly affect another; repeated transactions from the same customer, for instance, may require special care. The expected-count check supports using a normal approximation to the sampling distribution under the null. A large count alone does not repair a biased sample or dependence among observations.

The Statistic, Its Null Standard Error, and the P-Value

Under \(H_0\), the success count \(X\) has a binomial model with \(n\) trials and success probability \(p_0\), provided the outcomes are independent and share the same success probability. Thus \(E(X)=np_0\) and \(\operatorname{Var}(X)=np_0(1-p_0)\). Since \(\hat{p}=X/n\), dividing the count by \(n\) gives the null mean and standard deviation of the sample proportion.

When the expected success and failure counts are sufficiently large, the sampling distribution of \(\hat{p}\) under the null is approximated by a normal distribution with mean \(p_0\) and standard error \(SE_0\). Standardizing the observed sample proportion by subtracting its null mean and dividing by its null standard error gives the z statistic:

For an upper-tail test, the p-value is the probability, assuming \(H_0\), of obtaining a z statistic at least as large as the observed value. In symbols, it is \(P(Z\geq z_{\text{obs}})\), where \(Z\) follows the standard normal distribution under the approximation. A small p-value means that a sample result this far above the benchmark would be unusual if the true rate were \(p_0\). It is not the probability that the null hypothesis is true.

Choose a significance level \(\alpha\) before interpreting the result. If the p-value is at most \(\alpha\), reject \(H_0\); the data provide statistically significant evidence in the direction of \(H_a\). If the p-value exceeds \(\alpha\), fail to reject \(H_0\); the sample does not provide sufficient evidence for the stated alternative at that level. Neither decision proves the true rate equals the benchmark or establishes that the service is performing well or poorly in every practical sense.

Putting the Result in Context

A numerical decision is only part of a useful test conclusion. State the direction of the alternative, the population or service being discussed, and the strength of evidence indicated by the decision. For a small p-value, say there is evidence that the true success rate exceeds the advertised benchmark. For a larger p-value, say the sample did not provide sufficient evidence of an increase at the chosen significance level.

The p-value measures compatibility between the data and the null model in the direction specified by the alternative. It does not measure how important the difference would be to customers, and it does not give the probability that the advertised benchmark is correct. A very small departure can produce a small p-value with a sufficiently large sample; statistical significance alone does not establish practical importance.

A Common Mistake: Reversing the Tail

A frequent error is to see that the sample proportion is above the advertised benchmark and then report a two-sided p-value, or to use a lower-tail probability. The tail is determined by the alternative hypothesis, not selected after inspecting the sample. For \(H_a:p>p_0\), large positive z statistics are the evidence of interest, so the p-value is the area to the right of \(z_{\text{obs}}\).

Another mistake is to treat a non-significant result as proof that the service meets the advertised rate. Failing to reject \(H_0\) means only that the data did not provide sufficient evidence for \(p>p_0\) at the selected significance level. It does not confirm equality, and it does not rule out a higher rate.