Auditing a Claimed Product-Defect Distribution

Why Audit a Claimed Defect Distribution?

A manufacturer may specify not only how many products are expected to have a defect, but also what types of defects should occur. For example, its engineering model might say that defects are more likely to be surface flaws than electrical faults. A quality-control sample can be used to assess whether the observed mix of defect types is consistent with that specified distribution.

The question is about the whole categorical distribution, not just whether one defect type is unusually common. The chi-square goodness-of-fit test compares the observed counts with the counts expected under the manufacturer’s specified probabilities. Its category contributions then help describe which differences account most for the overall mismatch.

Definition: Suppose a fixed sample of \(n\) products is classified into \(k\) mutually exclusive and exhaustive defect categories. The manufacturer specifies probabilities \(p_1,\ldots,p_k\), with \(p_i>0\) for every included category and \(\sum_{i=1}^{k}p_i=1\). The null hypothesis is \(H_0\): the probabilities of the categories are \(p_1,\ldots,p_k\). The alternative is \(H_a\): at least one category probability differs from its specified value. Under \(H_0\), the expected count in category \(i\) is \(E_i=np_i\), and the Pearson statistic is \(\chi^2=\sum_{i=1}^{k}(O_i-E_i)^2/E_i\), where \(O_i\) is the observed count.

Set Up the Audit Before Calculating

The probability vector must describe categories that cover all possible outcomes being counted, with each product assigned to exactly one category. The sample size is fixed, and under the null model each product’s category is drawn independently with the same specified probabilities. These conditions give a multinomial model for the category counts. If products are clustered, repeatedly measured, or sampled in some other dependent way, the ordinary goodness-of-fit calculation may not be appropriate.

For the standard chi-square approximation, expected counts should be sufficiently large; a commonly used practical condition is that every \(E_i\) is at least 5. This is a condition on expected counts under \(H_0\), not on the observed counts. When it fails, the usual chi-square p-value may be unreliable; alternative methods for sparse counts may be needed.

When all included probabilities are positive and no probability has been estimated from the sample, the degrees of freedom are \(k-1\). If the model specifies a structural zero, that category is not an ordinary positive-probability category in this statistic: a sample observation there contradicts the model, while no observations there means it must be handled separately. For the chi-square approximation, the degrees of freedom are the number of positive-probability categories minus one, subject to the model’s structural-zero constraints.

Worked Example: Testing a Manufacturer’s Specified Defect Mix

Suppose a manufacturer claims that, among defective units, 40.000% have a surface flaw, 30.000% have a loose connection, 21.4968% have a calibration fault, and 8.5032% have a casing fault. In a hypothetical random sample of \(n=200\) defective units, inspectors record counts of 96, 46, 37, and 21 in those categories, respectively. Test the claim at significance level \(\alpha=0.05\).

Hypotheses and model. Under \(H_0\), the four probabilities equal the specified values; under \(H_a\), at least one differs. Assume the 200 sampled units are independent and each belongs to exactly one of the four categories. The probabilities are positive and sum to 1. Expected counts will all exceed 5, as the calculation below verifies.

Expected counts. Multiply each specified probability by 200:

$$ (E_1,E_2,E_3,E_4) =200(0.40000,0.30000,0.214968,0.085032) =(80,60,42.9936,17.0064). $$

The expected counts sum to 200, and even the smallest, 17.0064, exceeds 5. The approximation condition is satisfied.

Statistic. Substituting the observed and expected counts gives:

\begin{aligned} \chi^2 &=\frac{(96-80)^2}{80} +\frac{(46-60)^2}{60} +\frac{(37-42.9936)^2}{42.9936} +\frac{(21-17.0064)^2}{17.0064}\\ &=3.2000+3.2667+0.8356+0.9378\\ &\approx 8.240. \end{aligned}

There are \(k=4\) positive-probability categories, and no probabilities were estimated from these data, so \(df=k-1=3\). The right-tail p-value is \(P(\chi^2_3\geq 8.240)\approx 0.0413\). Since \(0.0413<0.05\), reject \(H_0\). At the 5% significance level, the sample provides evidence that the distribution of defect types differs from the manufacturer’s specified distribution. The test does not establish why the distribution differs.

Why the Degrees of Freedom Are \(k-1\)

The degrees of freedom reflect how many category-count discrepancies can vary independently once the total sample size is fixed. Under the null model, the expected counts add to the sample size:

\sum_{i=1}^{k}E_i =n\sum_{i=1}^{k}p_i=n.

The observed counts also add to \(n\), so their differences \(O_i-E_i\) must sum to zero. If the first \(k-1\) differences are known, the final one is forced to be their negative sum. Thus there are \(k-1\) independent discrepancies. With probabilities specified in advance, this gives \(df=k-1\) for the chi-square approximation. Estimating probabilities from the same data changes the degrees-of-freedom calculation; that is not the situation in this audit.

Under the null model and the stated approximation conditions, the Pearson statistic is approximately distributed as a chi-square random variable with these degrees of freedom. The p-value is the probability, assuming \(H_0\), of obtaining a statistic at least as large as the one observed. Because the statistic adds squared discrepancies, evidence against \(H_0\) is in the right tail.

Worked Example: Locating the Main Contributions

Use the audit above to identify which categories contribute most to the total statistic and what direction their observed counts differ from expectation.

The category contributions are the four terms in the calculation: 3.2000 for surface flaws, 3.2667 for loose connections, 0.8356 for calibration faults, and 0.9378 for casing faults. They sum to approximately 8.240. The loose-connection category has the largest contribution: its observed count of 46 is 14 below its expected count of 60. Surface flaws contribute almost as much, but in the opposite direction: 96 were observed compared with 80 expected, an excess of 16.

The contributions identify where the arithmetic mismatch is concentrated; they are not separate chi-square tests. In particular, the overall p-value of 0.0413 applies to the distribution as a whole. It does not mean that each category has independently been shown to differ significantly, nor does the statistic alone explain why the observed counts differ.

A Second Audit: A Sample Consistent With the Claim

A non-significant result does not prove that the manufacturer’s distribution is exactly correct. It says that, at the chosen significance level, the observed discrepancies do not provide strong enough evidence against the specified probabilities. Expected variation from sample to sample can produce differences even when the null model holds.

Worked Example: Interpreting a Smaller Mismatch

In another hypothetical audit, suppose the manufacturer specifies probabilities \(0.50, 0.30, 0.15,\) and \(0.05\) for four defect types. A random sample of 1,000 defective units gives observed counts \(515, 285, 145,\) and \(55\). Assess the fit using the chi-square approximation.

Assume the sampled units are independent, the categories are mutually exclusive and exhaustive, and the specified probabilities are fixed. The expected counts are \(1000(0.50,0.30,0.15,0.05)=(500,300,150,50)\). All are at least 5, so the approximation condition holds.

The statistic is:

\chi^2 =\frac{(515-500)^2}{500} +\frac{(285-300)^2}{300} +\frac{(145-150)^2}{150} +\frac{(55-50)^2}{50} =0.4500+0.7500+0.1667+0.5000 \approx 1.867.

Here \(df=4-1=3\), and the right-tail p-value is approximately 0.600. At \(\alpha=0.05\), do not reject \(H_0\): this sample does not provide evidence of a departure from the specified distribution. It does not confirm the probabilities as exact or rule out smaller departures that this sample could not detect.

Common Mistakes in a Distribution Audit

Comparing raw counts without accounting for the claimed probabilities. A common type may have a larger expected count simply because the manufacturer assigns it a higher probability. Compare each observed count with \(E_i=np_i\), not with the other observed counts alone.
Using \(n\) instead of \(np_i\) as every expected count. That would treat the categories as equally likely, which is a different null hypothesis unless all specified probabilities are equal.
Checking the observed counts instead of expected counts. The chi-square approximation condition concerns the \(E_i\) values under \(H_0\). A small observed count does not by itself show that this condition fails.
Treating a category contribution as a separate test. Contributions describe how terms add to the overall statistic. They do not each have the overall test’s p-value, and they do not by themselves establish a category-specific effect.
Including a zero-probability category in the ordinary formula. If \(p_i=0\), then \(E_i=0\), making \((O_i-E_i)^2/E_i\) undefined. Handle structural-zero categories separately rather than dividing by zero or pretending they are ordinary categories.
Calling a non-significant result proof of a perfect fit. Failure to reject means the data do not give sufficient evidence against \(H_0\) at the selected significance level; it is not proof that \(H_0\) is true.

Key takeaway: For a quality-control goodness-of-fit audit, first confirm the sampling and category assumptions, then calculate \(E_i=np_i\) and check the expected-count condition. The overall statistic tests the specified distribution; its category contributions describe which observed-versus-expected gaps account most for the mismatch.

Check Your Understanding

Use these questions to check your understanding of a product-defect distribution audit.

A manufacturer specifies probabilities \(0.60, 0.25,\) and \(0.15\) for three defect types. What are the expected counts in a sample of 400 defective units?
Why does a goodness-of-fit test with five positive-probability categories and probabilities specified in advance use \(df=4\)?
In a test, one category contributes 4.2 to the overall chi-square statistic. What can this contribution tell you, and what can it not establish by itself?
Why is a category with specified probability zero not inserted into the Pearson statistic in the same way as a category with positive probability?
A test gives \(p=0.18\) at significance level 0.05. State the conclusion in context without claiming that the manufacturer’s distribution has been proven correct.