A Worked MLE Workshop for Discrete Samples

Why Discrete-Sample MLEs Need Care

Maximum likelihood estimation chooses the parameter value that makes the observed sample most plausible under a specified model. For discrete observations, the likelihood is built from probability mass functions. The calculations are often compact, but they depend on identifying what was observed, stating the parameter space, and respecting the way the observations were collected.

This workshop develops the likelihoods for four common models: Bernoulli, binomial, Poisson, and geometric. The observations in each model are assumed independent, and the model and its parameter space are stated before maximizing. The resulting estimate must also be checked against the allowed parameter values: a score equation may give an interior candidate, while the actual maximum can occur at a boundary.

Definition: For observations \(x_1,\ldots,x_n\) with joint probability mass function \(L(\theta;x_1,\ldots,x_n)\), the likelihood as a function of \(\theta\) is \(L(\theta)=L(\theta;x_1,\ldots,x_n)\), with the observed data held fixed. A maximum likelihood estimate is any value \(\hat{\theta}\) in the parameter space that maximizes \(L(\theta)\). Because the logarithm is strictly increasing, maximizing the log-likelihood \(\ell(\theta)=\log L(\theta)\) gives the same maximizer.

Bernoulli Samples

Suppose \(X_1,\ldots,X_n\) are independent Bernoulli observations with common success probability \(p\), where \(P(X_i=1)=p\), \(P(X_i=0)=1-p\), and \(0\leq p\leq1\). Write \(s=\sum_{i=1}^n x_i\) for the observed number of successes. Independence gives the likelihood and log-likelihood below.

L(p)=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} =p^s(1-p)^{n-s}, \qquad \ell(p)=s\log p+(n-s)\log(1-p).

For \(0<p<1\), differentiating gives \(\ell'(p)=s/p-(n-s)/(1-p)\). Setting this derivative to zero yields \(s(1-p)=(n-s)p\), so \(s=np\) and \(p=s/n\). This candidate also handles the data patterns at the parameter-space boundaries: if \(s=0\), then \(L(p)=(1-p)^n\) is maximized at \(p=0\); if \(s=n\), then \(L(p)=p^n\) is maximized at \(p=1\).

Result: For an independent Bernoulli sample with common \(p\in[0,1]\), the MLE is \(\hat p=s/n=\bar X\), including the boundary values \(0\) when all observations are failures and \(1\) when all are successes.

Worked Example: Bernoulli Trial Records

A hypothetical monitoring system records whether each of 12 independent inspections detects a specified defect. Model each record as Bernoulli with common detection probability \(p\in[0,1]\). The records contain 7 detections and 5 non-detections. Find the MLE and verify that it maximizes the likelihood.

Method. Let \(s=7\) and \(n-s=5\). The likelihood is \(L(p)=p^7(1-p)^5\), so for \(0<p<1\), the log-likelihood is \(\ell(p)=7\log p+5\log(1-p)\).

Its derivative is \(\ell'(p)=7/p-5/(1-p)\). Setting it to zero gives \(7(1-p)=5p\), hence \(7=12p\) and \(\hat p=7/12\approx0.5833\). The second derivative is \(\ell''(p)=-7/p^2-5/(1-p)^2<0\) throughout the interior, so the log-likelihood is strictly concave there. At the boundaries \(L(0)=0\) and \(L(1)=0\), while \(L(7/12)>0\); the interior candidate is therefore the maximizer on the full parameter space.

Check. The estimate equals the observed success fraction, \(7/12\). The method avoids treating a boundary as the answer merely because the parameter space includes 0 and 1; neither boundary fits this mixed sample.

Binomial Counts, Including Unequal Trial Totals

A binomial observation counts successes in a fixed number of independent Bernoulli trials. Suppose \(X_i\sim\operatorname{Binomial}(m_i,p)\) independently, where each trial count \(m_i\) is fixed and known, the success probability \(p\in[0,1]\) is common, and \(i=1,\ldots,k\). The probability mass function for each observation includes a binomial coefficient. Those coefficients do not depend on \(p\), so they can be retained in the likelihood or omitted when finding its maximizer.

L(p)=\prod_{i=1}^k {m_i\choose x_i}p^{x_i}(1-p)^{m_i-x_i} \ \propto\ p^{\sum_i x_i}(1-p)^{\sum_i(m_i-x_i)}.

Let \(S=\sum_i x_i\) be the total number of observed successes and \(M=\sum_i m_i\) the total number of trials. The parameter-dependent log-likelihood is \(S\log p+(M-S)\log(1-p)\), giving the interior candidate \(S/M\). If \(S=0\), the likelihood is maximized at \(p=0\); if \(S=M\), it is maximized at \(p=1\). Thus the estimate pools successes over all trials, rather than treating each group proportion as equally informative regardless of group size.

Result: For independent binomial counts with a common \(p\in[0,1]\) and known trial totals \(m_i\), \(\hat p=\sum_i x_i/\sum_i m_i\). The denominator is the total number of trials, not the number of observed groups.

Worked Example: Pooling Binomial Inspection Batches

A hypothetical laboratory tests three batches using independent Bernoulli trials with the same unknown defect-detection probability \(p\). The batches record 3 detections in 5 trials, 6 in 8 trials, and 1 in 4 trials. Treat the three counts as independent binomial observations, with the trial totals fixed and \(p\in[0,1]\). Find the MLE and compare it with the unweighted average of the three observed proportions.

Method. The total number of detections is \(S=3+6+1=10\), and the total number of trials is \(M=5+8+4=17\). Therefore, apart from factors independent of \(p\), \(L(p)=p^{10}(1-p)^7\). The log-likelihood derivative is \(10/p-7/(1-p)\). Setting it to zero gives \(10(1-p)=7p\), or \(10=17p\), so \(\hat p=10/17\approx0.5882\).

The second derivative is \(-10/p^2-7/(1-p)^2<0\) for \(0<p<1\), and the likelihood is zero at \(p=0\) and \(p=1\). Since the sample includes both detections and non-detections, the interior maximizer \(10/17\) is the global maximizer.

The unweighted average of the batch proportions is \((3/5+6/8+1/4)/3=(0.6+0.75+0.25)/3=1.6/3=8/15\approx0.5333\). It is not the MLE: it assigns the same weight to a batch of 4 trials as to a batch of 8 trials. The pooled estimate weights each batch in proportion to its number of trials.

Check. The pooled estimate has denominator 17 total trials and numerator 10 total detections. This method avoids confusing an average of group proportions with the likelihood maximizer.

Poisson Samples

Suppose \(X_1,\ldots,X_n\) are independent observations from a Poisson distribution with common rate \(\lambda\geq0\), so \(P(X_i=x_i)=e^{-\lambda}\lambda^{x_i}/x_i!\) for nonnegative integer \(x_i\) when \(\lambda>0\). At \(\lambda=0\), the distribution is concentrated at zero. The likelihood for a sample is

L(\lambda)=\frac{e^{-n\lambda}\lambda^{\sum_i x_i}}{\prod_i x_i!}, \qquad \ell(\lambda)=-n\lambda+\Big(\sum_i x_i\Big)\log\lambda -\sum_i\log(x_i!), \quad \lambda>0.

Writing \(T=\sum_i x_i\), the interior score equation is \(-n+T/\lambda=0\), which gives \(\hat\lambda=T/n=\bar X\) when \(T>0\). The second derivative is \(-T/\lambda^2<0\), confirming a strict interior maximum. If \(T=0\), every observation is zero and \(L(\lambda)=e^{-n\lambda}\), which is maximized at the boundary \(\lambda=0\). Thus the same formula \(\hat\lambda=\bar X\) also gives the boundary estimate in that case.

Justification: In the Poisson likelihood, the parameter appears through the total count \(T\) and the sample size \(n\): its log-likelihood, apart from a constant, is \(-n\lambda+T\log\lambda\). Balancing the rate penalty \(-n\lambda\) against the count contribution \(T\log\lambda\) gives \(T/n\). The factorial terms are fixed by the observed data and cannot change the maximizing value.

Worked Example: Estimating a Poisson Event Rate

A hypothetical sensor counts events in five equal observation windows. Assume the counts are independent and identically distributed Poisson random variables with unknown mean count \(\lambda\geq0\) per window. The observed counts are \(2,0,3,1,4\). Find the MLE and check whether it is on the boundary.

Method. The total count is \(T=2+0+3+1+4=10\), with \(n=5\) windows. The likelihood is proportional to \(e^{-5\lambda}\lambda^{10}\), and the log-likelihood apart from constants is \(\ell(\lambda)=-5\lambda+10\log\lambda\) for \(\lambda>0\).

Differentiating gives \(\ell'(\lambda)=-5+10/\lambda\). The score equation gives \(10/\lambda=5\), hence \(\hat\lambda=2\). Also, \(\ell''(\lambda)=-10/\lambda^2<0\), so this is a strict interior maximum. At \(\lambda=0\), the likelihood is zero because the sample contains positive counts, while \(L(2)>0\).

Check. The estimate is the observed mean count, \(10/5=2\) events per window. The method avoids dividing the total count by the number of events; the appropriate denominator is the five independent observation windows.

Geometric Samples

Here use the geometric convention in which \(X_i\) is the trial number of the first success, so \(X_i\in\{1,2,\ldots\}\). Suppose \(X_1,\ldots,X_n\) are independent with common success probability \(p\in(0,1]\). Then \(P(X_i=x_i)=p(1-p)^{x_i-1}\). This parameterization counts the successful trial; a convention that counts failures before the first success has a different support and likelihood.

L(p)=p^n(1-p)^{\sum_i x_i-n}.

Set \(R=\sum_i x_i-n=\sum_i(x_i-1)\), the total number of failures observed before the first successes. For \(0<p<1\), the log-likelihood is \(\ell(p)=n\log p+R\log(1-p)\), with derivative \(n/p-R/(1-p)\). If \(R>0\), solving \(n(1-p)=Rp\) gives \(\hat p=n/(n+R)=n/\sum_i x_i\). The second derivative \(-n/p^2-R/(1-p)^2\) is negative, so this is a strict interior maximum. If \(R=0\), all observations equal 1; then the likelihood is \(p^n\) and is maximized at \(p=1\), again agreeing with \(n/\sum_i x_i\).

Worked Example: Geometric Waiting Times

A hypothetical call-routing system is observed over five independent attempts. For each attempt, the recorded value is the trial number on which the first successful connection occurs. Assume a geometric model with common success probability \(p\in(0,1]\), and suppose the observed values are \(2,1,4,3,5\). Estimate \(p\) by maximum likelihood and check the parameter-space boundary.

Method. There are \(n=5\) observations and their sum is \(2+1+4+3+5=15\). Hence \(R=15-5=10\) failures in total before the five first successes. The likelihood is \(L(p)=p^5(1-p)^{10}\), and the log-likelihood derivative is \(5/p-10/(1-p)\).

Setting the derivative to zero gives \(5(1-p)=10p\), so \(5=15p\) and \(\hat p=1/3\). The second derivative is \(-5/p^2-10/(1-p)^2<0\) for \(0<p<1\), establishing a strict interior maximum. At \(p=1\), the likelihood is zero because some waiting times exceed 1. Thus \(1/3\) maximizes the likelihood on the stated parameter space.

Check. The estimate is \(n/\sum_i x_i=5/15=1/3\). The method avoids using the number of failures alone as the numerator: the numerator is the number of observed first successes, one per waiting time.

Common Mistakes and Exam Checks

The central habit is to write down the data model and parameter space before taking derivatives. For each sample, identify what counts as one independent observation and whether its trial total is fixed. These choices determine both the likelihood and the meaning of the denominator in the estimate.

Averaging unequal-group proportions. For binomial groups with different trial totals, use total successes divided by total trials. An unweighted average of proportions generally does not maximize the likelihood.
Forgetting the boundary. An interior score equation does not cover samples with all successes, all failures, or all zero counts. Evaluate the likelihood at feasible endpoints when the sample structure calls for it.
Using the wrong geometric convention. If \(X\) counts the trial of the first success, its support begins at 1 and the likelihood contains \((1-p)^{\sum x_i-n}\). A model counting failures begins at 0 instead.
Treating constants as parameter-dependent. Binomial coefficients and Poisson factorials can be dropped when maximizing over the parameter, but not because they are probabilities equal to one; they are simply fixed positive factors for the observed data.

A useful final check is to substitute the estimate into the score equation when it is interior, verify concavity or another global-maximization argument, and compare with relevant boundary values. For discrete models, this disciplined check is often as important as obtaining the candidate estimate.

Check Your Understanding

For each problem, identify the relevant model and use its stated parameter space. The answers are included for checking.

An independent Bernoulli sample has 9 observations, of which 4 are successes. Find the MLE of the success probability. (Answer: \(\hat p=4/9\).)
Two independent binomial counts share a common success probability \(p\in[0,1]\). The first records 2 successes in 6 trials and the second records 5 successes in 10 trials. Find the MLE. (Answer: \(\hat p=7/16=0.4375\).)
Six independent Poisson observations with common rate \(\lambda\geq0\) are \(0,2,2,5,1,2\). Find the MLE and state whether it is interior. (Answer: \(\hat\lambda=12/6=2\), an interior estimate.)
Independent geometric observations count the trial of the first success and are \(1,3,2,6\), with common \(p\in(0,1]\). Find the MLE. (Answer: \(\hat p=4/(1+3+2+6)=1/3\), an interior estimate.)
An independent Bernoulli sample has 8 observations, all of which are failures, and \(p\in[0,1]\). Find the MLE and explain its location. (Answer: \(\hat p=0\), a boundary maximum.)

What You'll Learn