Why Response Coding Matters
A GLM’s response is not just a column supplied to R: its representation tells the fitting function what the observed values mean. A column of zeroes and ones describes individual binary outcomes; a pair of columns can describe grouped successes and failures; a count response records event totals. These forms are not interchangeable, even when they happen to contain similar numbers.
The response should be prepared and checked before fitting. A model can run with a wrongly coded response and return plausible-looking output. The important question is whether the values supplied to R represent the intended outcome under the selected family.
The earlier material on GLM families and links establishes that different families describe different response distributions. Here the emphasis is practical: how to represent the observed response in R, and how to detect values or data types that do not match that representation. The right-hand side of a model formula does not correct a response that has been misencoded.
Individual Binary Outcomes
For a Bernoulli response, each row represents one trial and the response records whether the event of interest occurred. The safest general representation is a numeric variable containing only 0 and 1, with 1 explicitly defined as the event or “success.” For example, in a hypothetical renewal study, 1 might mean that a policy renewed and 0 that it did not.
An intercept-only binomial GLM can be written as glm(renewed ~ 1, family = binomial(link = "logit"), data = records). The intercept-only formula is useful for checking response coding; it does not include predictors. Before fitting, inspect missingness and values—for example, with is.na(renewed) and unique(renewed)—and verify that all observed values are 0 or 1.
A two-level factor is also accepted by R’s binomial family, but it has an important convention: the first factor level is treated as failure, and the other level is treated as success. The ordering of levels therefore determines which outcome is modeled as success. Because default ordering may not match the analyst’s intended definition, numeric zero-one coding is often clearer. If a factor is used, set and verify its levels deliberately.
Worked Example: Checking a Binary Renewal Response
In a hypothetical sample of eight policies, the recorded renewal outcomes in policy order are 1, 0, 1, 1, 0, 1, 1, 0, where 1 means renewed. An analyst also receives a factor version with levels ordered as “Yes,” then “No.” Prepare the response for an intercept-only binomial GLM, calculate the observed renewal proportion and fitted intercept, and explain the factor trap.
Method. Confirm the event definition, count the zeroes and ones, then use the numeric response. There are five ones and three zeroes, so \(n=8\), the observed renewal proportion is \(5/8=0.6250\), and the observed nonrenewal proportion is \(3/8=0.3750\).
For an intercept-only logit model, the fitted probability is the observed proportion. The intercept is therefore the log odds of renewal:
The numeric response can be fitted with glm(renewed ~ 1, family = binomial(link = "logit"), data = records), after confirming that renewed contains only 0 and 1. With the factor levels ordered “Yes,” then “No,” R treats “Yes” as the failure level and “No” as success. The fitted probability would then refer to nonrenewal: \(3/8=0.3750\), with intercept \(\log(3/5)=-0.5108\).
Check. The two probabilities sum to \(0.6250+0.3750=1.0000\), and the two intercepts have opposite signs because the success and failure categories were reversed. The common mistake is to assume that “Yes” is automatically success. Use numeric coding or explicitly verify factor-level order before fitting.
Grouped Binomial Outcomes
Sometimes each row summarizes several Bernoulli trials with the same predictor values. For row \(i\), let \(s_i\) be the number of successes and \(n_i\) the total number of trials. Then the number of failures is \(n_i-s_i\). In R, a standard representation is cbind(successes, failures) as the response to a binomial GLM. The column order matters: successes come first.
Another valid representation is the observed proportion \(s_i/n_i\) with prior weights \(n_i\). The weights tell the binomial fit how many trials the proportion represents. A proportion without its trial count does not convey the same information: a rate of 0.5 from 2 trials is not as informative as 0.5 from 200 trials.
The equivalence of these representations follows from the grouped binomial likelihood. If a row has \(s_i\) successes out of \(n_i\) trials and success probability \(p_i\), its likelihood is:
Taking logarithms gives a term involving \(s_i\log(p_i)+(n_i-s_i)\log(1-p_i)\), plus \(\log\binom{n_i}{s_i}\). The latter does not depend on \(p_i\). With a proportion response and weight \(n_i\), the binomial contribution to the likelihood has the same terms that depend on \(p_i\). Thus both forms encode the same information about the model’s success probability, provided the trial counts are used correctly.
Worked Example: Encoding Grouped Claim Notifications
A hypothetical portfolio summary has three groups of otherwise comparable policies. The groups record 3 notifications among 10 policies, 8 among 20, and 6 among 10. Prepare both accepted grouped-binomial response forms and calculate the intercept-only fitted probability and logit intercept.
Method. For each group, calculate failures by subtracting successes from trials. The failure counts are \(10-3=7\), \(20-8=12\), and \(10-6=4\). Thus the response matrix supplied as cbind(successes, failures) has rows (3, 7), (8, 12), and (6, 4).
The alternative success proportions are \(3/10=0.30\), \(8/20=0.40\), and \(6/10=0.60\), with weights 10, 20, and 10. An R fit can use cbind(successes, failures) ~ 1, or successes/trials ~ 1 with weights = trials, together with the binomial family.
There are \(3+8+6=17\) successes and \(10+20+10=40\) trials, so the intercept-only fitted probability is \(17/40=0.4250\). The intercept is:
Check. The total number of failures is \(7+12+4=23\), and \(17+23=40\), matching the total trial count. The proportion \(0.4250\) is also the trial-weighted average of the group proportions: \((10(0.30)+20(0.40)+10(0.60))/40=17/40\). The common mistake is to average the three proportions equally, which gives \(0.4333\) and incorrectly gives each group equal influence regardless of size.
Count Responses
A count response records the number of events observed for each unit, such as the number of service calls in a month. For a Poisson GLM, the response must be nonnegative and integer-valued. A count of zero is valid. A fraction such as 1.5 calls is not an observed event count, even if it arose from dividing or averaging other values.
In R, a count column should be numeric and should contain values satisfying \(y_i\geq 0\) and \(y_i=\lfloor y_i\rfloor\) for every observed response. A numeric column storing 2.0 is integer-valued in meaning, even if R stores it using a floating-point type. A character or factor column should be converted carefully: as.numeric(factor_variable) returns the factor’s internal level codes, not necessarily the displayed values. For text containing numeric counts, a safer conversion is as.numeric(as.character(variable)), followed by checks for conversion failures, missing values, and invalid values.
A count response is not the same as a frequency or rate. For example, a count divided by exposure may be useful in an analysis, but it is not itself an integer event count. Keep the recorded count as the response; exposure adjustments are handled separately from response coding.
Worked Example: Recovering Counts Stored as a Factor
A hypothetical service log records event counts for four districts as the text values “0,” “2,” “1,” and “4.” The column was imported as a factor. Identify the conversion error, recover the count response, and calculate the intercept-only Poisson fitted mean.
Method. The factor levels, in default sorted order, are “0,” “1,” “2,” and “4.” Converting the factor directly with as.numeric gives the internal codes 1, 3, 2, and 4—not the event counts. The erroneous sum is \(1+3+2+4=10\), and the erroneous mean is \(10/4=2.50\).
Convert through the displayed text values instead: as.numeric(as.character(events)). This gives the intended responses 0, 2, 1, and 4. They are all nonnegative integers. Their sum is \(0+2+1+4=7\), so the intercept-only Poisson fitted mean is \(7/4=1.75\). With the log link, the intercept is \(\log(1.75)=0.5596\).
Check. The corrected mean, 1.75, lies between the smallest and largest observed counts, 0 and 4. The direct factor conversion has changed the observations, not merely their storage type. The common mistake is to treat factor codes as the values printed in the data; always inspect the levels and validate the converted values.
Continuous Responses and Family-Specific Checks
A continuous response should normally be stored as a numeric vector, not as a factor or character labels. The range of valid observations depends on the chosen family. A Gaussian response can take any real value. Gamma and inverse Gaussian responses must be strictly positive. A Tweedie response used for compound costs may allow zero as well as positive values under the relevant parameterization, but negative values are not valid for that nonnegative-cost interpretation. These are domain checks, not a substitute for choosing a family appropriate to the modeling target.
Before fitting, check whether the response contains missing values, whether it converted to numeric without introducing new missing values, and whether its observed range is allowed by the chosen family. Do not silently replace an invalid value merely to make a fit run. First determine whether the value reflects a data error, a different outcome definition, or a mismatch between the response and family.
Worked Example: Validating a Continuous Cost Response
A hypothetical sample records monthly administrative costs of 120, 150, 130, and 160 dollars. An analyst plans an intercept-only Gaussian GLM, while a colleague proposes a Gamma GLM. Check the numeric response, calculate the Gaussian fitted mean, and assess whether the same values could be used unchanged for Gamma.
Method. All four amounts are numeric and strictly positive. Their sum is \(120+150+130+160=560\), so the Gaussian intercept-only fitted mean is \(560/4=140\) dollars. The deviations from the mean are \(-20, 10, -10,\) and \(20\) dollars, which sum to zero, as expected for an intercept-only Gaussian fit.
These observations are also strictly positive, so they satisfy the basic response-domain requirement for a Gamma GLM. That check does not establish that Gamma is the best model; it only shows that the observed values are within its required response range. If a later record were zero, it would still be a valid Gaussian response but would violate the strict positivity requirement for Gamma.
Check. The computed mean, 140 dollars, lies within the observed range from 120 to 160 dollars, and all four values meet the Gamma positivity condition. The common mistake is to assume that every continuous response is valid for every continuous-outcome family. Check the chosen family’s domain against the actual recorded values.
A Practical Response-Validation Sequence
A short sequence of checks catches many response-preparation problems before model fitting. The checks below concern the response column; they do not replace broader data exploration, predictor checks, or model diagnostics.
Determine whether a row is one Bernoulli trial, a group of trials, a count observation, or a continuous measurement. The intended observational unit determines the correct response form.
Inspect the response class and levels, and count missing values. Convert factors through their displayed labels rather than directly to numeric codes.
For binary data, confirm only 0 and 1. For grouped binomial data, check successes and failures are nonnegative integers and their totals are consistent. For counts, check nonnegative integer values. For continuous responses, check numeric storage and the chosen family’s domain.
Use the representation that matches the observation unit and family. Recheck that success is the intended event and, for grouped data, that the success and failure columns are in the correct order.
Common Response-Coding Traps
- Reversed binary success. A factor’s first level is treated as failure in a binomial fit. Explicitly define success, and verify the factor-level order if using a factor.
- Grouped data entered as unweighted proportions. A proportion alone omits the trial count. Use the two-column success/failure response or supply the number of trials as weights.
- Factor codes mistaken for counts. Direct numeric conversion of a factor returns internal codes. Convert through character values and then check the result.
- Noninteger count responses. A rate, average, or fractional allocation is not an event count. Confirm that the response represents actual counts before using a count family.
- Ignoring the family’s range. Zero is valid for a Gaussian response but not for Gamma. Check the selected family’s requirements rather than relying on the response’s general label as “continuous.”
- Assuming a successful fit proves correct coding. A fit can run after an outcome reversal or unintended conversion. Validate the response’s meaning as well as whether R accepts its storage type.
Check Your Understanding
For each situation, identify the response representation or validation issue and check the numerical result.
- A hypothetical survey records 7 positive responses among 12 individual trials. If positive is coded 1, what is the intercept-only binomial fitted probability and logit intercept? (Answer: Probability \(7/12=0.5833\); intercept \(\log(7/5)=0.3365\).)
- Two groups record 4 successes out of 8 and 3 successes out of 12. Give the success/failure rows for the grouped response and the intercept-only fitted probability. (Answer: Rows (4, 4) and (3, 9); fitted probability \(7/20=0.3500\).)
- A factor has displayed values “1,” “3,” and “5,” with factor levels in that order, and one observation of each value. What does direct as.numeric conversion produce, and what is the correct mean count after conversion through character? (Answer: Direct conversion gives 1, 2, 3; correct counts are 1, 3, 5 with mean 3.00.)
- A grouped-binomial row has 9 trials and 10 recorded successes. Is it valid? Give the correct bound that should have been checked. (Answer: No; successes must satisfy \(0\leq s\leq n\), so for 9 trials the success count cannot exceed 9.)
- A continuous response contains 0, 40, and 80. Is zero allowed for a Gaussian response and for a Gamma response? (Answer: Yes for Gaussian; no for Gamma, whose response must be strictly positive.)