Why Reporting Is Part of the Analysis
A regression test does not speak for itself. A statement such as “the result was significant” leaves out what was compared, how large the estimated difference was, and how uncertain that estimate is. Responsible reporting connects the statistical calculation to the scientific question: what quantity was estimated, in what units, and what does the evidence justify saying about it?
This applies both to an individual regression coefficient and to a linear contrast of several coefficients. For instance, a contrast might compare two adjusted group means or compare the slopes for two groups. The reporting goal is the same: name the comparison, give its estimate and uncertainty, and describe the test without implying more than it establishes.
What a Responsible Report Needs
Before reporting a test, state the scientific comparison in words. Then identify the estimated coefficient or contrast, its units, and the null value being tested. For a test of no difference, the null value is often zero, but a test of a specified benchmark can use another value. The report should make that value explicit.
For a two-sided test of \(H_0:L=L_0\), the test statistic is \(t=(\hat{L}-L_0)/SE(\hat{L})\), where \(SE(\hat{L})\) is the standard error of the estimated contrast. Report the statistic, its degrees of freedom, and the p-value. Also report a confidence interval at a stated confidence level. When the interval and test are based on the same model and standard error, the interval shows a range of values compatible with the data under the procedure, while the test evaluates the specified null value.
For the usual ordinary least-squares t procedures, the model must be suitable for the scientific question and its mean structure. In addition, the errors are assumed independent, to have mean zero and a common variance, and to be normally distributed for the exact small-sample t distribution. The design matrix must permit estimation of the contrast. With \(n\) observations and \(m\) estimated coefficients in a full-rank model, the residual degrees of freedom are \(df=n-m\). If these conditions are doubtful, a conventional t-based p-value and interval may not have their stated calibration.
The exact t interval for \(L\) at confidence level \(1-\alpha\) is \(\hat{L}\pm t_{1-\alpha/2,df}SE(\hat{L})\). It is important to report the confidence level and the units. A p-value is the probability, assuming the null hypothesis and model conditions, of obtaining a test statistic at least as extreme as the observed one in the direction or directions specified by the alternative. It is not the probability that the null hypothesis is true.
Why the Test and Interval Tell a Consistent Story
For a two-sided test at level \(\alpha\), the null value \(L_0\) is rejected when \(|t|>t_{1-\alpha/2,df}\). Substituting the definition of \(t\), this condition is equivalent to \(|\hat{L}-L_0|>t_{1-\alpha/2,df}SE(\hat{L})\). That says \(L_0\) lies outside the interval \(\hat{L}\pm t_{1-\alpha/2,df}SE(\hat{L})\). Thus, for a matching two-sided test and confidence interval, the null value is excluded from the \(100(1-\alpha)\%\) interval exactly when the test rejects at level \(\alpha\).
This equivalence is a useful reporting check, not a reason to treat the interval and test as separate discoveries. If a report gives a 95% interval containing zero but claims that the corresponding two-sided test rejects \(H_0:L=0\) at \(\alpha=0.05\), some part of the analysis or reporting is inconsistent. The check depends on using the same model, standard error, degrees of freedom, and two-sided procedure for both.
Worked Example: Reporting an Adjusted Difference Between Groups
Hypothetical example. A regression model relates a follow-up score, measured in points, to treatment group, baseline score, age, and site. The treatment contrast is defined as the adjusted mean score for Group A minus that for Group B. There are 23 independent observations and five estimated coefficients, so \(df=23-5=18\). Suppose the fitted contrast is \(\hat{L}=6.0\) points with \(SE(\hat{L})=2.0\) points. Consider the two-sided test \(H_0:L=0\) at \(\alpha=0.05\).
The statistic is \(t=(6.0-0)/2.0=3.00\). With 18 degrees of freedom, the two-sided p-value is approximately \(0.007685\). Since this is below \(0.05\), the data provide evidence against a zero adjusted difference under the stated model and test assumptions.
For a 95% interval, \(t_{0.975,18}\approx2.1009\). The margin of error is \(2.1009(2.0)=4.2018\) points, giving \(6.0\pm4.2018=(1.798,10.202)\) points. This interval excludes zero, consistently with the two-sided test.
A concise report could read: “The adjusted mean follow-up score was estimated to be 6.0 points higher in Group A than Group B (95% CI 1.80 to 10.20; \(t(18)=3.00\), \(p=0.007685\)).” This describes an adjusted association. Unless the study design and assumptions justify a causal interpretation, it does not establish that treatment caused the difference.
Reporting the Direction and Scale of a Contrast
The sign of an estimate depends on the order and weights in the contrast. A negative estimate for “slope in Group A minus slope in Group B” means the estimated slope in Group A is lower; it does not mean that both slopes are negative. State the contrast order explicitly, and include the units of the contrast. For a slope difference, these units are outcome units per predictor unit. For an adjusted mean difference, they are outcome units.
Avoid translating “statistically significant” into “large,” “important,” or “clinically meaningful” without a relevant scale or benchmark. A small effect can have a small p-value when estimated precisely; a practically important effect can have a large p-value when estimated imprecisely. Report the estimate and interval so readers can assess magnitude and uncertainty in context.
Worked Example: Reporting a Difference Between Slopes
Hypothetical example. A fitted model estimates the association between weekly activity and a health score separately for two groups. The contrast is the slope for Group A minus the slope for Group B, measured in score points per additional activity hour per week. With 30 observations and six estimated coefficients, \(df=24\). Suppose \(\hat{L}=-4.0\) score points per hour and \(SE(\hat{L})=1.6\) score points per hour. Test \(H_0:L=0\) against a two-sided alternative.
The test statistic is \(t=(-4.0-0)/1.6=-2.50\). For \(df=24\), the two-sided p-value is approximately \(0.01965\). At \(\alpha=0.05\), this provides evidence that the two slopes differ under the model assumptions.
For a 95% interval, \(t_{0.975,24}\approx2.0639\). The margin is \(2.0639(1.6)=3.3022\) score points per hour, so the interval is \(-4.0\pm3.3022=(-7.302,-0.698)\) score points per hour.
A suitable report is: “The estimated slope in Group A was 4.0 score points per additional weekly activity hour lower than the slope in Group B (95% CI for Group A minus Group B: \(-7.30\) to \(-0.698\) score points per hour; \(t(24)=-2.50\), \(p=0.01965\)).” This is evidence of a difference in the modeled associations. It does not show that increasing activity causes the score to change.
When the Result Is Not Statistically Significant
A p-value above the chosen significance level means the data do not provide sufficient evidence to reject the specified null using that test. It does not prove the null, establish that the effect is zero, or show that the effect is unimportant. The confidence interval is especially useful here: it displays which effect sizes remain compatible with the estimate and its uncertainty.
Worked Example: Describing an Uncertain Estimate
Hypothetical example. A regression estimates a contrast in mean recovery time between two groups, measured in days, with a specified order of Group A minus Group B. There are 36 observations and six estimated coefficients, giving \(df=30\). Suppose the estimated contrast is \(\hat{L}=1.2\) days and its standard error is \(0.8\) days. Test \(H_0:L=0\) using a two-sided alternative.
The test statistic is \(t=(1.2-0)/0.8=1.50\). For 30 degrees of freedom, the two-sided p-value is approximately \(0.144\), which is greater than \(0.05\). The test therefore does not reject the null at the 5% level.
Using \(t_{0.975,30}\approx2.0423\), the margin of error is \(2.0423(0.8)=1.6338\) days. The 95% interval is \(1.2\pm1.6338=(-0.434,2.834)\) days. The interval includes zero but also includes differences that may matter scientifically, depending on the context.
A responsible summary is: “The estimated Group A minus Group B difference was 1.2 days (95% CI \(-0.434\) to \(2.834\); \(t(30)=1.50\), \(p\approx0.144\)). The data do not provide sufficient evidence of a nonzero difference at the 5% level, but the interval remains compatible with both a small negative difference and a positive difference of several days.” This does not claim that the groups are equivalent.
Common Reporting Pitfalls
- Reporting only a p-value. “\(p<0.05\)” does not name the estimate, comparison, direction, magnitude, or units. Give the contrast and its interval as well.
- Calling a p-value the chance the null is true. A p-value is calculated under the null and model assumptions; it is not a posterior probability for the hypothesis.
- Treating a non-significant result as proof of no effect. Failure to reject is not evidence that the contrast equals zero. Examine the interval and consider whether it rules out effects large enough to matter.
- Overstating causality. Regression adjustment alone does not establish that a predictor caused an outcome difference. The design and assumptions must support that claim.
- Omitting the contrast direction or units. “The difference was \(-4\)” is ambiguous. Say which quantity was subtracted from which, and give the units.
- Ignoring how many tests were considered. If many contrasts were examined, say how the analyses were selected and whether an appropriate multiplicity procedure was used. A lone unadjusted p-value can overstate the strength of evidence after extensive searching.
A Practical Reporting Sequence
A report can be concise without being cryptic. Check that the reader can identify the question, the scale of the estimate, and the uncertainty. Use this sequence:
Specify the coefficient or contrast, including the order of subtraction, the modeled adjustment, and the units.
Report the contrast estimate and a confidence interval at a stated confidence level, using matching units.
State the null value, test direction, test statistic, degrees of freedom, and p-value at sensible precision.
Describe the strength of evidence and the size of the estimate without claiming proof, practical importance, or causation that the analysis does not establish.
Check Your Understanding
Use the reporting principles in this tutorial to answer the following.
- A report gives \(p=0.03\) for a contrast but omits the contrast estimate and units. What important information is missing?
- In the contrast “slope for Group B minus slope for Group A,” the estimate is negative. What does its sign say, and what does it not say about the individual slopes?
- A two-sided test of \(L=0\) has \(p=0.20\), and its 95% confidence interval includes both zero and effects considered practically important. What conclusion is justified, and what conclusion is not?
- Why should a report state the degrees of freedom along with a t statistic?
- What additional evidence or design features would be needed before describing a regression association as a causal effect?