Diagnosing Misspecification in Segregation Analysis

Why Diagnose a Fitted Segregation Model?

A fitted segregation model compresses several assumptions into one likelihood: how alleles are transmitted, how genotype and age relate to phenotype, how often disease occurs without the modeled risk genotype, and how families entered the study. A model may estimate its parameters successfully and still fail to reproduce important features of the observed pedigrees. Diagnosis asks where the model's predictions depart from the data, not merely whether one parameter estimate looks plausible.

A discrepancy is a warning, not an automatic explanation. Too many affected noncarriers might reflect an underestimated phenocopy rate, but could also arise from inaccurate genotypes, a second disease cause, or an incorrect ascertainment correction. Similarly, an age-pattern mismatch can indicate a penetrance problem without establishing which biological mechanism produced it. The goal is to find the assumptions that deserve scrutiny and to check whether conclusions remain stable under reasonable alternatives.

Definition: A segregation model is misspecified when its assumptions do not adequately represent the data-generating process relevant to the observed pedigree data. A model diagnostic compares an observed feature with the distribution or expected value predicted by the fitted model. A diagnostic discrepancy can indicate misspecification, but by itself does not identify its cause.

What the Model Predicts

Let \(G_i\) denote the genotype of individual \(i\), \(Y_i\) the recorded disease phenotype, \(a_i\) the age or follow-up information, and \(S\) the event that a pedigree was selected for study. A general pedigree likelihood combines transmission probabilities, phenotype probabilities, and any relevant age or censoring information. If genotypes are not observed, the likelihood sums over their possible values. For an ascertained sample, the likelihood must also account for the selection event, as in the earlier material on conditioning likelihoods on proband selection.

A useful diagnostic starts with the model's fitted parameters and asks what the model predicts for a feature such as the number of affected relatives in a genotype group or the distribution of onset ages. For an ascertained sample, those predictions must use the same conditioning on selection as the analysis. Comparing unconditioned expectations with pedigrees selected through affected probands can create an apparent discrepancy even when the model is correct.

Model-checking principle: State the fitted assumptions before interpreting a discrepancy. Relevant assumptions include the transmission probabilities, age-specific penetrance, phenocopy probability, censoring model, and ascertainment mechanism. A diagnostic is meaningful only when its expected values and variability are calculated under assumptions appropriate to the observed sampling scheme.

For a simple count \(O\) with model-predicted mean \(E(O)\) and variance \(\operatorname{Var}(O)\), a standardized residual is

r=\frac{O-E(O)}{\sqrt{\operatorname{Var}(O)}}.

Its sign indicates direction: positive values mean more observations than predicted, and negative values mean fewer. Its magnitude expresses the discrepancy in standard-deviation units under the stated model. In a simple independent binomial setting, the variance is \(np(1-p)\). In pedigrees, relatives are generally not independent, so a count's variance must reflect dependence, uncertain genotypes, and ascertainment. A residual computed as though relatives were independent can overstate the evidence for misfit.

Worked Checks for Specific Assumptions

Worked Example: An Age-Specific Penetrance Discrepancy

Suppose a hypothetical, unascertained cohort contains 40 unrelated people known to carry a specified genotype. Each has complete follow-up to age 60. A fitted model predicts that the probability of being affected by age 60 is \(0.60\), and 30 of the 40 are affected by that age. Assume for this check that the people are independent and the fitted probability applies equally to all 40.

Under these assumptions, the affected count \(O\) has a binomial distribution with \(n=40\) and \(p=0.60\). Its expected value and standard deviation are

E(O)=np=40(0.60)=24,\qquad SD(O)=\sqrt{np(1-p)}=\sqrt{40(0.60)(0.40)}=\sqrt{9.6}=3.098.

The standardized residual is

r=\frac{30-24}{3.098}=1.937.

The observed count is about 1.94 standard deviations above the model expectation. This is a diagnostic signal that the fitted penetrance by age 60 may be too low for this group. It is not proof of that explanation: the model may have omitted an age-related covariate, and the result would require a different calculation if the cohort had been selected through affected family members.

Worked Example: More Affected Noncarriers Than Predicted

In a second hypothetical check, an unascertained sample contains 100 unrelated noncarriers with complete phenotype assessment. The fitted model assigns each a phenocopy probability of \(0.02\), but 8 are affected. Assume independent outcomes and that the same probability applies to each person.

The affected count has expected value \(100(0.02)=2\), and its standard deviation is

\sqrt{100(0.02)(0.98)}=\sqrt{1.96}=1.400.

Thus the standardized residual is

r=\frac{8-2}{1.400}=4.286.

This is a substantial excess relative to this simple model. It prompts a review of the phenocopy assumption, but other explanations remain possible, including genotype misclassification or an additional cause of disease. The calculation relies on independent sampling: if these noncarriers are relatives from a small number of pedigrees, their outcomes may be correlated and this binomial standard deviation may be too small.

Worked Example: Checking a Transmission Probability

Consider a hypothetical set of 100 informative meioses from heterozygous parents. For this diagnostic, assume the meioses are independent, the transmitted allele is known without error, and families were not selected according to offspring genotype. Under Mendelian transmission the probability of transmitting the allele is \(0.50\). Suppose it is transmitted in 62 meioses.

Under the null transmission model, the expected count is \(100(0.50)=50\), with standard deviation

\sqrt{100(0.50)(0.50)}=5.

The standardized residual is \(r=(62-50)/5=2.400\). The observed transmission count is 12 above expectation, or 2.40 standard deviations under this binomial calculation. This motivates checking the transmission assumption and the data coding. It does not establish transmission distortion: dependence among meioses, selection, or genotyping error would invalidate the simple calculation and require a model that represents those features.

Patterns Matter More Than One Residual

A single unusual count is rarely enough to diagnose a complicated model. Look for structured discrepancies that match a particular assumption. For penetrance, compare observed and predicted disease patterns across genotype and age groups. For phenocopies, inspect affected individuals who are not assigned the modeled risk genotype, while accounting for uncertainty in those assignments. For transmission, examine counts by informative parent, sex, or other prespecified grouping when the model makes relevant predictions. These comparisons should be planned with care: splitting data into many groups can produce chance extremes.

Age deserves special attention. A person who is unaffected at a young age may still develop disease later, so treating all currently unaffected relatives as permanently unaffected can distort the diagnostic. Use the age-specific penetrance or censoring assumptions used in the fitted model. For example, if the model gives the probability of remaining unaffected through age \(a\), a person last observed disease-free at age \(a\) contributes that age-dependent information; the person is not equivalent to someone known to remain unaffected for life.

When several related observations are checked together, their dependence matters. Let \(\mathbf{O}\) be a vector of observed counts and \(\boldsymbol{\mu}\) the vector of model-predicted means. Let \(\boldsymbol{\Sigma}\) be the covariance matrix predicted for those counts under the fitted model. A joint discrepancy can be summarized by

T=(\mathbf{O}-\boldsymbol{\mu})^\mathsf{T} \boldsymbol{\Sigma}^{-1}(\mathbf{O}-\boldsymbol{\mu}).

This expression accounts for covariance only if \(\boldsymbol{\Sigma}\) is correctly specified and invertible. Its value is not automatically a calibrated test statistic: a reference distribution must also be justified, often by simulation under the fitted model and the actual sampling design. In practice, the direction and pattern of discrepancies can be informative even when a formal reference distribution is not available.

Checking Ascertainment and Sensitivity

Ascertainment assumptions deserve their own diagnostic because selection changes the distribution of observed pedigrees. If \(D\) denotes the pedigree data and \(A\) the selection event, the conditional likelihood is

P(D\mid A,\theta)=\frac{P(D,A\mid\theta)}{P(A\mid\theta)},

where \(\theta\) denotes model parameters. The denominator is not a technical decoration: it adjusts for how likely a pedigree would be selected under the model. Before interpreting a phenotype residual, verify that the conditioning event matches the actual recruitment rule, such as selection through one proband or through any qualifying affected relative. Also check that the analysis treats multiple probands consistently with the sampling design.

A sensitivity analysis fits or evaluates plausible alternative assumptions and records which conclusions change. Alternatives should be scientifically defensible, not chosen only because they improve fit. Useful variations include a different age-specific penetrance curve, a wider plausible range of phenocopy probabilities, an alternative transmission probability when justified, or a different ascertainment rule that matches an uncertainty in recruitment records.

Worked Example: Sensitivity to the Ascertainment Rule

Suppose two hypothetical segregation models, \(M_0\) and \(M_1\), are compared for a pedigree pattern \(D\). Using one proposed selection event \(A_1\), a calculation gives \(P(D,A_1\mid M_0)=0.012\) and \(P(A_1\mid M_0)=0.080\); for \(M_1\), the corresponding values are \(0.009\) and \(0.030\). The conditional likelihoods are

P(D\mid A_1,M_0)=\frac{0.012}{0.080}=0.150,\qquad P(D\mid A_1,M_1)=\frac{0.009}{0.030}=0.300.

Under \(A_1\), the conditional likelihood for this pattern is twice as large under \(M_1\) as under \(M_0\). Now suppose recruitment records leave open a second plausible selection event \(A_2\). Recalculation gives \(P(D,A_2\mid M_0)=0.010\), \(P(A_2\mid M_0)=0.040\), \(P(D,A_2\mid M_1)=0.012\), and \(P(A_2\mid M_1)=0.060\). Then

P(D\mid A_2,M_0)=\frac{0.010}{0.040}=0.250,\qquad P(D\mid A_2,M_1)=\frac{0.012}{0.060}=0.200.

The relative ordering reverses under \(A_2\). These illustrative likelihoods show why an apparent model preference can depend on the selection mechanism. They do not establish which event occurred; that must be resolved from how the families were recruited. If the recruitment rule is uncertain, report the sensitivity rather than presenting one conditional result as definitive.

A Practical Diagnostic Sequence

Write down the fitted assumptions.
Record the transmission, penetrance, phenocopy, censoring, and ascertainment assumptions relevant to the feature being checked.

Choose a targeted feature.
For example, compare affected counts by age and genotype, or examine transmissions in informative meioses. Avoid searching many unplanned subgroups and highlighting only the most extreme result.

Calculate expected values under the actual sampling design.
Account for pedigree dependence, genotype uncertainty, censoring, and ascertainment rather than treating relatives as an independent random sample.

Trace the pattern to possible assumptions.
Ask which assumption could generate the direction and age or genotype pattern observed, while keeping alternative explanations in view.

Check sensitivity and report uncertainty.
Evaluate plausible alternatives and state whether parameter estimates or model comparisons change. A diagnostic flags vulnerability; it does not prove a biological cause.

Common Pitfalls and Why the Diagnosis Matters

Using the wrong reference population. Comparing an ascertained pedigree count with an unconditioned expectation ignores selection. Calculate predictions conditional on the relevant ascertainment event.
Counting relatives as independent. Shared inheritance and family environment create dependence. A binomial standard deviation can be misleading when it ignores that structure.
Calling every discrepancy a penetrance failure. An age-pattern mismatch may also reflect censoring, phenotype classification, or selection. Check the assumptions jointly before assigning a cause.
Treating a large residual as proof. A residual measures discrepancy relative to a specified model, not the probability that the model is true. Consider data quality, alternative explanations, and whether the residual was selected after many comparisons.
Changing many assumptions at once. If the fit improves, it is then unclear which change mattered. Vary assumptions in a targeted, documented way and distinguish exploratory checks from confirmatory conclusions.

Good diagnostics make segregation analysis more credible because they expose what its conclusions depend on. A fitted model that reproduces the broad phenotype and age patterns, and whose conclusions are stable to reasonable ascertainment and parameter alternatives, is more reassuring than one judged only by its optimized likelihood. Even then, model checking supports adequacy for a purpose; it does not prove that every genetic or clinical mechanism has been represented.

Key takeaway: Diagnose misspecification by comparing observed pedigree patterns with model-based expectations that respect age, dependence, genotype uncertainty, and ascertainment. Use discrepancies to identify assumptions for review, and use sensitivity analysis to determine whether substantive conclusions depend on them.

Check Your Understanding

For each question, distinguish what a diagnostic can show from what it cannot establish.

A fitted model predicts 20 affected individuals in a group, but 27 are observed. What additional information is needed before calculating and interpreting a standardized residual?
Why can a young unaffected relative not always be treated as evidence against a model's penetrance assumption?
A pedigree contains several affected noncarriers. Give two possible explanations other than an underestimated phenocopy probability.
Explain why the conditional likelihood divides by the probability of the ascertainment event.
A model comparison changes direction under two plausible ascertainment rules. What should be reported, and what information might resolve the uncertainty?