Why Source Weighting Needs Care
A QSP calibration may draw on several studies, assays, or experimental conditions. A source with many observations can legitimately contribute more information than a small source, while a noisy measurement should generally count less than a precise one. But a data set can also appear informative simply because the same underlying evidence has been recorded more than once: a reported mean and its component observations, overlapping summaries, or multiple correlated endpoints may all describe much of the same signal.
Choosing source weights is therefore not just a matter of deciding which study seems important. The weight should reflect the uncertainty and dependence structure of the evidence actually used. This tutorial develops that principle without prescribing a particular full calibration objective.
Uncertainty and Sample Size
Let source \(s\) contain an observation \(y_s\) and a model prediction \(m_s(\theta)\), where \(\theta\) denotes the model parameters. The residual is \(r_s=y_s-m_s(\theta)\). If the observation has known variance \(\sigma_s^2\), its squared residual scaled by uncertainty is \(r_s^2/\sigma_s^2\). A residual of a given size should contribute less when it is compatible with the source’s ordinary measurement variability.
For a source represented by a sample mean \(\bar{x}\) from \(n\) independent observations with common variance \(\sigma^2\), the variance of the mean is \(\sigma^2/n\), and its standard error is \(\sigma/\sqrt{n}\). Thus, a discrepancy between a predicted mean and \(\bar{x}\) is scaled by the variance of the mean, not by the individual-observation variance. If replicate correlation or other dependence is present, this simple standard-error expression may not apply; the uncertainty must reflect that dependence.
This distinction prevents two opposite errors. If the data are reduced to a source mean, using the individual-observation variance for that mean understates the information in the mean by a factor of \(n\). Conversely, if all individual observations are already included, multiplying their total contribution by \(n\) again counts sample size twice.
Worked Example: Comparing Two Source Means
Suppose two hypothetical sources report mean biomarker concentrations. Source A has \(\bar{x}_A=12\) ng/mL with \(SE_A=2\) ng/mL; the model predicts \(m_A(\theta)=8\) ng/mL. Source B has \(\bar{x}_B=21\) ng/mL with \(SE_B=1.5\) ng/mL; the model predicts \(m_B(\theta)=18\) ng/mL. Treat the reported standard errors as valid and the two source means as independent.
For Source A, the residual is \(12-8=4\) ng/mL, so its uncertainty-scaled contribution is \(q_A=4^2/2^2=16/4=4\).
For Source B, the residual is \(21-18=3\) ng/mL, giving \(q_B=3^2/1.5^2=9/2.25=4\).
The two sources are each two standard errors from their model predictions, so their squared standardized discrepancies are equal. Comparing the raw squared residuals alone would give 16 versus 9 and ignore that the sources have different precision. These calculations describe contributions under the stated uncertainty assumptions; they do not establish that the model is adequate overall.
Use Sample Size Once
How sample size enters depends on what has been supplied to the calibration. If the individual observations are available, an independent-observation likelihood or sum of standardized residual squares already accumulates information across the \(n\) observations. If only a sample mean is available, its standard error carries the sample-size information. The analyst must identify which representation is being used before adding any source-level multiplier.
For example, if \(n\) independent measurements have common variance \(\sigma^2\), then the sum of their squared deviations from a fixed model prediction contains \(n\) terms. If instead the only retained datum is their mean, the mean’s variance is \(\sigma^2/n\). For observations whose errors are independent and normally distributed with known variance, the mean-based squared standardized discrepancy is the part of the full sum that measures mismatch in the sample average. The full observations may also carry information about variation, but that is not the same as a second copy of the mean discrepancy.
Determine whether the source enters as individual observations, a summary statistic, or both.
For individual data, use individual measurement variability; for a summary, use the summary’s sampling uncertainty, including dependence where relevant.
Do not add a summary contribution to its component observations as if it were new independent evidence.
Worked Example: A Mean and Its Component Observations
A hypothetical assay source has \(n=9\) independent measurements, each with known standard deviation \(\sigma=3\) units. Their sample mean is 10 units, while the model predicts 12 units. For illustration, suppose the nine individual measurements are all 10 units. Compare the mean-based contribution with the contribution from all nine observations, and explain what goes wrong if both are added.
The standard error of the mean is \(SE=3/\sqrt{9}=1\) unit. The mean-based contribution is \(q_{\text{mean}}=(10-12)^2/1^2=4\).
For the individual observations, each residual is \(10-12=-2\) units. The sum of squared standardized residuals is \(q_{\text{individual}}=9((-2)^2/3^2)=9(4/9)=4\).
The agreement here follows because all nine observations are identical: all their mismatch is in the mean, with no additional spread around it. Adding both contributions would give \(4+4=8\), treating the same mean mismatch as two independent pieces of evidence. It would overstate the information. In real data, individual observations may also inform variability or other features, but the mean summary remains derived from those observations and is not an independent replicate.
Correlated Endpoints and Duplicate Summaries
A source may report several endpoints measured on the same subjects or samples. If their errors are correlated, treating their residuals as independent exaggerates the amount of distinct information. Let \(\mathbf{y}_s\) be a vector of observations, \(\mathbf{m}_s(\theta)\) the corresponding prediction vector, and \(\Sigma_s\) their error covariance matrix. When this covariance is positive definite, the uncertainty-scaled quadratic discrepancy is \(\mathbf{r}_s^\mathsf{T}\Sigma_s^{-1}\mathbf{r}_s\), where \(\mathbf{r}_s=\mathbf{y}_s-\mathbf{m}_s(\theta)\).
This expression accounts for both the scale of each endpoint and their correlation. If two endpoints are exact duplicates, their covariance matrix is singular: there is no valid inverse that would treat them as two independent measurements. The correct response is not to force an inverse or assign an arbitrary small variance. Instead, represent the unique information once, or use a model that explicitly handles the dependence and any measurement error.
Worked Example: Two Correlated Endpoints
A hypothetical study reports two endpoints, each standardized to have error variance 1, with error correlation \(\rho=0.8\). At a parameter value under consideration, both observed-minus-predicted residuals are 2 standardized units. Thus, \(\mathbf{r}=(2,2)^\mathsf{T}\) and \(\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}\). Assume this covariance adequately describes the errors.
For this two-endpoint covariance, the inverse is \(\Sigma^{-1}=\frac{1}{1-\rho^2}\begin{pmatrix}1&-\rho\\-\rho&1\end{pmatrix}\). Substitution gives \(q=(4+4-2(0.8)(2)(2))/(1-0.8^2)=(8-6.4)/0.36=1.6/0.36=4.444\).
If the endpoints were incorrectly treated as independent with unit variance, the contribution would be \(2^2+2^2=8\). Their shared error means they provide less than two independent copies of the same-direction discrepancy, so the covariance-adjusted contribution is smaller. It is still positive: accounting for correlation does not erase the observed mismatch.
Choosing a Weighting Principle
Uncertainty-based weighting is appropriate when the stated measurement model and uncertainty estimates are credible. In that setting, precision determines relative influence: a more precise source generally contributes more for a given raw discrepancy. There are situations where an analyst instead wants sources to have similar influence, for example to guard against a large study dominating a synthesis intended to balance evidence streams. Such equal-source weighting is a deliberate change in the calibration target, not a correction that automatically makes the analysis more statistically valid.
A useful comparison is:
| Approach | What determines influence? | Main caution |
|---|---|---|
| Observation-level uncertainty weighting | Each observation’s variance and the number of independent observations | Do not multiply by sample size again after including the observations. |
| Summary-level uncertainty weighting | The summary’s standard error or covariance | Use uncertainty for the summary, not the raw measurement SD. |
| Equal-source weighting | A chosen policy that gives each source a similar prescribed contribution scale | It may discount precision and does not by itself resolve within-source dependence. |
| Covariance-aware weighting | The full error covariance among endpoints or summaries | Covariance estimates can be uncertain; exact duplication makes the covariance singular. |
A global rescaling of every source contribution by the same positive constant does not change the minimizer of a criterion consisting only of those contributions. Relative weights do matter: multiplying just one source by a constant changes its influence and can shift the fitted parameters. If weights are selected for policy reasons rather than derived from a sampling model, uncertainty summaries from the resulting fit should not be interpreted as if the original likelihood assumptions still applied unchanged.
Diagnostics and Common Failure Modes
Before calibration, create an evidence accounting table that records, for each source, the observations or summaries used, their units, the uncertainty quantity and its basis, sample size, and known within-source dependence. Then inspect the proposed contributions at plausible parameter values. Ask whether large influence comes from genuinely precise, distinct evidence or from repeated representation of the same measurements.
- Red flag: raw residuals determine influence across different scales. Compare standardized residuals, not just differences in units such as ng/mL and hours.
- Red flag: sample size appears twice. Check whether \(n\) is already reflected in the number of observations or in the standard error of a summary.
- Red flag: a reported summary and its underlying data both enter independently. Keep one representation, or model their dependence explicitly.
- Red flag: correlated endpoints are treated as independent. Use a defensible covariance structure or avoid counting redundant endpoints as separate evidence.
- Red flag: a source dominates after a small change in its weight. Compare fits under defensible alternative assumptions and report the sensitivity; do not hide a consequential weighting choice.
These checks cannot establish that the QSP model is correct. A source can receive a small uncertainty-based contribution because it is noisy, yet still reveal systematic model misspecification. Conversely, a highly precise source can dominate because of genuine precision, but also because its uncertainty was underestimated. Weighting is limited by the quality of the error model, the independence assumptions, and the completeness of the evidence accounting.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A source mean has a standard error of 0.5 units and differs from its model prediction by 1 unit. What is its squared standardized contribution?
- For independent measurements with standard deviation 4 units, what is the standard error of their mean when \(n=16\)?
- Why is it generally inappropriate to add a sample-mean discrepancy to the contribution from all individual observations that produced that mean?
- Two standardized endpoints each have residual 1, variance 1, and correlation 0.5. Find their covariance-adjusted quadratic discrepancy.
- What should be done if two reported summaries are exact duplicates and their covariance matrix is singular?
Solutions. (1) The contribution is \(1^2/0.5^2=1/0.25=4\). (2) \(SE=4/\sqrt{16}=1\) unit. (3) The mean is computed from those observations, so its mismatch is not an independent replication; adding both as independent evidence can double count the same information. (4) With \(\rho=0.5\), \(q=(1+1-2(0.5)(1)(1))/(1-0.5^2)=1/0.75=1.333\). (5) Do not invert the singular matrix as though the duplicates were independent. Represent the unique information once or specify a model that accounts for the exact dependence.