Why Calibration Decisions Need a Record
A calibration result is not reproducible from a parameter estimate alone. The result also depends on which source records were used, how values were processed, how predictions were matched to observations, and how each observation influenced the fit. These choices can be reasonable and still lead another analyst to a different result if they are not made explicit.
A calibration record is an auditable account of that path. It links each source to its original data, processing steps, analysis-ready observations, observation model, inclusion or exclusion decisions, and weighting contribution. Its purpose is not to assert that a decision is correct; it is to make the decision inspectable and the analysis reconstructable.
For a multi-source calibration, assign a stable source identifier to every distinct data source or protocol. Keep the original files unchanged and identify them by filename, version or date, and preferably a checksum. Store processed data separately, with a script or explicit processing specification that can recreate them from the originals. A checksum helps detect an altered file; it does not establish that the file is accurate or that its contents are appropriate for calibration.
A Traceable Record from Source to Objective
A practical record should allow a reviewer to follow both the data and the reasoning. For each source, document its provenance and design (for example, population, species, treatment, protocol, and sampling times); the raw-file location and version; the extraction and processing code; and the identifiers that link each analysis row to its source record. State units at each stage. Record transformations as formulas or versioned code, not as unexplained replacement values.
The record must also state how processed observations are represented in the calibration. Let \(y_j\) denote the raw data from source \(j\), and let \(T_j\) denote the documented processing operation. The analysis-ready data are \(z_j=T_j(y_j)\). Let \(\mu_j(\theta)\) be the model predictions corresponding to the same endpoints, conditions, and times, where \(\theta\) is the parameter vector. The record specifies how \(z_j\) is related to \(\mu_j(\theta)\), including the observation distribution, error scale, covariance assumptions, and any censoring or aggregation.
For example, if processed observations are modeled as normally distributed with fixed covariance matrix \(\Sigma_j\), the residual vector is \(e_j=z_j-\mu_j(\theta)\). A weighted least-squares contribution is
This expression is appropriate only when the stated covariance describes the observation errors on the scale of \(z_j\), and \(\Sigma_j\) is positive definite. If sources are independent, their contributions can be added. If observations from different sources are correlated, a source-by-source sum that ignores cross-source covariance is not the corresponding joint objective; the record must specify the joint covariance or other joint observation model.
The connection to likelihood explains why the assumptions and scale belong in the record. Under a multivariate normal observation model with fixed \(\Sigma_j\), the negative log-likelihood for source \(j\), up to terms constant in \(\theta\), is \(Q_j(\theta)/2\). If the covariance depends on \(\theta\), the log-determinant term also depends on \(\theta\) and cannot be dropped. A documented weighted objective is therefore not interchangeable with a likelihood unless its assumptions and omitted terms are clear.
Worked Examples
Worked Example: Recording a Dilution and Unit Conversion
Suppose a hypothetical assay reports a measured concentration of \(4\) ng/mL for a sample diluted fivefold before measurement. The analyte has molecular weight \(500\) g/mol. The calibration model predicts concentration in nM, and the observation error is modeled on the natural-log scale with standard deviation \(0.2\) log units.
The record should retain the reported value and document the dilution correction. Under the stated convention, the corrected concentration is \(4\times5=20\) ng/mL. Since \(1\) ng/mL equals \(1\) microgram/L, conversion to molar concentration gives
If the corresponding model prediction is \(36\) nM, the log-scale residual, defined here as observed minus predicted, is
With the stated log-scale standard deviation, the squared standardized residual is \(0.1054^2/0.2^2\approx0.2775\). The record should identify the original assay row, the fivefold correction, the molecular-weight conversion, the log-scale observation model, and the predicted endpoint. This example assumes the dilution factor is known and that the log-scale error model applies to the corrected concentration; the conversion itself does not establish either assumption.
Worked Example: Documenting Replicate Aggregation and Weight
A hypothetical source has three independent replicate measurements, \(8\), \(10\), and \(12\) units. Assume each replicate has a known measurement standard deviation of \(2\) units and shares the same model prediction, \(11\) units. The analyst chooses to represent these replicates by their mean.
The mean is \((8+10+12)/3=10\) units. Under the stated independence and equal-variance assumptions, the standard error of the mean is \(2/\sqrt{3}\approx1.1547\) units. The aggregate observation's weighted least-squares contribution is
The record must preserve the three replicate values, not just the mean, and state that the replicates are independent, the variance is treated as known and equal, and all three share the same prediction. If the raw replicates were retained individually, their contribution would be \(\big((8-11)^2+(10-11)^2+(12-11)^2\big)/2^2=11/4=2.75\). The difference from \(0.75\) is \(8/4=2\), the within-replicate variation about the observed mean. With this fixed variance and a common prediction, that difference is constant in the model parameters, so the two representations have the same parameter-dependent part of the objective. They do not preserve the same information about replicate variation, and this equivalence need not hold if the variance is estimated or predictions differ across replicates.
Worked Example: Representing a Below-Quantification Result
Suppose an assay has a lower limit of quantification (LLOQ) of \(5\) units. A hypothetical sample is reported as below the LLOQ. Assume, for this example, that the latent measurement \(Y\) is normally distributed with model-predicted mean \(4\) units and known standard deviation \(2\) units. The observation is the event \(Y<5\), not an exact measurement of zero or of \(2.5\) units.
The likelihood contribution for this left-censored observation is the probability of falling below the limit:
Here \(\Phi\) is the standard normal cumulative distribution function. The negative log-likelihood contribution is \(-\ln(0.6915)\approx0.369\). A reconstructable record states the LLOQ, the censoring direction, the assumed normal observation model and scale, and the model prediction used. Substituting half the LLOQ would instead treat \(2.5\) units as an exact observation and produce a different contribution. Excluding the result would discard its information. Neither alternative is equivalent to the documented censored-data likelihood.
How to Record Exclusions and Weighting Choices
An exclusion should be represented as a decision about a named record, not by silently omitting a row from the processed file. Record the source-row identifier, exclusion status, reason, decision date or analysis version, and the rule supporting the decision. Also record whether the rule was set before examining calibration residuals. Reasons might include a documented protocol deviation or a prespecified assay quality-control failure. A poor match to the fitted model, by itself, is not a data-processing reason to remove an observation.
For weighting, record the actual quantity used in the objective and its derivation. If weights are inverse variances, state which variance and observation scale they refer to, how they were obtained, and whether they were fixed or estimated. A source with many observations may contribute more simply because it has more rows; a separate source-level multiplier can change that balance. Document such a multiplier and its rationale explicitly. Do not apply a precision weight and an additional source weight without explaining that the combined weighting is intentional.
| Decision | Record explicitly | Check during reconstruction |
|---|---|---|
| Transformation | Formula or code version, input and output units, order of operations | Recreate processed values from the raw records |
| Observation model | Scale, distribution, error parameters, correlation assumptions, censoring | Confirm prediction and observation use matching endpoints and scales |
| Exclusion | Record identifier, reason, rule, and decision version | Reconcile included and excluded counts with the source |
| Weighting | Weight formula, estimated or assumed precision, any source multiplier | Recalculate source contributions and check for duplicate weighting |
Reconstruction, Diagnostics, and Limits
A useful audit is to rebuild the analysis from the recorded inputs rather than to inspect only the final fit. Recreate each processed data set from its identified raw version, check row counts and units, and recompute each source's objective contribution at a stated parameter vector. Compare the reconstructed predictions, residuals, weights, and source totals with the calibration output. If these do not agree within the declared rounding and software tolerances, the record is incomplete or the implementation differs from the described analysis.
Verify source identifiers, file versions, checksums where available, and the mapping from source rows to analysis rows.
Run the specified transformations and check units, row counts, aggregation, missingness, and inclusion status.
Match each processed value to the correct prediction and apply the stated observation model and weighting rule.
Compare reconstructed source-level contributions and their combination with the recorded calibration objective.
Several discrepancies are especially informative. A count mismatch can reveal lost, duplicated, or newly excluded rows. A unit mismatch can produce plausible-looking but systematically incorrect residuals. Unexpectedly dominant source contributions can reveal an unintended scale or weight, though dominance alone does not prove an error. Repeated source identifiers or identical records can signal double counting. If source contributions do not add to the reported total, check whether the objective includes cross-source covariance, penalties, likelihood normalization terms, or other documented components.
There are alternatives in how data are represented, and the record should make the chosen one distinguishable from the others. Raw replicates can be retained individually or summarized; measurements can be compared on the original or a transformed scale; and source contributions can use specified precision weights or an explicitly stated source-level weighting scheme. These choices are not interchangeable by default. Their consequences depend on the observation model, covariance assumptions, and objective being used. The point of documentation is not to prescribe one choice for every data set, but to let another analyst identify and reproduce the choice made.
A complete record cannot correct biased sampling, a flawed assay, an unsuitable observation model, or a biologically inadequate QSP model. It also cannot make an undocumented assumption defensible merely by recording it. Its value is narrower and essential: it separates the data and calibration decisions that were actually made from what a reader might otherwise assume, and it provides a basis for targeted review and later reanalysis.
Common Mistakes
- Replacing source data with processed values. Keep an immutable raw version and retain row-level identifiers so that every processed value can be traced back.
- Reporting a transformation without its scale or units. State the exact operation, input units, output units, and order of operations.
- Calling all low-quality fits “outliers.” Record the actual exclusion criterion and the evidence that it was met; do not hide a residual-driven exclusion as routine cleaning.
- Using a weight without specifying its meaning. State whether it represents inverse variance, a source-level multiplier, or another defined quantity, and show how it enters the objective.
- Treating a reproducible pipeline as proof of validity. Reproducibility makes assumptions inspectable; it does not establish that they are scientifically appropriate.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A source file has 120 rows. The processed data have 114 rows, and the record says six failed quality control. What additional information is needed to make the exclusions auditable?
- Three independent replicates have known standard deviation \(3\) units. What is the standard error of their mean, and what must be recorded if the mean is used for calibration?
- A result is reported below an assay's LLOQ. Why is replacing it by half the LLOQ not the same as recording a censored observation?
- Two source contributions do not add to the reported total objective. Give two documented features that could explain the difference.
Solutions. (1) The record needs the six source-row identifiers, the specific quality-control rule and evidence for each failure, the exclusion status and analysis version, and a count reconciliation showing which 114 rows remain. (2) The standard error is \(3/\sqrt{3}=\sqrt{3}\approx1.732\) units. The record should preserve all replicate values and state independence, the common variance, the aggregation rule, units, and how the mean's uncertainty enters the observation model. (3) Half the LLOQ treats that substituted value as an exact observation; censoring instead contributes the probability of an observation falling below the LLOQ under the specified model. (4) The objective may include cross-source covariance, so the joint contribution is not a simple sum of separate terms; alternatively, it may include documented normalization terms, penalties, or other objective components.