Why Blind the Prediction?
A model can appear to predict well if its predictions are adjusted after the validation outcomes are seen. Even an apparently modest change—shifting a curve, changing a parameter, or selecting a different output time—can use information from the very outcomes intended to test the model. A blinded prediction exercise prevents this by recording the predictions and the rules for judging them before the relevant outcome data are revealed to the model team.
The aim is not merely to hide a spreadsheet. The exercise must define what counts as a prediction, which outcomes will be compared with it, how discrepancies will be summarized, and what happens if data are missing or unusable. The locked predictions then serve as a fixed reference for assessment. Any subsequent model change can be studied separately, but it cannot alter the record of the original predictive performance.
Blinding concerns access to information, not the biological plausibility of a prediction. Modelers may know the protocol, dose, or assay conditions needed to make a forecast. They must not have access to the validation outcomes, or to outcome-derived summaries that would reveal them, before the prediction record is finalized. The division of responsibilities described in the earlier material on independent validation by a separate team can support this arrangement, but a separate team alone does not guarantee that predictions were locked before disclosure.
Specify the Exercise Before Generating Predictions
A useful protocol turns a general claim such as “the model predicts the biomarker response” into a checkable statement. It should state the population or experimental system, treatment and dosing regimen, observation times, endpoint and units, and the data processing that maps raw measurements to the quantity being predicted. It should also state whether the target is an individual measurement, a summary for a group, or another explicitly defined quantity. These targets are not interchangeable.
The protocol must distinguish information that is allowed as model input from information that would leak the target. For example, a baseline measurement may be a legitimate input if the prediction claim is conditional on that baseline and the same rule is applied for every prediction. A post-dose measurement from the validation period is not a legitimate input when it contains information about the outcome being predicted. This distinction should be decided in advance, not after seeing which inputs improve agreement.
Before outcomes are disclosed, define the comparison and scoring rules. For point predictions, state the error sign convention and which summaries will be reported. For uncertainty intervals, specify their nominal level and how an observation inside or outside an interval will be counted. If there are multiple endpoints or time points, state whether each is reported separately, how they are combined if at all, and how repeated measurements from the same unit will be handled. Do not pool raw errors in different units into one score without a justified, prespecified transformation.
A prediction record should be sufficiently complete to reproduce the comparison. It commonly includes a unique identifier for each target, the predicted value and units, any prediction interval, the model and parameter-set version, the input data and processing version, and the prediction-generation date. Preserve the executable code, configuration, and an immutable copy of the output. A timestamped, access-controlled file or a cryptographic hash can help demonstrate that the file did not change after locking; neither substitutes for documenting how the predictions were generated.
Specify the system, conditions, endpoint, units, and observation schedule, along with what information may be used to generate each prediction.
Choose the error summaries, interval rules, exclusions, and handling of missing or invalid observations before revealing outcomes.
Run the stated model and inputs, preserve the prediction file and computational record, and document when and by whom they were locked.
Apply the prespecified data-processing and matching rules, then report discrepancies against the locked values without altering them.
Define and Interpret Predictive Error
For target \(i\), let \(y_i\) be the observed value and \(\hat{y}_i\) the locked model prediction, both expressed in the same units. Define the signed prediction error as prediction minus observation:
With this convention, a positive error means the model overpredicted and a negative error means it underpredicted. The mean error describes average signed discrepancy, while the mean absolute error (MAE) describes average error magnitude without cancellation. The root mean squared error (RMSE) also measures discrepancy magnitude but gives relatively greater weight to larger errors because errors are squared.
These are descriptive summaries of the evaluated predictions. Mean error near zero does not establish that individual predictions are accurate: positive and negative errors can cancel. MAE and RMSE do not indicate the direction of error, so they should generally be read alongside the signed errors or mean error. All three retain the units of the endpoint.
A short identity explains why mean error and RMSE provide different information. Write \(b=\frac{1}{n}\sum_{i=1}^{n}e_i\), the mean error. Since \(\sum_{i=1}^{n}(e_i-b)=0\), expanding the squared deviations gives:
Rearranging shows that the mean squared error equals the squared mean error plus the average squared deviation of the errors around their mean:
Thus, a set of errors can have little average bias but substantial variation around that average. This is an exact decomposition of the observed errors in the exercise, not a claim that the finite set of errors estimates a particular population distribution.
Worked Example: Score Four Locked Concentration Predictions
In a hypothetical blinded exercise, a model predicts plasma concentration at a prespecified time for four validation units. The predictions are locked in ng/mL before the corresponding observations are revealed. The observed and predicted values are 8 and 10, 12 and 11, 18 and 20, and 24 and 21 ng/mL, respectively. Assume all four predictions match the intended endpoint and no measurement is missing.
The signed errors, using prediction minus observation, are \(2,-1,2,-3\) ng/mL. Therefore, the mean error is:
The mean absolute error is:
The root mean squared error is:
The mean error is zero because the signed errors cancel, not because every prediction is exact. The MAE of 2.00 ng/mL and RMSE of 2.12 ng/mL show nonzero discrepancies; the RMSE is slightly larger because it weights the 3 ng/mL error more heavily than the absolute-error average does. These summaries describe these four predictions only.
Check the Match and the Pattern of Errors
Scoring is meaningful only if each prediction is compared with the outcome it was intended to predict. Check identifiers, units, time windows, assay definitions, and data-processing rules before calculating error. A mismatch such as comparing a predicted trough concentration with an observed value taken at a different time is not a model error summary; it is a failure to align the target. If an observation is unavailable or invalid, apply the prespecified rule and report the number of predictions actually scored. Do not silently replace, omit, or redefine a target based on whether its error is inconvenient.
After matching, inspect errors against relevant conditions and time. A table of individual errors or a plot of observed versus predicted values can reveal outliers, consistent overprediction at a dose level, or errors that change over time. Such patterns can indicate that a single aggregate score conceals a consequential discrepancy. Interpret them in the context of the exercise design: a pattern is evidence about predictive performance in the tested conditions, not by itself proof of a particular biological mechanism.
If multiple time points come from the same experimental unit, they are not automatically independent observations. A simple average over every row can give more influence to units with more measurements, and treating each row as an independent replicate can exaggerate how much information the exercise contains. Prespecify whether scoring is per unit, per time point, or by another summary, and show the individual-level pattern where feasible. When endpoint scales differ, report endpoint-specific results or use a scale transformation justified and declared before outcome disclosure.
Worked Example: Assess Prediction-Interval Coverage
In a hypothetical exercise, four locked predictions are accompanied by nominal 80% prediction intervals, all in the endpoint’s units. The intervals and revealed observations are: \([8,12]\) with observation 10, \([12,16]\) with observation 14, \([14,18]\) with observation 16, and \([20,24]\) with observation 25. Assume the intervals and observations refer to the same target definition and units, and the protocol counts endpoints as included.
The first three observations lie within their intervals. The fourth, 25, is above its interval’s upper limit of 24. The observed coverage proportion is therefore:
Coverage is 75% in these four cases, and the protocol’s nominal interval level was 80%. This small descriptive result does not establish that the model’s intervals have 80% coverage in a broader setting. The interval for the fourth prediction remains \([20,24]\); it must not be widened after observing 25 and then reported as the original prediction interval.
Blinding, Diagnostics, and Alternatives
A blinded workflow can fail even if a file is eventually labeled “locked.” Potential leakage includes access to partial outcomes, preliminary plots, treatment-response summaries, or data from the same validation set that reveal its outcome pattern. Record who can access each data stream and when. Keep a dated audit trail of prediction generation, locking, outcome release, and scoring. If an unexpected data-processing issue arises after release, document it and apply the prespecified rule where possible; any justified deviation should be reported rather than hidden.
A particularly important failure mode is silently changing the model after seeing discrepancies. Re-estimating parameters, shifting the prediction curve, removing difficult cases, or choosing a different endpoint can improve apparent agreement, but the resulting fit no longer evaluates the locked predictions. If such a change is scientifically warranted, label it as a model update and evaluate it on data that were not used to make that update. Do not overwrite the original prediction file or combine updated predictions with the original blind score.
Worked Example: Separate Scoring from Recalibration
Suppose a hypothetical exercise has three locked predicted biomarker changes of 4, 6, and 8 percentage points, while the revealed observations are 6, 8, and 10 percentage points. The errors are \(4-6=-2\), \(6-8=-2\), and \(8-10=-2\) percentage points. The mean error is \((-2-2-2)/3=-2.00\) percentage points, indicating underprediction on average under the stated sign convention.
Adding 2 percentage points to every prediction would make the revised values 6, 8, and 10, giving zero error for these observations. But this adjustment is calculated from the revealed outcomes. It is a recalibration based on the evaluation data, not evidence that the original locked predictions were accurate. The valid blind score remains a mean error of \(-2.00\) percentage points. The adjusted model may be investigated separately, with its development and evaluation clearly distinguished.
A blinded exercise is one way to guard against outcome-informed changes. A conventional holdout can serve a related purpose if it is genuinely inaccessible during model development and its predictions are fixed before evaluation. A retrospective analysis in which the team has already seen the outcomes may still be informative, but it does not have the same protection against information leakage. Recalibration has a different purpose: it adapts a model using data and should be reported as model development or updating, not as the untouched predictive assessment.
| Approach | What it can answer | Key distinction |
|---|---|---|
| Blinded prediction exercise | How well did the prespecified, locked predictions match the revealed outcomes? | Predictions and scoring rules are fixed before outcome disclosure. |
| Holdout evaluation | How well do predictions perform on data withheld from model development? | Protection depends on preserving the holdout from information leakage and changes. |
| Post hoc recalibration | Can model predictions be adjusted to better fit available outcomes? | Using the revealed outcomes changes the predictions; the adjusted fit is not the original blind score. |
A blind score also has limits. Few targets produce uncertain, potentially unstable summaries; an apparently good average can conceal important individual errors. The exercise tests only its specified endpoints, conditions, measurement procedures, and input assumptions. It cannot establish performance outside that scope, and low error does not prove that the model’s mechanisms are uniquely correct. Interpret scores with the design, data quality, and intended prediction claim in view.
Common Mistakes
- Calling predictions locked when only the final report is locked. The correction is to preserve the actual prediction values and their computational provenance before outcome access.
- Using a zero mean error as proof of accuracy. Opposite-signed errors can cancel. Report magnitude summaries and inspect the individual errors.
- Changing exclusions or targets after seeing which predictions missed. Apply the prespecified matching and exclusion rules, and report any unavoidable deviations transparently.
- Revising predictions and still calling the result blind validation. Preserve the original score and label any outcome-informed adjustment as recalibration or model updating.
- Interpreting a small exercise as broad evidence. State how many targets were scored and the conditions they represent; do not generalize beyond the evidence.
Check Your Understanding
Answer each question using the definitions and rules in this tutorial. Brief solutions follow.
- A prediction is changed after its corresponding outcome is revealed, then scored against that outcome. Is this a blind assessment of the original prediction? Explain.
- For three targets, the locked predictions are 5, 8, and 11 mg/L, and the observations are 6, 7, and 12 mg/L. Calculate the signed errors, mean error, MAE, and RMSE.
- Why can a mean error of zero coexist with poor individual predictions?
- Four of five observations fall inside their prespecified prediction intervals. State the observed coverage proportion and one reason not to interpret it as proof of nominal coverage.
Solutions. (1) No. The changed value uses the revealed outcome, so it cannot represent the original locked prediction; retain and score the original value, and label any adjustment as recalibration. (2) The errors are \(-1,1,-1\) mg/L. Mean error is \((-1+1-1)/3=-0.333\) mg/L; MAE is \((1+1+1)/3=1.00\) mg/L; RMSE is \(\sqrt{(1+1+1)/3}=1.00\) mg/L. (3) Positive and negative errors can cancel in the signed average even when each error has substantial magnitude, so magnitude summaries are also needed. (4) Observed coverage is \(4/5=0.80\), or 80%. With only five observations, this descriptive proportion is too limited to establish coverage in a broader setting.