Blinded Prediction Files And Audit Trails

Why a Prediction File Needs an Audit Trail

A PBPK prediction is useful for independent verification only if the record shows what was predicted before the corresponding validation observations were available to the modeling team. A table of predictions alone cannot establish that chronology. It might have been edited after the observations were seen, generated with a different model version, or joined to the validation records using information that was unavailable at prediction time.

The purpose of a blinded prediction file and its audit trail is therefore narrow and practical: preserve a reproducible, identifiable record of the model version, input data, execution, and predictions, together with evidence that the record was committed before unblinding. This complements prespecifying predictions; it does not replace a sound independent validation design.

“Blinded” must describe the actual information available to the people generating predictions, not merely a label on a file. The outcome data, outcome-derived summaries, and identifiers that reveal the answer must be withheld from those people until the package is locked. A designated custodian may hold the key linking blinded records to participants or studies, but the custodian’s role and access should be documented.

What the Locked Package Should Contain

The package should allow an independent reviewer to answer four questions: which model was used, what information went into the run, what predictions came out, and when was the package fixed relative to unblinding? A concise machine-readable manifest can record the answer while keeping the prediction table separate from the model files.

• Model identity: a unique version or release identifier; model structure and parameter-set identifiers; and references to any required configuration files. State explicitly whether parameters were frozen and whether any fitting or adjustment occurred for this validation exercise.
• Inputs: the exact input files or records used, their formats, units, record identifiers, and cryptographic hashes. Include protocol information required to run the model, such as dose, route, formulation, sampling schedule, and relevant population characteristics.
• Execution: analysis code and its version, the command or procedure used to run it, software and dependency versions, operating environment, random seeds if relevant, and execution start and completion times with time zone.
• Predictions: one row per prespecified prediction unit, with a stable blinded identifier, endpoint, time or summary measure, predicted value, unit, and any status needed to distinguish a valid result from a failed or missing output. Define the row structure and permitted values in advance.
• Lock and custody: a manifest, file hashes, a timestamped record of package creation, the identity or role of the person or service that committed it, and the planned date or procedure for unblinding. Record later access and any attempted or authorized changes.

Machine-readable means that the fields, types, units, identifiers, and allowed missing-value codes are defined, not that a file merely has a computer-readable extension. For example, a manifest might record the fields “package_id,” “model_version,” “input_sha256,” “code_commit,” “runtime_version,” “prediction_file_sha256,” “created_utc,” “timestamp_service,” and “unblinding_utc.” Document the schema and version it. Use stable identifiers rather than names or other direct identifiers in prediction files; keep any re-identification key separately under controlled access.

Hash values identify the bytes of particular files. If even one byte changes, a cryptographic hash will almost certainly change as well, provided the hash function is appropriate and the file representation is fixed. To make this check meaningful, specify the exact files hashed and their encoding or canonicalization rules. A hash does not prove that the contents are accurate, that the model is scientifically valid, or that the file existed at the claimed time. A trusted timestamp or controlled repository record can provide evidence of when a particular hash was committed; it still does not establish that the underlying prediction process was free from bias.

Chronology and Integrity Checks

A defensible lock requires more than putting a date in a spreadsheet. Keep timestamps in a stated time standard, preferably Coordinated Universal Time (UTC), and record the source of the timestamp. Retain the original committed package and its manifest in storage with access controls and version history; do not overwrite it with a “corrected” file. If an error requires a new package, give it a new identifier and preserve the old one and the reason for replacement.

Let \(t_L\) denote the time the package was committed and \(t_U\) the time outcomes were unblinded. The basic chronology check is:

That inequality is necessary, but it is not sufficient. The times must come from credible, auditable records; the committed hash must match the package reviewed; and the people generating predictions must not have had earlier access to outcomes through another route. If unblinding occurred before the lock, a later timestamp cannot restore the intended blinding.

A practical audit also checks that the prediction table conforms to its prespecified structure. If there are \(N\) blinded participants and \(K\) predictions per participant, the expected number of rows is \(N K\), provided every participant has all \(K\) required predictions. Check for missing and duplicate keys, unexpected endpoints, inconsistent units, and records not in the prespecified analysis set. These are data-integrity checks, not assessments of predictive performance.

Linking Predictions to Inputs and Verifying the Package

A prediction is auditable only if a reviewer can trace it to the intended blinded record and model run. Use a stable key that remains the same across the input file, prediction file, and manifest, while not exposing identity to the modeling team. Define the prediction unit precisely: for example, one participant-endpoint-time combination, not simply “one patient.” Record units explicitly, because a value without a unit can be misread even when its numeric value is unchanged.

Before lock, reconcile expected and observed output rows. Suppose a study has \(N=8\) blinded participants and \(K=3\) prespecified sampling-time predictions per participant. If all are required, the expected count is:

A count of 24 alone does not guarantee correct linkage: one missing row and one duplicate row would still give 24. Therefore, check uniqueness of the composite key, such as participant ID, endpoint, and time, as well as the full expected key set. Compare units and endpoint labels to the schema. Verify that every file hash in the manifest matches the file actually reviewed.

Reproducibility, Alternatives, and Limits

Re-running the recorded code can help establish that the inputs and specified execution produce the stored predictions. Exact equality may depend on software versions, numerical libraries, hardware, parallel execution, or random-number handling. Where exact reproducibility is not guaranteed, define the expected reproducibility check in advance—for example, exact equality for deterministic output or a documented numerical tolerance—and retain the original output as the authoritative locked prediction file. A tolerance should not be invented after observing a discrepancy.

A useful audit trail links files in a provenance chain: input files and model configuration lead to a recorded execution, which leads to a prediction file and manifest. Hashes can detect later byte changes in those files; a trusted repository timestamp can strengthen evidence of when a hash was committed. Neither alternative alone is enough. A spreadsheet date is easy to alter; a file hash without a trusted record does not establish when the file existed; and a timestamped hash does not reveal whether a person had already seen the outcomes.

The approach also has limits. It cannot establish that the PBPK model is biologically appropriate, that all relevant inputs were recorded, or that validation data are independent. It cannot prevent collusion or access outside the logged system. It cannot turn a post hoc prediction into a prospectively blinded one. Its value is that it makes key claims inspectable and helps detect inconsistencies, unauthorized changes, and ambiguous provenance.

Common Pitfalls

• Wrong reasoning: “The file is named final_predictions.csv, so it was finalized before unblinding.” Correction: Names are not chronology evidence. Check a credible timestamped commit and its hash against the exact file reviewed.
• Wrong reasoning: “The file hash matches, so the prediction process was unbiased.” Correction: A matching hash supports byte-level integrity. Review blinding, access, and custody records separately.
• Wrong reasoning: “The table has the expected number of rows, so all predictions are present.” Correction: Also check key uniqueness and coverage of every expected participant-endpoint-time combination.
• Wrong reasoning: “We can overwrite the output to fix a formatting error because the values are unchanged.” Correction: Preserve the committed artifact; create a versioned replacement, recompute hashes, and document the change and its timing.
• Wrong reasoning: “A reproducible run proves the model is valid.” Correction: Reproducibility concerns the ability to recover specified outputs from recorded inputs and code. It is distinct from independent predictive performance.