Why Use a Limitations Register?
A report can list caveats and still leave a decision-maker unsure what to do. “The data may not be representative” does not say which predictions are affected, whether the model can be used for a particular decision, or what safeguard is needed. A limitations register turns a known weakness into a traceable decision statement: what is uncertain, what evidence supports that concern, which predictions it may affect, what uses remain reasonable, and what caveat or action follows.
The register is not a promise that every weakness can be fixed before a model is used. It is a structured record of what is known, what the limitation means for a specific use, and what should not be inferred. It can support a decision to proceed with restrictions, gather more evidence, revise the model, or pause a proposed use. This practice is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Build Each Entry Around a Decision
A useful entry is specific enough to guide an action. Record the limitation in plain language, but do not stop there. State what evidence identifies it, which part of the target population or workflow is affected, and how that may change the interpretation of predictions. Then state the permitted use and the practical caveat. If an action or review is needed, assign an owner and a trigger for revisiting the entry.
A compact register can use the following fields. The exact format may be a table, spreadsheet, or report appendix; consistency matters more than the software used.
| Field | What to record | Useful test |
|---|---|---|
| Limitation | A specific weakness, not a general warning. | Could a reader tell what is uncertain? |
| Evidence and scope | The observed facts, affected population, time period, or process. | Can the concern be traced to something known? |
| Affected predictions | The predictions, groups, or decision points where the weakness matters. | Is the impact bounded rather than implied to affect everything? |
| Decision consequence | How the weakness could change an interpretation or action. | Could it alter a decision, not just a technical statistic? |
| Permitted use | Uses that remain reasonable and any use that is restricted. | Does the entry say what users may do now? |
| Caveat and action | Plain-language qualification, safeguard, owner, and review trigger. | Is there a concrete next step or boundary? |
“Evidence and scope” should separate an observed fact from a concern about its implications. For example, “the historical records contain only web applications” is a statement about the data. “Predictions for phone applicants are not established by those records” is a limitation on interpretation. This distinction prevents a register from presenting speculation as fact, while still making the potential decision risk visible.
Quantify Scope When the Counts Support It
Where a limitation concerns coverage, a simple count can clarify its scale. Suppose the records divide into two non-overlapping, exhaustive groups: cases within the supported scope and cases outside it. If \(N\) is the total number of cases and \(n_{\text{outside}}\) is the number outside the supported scope, then the outside-scope fraction is:
This follows because each of the \(N\) cases belongs to exactly one of the two groups, and \(n_{\text{outside}}\) counts the cases whose predictions are not supported by the stated scope. The fraction describes how much of the specified population lies outside that scope. It does not estimate the model’s error rate for those cases. That distinction matters: a coverage count can justify a use restriction, but it cannot establish how inaccurate unsupported predictions will be.
Worked Example: A Model Trained on One Application Channel
Situation. In a hypothetical lending program, a model was developed using 2,400 historical web applications. A proposed deployment is expected to receive 10,000 applications: 40% through the web channel and 60% by phone. The model has not been assessed on phone applications. Build the relevant register entry and determine the scope of the proposed use.
Method. Separate the deployment population by channel, calculate the count outside the model’s observed channel, and link that scope to a use restriction.
Step 1: Calculate the expected channel counts. The expected web count is \(10{,}000 \times 0.40=4{,}000\). The expected phone count is \(10{,}000 \times 0.60=6{,}000\). These add to \(4{,}000+6{,}000=10{,}000\), as required.
Step 2: Calculate the outside-scope fraction. Phone applications are outside the observed channel, so the fraction is \(6{,}000/10{,}000=0.60\), or 60%. This calculation describes deployment scope, not a 60% error rate.
Step 3: State the limitation and consequence. The historical development records are web applications only; phone applications make up an expected 6,000 of 10,000 proposed cases. The model’s predictions for phone applications are not established by the available development experience. Treating those scores as equally supported could make a decision appear more evidence-based than it is.
Step 4: Record permitted use and caveat. A reasonable interim restriction is to use the model only for web applications, with the existing process used for phone applications. The caveat is: “This model has been developed using web applications and has not been assessed for phone applications; phone cases should follow the separate process.” The entry can be reconsidered after phone-channel evidence has been reviewed.
Check and trap avoided. The two channel counts sum to the proposed total, and the 60% figure is correctly described as the share outside scope. A common mistake is to report that “60% of predictions will be wrong.” The calculation provides no basis for that claim; it identifies where the evidence does not yet support the use.
Translate Uncertainty Into a Use Boundary
Not all limitations are about coverage. A target label may be provisional, or a predictor may be recorded in a way that does not support the intended interpretation. The register should identify the specific consequence for the decision. Avoid broad claims such as “the model is unreliable” unless there is evidence for that conclusion. Prefer a bounded statement: “The outcome status for this subset is provisional, so measured performance for this subset should not be treated as settled.”
A use boundary is not necessarily a ban. A model with uncertain outcomes might still help organize a review queue, while not being suitable as the sole basis for an adverse decision. The point is to state what the current evidence can support. The practical caveat should travel with the score or recommendation so that the restriction is not lost when the model is put into operational use.
Worked Example: Provisional Outcomes in a Review Queue
Situation. A hypothetical service team uses a model to identify accounts for follow-up. Among 800 accounts assigned to the high-priority queue, 160 have an outcome status still marked provisional; the remaining 640 have confirmed statuses. A manager asks whether the model has been shown to identify the outcome for all 800 accounts. Create a register entry.
Method. Quantify the share with provisional status, then distinguish a queue-prioritization use from a claim that the outcomes or model performance are settled.
Step 1: Check the counts. The groups total \(160+640=800\), matching the high-priority queue.
Step 2: Calculate the provisional share. The share is \(160/800=0.20\), or 20%. This means one in five accounts in this queue has a provisional status. It does not show whether the model is correct or incorrect on those accounts.
Step 3: Identify the affected interpretation. Any statement about observed outcomes or performance for all 800 accounts would treat the 160 provisional statuses as if they were settled. The limitation affects that interpretation; it does not, by itself, show that the model is unsuitable for arranging follow-up.
Step 4: Set a permitted use and caveat. The model may be used as one input to prioritize human follow-up, provided reviewers do not treat the score as a confirmed outcome or as the sole basis for a consequential decision. A suitable caveat is: “Twenty percent of accounts in this queue have provisional outcome statuses; conclusions about outcomes for the full queue are preliminary.” The register should identify who will revisit the entry when the statuses are resolved.
Check and trap avoided. The 20% is calculated from the 800 accounts in the stated queue, not from an unstated population. A common mistake is to call all 160 provisional cases failures, or to exclude them silently and describe a result as applying to all 800. Neither conclusion follows from the information given.
Record Operational Constraints as Limitations
A model can be statistically usable in one workflow and unusable in another if the required information is unavailable at the relevant decision time. The register should name the decision point, not merely say that a field is “sometimes missing.” In particular, a field found in a historical extract is not automatically available when a future prediction must be made. If the field is not available at that point, users need a restriction or a model designed and assessed for the intended workflow.
Worked Example: A Predictor Recorded After the Decision Point
Situation. A hypothetical claims team wants a model to prioritize cases at first notice. In a retrospective extract of 1,200 past cases, an inspection code is present for 1,020 cases and absent for 180. The code is entered after an inspection, which happens after first notice. The proposed model uses this code. Decide what belongs in the register.
Method. Calculate the historical presence rate, then compare when the field was recorded with when the prediction is needed.
Step 1: Check the extract. The counts reconcile: \(1{,}020+180=1{,}200\).
Step 2: Calculate the historical presence rate. The code is present in \(1{,}020/1{,}200=0.85\), or 85%, of the retrospective records. It is absent in \(180/1{,}200=0.15\), or 15%. These percentages describe the extract.
Step 3: Identify the decision-time mismatch. At first notice, the inspection has not yet happened, so the code is unavailable at the proposed prediction time. The historical 85% presence rate does not make the code available at first notice; its timing is the decisive limitation.
Step 4: State the restriction and action. The model should not be used at first notice with this predictor treated as available. The team could instead consider a use after inspection, or develop and assess a version that does not rely on the code before proposing first-notice use. The caveat should say: “The inspection code is recorded after inspection and is not available at first notice; this model is not supported for first-notice prioritization.”
Check and trap avoided. The presence and absence percentages sum to 100%, but the 85% figure is not evidence of availability at first notice. A common mistake is to fill the field using a default or assume its retrospective presence means it can be used operationally. That would not resolve the timing mismatch.
Prioritize, Phrase, and Maintain the Register
A register can become long, so distinguish limitations by the consequences they could have for the intended decision. Give priority to issues that affect a consequential decision, a large share of the intended use, or a use for which a practical safeguard is not yet available. This is a reasoned ordering, not a substitute for evidence or a precise numerical risk score. If two limitations interact, record that connection when it changes the permitted use.
Write entries so they remain useful outside the modeling team. A reader should be able to identify what is known, what is uncertain, and what action follows. Avoid false precision: do not assign an exact probability of harm when the available evidence cannot support one. Equally, avoid a caveat so general that it imposes no boundary at all.
Describe one specific limitation in neutral language, such as a channel not represented in the development records or a predictor unavailable at the intended decision time.
Record the relevant counts, process facts, or affected cases. State what the evidence does and does not establish.
Specify which predictions or interpretations may be affected and how the limitation could matter to the proposed action.
Say whether the model may be used with a safeguard, only for a narrower purpose, or not for the proposed decision on current evidence.
Write a practical sentence for users and identify the action, responsible owner, or new evidence that would prompt review.
A final check is traceability: every restriction should connect to a named limitation, and every material limitation should lead to a clear consequence or explain why it does not change current use. This helps avoid two opposing errors—ignoring a weakness because it is inconvenient, and overstating a weakness until no decision can be made. A well-maintained register supports proportionate decisions while keeping the boundaries of the evidence visible.
Check Your Understanding
For each hypothetical situation, identify the relevant limitation and state a defensible use boundary or caveat.
- A hypothetical model was developed on 3,000 online renewals. A proposed population is 5,000 renewals, of which 1,500 are by phone and have not been assessed. What fraction of the proposed population is outside the observed channel? (Answer: 1,500/5,000 = 30%; phone renewals are outside the assessed scope, not necessarily wrong predictions.)
- In a hypothetical priority queue of 600 cases, 90 have provisional outcomes and 510 have confirmed outcomes. What is the provisional share, and what should the caveat avoid claiming? (Answer: 90/600 = 15%; avoid presenting conclusions about all 600 outcomes as settled.)
- A hypothetical model is proposed for decisions at application submission, but one input is entered only after an eligibility interview. What is the central limitation and a suitable restriction? (Answer: The input is unavailable at submission; do not use the model for submission-time decisions with that input assumed available.)
- A hypothetical deployment expects 2,400 cases. Of these, 1,800 are within the model’s assessed scope and 600 are outside it. A manager says the model will be wrong on 25% of cases. Is that conclusion supported, and what should the register say? (Answer: No; 600/2,400 = 25% is the outside-scope fraction, not an error rate. Restrict or qualify use for those 600 cases.)
- A hypothetical report states, “Results may be affected by data quality,” but gives no affected cases or action. Name two changes that would make this entry useful. (Answer: Specify the observed data-quality issue and the affected predictions or population; then state a practical use restriction, safeguard, caveat, or follow-up action.)