ROC Curve Analysis Explained

Introduction

Many clinical and biomedical analyses involve a continuous measurement that must be used to distinguish between two outcome states.

Examples include a laboratory biomarker used to identify disease, a diagnostic score used to classify patients, a risk prediction model used to identify future events, or an imaging measurement used to distinguish responders from nonresponders.

The challenge is that a continuous marker does not usually produce a single yes-or-no classification automatically.

Instead, the analyst must choose a threshold.

Patients above the threshold might be classified as positive, while patients below it might be classified as negative.

Changing the threshold changes the number of true positives and false positives.

A receiver operating characteristic (ROC) curve summarizes this trade-off across possible thresholds.

What Is an ROC Curve?

Suppose a diagnostic biomarker produces a continuous value for every patient. The reference standard determines whether each patient truly has the disease.

Choose a threshold \(c\).

A simple rule might be:

• Marker value \(\ge c\): classify as positive
• Marker value \(< c\): classify as negative

For that threshold, four possible classification outcomes exist.

These four quantities form the foundation of ROC analysis.

The Confusion Matrix

A binary classification problem can be represented as:

where:

• TP = true positives
• FP = false positives
• FN = false negatives
• TN = true negatives

Sensitivity

Sensitivity measures the probability that a truly positive patient is classified as positive.

Sensitivity is also called the true-positive rate.

A highly sensitive diagnostic test misses relatively few patients who truly have the condition.

Specificity

Specificity measures the probability that a truly negative patient is correctly classified as negative.

A highly specific test produces relatively few false-positive results among patients who do not have the condition.

False-Positive Rate

The ROC curve does not plot specificity directly.

It plots the false-positive rate.

Equivalently:

Therefore, the standard ROC plot uses:

or:

The Basic ROC Plot

The following interactive example illustrates a simulated diagnostic marker. The diagonal line represents random discrimination.

How an ROC Curve Is Constructed

Suppose a biomarker has values ranging from low to high.

Start with an extremely high threshold.

Almost nobody will test positive.

As the threshold is gradually lowered, more patients are classified as positive.

This usually increases both:

• Sensitivity
• False-positive rate

Each threshold therefore corresponds to a point on the ROC curve.

Connecting the operating points produces the ROC curve.

What Happens at Extreme Thresholds?

Consider a marker where higher values indicate greater disease probability.

If the threshold is set above every observed marker value:

Nobody is classified as positive.

At the other extreme, if the threshold is set below every observed value:

Everybody is classified as positive.

These extreme operating points help define the ROC curve.

Why the Diagonal Line Represents Random Classification

The diagonal line runs from:

At any point on this line:

This corresponds to discrimination that is no better than random guessing, under the standard binary-classification interpretation.

The area under the diagonal is:

What Does AUC Mean?

The area under the ROC curve (AUC) summarizes discrimination across thresholds.

A useful probabilistic interpretation is:

when larger marker values correspond to greater likelihood of the positive condition and ties are handled appropriately.

In words, the AUC can be interpreted as the probability that a randomly chosen positive individual receives a higher marker value than a randomly chosen negative individual.

Interpreting Common AUC Values

These descriptive ranges should not be treated as universal regulatory or clinical standards.

Whether an AUC is clinically useful depends on the disease, intended use, consequences of classification errors, comparator, population, and clinical decision threshold.

AUC Is Not Accuracy

A common mistake is to describe the AUC as "the percentage of patients correctly classified."

That is generally incorrect.

AUC is a measure of discrimination across thresholds.

Accuracy at a particular threshold is:

Accuracy therefore depends on a selected threshold and can also be strongly affected by disease prevalence.

Sensitivity and Specificity Trade Off

Lowering a threshold usually makes it easier for a patient to test positive.

This often:

• Increases sensitivity
• Decreases specificity
• Increases the false-positive rate

Raising a threshold usually does the opposite.

ROC analysis makes this trade-off explicit.

An Example With 100 Patients

Suppose a study includes:

• 40 patients with disease
• 60 patients without disease

At a selected threshold, suppose:

• TP = 32
• FN = 8
• FP = 12
• TN = 48

Then sensitivity is:

or 80%.

Specificity is:

or 80%.

The false-positive rate is therefore:

or 20%.

The corresponding ROC point is:

Threshold Selection

An ROC curve describes many possible thresholds, but clinical decisions usually require one or more operating thresholds.

The appropriate threshold depends on the intended application.

For example, a screening test may prioritize sensitivity because missing a true case has serious consequences.

A confirmatory test may prioritize specificity because false-positive classification may lead to unnecessary invasive testing or treatment.

Youden's Index

One common threshold-selection criterion is Youden's index.

Equivalently:

The threshold maximizing \(J\) identifies the point with the largest vertical distance above the diagonal reference line.

Example of Youden's Index

Suppose three candidate thresholds produce:

Thresholds 3.0 and 4.0 have the same Youden index.

This illustrates an important point: a single optimization criterion does not necessarily identify a unique threshold.

Why Youden's Index Is Not Always the Right Choice

Youden's index gives equal weight to sensitivity and specificity.

Clinical decisions often do not have equal consequences for false negatives and false positives.

For example:

• A missed cancer diagnosis may be substantially more harmful than an unnecessary follow-up test.
• A false-positive treatment classification may have substantial toxicity or cost.
• A screening program may intentionally choose a threshold with very high sensitivity.

In these settings, a threshold selected by Youden's index may not be clinically optimal.

Cost-Based Threshold Selection

A decision-theoretic approach can explicitly incorporate the costs of classification errors.

For example, define:

• \(C_{FN}\) = cost of a false negative
• \(C_{FP}\) = cost of a false positive

Then the preferred threshold can depend on both error costs and disease prevalence.

This is often more clinically meaningful than simply maximizing Youden's index.

ROC Curve and Clinical Decision Thresholds

The ROC curve answers:

It does not by itself answer:

That second question requires clinical context.

Interactive Threshold Example

The following example illustrates a selected operating point on an ROC curve. The point corresponds to an illustrative threshold.

Specificity Versus False-Positive Rate

These two quantities contain the same information but are displayed differently.

For example:

ROC Curves and Direction of the Marker

The standard ROC setup assumes that larger marker values correspond to greater likelihood of the positive outcome.

But some biomarkers behave in the opposite direction.

For example, lower concentrations may indicate disease.

If the direction is reversed, the ROC curve can appear below the diagonal and produce an AUC below 0.5.

The analyst should therefore confirm the biological and statistical direction of the marker before interpreting the ROC curve.

Confidence Interval for AUC

An AUC estimated from a sample is subject to sampling variability.

Therefore, an AUC should generally be reported with a confidence interval when formal inference is required.

For example:

The confidence interval communicates uncertainty around the estimated discrimination.

A narrow interval generally indicates greater precision than a wide interval, although precision also depends on the study design and number of positive and negative observations.

Testing Whether AUC Differs From 0.5

One common inferential question is:

against:

A statistically significant difference from 0.5 indicates evidence of discrimination beyond the null reference under the specified analysis.

However, statistical significance does not automatically imply clinical usefulness.

Statistical Significance Versus Clinical Utility

Suppose a very large study estimates:

The confidence interval might exclude 0.5 because the sample is very large.

Nevertheless, an AUC of 0.57 may have limited clinical usefulness.

Conversely, a clinically promising marker in a small study may have an imprecise confidence interval.

Comparing Two ROC Curves

Suppose two biomarkers are evaluated in the same patients.

For example:

• Model A: standard clinical predictors
• Model B: standard predictors plus a new biomarker

The analyst may want to determine whether their AUCs differ.

A paired comparison is appropriate when both markers are evaluated on the same individuals.

Methods such as the DeLong approach can be used for comparing correlated ROC curves.

The key issue is that the ROC curves are not independent when they are calculated from the same patients.

Independent Versus Paired ROC Curves

Using an inappropriate independence assumption can produce incorrect standard errors and inference.

Does a Higher AUC Always Mean a Better Model?

Not necessarily.

Suppose:

and:

The difference may be small and statistically uncertain.

Even if statistically significant, the improvement may not alter clinical decisions.

A model with a slightly higher AUC may therefore have little practical advantage.

AUC Does Not Evaluate Calibration

ROC analysis primarily evaluates discrimination.

A risk model can distinguish high-risk from low-risk patients well while producing poorly calibrated absolute risk estimates.

For example, a model might rank patients correctly but systematically predict 10-year risks that are too high.

Therefore, prediction-model evaluation should generally consider both:

• Discrimination
• Calibration

and, where appropriate, clinical utility.

ROC Analysis Versus Precision-Recall Analysis

ROC curves are useful for binary discrimination, but they may not provide the most informative summary when the positive outcome is extremely rare.

In highly imbalanced datasets, precision-recall curves can provide useful additional information.

Precision is:

which is also called the positive predictive value in many diagnostic settings.

Precision-recall analysis therefore focuses directly on the positive classification burden.

Predictive Values Are Different From Sensitivity and Specificity

Sensitivity and specificity condition on the true disease state.

Predictive values condition on the test result.

Positive predictive value is:

Negative predictive value is:

Unlike sensitivity and specificity, predictive values depend strongly on disease prevalence.

The Prevalence Problem

Suppose a diagnostic test has:

and:

Those properties may remain the same in two populations, but the PPV can be very different if disease prevalence differs.

For example, a screening population may have a much lower disease prevalence than a specialty clinic.

Consequently:

Partial AUC

Sometimes the entire ROC curve is not equally relevant.

A clinical application may care specifically about a region with a very low false-positive rate.

For example, a screening program might only consider:

In that situation, a partial AUC (pAUC) can summarize performance over the prespecified region.

Partial AUC can be particularly useful when only a clinically relevant portion of the ROC curve matters.

Why Partial AUC Can Be More Informative

Imagine two markers:

• Marker A performs exceptionally well at very low false-positive rates.
• Marker B performs better only when false-positive rates become relatively high.

Their total AUCs may be similar.

But if the clinical application requires extremely high specificity, Marker A may be preferable.

ROC Curves and Clinical Trial Biomarkers

ROC analysis is common in biomarker development.

A biomarker may be evaluated for:

• Disease detection
• Diagnostic classification
• Patient selection
• Prognostic discrimination
• Prediction of treatment response
• Risk stratification

The intended use determines the appropriate reference standard, population, time horizon, and interpretation.

Diagnostic Versus Prognostic ROC Analysis

A diagnostic ROC analysis typically evaluates whether a marker distinguishes patients with and without a condition at a defined assessment.

A prognostic analysis may instead evaluate whether a marker distinguishes patients according to a future event.

For example:

• Death within 1 year
• Recurrence within 2 years
• Development of disease within 5 years

For time-to-event outcomes, ordinary binary ROC analysis may not be sufficient. Time-dependent ROC methods may be more appropriate.

Time-Dependent ROC Curves

When the outcome involves time to an event and censoring, a binary definition such as "event by 12 months" can discard information and introduce additional assumptions.

Time-dependent ROC methods can evaluate discrimination at a specified time point while accounting for censoring.

For example:

can represent discrimination at time \(t\).

The exact estimator should be selected according to the estimand, censoring assumptions, and study design.

Repeated Measurements

A common complication occurs when each patient has multiple biomarker measurements.

Standard ROC analysis generally assumes one relevant predictor measurement per subject for the specified analysis.

Repeated measurements create correlation.

The analysis may therefore require:

• A prespecified landmark time point
• A baseline marker
• Longitudinal modeling
• Time-dependent ROC methods
• Methods accounting for within-subject correlation

Simply treating repeated observations as independent patients is generally inappropriate.

ROC Curve Data Structure

A basic binary-outcome ROC analysis generally requires:

The exact variable names depend on the analysis dataset and programming standards.

Reference Standard Matters

ROC analysis does not establish whether the reference standard itself is correct.

The outcome classification is assumed to define the target condition for the analysis.

If the reference standard is imperfect, estimates of diagnostic performance can be biased.

Examples include:

• An imperfect laboratory reference
• Subjective clinical diagnosis
• Inaccurate adjudication
• Composite reference standards
• Verification bias

Verification Bias

Suppose only patients with a positive screening test receive the definitive reference test.

The reference outcome is then missing selectively among many patients.

This can distort sensitivity, specificity, and ROC estimates.

Diagnostic-study design should therefore specify how the reference standard is obtained and how incomplete verification is handled.

Spectrum Effects

Diagnostic performance can vary across patient populations.

A biomarker may perform extremely well when comparing:

• Healthy volunteers
• Patients with severe disease

but perform less well in the clinically relevant population containing borderline cases and related diseases.

This is one reason why external validation is important.

Internal Versus External Validation

AUC estimated in the development dataset can be optimistic, especially when a flexible model has been heavily tuned to the same data.

Overfitting and ROC Analysis

A model can achieve an impressive apparent AUC while performing poorly in new patients.

This can occur when the model is too flexible relative to the available sample size or when many candidate predictors are evaluated.

Potential safeguards include:

• Prespecified modeling strategies
• Internal validation
• Bootstrap validation
• Cross-validation
• External validation
• Appropriate regularization

ROC Curves From Prediction Models

Suppose a logistic regression model estimates:

The predicted probabilities can be evaluated using ROC analysis.

The ROC curve then asks how well the predicted score discriminates between positive and negative outcomes across probability thresholds.

Logistic Regression and ROC Analysis

model <- glm(
  outcome ~ age + biomarker + treatment,
  data = data,
  family = binomial()
)

data$predicted_risk <- predict(
  model,
  type = "response"
)

The predicted risk can then be used as the ROC predictor.

library(pROC)

roc_obj <- roc(
  response = data$outcome,
  predictor = data$predicted_risk
)

auc(roc_obj)

ROC Analysis in R

The pROC package provides a convenient framework for standard ROC analysis.

library(pROC)

roc_obj <- roc(
  response = data$outcome,
  predictor = data$marker,
  direction = ">"
)

auc(roc_obj)

The direction argument should be checked carefully against the actual marker and outcome coding.

Plotting an ROC Curve in R

plot(
  roc_obj,
  legacy.axes = TRUE,
  print.auc = TRUE,
  main = "ROC Curve"
)

The resulting figure displays the sensitivity versus false-positive-rate relationship.

Obtaining the AUC Confidence Interval

ci.auc(
  roc_obj,
  conf.level = 0.95
)

The confidence interval method should be selected consistently with the analysis plan and study design.

Finding a Threshold Using Youden's Index

coords(
  roc_obj,
  x = "best",
  best.method = "youden",
  ret = c(
    "threshold",
    "sensitivity",
    "specificity"
  )
)

This returns an operating point based on the selected optimization criterion.

Reporting a Prespecified Sensitivity

Sometimes the analysis objective is to identify the threshold achieving at least a specified sensitivity.

For example, a screening application may target:

The corresponding specificity can then be reported.

coords(
  roc_obj,
  x = 0.95,
  input = "sensitivity",
  ret = c(
    "threshold",
    "sensitivity",
    "specificity"
  )
)

The exact implementation should be checked against the software version and analysis specification.

Comparing Two ROC Curves in R

When two markers are measured on the same patients, a paired comparison can be performed.

roc_a <- roc(
  data$outcome,
  data$marker_a
)

roc_b <- roc(
  data$outcome,
  data$marker_b
)

roc.test(
  roc_a,
  roc_b,
  paired = TRUE
)

The paired option is important when the two markers are obtained from the same subjects.

AUC Confidence Intervals by Bootstrap

Bootstrap methods can also be used to estimate uncertainty.

roc_boot <- roc(
  data$outcome,
  data$marker
)

ci(
  roc_boot,
  of = "auc",
  method = "bootstrap",
  boot.n = 2000
)

The number of bootstrap replicates should be selected according to the analysis requirements and computational considerations.

Partial AUC in R

A partial AUC can be requested for a prespecified specificity or false-positive rate region.

roc_obj <- roc(
  data$outcome,
  data$marker
)

auc(
  roc_obj,
  partial.auc = c(
    1.0,
    0.9
  ),
  partial.auc.focus = "specificity"
)

The direction and interpretation of the partial interval should be verified carefully because software functions may express the region using specificity rather than false-positive rate.

Creating a Publication-Quality ROC Figure

A publication-quality figure should clearly identify:

• The outcome definition
• The predictor or biomarker
• The analysis population
• The AUC
• The AUC confidence interval where appropriate
• The reference diagonal
• The direction of the marker

If an operating threshold is shown, its sensitivity and specificity should also be clearly identified.

Example ROC Reporting Table

The inclusion of PPV and NPV is useful when the study population prevalence and intended clinical application make predictive values relevant.

ROC Curve Versus Confusion Matrix

The two tools operate at different levels.

An ROC curve therefore provides a global view of threshold-dependent performance, while the confusion matrix describes classification at a specific operating point.

ROC Analysis and Sample Size

ROC precision depends heavily on the number of positive and negative observations.

A study with very few positive cases may produce a highly uncertain AUC even if the point estimate appears impressive.

Similarly, few negative cases can make specificity estimates unstable.

Sample-size planning should therefore consider:

• Expected AUC
• Null AUC
• Desired confidence-interval precision
• Number of positive subjects
• Number of negative subjects
• Expected prevalence
• Study design

Confidence Intervals for Sensitivity and Specificity

Sensitivity and specificity at a selected threshold are also estimates and should generally have confidence intervals when inferential reporting is required.

For sensitivity:

The denominator is the number of reference-positive subjects.

For specificity:

The denominator is the number of reference-negative subjects.

Confidence intervals should use an appropriate binomial method rather than relying automatically on a simple normal approximation, especially with small samples or proportions near 0 or 1.

ROC Analysis and Missing Data

Missing predictor values or missing reference-standard outcomes can change the analysis population.

The analyst should determine:

• Which subjects are included
• Why observations are missing
• Whether missingness differs by outcome
• Whether complete-case analysis is appropriate
• Whether imputation is justified

Missingness should be handled according to the prespecified statistical analysis plan.

ROC Analysis and Verification Bias

If the reference standard is unavailable for a non-random subset of patients, the estimated ROC curve may be biased.

This is especially important in diagnostic studies where invasive or expensive reference standards are not applied uniformly.

Study design should therefore minimize differential verification whenever possible.

ROC Curves for Machine-Learning Models

ROC analysis is frequently used to compare machine-learning classifiers.

For example:

• Logistic regression
• Random forests
• Gradient boosting
• Support-vector machines
• Neural networks

Each model produces a score or predicted probability.

ROC analysis evaluates the ranking and discrimination of that score.

Training AUC Versus Validation AUC

External validation is particularly important when a prediction model is intended for use beyond the development institution or study population.

ROC Curves and Cutoff Optimization

Selecting a cutoff after inspecting the same data used to estimate performance can lead to optimistic estimates.

This is particularly relevant when many candidate thresholds are examined and the "best" threshold is selected retrospectively.

A robust workflow separates:

Common ROC Curve Mistakes

1. Confusing sensitivity with specificity. Sensitivity is the true-positive rate; specificity is the true-negative rate.
2. Forgetting that the ROC x-axis is 1 − specificity. Specificity itself is not the usual x-axis quantity.
3. Calling AUC "accuracy." AUC summarizes discrimination across thresholds and is not the percentage correctly classified at a single cutoff.
4. Choosing the Youden threshold automatically. The optimal clinical threshold depends on consequences and intended use.
5. Ignoring disease prevalence. Predictive values can change substantially when prevalence changes.
6. Comparing correlated ROC curves as independent. Two markers measured in the same subjects generally produce correlated ROC curves.
7. Reporting only the AUC. Threshold-specific sensitivity, specificity, and confidence intervals may also be clinically important.
8. Using an inappropriate reference standard. ROC analysis cannot compensate for systematic reference-standard problems.
9. Ignoring marker direction. AUC below 0.5 can result from reverse coding.
10. Reporting training-set AUC as validation performance. This can substantially overstate predictive performance.
11. Ignoring class imbalance. For rare outcomes, precision-recall analysis and predictive values may provide important complementary information.
12. Choosing thresholds after looking at validation results. This can introduce optimism and compromise the validity of the reported performance.

A Practical ROC Analysis Workflow

ROC Analysis Quality Control

Before finalizing an ROC analysis, verify:

• The positive outcome is coded correctly.
• The negative outcome is coded correctly.
• The biomarker direction is correct.
• Duplicate patients are not present.
• Repeated observations are handled appropriately.
• Missing outcomes are handled according to the analysis plan.
• The analysis population is correct.
• The AUC is calculated using the intended estimator.
• Confidence intervals use the specified method.
• The threshold is defined correctly.
• Sensitivity and specificity correspond to the reported threshold.
• Paired comparisons use a method appropriate for correlated observations.
• The ROC plot matches the underlying data.

Manual Validation Example

Consider the following simplified dataset:

At a threshold of 6.0:

• Patients 001–003 are positive.
• Patients 004–006 are negative.

The classification is therefore perfect for this small illustrative dataset:

The corresponding ROC curve would approach the ideal upper-left corner.

In a real study, however, perfect discrimination is uncommon and should prompt careful examination for sample size, spectrum restriction, leakage, overfitting, or other design issues.

The Ideal ROC Curve

A perfect classifier reaches:

That means:

• False-positive rate = 0%
• Sensitivity = 100%
• Specificity = 100%

The AUC is:

A perfect classifier is a theoretical reference. Real-world biomarkers usually involve overlap between positive and negative distributions.

Visualizing Marker Distributions

ROC curves summarize classification performance, but they do not directly show the underlying marker distributions.

It can therefore be useful to display:

• Density plots
• Box plots
• Violin plots
• Histograms

These displays can help explain why the ROC curve has its particular shape.

What the ROC Curve Is Really Measuring

A useful way to think about an ROC curve is as a summary of the overlap between the marker distributions for the two outcome classes.

If positive and negative subjects have strongly separated marker values, the ROC curve tends toward the upper-left corner.

If the distributions substantially overlap, the ROC curve tends toward the diagonal.

ROC Curve Shape

A curve that rises rapidly toward the upper-left corner indicates strong discrimination.

A curve close to the diagonal indicates weak discrimination.

The shape can also reveal whether discrimination is concentrated in a particular false-positive-rate region.

This is one reason why the total AUC should not always be the only reported ROC summary.

Example of Two Markers

Nested Prediction Models

A common application is comparing:

with:

The analyst may ask whether adding the biomarker improves discrimination.

An AUC comparison can address one aspect of this question.

However, model improvement can also involve:

• Calibration
• Reclassification
• Decision-curve utility
• Clinical consequences

Therefore, an AUC comparison should not automatically be interpreted as a complete assessment of incremental clinical value.

ROC Analysis and Clinical Utility

Suppose a new biomarker increases AUC from:

That may represent a statistically detectable improvement.

But clinicians may still use the same treatment decisions at all relevant thresholds.

A small AUC improvement can therefore have little practical impact.

Clinical utility methods, such as decision-curve analysis, can complement ROC analysis when the question concerns whether using the model improves decisions.

ROC Analysis in Regulatory and Clinical Research

In regulated clinical research, the analysis should be connected to a clearly defined estimand and intended use.

Important questions include:

• What is the target condition?
• What is the reference standard?
• What population is being evaluated?
• What is the prediction horizon?
• What constitutes a positive classification?
• Which threshold is clinically actionable?
• Is the marker being evaluated prospectively or retrospectively?
• Was the threshold prespecified?
• Is external validation available?

Suggested ROC Analysis Specification

Recommended Reporting Format

A concise ROC-analysis result might be reported as:

How to Report an ROC Curve Figure

A useful figure title might be:

A useful footnote might state:

What an ROC Curve Does Not Tell You

An ROC curve alone does not establish:

• Clinical benefit
• Patient outcomes
• Whether a treatment improves survival
• Whether a diagnostic test changes management
• Whether a threshold is clinically acceptable
• Whether the model is well calibrated
• Whether performance generalizes to another population
• Whether the reference standard is valid

These questions require additional analyses and clinical evidence.

ROC Analysis Checklist

ROC Curve Analysis: The Core Concepts

The Most Important Concept

The central idea behind ROC analysis is that a continuous predictor does not have one inherent classification threshold.

Every threshold produces a different balance between:

• True positives
• False positives
• True negatives
• False negatives

The ROC curve displays that trade-off across thresholds.

The AUC summarizes overall discrimination, while sensitivity and specificity describe performance at particular operating points.

Clinical threshold selection then adds another layer: the consequences of classification errors and the intended use of the test.

References

Hanley, J.A., & McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36.
DeLong, E.R., DeLong, D.M., & Clarke-Pearson, D.L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845.
Zou, K.H., O'Malley, A.J., & Mauri, L. (2007). Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation, 115(5), 654–657.
Pepe, M.S. (2003). The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford University Press.
Vickers, A.J., & Elkin, E.B. (2006). Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making, 26(6), 565–574.