ROC Curves in NCSS
NCSS contains procedures for single sample ROC curve analysis and for comparing two ROC curves. Use the links below to jump to a ROC Curve topic. To see how these tools can benefit you, we recommend you download and install the free trial of NCSS. Jump to:- Introduction and Discussion
- Other Technical Details
- One ROC Curve and Cutoff Analysis
- Comparing Two ROC Curves - Paired Design
- Comparing Two ROC Curves - Independent Groups Design
Introduction and Discussion
Although ROC curve analysis can be used for a variety of applications across a number of research fields, we will examine ROC curves through the lens of diagnostic testing. In a typical diagnostic test, each unit (e.g., individual or patient) is measured on some scale or given a score with the intent that the measurement or score will be useful in classifying the unit into one of two conditions (e.g., Positive / Negative, Yes / No, Diseased / Non-diseased). Based on a (hopefully large) number of individuals for which the score and condition is known, researchers may use ROC curve analysis to determine the ability of the score to classify or predict the condition. The analysis may also be used to determine the optimal cutoff value (optimal decision threshold). For a given cutoff value, a positive or negative diagnosis is made for each unit by comparing the measurement to the cutoff value. If the measurement is less (or greater, as the case may be) than the cutoff, the predicted condition is negative. Otherwise, the predicted condition is positive. However, the predicted condition doesn’t necessarily match the true condition of the experimental unit (patient). There are four possible outcomes: true positive, true negative, false positive, false negative.![ROC Curves Classification Table 1 ROC Curves Classification Table 1](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Classification-Table-1.png)
![ROC Curves Classification Table 2 ROC Curves Classification Table 2](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Classification-Table-2.png)
Rates Assuming a True Condition
The following rates assume one of the two true conditions.True Positive Rate (TPR) or Sensitivity = A / (A + C)
The true positive rate is the proportion of the units with a known positive condition for which the predicted condition is positive. This rate is often called the sensitivity, and constitutes the Y axis on the ROC curve.![True Positive Rate True Positive Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-TPR-3.png)
True Negative Rate (TNR) or Specificity = D / (B + D)
The true negative rate is the proportion of the units with a known negative condition for which the predicted condition is negative. This rate is often called the specificity. One minus this value constitutes the X axis on the ROC curve.![True Negative Rate True Negative Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-TNR-4.png)
False Negative Rate (FNR) or Miss Rate = C / (A + C)
The false negative rate is the proportion of the units with a known positive condition for which the predicted condition is negative. This rate is sometimes called the miss rate.![False Negative Rate False Negative Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-FNR-5.png)
False Positive Rate (FPR) or Fall-out = B / (B + D)
The false positive rate is the proportion of the units with a known negative condition for which the predicted condition is positive. This rate is sometimes called the fall-out, and constitutes the X axis on the ROC curve.![False Positive Rate False Positive Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-FPR-6.png)
Rates Assuming a Predicted Condition
The following rates assume one of the two predicted conditions.Positive Predictive Value (PPV) or Precision = A / (A + B)
The positive predictive value is the proportion of the units with a predicted positive condition for which the true condition is positive. This rate is sometimes called the precision.![Positive Predictive Value Positive Predictive Value](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-PPV-7.png)
Positive Predictive Value Adjusted for Known Prevalence
When the prevalence (or pre-test probability of a positive condition) is known for the experimental units, an adjusted formula for positive predictive value, based on the known prevalence value, can be used. Using Bayes theorem, adjusted values of PPV are calculated based on known prevalence values as follows:![Positive Predictive Value Adjusted for Known Prevalence Positive Predictive Value Adjusted for Known Prevalence](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Adjusted-PPV.png)
Negative Predictive Value (NPV) = D / (C + D)
The negative predictive value is the proportion of the units with a predicted negative condition for which the true condition is negative.![Negative Predictive Value Negative Predictive Value](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-NPV-8.png)
Negative Predictive Value Adjusted for Known Prevalence
When the prevalence (or pre-test probability of a positive condition) is known for the experimental units, an adjusted formula for negative predictive value, based on the known prevalence value, can be used. Using Bayes theorem, adjusted values of NPV are calculated based on known prevalence values as follows:![Negative Predictive Value Adjusted for Known Prevalence Negative Predictive Value Adjusted for Known Prevalence](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Adjusted-NPV.png)
False Omission Rate (FOR) = C / (C + D)
The false omission rate is the proportion of the units with a predicted negative condition for which the true condition is positive.![False Omission Rate False Omission Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-FOR-9.png)
False Discovery Rate (FDR) = B / (A + B)
The false discovery rate is the proportion of the units with a predicted positive condition for which the true condition is negative.![False Discovery Rate False Discovery Rate](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-FDR-10.png)
Whole Table Rates
The following rates are proportions based on all the units.Prevalence = (A + C) / (A + B + C + D)
The prevalence may be estimated from the table if all the units are randomly sampled from the population.![Prevalence Prevalence](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Prevalence-11.png)
Accuracy or Proportion Correctly Classified = (A + D) / (A + B + C + D)
The accuracy reflects the total proportion of units that are correctly predicted or classified.![Proportion Correctly Classified Accuracy](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Accuracy-12.png)
Proportion Incorrectly Classified = (B + C) / (A + B + C + D)
The proportion incorrectly classified reflects the total proportion of units that are incorrectly predicted or classified.![Proportion Incorrectly Classified Proportion Incorrectly Classified](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-PIC-13.png)
Confidence Intervals for Rates (Proportions)
Confidence limits for the above rates are calculated using the exact (Binomial distribution) methods described in the One Proportion chapter of the documentation.Other Diagnostic Accuracy Indices
Over the past several decades, a number of table summary indices have been considered, above those described above. Those available in NCSS are described below.Youden Index
Conceptually, the Youden index is the vertical distance between the 45 degree line and the point on the ROC curve. The formula for the Youden index is![Youden Index Youden Index](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Youden-14.png)
Sensitivity + Specificity
The addition of the sensitivity and the specificity gives essentially the same information as the Youden index, but may be slightly more intuitive for interpretation. Higher values of sensitivity plus specificity are better than lower values.Distance to Corner
The distance to the top-left corner of the ROC curve for each cutoff value is given by![ROC Distance to Corner ROC Distance to Corner](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Distance-to-Corner-15.png)
Positive Likelihood Ratio (LR+) = TPR / FPR
The positive likelihood ratio is the ratio of the true positive rate (sensitivity) to the false positive rate (1 – specificity). This likelihood ratio statistic measures the value of the test for increasing certainty about a positive diagnosis.![ROC Positive Likelihood Ratio ROC Positive Likelihood Ratio](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-LRPos-16.png)
Negative Likelihood Ratio (LR-) = FNR / TNR
The negative likelihood ratio is the ratio of the false negative rate to the true negative rate (specificity).![ROC Negative Likelihood Ratio ROC Negative Likelihood Ratio](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-LRNeg-17.png)
Diagnostic Odds Ratio (DOR) = LR+ / LR-
The diagnostic odds ratio is the ratio of the positive likelihood ratio to the negative likelihood ratio.![ROC Diagnostic Odds Ratio ROC Diagnostic Odds Ratio](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-DOR.png)
Cost
A cost approach is sometimes used when seeking to determine the optimal cutoff value. This approach is based on an analysis of the costs of the four possible outcomes of a diagnostic test: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). If the cost of each of these outcomes is known. The average overall cost C of performing a test at a given cutoff is given by![ROC Cost Index ROC Cost Index](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Cost-18.png)
ROC Curves
Each of the rates above are calculated for a given table, based on a single cutoff value. A receiver operating characteristic (ROC) curve plots the true positive rate (sensitivity) against the false positive rate (1 – specificity) for all possible cutoff values. Two types of ROC curves can be generated in NCSS: the empirical ROC curve and the binormal ROC curve.Empirical ROC Curve
The empirical ROC curve is the more common version of the ROC curve. The empirical ROC curve is a plot of the true positive rate versus the false positive rate for all possible cut-off values.![Empirical ROC Curve Empirical ROC Curve](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Empirical-19.png)
Binormal ROC Curve
The Binormal ROC curve is based on the assumption that the diagnostic test scores corresponding to the positive condition and the scores corresponding to the negative condition can each be represented by a Normal distribution. To estimate the Binormal ROC curve, the sample mean and sample standard deviation are estimated from the known positive group, and again for the known negative group. These sample means and sample standard deviations are used to specify two Normal distributions. The Binormal ROC curve is then generated from the two Normal distributions. When the two Normal distributions closely overlap, the Binormal ROC curve is closer to the 45 degree diagonal line. When the two Normal distributions overlap only in the tails, the Binormal ROC curve has a much greater distance from the 45 degree diagonal line.![Binormal ROC Curve Binormal ROC Curve](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Binormal-20.png)
Area under the ROC Curve (AUC)
The area under an ROC curve (AUC) is a popular measure of the accuracy of a diagnostic test. In general higher AUC values indicate better test performance. The possible values of AUC range from 0.5 (no diagnostic ability) to 1.0 (perfect diagnostic ability). The AUC has a physical interpretation. The AUC is the probability that the criterion value of an individual drawn at random from the population of those with a positive condition is larger than the criterion value of another individual drawn at random from the population of those where the condition is negative. Another interpretation of AUC is the average true positive rate (average sensitivity) across all possible false positive rates. Two methods are commonly used to estimate the AUC. One method is the empirical (nonparametric) method. This method has become popular because it does not make the strong normality assumptions that the Binormal method makes. The other method is the Binormal method. This method results in a smooth ROC curve from which the complete (and partial) AUC may be calculated.Other Technical Details
The discussion above gives a general overview of many of the diagnostic test summary statistics. If you would like to examine the formulas and technical details relating to a specific NCSS procedure, click on the corresponding '[Documentation PDF]' link under each heading to load the complete procedure documentation. There you will find formulas, references, discussions, and examples or tutorials describing the procedure in detail.One ROC Curve and Cutoff Analysis
[Documentation PDF]The One ROC Curve and Cutoff Analysis procedure generates empirical (nonparametric) and Binormal ROC curves. It also gives the area under the ROC curve (AUC), the corresponding confidence interval of AUC, and a statistical test to determine if AUC is greater than a specified value. Summary measures for a desired (user-specified) list of cutoff values are also available. Some of these measures include sensitivity, specificity, proportion correctly specified, table counts, positive predictive value, cost analysis, likelihood ratios, and the Youden index. These measure are often used to determine the optimal cutoff value (optimal decision threshold).
ROC Curve Analysis Example Dataset
![ROC Curves Example Data ROC Curves Data](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Example-Data.png)
Example Setup of the One ROC Curve and Cutoff Analysis Procedure
![ROC Curves Procedure Window ROC Curves Procedure Window](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Procedure-Window.png)
Example Output for the One ROC Curve and Cutoff Analysis Procedure
![One ROC Curve and Cutoff Analysis Output Window One ROC Curve and Cutoff Analysis Output Window](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Output-Window.png)
Comparing Two ROC Curves - Paired Design
[Documentation PDF]This procedure is used to compare two ROC curves for the paired sample case wherein each subject has a known condition value and test values (or scores) from two diagnostic tests. The test values are paired because they are measured on the same subject. In addition to producing a wide range of cutoff value summary rates for each criterion, this procedure produces difference tests, equivalence tests, non-inferiority tests, and confidence intervals for the difference in the area under the ROC curve. This procedure includes analyses for both empirical (nonparametric) and Binormal ROC curve estimation.
Comparing Two ROC Curves - Independent Groups Design
[Documentation PDF]This procedure is used to compare two ROC curves generated from data from two independent groups. In addition to producing a wide range of cutoff value summary rates for each group, this procedure produces difference tests, equivalence tests, non-inferiority tests, and confidence intervals for the difference in the area under the ROC curve. This procedure includes analyses for both empirical (nonparametric) and Binormal ROC curve estimation.
![Comparing Two ROC Curves Output Comparing Two ROC Curves Output](https://www.ncss.com/wp-content/uploads/2016/06/ROC-Curves-Output-Window-Groups.png)