Net reclassification improvement

Net reclassification improvement (NRI) is an index that attempts to quantify how well a new model reclassifies subjects - either appropriately or inappropriately - as compared to an old model. While c-statistics or AUC has been the standard metric for quantifying improvements over the last few decades, several studies have analyzed the limitations of this metric including lack of clinical relevance and difficulty in interpretation of small magnitude changes. This limitation can be best seen in the example of HDL and Framingham Risk Score (FRS). When models, with and without HDL, were analyzed with AUC regarding effect of HDL of modifying FRS, HDL was found not to have a statistical significant effect. However, when analyzed in terms of outcomes, HDL was found to be a significant predictor of heart disease and thus should affect FRS. To overcome this limitation the concept of reclassification, that is how well a new model correctly reclassifies cases, was introduced through the metric of NRI.

Basic Concept
NRI attempts to quantify how well a new model correctly reclassifies subjects. Typically this comparison is between an original model (e.g. hip fractures as a function age and sex) and a new model which is the original model plus one additional component (e.g. hip fractures as a function of age, sex, and a genetic or proteomic biomarker). NRI is composed of two components, subjects without events and subjects with events. Subject without events who are correctly reclassified lower risk are assigned a +1, as are subjects with an event who are correctly reclassified higher risk. Subjects without events who are incorrectly classified as higher risk are assigned a -1, as are subjects with events who are assigned a lower risk. Subjects not reassigned (i.e., those who remain in the same risk category) are assigned a 0. Sum the scores in each group and divide by the number of subjects in that group. The sum of these two values is the NRI.

Example
In a perfect test, all subjects with events would be classified as abnormal and all subjects without events would be classified as normal. Bold indicates subjects correctly classified by both tests. White indicates subjects incorrectly classified by both tests. Green indicates subjects correctly reclassified by test 2. Red indicates subjects incorrectly reclassified by test 2. NRIe = (4-2)/30 = 0.067. NRIne = (8-4)/70 = 0.057. NRI is the sum which is approximately 0.12.

Continuous Net Reclassification Improvement
Continuous net reclassification improvement is a category-less and more objective form of net reclassification improvement. Furthermore, continuous NRI is less affected by event rates.

Disconfirmation
NRI has been demonstrated to be invalid. Specifically, the NRI is likely to be positive even for uninformative markers. This is not the case for other metrics such as area-under-the-curve, Brier score or net benefit.

R Packages for Statistical Coding
PredictABEL: an R package for the assessment of risk prediction models.

survIDINRI: IDI and NRI for comparing competing risk prediction models with censored survival data.

nricens: calculate the net reclassification improvement (NRI) for risk prediction models with time to event and binary data.