“Neighborhood socioeconomic status (nSES) variables have been reported to improve predictions in some cases but not others. For instance, Molshatzki et al found that nSES variables improved the prediction of long-term mortality after myocardial infarction but Bhavsar et al found that an nSES index did not improve prediction of a variety of health care use measures within a 3-year window. Another study suggests that neighborhood-level indicators are not predictive above and beyond individual-level indicators. Whether these predictors are useful for risk stratification, and, if so, in what contexts remains unclear.
[..] We assess the addition of diverse nSES predictors to various risk models to predict 1-year health care use, hospitalization, and mortality in a large, heterogeneous population. However, while our prediction tasks and cohort are diverse, they are not exhaustive. We limit ourselves to 1-year prediction windows, but it is possible that nSES predictors may be more useful in longer term.
We included 3 types of predictors (claims, clinical, and nSES) in prediction models that use 3 complementary machine learning modeling approaches (penalized regression, random forests, and neural networks). The setting for our investigation is Kaiser Permanente Northern California (KPNC), an integrated health care delivery system with comprehensive information systems.
[..] The first set of predictors (administrative) includes demographic characteristics, categorical aggregated diagnosis codes (related clinical conditions [RCCs]), and a derived real-valued score predictive of future cost (diagnosis cost group). Both RCCs and diagnosis cost group scores are routinely assigned to all patients with a KPNC medical record number on a monthly basis. With respect to RCCs, any accrual of the relevant diagnosis during the last year of the preperiod month sets the value of that RCC to 1, otherwise it is set to 0. Diagnosis cost group scores are likewise assigned monthly based on data from the preceding 12 months.
The second set of predictors (EHR) includes a comorbidity score, Comorbidity Point Score, version 2 that is assigned on a monthly basis to all KPNC adult patients and in real time on hospital admission. This score is calculated based on administrative data, but is integrated with KPNC’s EHR for real-time use. In addition, we include a composite laboratory index, the abbreviated LAPS (Laboratory-based Acute Physiology Score, abLAPS), which is based on the lowest (indicating maximal physiologic derangement) value for 14 laboratory test results over the preceding month; this score is an outpatient modification of a previously reported hospital score and is assigned to all KPNC adults each month. We also included BMI and hemoglobin A1c (HbA1c). Last, we included an indicator of whether a patient had registered with the KPNC online patient portal.
The third set of predictors (nSES) includes 27 indicators of neighborhood socioeconomic status relating to transportation, housing, jobs, food access, crime, environment, and walkability. These variables were obtained from the US Department of Agriculture, US Environmental Protection Agency, California Environmental Protection Agency and the National Oceanic and Atmospheric Administration.
[..] We defined 7 dependent variables for the 1-year post-period for our analyses: 5 encounter counts (doctor office visits, virtual visits, emergency department [ED] visits, elective hospitalizations [those that did not begin in the ED], and nonelective hospitalizations [those that began in the ED]), cost, and mortality. [..] To make all analyses consistent with the binary analysis for mortality, encounter counts and cost were binarized at the 80th percentile of their distributions for the main analysis. This cutoff was chosen by clinical collaborators based on heuristic calculations of how many patients would be flagged as at risk each year at a given threshold and how many patients the health system would have the capacity to include.
[..] The areas under the receiver operator curve ranged from 0.71 for emergency department use (using the LASSO method and electronic health record predictors [area under the precision-recall curve, 0.36; Brier Score, .011; and McFadden pseudo-R2, 0.11]) to 0.94 for mortality (using the random forest method and electronic health record predictors [area under the precision-recall curve, 0.23; Brier Score, .006; and McFadden pseudo-R2, 0.37]). [..] the neural network model [..] had the best performance in all cases. [..]
The nSES predictors did not improve the models, regardless of which machine learning method was used. Our EHR predictors also do not meaningfully improve the models. Our results are consistent across each of our performance measures (area under the receiver operator curve, area under the precision-recall curve, McFadden pseudo-R2, and Brier Score). Although models perform quite differently for different outcomes (because different outcomes may be more difficult or easier to predict), the performance differences between predictor sets and methods are largely consistent within each outcome.
[..] Most studies of nSES only show that these variables are statistically associated with outcomes, not that they improve the ability to risk stratify patients. Predictors that show strong statistical associations with an outcome are not guaranteed to improve the prediction of that outcome relative to some baseline model.”
Full article, Schuler AS, O’Suilleabhain L, Rinetti-Vargas G et al. JAMA Network Open 2020.10.22