Discriminative Accuracy of Physician and Nurse Predictions for Survival and Functional Outcomes 6 Months After an ICU Admission

Detsky M. JAMA 2017; Published online May 21, 2017. doi:10.1001/jama.2017.4078

Clinical Question

  • What is the discriminative accuracy of physicians and nurses in predicting 6-month mortality and functional outcomes of critically ill patients?


  • Clinicians’ abilities to discriminate between patients who will or will not develop unfavourable outcomes, other than mortality, are important for several reasons:
    • knowledge of future function is important to ICU patients, their families, and clinicians
    • multidisciplinary family meetings commonly focus on function and quality of life
    • predictions of future function may influence clinician behaviour, as physicians are more likely to offer the withdrawal of life support when they believe the patient will experience future dysfunction


  • Prospective cohort study
  • Physicians’ and nurses were asked to individually predict 6 patient outcomes (included mortality, independent living, ambulation and cognition) in ICU patients. Each outcome required two responses:
    • Would the outcome be achieved? Yes or No
    • Confidence in each prediction using a 5-point Likert scale ranging from 1 (not confident at all) to 5 (very confident)
    • Predictions were completed within 24 hours of patient enrolment
    • Physicians were required to have been responsible for patient care for at least 2 calendar days; nurses were required to have one day of contact, owing to greater variability on their daily schedule and more frequent contact with patients on a given day
      • Predictions were made by 47 physicians and 128 nurses
  • For each outcome, 8 operating characteristics were calculated separately for physicians’ and nurses’ predictions: sensitivity, specificity, C (concordance) statistic, positive predictive value, negative predictive value, positive likelihood ratio (LR), negative LR, and diagnostic odds ratio
  • In-hospital mortality was determined by reviewing the electronic health record. If patients were not confirmed to be dead at 6 months, a trained research assistant unaware of the clinicians’ baseline predictions attempted to contact survivors or their surrogate via telephone or email to determine the patient’s vital status and function. Patients were considered lost to follow-up after 5 failed contact attempts
  • For some outcomes, clinicians were asked to make predictions on the assumption that the patient was alive at 6 months
  • A target sample size of 300 patients was chosen to provide 80% power to detect 95% CIs no wider than 20% around sensitivity and specificity if the operating characteristics were approximately 75%


  • 5 ICU’s (3 medical and 2 surgical) in 3 hospitals in the US
  • October 2013 – May 2014


  • Inclusion: Patients who spent at least 3 days in the ICU and received mechanical ventilation for more than 48 consecutive hours, vasoactive infusions for more than 24 consecutive hours, or both
  • Exclusion: none specifically, beyond not meeting inclusion criteria
  • 303 patients enrolled
  • Baseline characteristics: Median age 62; hospitalised in prior year 70%; preponderance of medical ICU patient’s 63%; admitting diagnosis: 27% respiratory failure, 22% sepsis; 18% post nonemergency surgery; 11% post emergency surgery


  • None


  • N/A

Outcome – overall predictions: Yes or No

  • In-hospital mortality
    • Positive LR: physicians 4.81; nurses 4.71
    • Negative LR: physicians 0.64; nurses 0.61

  • Mortality at 6 months: physicians predictions were statistically significantly better than nurses
    • Positive LR: physicians 5.91; nurses: 4.23
    • Negative LR: physicians 0.41; nurses 0.56
    • C statistic (the area under the receiver operator curve), 0.76 [95% CI 0.72-0.81] vs 0.69 [95% CI 0.64-0.74]; P=0.02


  • unable to return to original residence at 6 months
    • Positive LR: physicians 3.20; nurses: 2.06
    • Negative LR: physicians 0.49; nurses: 0.51
  • unable to toilet independently at 6 months
    • Positive LR: physicians 6.00; nurses: 2.61
    • Negative LR: physicians 0.51; nurses: 0.48
  • unable to ambulate up 10 stairs independently at 6 months
    • Positive LR: physicians 2.18; nurses 2.04
    • Negative LR: physicians 0.51; nurses 0.48
  • abnormal cognitive function (ability to “remember most things, think clearly, and solve day-to-day problems”) at 6 months
    • Positive LR: physicians 2.36; nurses 1.50
    • Negative LR: physicians 0.75; nurses 0.88

Outcome – When confident in each prediction

  • In-hospital mortality: when physicians and nurses predicted a patient would die during their hospital admission, there was a ‘modest shift’ in the probability of this occurring. There was a higher discriminatory accuracy when predictions were combined and when physicians and nurses were more confident in their prediction
    • Positive LR: physicians 7.32nurses 6.70
    • Negative LR: physicians 0.61; nurses 0.54
    • Concordant predictions: Positive LR 17.33
    • C statistic for predictions: Confident vs Non-confident
      • Physicians 0.68 vs 0.64; P=0.59
      • Nurses 0.71 vs 0.65; P=0.36
  • Outcomes at 6 months
    • Mortality: when physicians predicted a patient would be dead by 6 months, the likelihood ratio of this occurring was 33.0, which is a ‘conclusive change’ in the probability
      • Positive LR: physicians 33.00nurses 9.90
      • Negative LR: physicians 0.18; nurses 0.42
      • Concordant predictions: Positive LR 40.35
      • C statistic for predictions: Confident vs Non-confident
        • Physicians 0.90 vs 0.70; P=<0.001
        • Nurses 0.77 vs 0.63; P=0.006
    • unable to return to original residence
      • Positive LR: physicians 5.62; nurses 3.33
      • Negative LR: physicians 0.28; nurses 0.25
      • Concordant predictions: Positive LR 15.24
      • C statistic for predictions: Confident vs Non-confident
        • Physicians 0.81 vs 0.62; P=<0.001
        • Nurses 0.78 vs 0.54; P=<0.001
    • unable to toilet independently
      • Positive LR: physicians 22.94; nurses 3.28
      • Negative LR: physicians 0.30; nurses 0.40
      • Concordant predictions: Positive LR 15.75
      • C statistic for predictions: Confident vs Non-confident
        • Physicians 0.84 vs 0.58; P=0.004
        • Nurses 0.74 vs 0.66; P=0.44
    • unable to ambulate up 10 stairs independently
      • Positive LR: physicians 22.94; nurses 2.28
      • Negative LR: physicians 0.30; nurses 0.25
      • Concordant predictions: Positive LR 5.36
      • C statistic for predictions: Confident vs Non-confident
        • Physicians 0.76 vs 0.59; P=0.08
        • Nurses 0.74 vs 0.60; P=0.08
    • abnormal cognitive function
      • Positive LR: physicians 4.53; nurses 2.44
      • Negative LR: physicians 0.74; nurses 0.82
      • Concordant predictions: Positive LR 12.35
      • C statistic for predictions: Confident vs Non-confident
        • Physicians 0.62 vs 0.58; P=0.94
        • Nurses 0.58 vs 0.51; P=0.48
  • Compared with a predictive model including objective clinical variables, a model that also included physician and nurse predictions had significantly higher discriminative accuracy for in-hospital mortality, 6-month mortality, and return to original residence (P =<0 .01 for all)

More details about Likelihood Ratios can be found at the Centre for Evidence Based Medicine (CEBM) site

Authors’ Conclusions

  • ICU physicians and nurses’ discriminative accuracy in predicting 6-month outcomes of critically ill patients varied depending on the outcome being predicted and confidence of the predictors


  • The study addresses an important clinical question about the accuracy of clinicians outcome predictions
  • Phone interviewers used a standardised script
  • Secondary analyses were specified a priori
  • Outcome data was obtained at 6 months for 99% of enrolled patients


  • A high percentage of patients were unable to perform some of the outcome measures prior to hospital admission. 19% were unable to ambulate up 10 stairs; 12% were not toileting independently; 20% had abnormal cognition before hospitalisation. This will clearly influence the clinicians predictive accuracy
  • These data apply only to predictions made on ICU days 3 to 6
  • The confidence intervals surrounding the likelihood ratio’s are wide because of small sample sizes (43% of patients had died by 6 months)

The Bottom Line

  • When ICU physicians and nurses agreed in their predictions and both were confident, the discrimination was typically maximised. Combined with objective clinical variables and illness severity scores, this is likely to have an even higher discriminative accuracy for predicting long-term outcomes
  • I’m reassured about predicting mortality after I’ve cared for a patient for 48 hours but I will no longer make predictions for the family regarding cognition. I will continue to be cautious about making any prediction before 48 hours, but this study doesn’t address this early time period
  • Further research is needed to better understand and refine predictive outcome modelling

External Links


Summary author: Steve Mathieu
Summary date: 13th June 2017
Peer-review editor: Duncan Chambler

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.