Performance of the early warning system score in predicting postoperative complications in older versus younger patients

Abstract

Background

Early warning system (EWS) scores are implemented on surgical wards to identify patients at high risk of postoperative clinical deterioration, but its predictive value in older patients is unclear. This study assessed the prognostic value of EWS scores to predict severe postoperative complications in older patients compared to younger patients.

Methods

This study utilized data from the TRACE study. EWS scores were routinely measured on postoperative days one (POD1) and three (POD3). The cohort was divided by age: < 70 years and ≥ 70 years. Performance measures of EWS scores on POD1 and POD3 were assessed to predict severe postoperative complications. Missed event rates (proportion of events not detected by the EWS threshold) and nonevent rates (proportion of EWS values above the threshold without an adverse event) were calculated.

Results

Among 4866 patients, 39.3% were ≥ 70 years old. Severe complications occurred in 6.1% of older compared to 5.8% of younger patients (P = 0.658). EWS scores on POD1 and POD3 did not differ between age groups. For severe complications, EWS showed moderate discrimination in both older (POD1: C-statistic 0.65 (95%CI 0.59–0.70); POD3: 0.63 (95%CI 0.57–0.69)) and younger patients (POD1: 0.68 (95%CI 0.65–0.72); POD3: 0.65 (95%CI 0.61–0.70)). Overall, calibration was good. For EWS score ≥ 3, the missed event rate was at least 69% and nonevent rate 75%.

Conclusions

Predicted performance of the EWS score was moderate among older and younger patients. A limitation of the EWS score is the high rate of missed events and nonevents.

Introduction

The European population has been ageing for decades. In 2021, 20% of the residents was 65 years or older, and this proportion will likely increase to approximately 30% in 2050.(European Commission et al. 2023) Innovations in perioperative care have reduced the threshold for surgery in older patients. Thus, the number of elderly undergoing major surgery will steadily increase in the forthcoming years. Older surgical patients are often frail and suffer from comorbidities, which increases the risk of postoperative complications, prolonged hospital stay and poor quality of life.(Richards et al. 2018; Watt et al. 2018; McIsaac et al. 2020; Han et al. 2019).

Prompt identification of clinical deterioration during the postoperative course aims to mitigate unexpected and preventable adverse events, also known as failure to rescue.(Schmid et al. 2007) Early warning systems (EWS) assign scores to vital parameters to identify patients at risk of clinical deterioration on the ward. These scores trigger appropriate interventions when certain thresholds are reached, for example the activation of a medical emergency team. EWS scores were originally derived in the medical population, but show good discrimination for adverse events in surgical populations as well.(Grooth et al. 2018).

Despite wide implementation of EWS scores on surgical wards, its prognostic value for adverse postoperative events in older patients is unclear. Elderly may exhibit a distinct vital sign pattern during the postoperative course because of, for example, frailty, comorbidities and polypharmacy.(Flint et al. 2023) This may impede the ability of EWS scores to accurately predict postoperative deterioration. This study assessed the prognostic value of routinely measured EWS scores to predict severe complications in older patients compared to younger patients. We hypothesize that the predictive accuracy of the EWS score in older patients is worse compared to younger patients.

Methods

Study design and participants

This study utilized data from the TRACE (routine posTsuRgical Anesthesia visit to improve patient outComE) study, of which the study protocol and results has been previously published.(Smit-Fun et al. 2018; The TRACE Study Investigators Group 2023) In short, TRACE was a prospective, multicenter, stepped-wedge, cluster-randomized interventional study conducted between 2016 and 2019 across nine academic and non-academic hospitals in the Netherlands. TRACE investigated the impact of a standardized, EWS score based, postoperative visit by an anesthesiologist on postoperative day one (POD1) and postoperative day three (POD3) on the occurrence of postoperative complications and mortality. Patients in the control group received standard care, which included EWS score measurements and subsequent actions according to the hospitals’ protocol. Because no substantial effect of the TRACE intervention on study outcomes was demonstrated (except for a small effect on renal complications), patients from both the control and intervention group were included in this study.(The TRACE Study Investigators Group 2023) Ethical approval was obtained from the Human Subjects Committee of Amsterdam UMC, location VUmc Amsterdam (number NL56004.029.16, 29–06–2016) and TRACE was registered at the Netherlands Trial Register (NTR5506). The Clinical Research Unit of the Amsterdam UMC monitored patient inclusion and data registration, with all participants providing informed consent. This study followed the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement.(Collins et al. 2015).

Study data and variables

For all participants enrolled in the TRACE study, vital parameters (including respiratory rate (RR), heart rate (HR), systolic blood pressure (SBP), oxygen therapy, oxygen saturation, temperature, level of consciousness and urine production) and the corresponding EWS score were recorded on POD1 and POD3. In cases of multiple EWS scores on the same postoperative day, the first recorded score of that day was used for this study. For participants with a missing EWS score due to one missing vital parameter, the respective missing vital parameter was scored as normal (zero points). Study participants with two or more missing vital parameters, or with missing data regarding postoperative complications, were excluded. Appendix A presents the EWS score used in the TRACE study.

Outcomes

The primary study endpoint was a severe in-hospital postoperative complication occurring until postoperative day thirty, defined as Clavien-Dindo grade III or higher.(Dindo et al. 2004) The secondary study endpoint was in-hospital mortality.

Statistical analysis

The sample size was based on data availability of the TRACE study. The cohort was divided based on age: < 70 years and ≥ 70 years. Continuous variables were presented as mean ± standard deviation (SD) or median with interquartile range (IQR) in case of a non-normal distribution. Categorical variables were described using frequencies and percentages. Differences between groups were tested by an independent samples t-test or Mann–Whitney U test for continuous variables, and by a Pearson-Chi-Square test or Fisher Exact test for categorical variables. Distribution of vital parameters on POD1 and POD3 were calculated. Discrimination of the EWS score for severe complication and in-hospital mortality was assessed with the receiver operating characteristic (ROC) curves with corresponding C-statistics and 95% confidence intervals (CI). Calibration was assessed by construction of calibration plots for severe complication. To evaluate clinical relevance of the EWS score sensitivity, specificity, negative predictive values (NPV) and positive predictive values (PPV) were calculated for several thresholds of the EWS score. Furthermore, we computed the missed event rate (i.e. proportion of events that were not detected by the EWS score threshold) and nonevent rate (i.e. proportion of EWS values above the threshold without an adverse event).(Grooth et al. 2018) All analyses were performed using SPSS Statistics for Windows, version 28.0 (Armonk, NY: IBM Corp, USA), except for the calibration plots which were constructed using R, version 4.3.0 – © 2020–10-10, R, Inc., for Windows. Figures were created using GraphPad Prism version 9.6.1 for Windows (GraphPad Software, Boston, Massachusetts, USA).

Results

Study population

The TRACE study cohort consisted of 5190 patients after excluding drop-outs. In 405 (7.8%) study participants, one vital parameter of the EWS score was missing and was considered normal. 324 (6.2%) study participants were excluded because of missing data for multiple vital parameters or postoperative complications. In total 4866 patients were included in the POD1 analysis, and 2582 patients in the POD3 analysis because of discharge before POD3 (Appendix B). 39.3% of the study participants (N = 1910) was 70 years or older (Table 1). The proportion of patients with ASA class III or IV was higher in older patients. Older patients underwent high-risk surgery, less frequently (N = 658, 34.5%) compared to younger patients (N = 1378, 46.6%) (Appendix C). The incidence of severe postoperative complications was similar in older and younger patients (6.1% versus 5.8% respectively, P = 0.658), however, the proportion of patients with one or more postoperative complications was higher in older patients (29.8% versus 24.8%, P < 0.001) (Table 2). Eight (0.4%) older patients died during hospital stay compared to six (0.2%) younger patients (P = 0.170).

Table 1 Study characteristics

Table 2 Primary and secondary outcome values

Postoperative EWS score in older and younger patients

Overall, the majority of patients had an EWS score < 3 on POD1 (N = 4301, 88.4%) and POD3 (N = 2125, 82.3%), which was not different between older and younger patients (Appendix D). In Appendix E the distributions of six vital parameters in both groups are shown. Older patients showed higher RRs (P = 0.034), lower oxygen saturations (P < 0.001) and less urine output (P = 0.026) on POD1, and a lower level of consciousness on POD3 (P = 0.007) (Appendix F). The remaining vital parameters were similar among older and younger patients.

Predictive performance of EWS score for postoperative complications

The EWS score on POD1 and POD3 showed moderate discrimination for severe postoperative complications in older patients (C-statistic 0.65 (95% CI 0.59—0.70) and 0.63 (95% CI 0.57 – 0.69) respectively) and younger patients (0.68 (95% CI 0.65 – 0.72) and 0.65 (95% CI 0.61 – 0.70), respectively (Table 3). Model performance was better for in-hospital mortality, however, the incidence was low (< 0.5%). Overall calibration was good, despite that reliable calibration for higher risk groups could not be achieved due to limited variability in EWS scores (i.e. a majority of patients had EWS scores 0 or 1) (Fig. 1). Nevertheless, for patients at high risk for severe complications the model tended to underestimate risk in older patients, and overestimated risk in younger patients.

Table 3 Discrimination for total EWS score for outcome values severe complication and in-hospital mortality

Fig. 1

Calibration plots for severe complication on (A) POD1 and (B) POD3. A1) Patients aged < 70 years. A2) Patients aged ≥ 70 years. B1) Patients aged < 70 years. B2) Patients aged ≥ 70 years. Abbreviations: POD1 = postoperative day 1, POD3 = postoperative day 3

The performance of the EWS score in terms of sensitivity and specificity was comparable between older and younger patients (Appendices G.1 and G.2). A threshold of EWS score ≥ 3 resulted in a sensitivity of at least 26% and a specificity of at least 88% on POD1, and a sensitivity of at least 28% and a specificity of at least 87% on POD3. In addition, the NPV and PPV rates were similar among younger and older patients, except for higher PPV values that were observed for EWS scores ≥ 5 in older patients (Fig. 2). Using a threshold of EWS score ≥ 3, the missed event rate was high, meaning that at least 69% of patients with a severe complication had an EWS score < 3, and were therefore not detected by the EWS. In addition, the nonevent rates were high, which means that for patients with an EWS score ≥ 3 at least 75% did not develop a severe complication. When raising the EWS score threshold, the nonevent rates generally decreased, with a lowest measured rate of 42% on POD3 for patients aged ≥ 70 years and an EWS score threshold ≥ 6. However, the missed event rates simultaneously increased to at least 90%.

Fig. 2

NPV and PPV of different EWS scores for the prediction of severe complication on (A) POD1 and (B) POD3. Abbreviations: NPV = negative predictive value, PPV = positive predictive value, POD1 = postoperative day 1, POD3 = postoperative day 3

Discussion

In a large prospective and heterogenous cohort of 4866 surgical patients, this study assessed the prognostic value of the EWS score to predict severe postoperative complications in older versus younger patients. We found moderate discrimination and good calibration in both older and younger patients. An important limitation of the EWS score in this study was the high rate of missed events and nonevents.

In literature, good to excellent performance of the EWS score for the discrimination of postoperative adverse events is reported.(Grooth et al. 2018; Bartkowiak et al. 2019; Kellett and Kim 2012; Kovacs et al. 2016; Smith et al. 2012; Hollis et al. 2016) The studies varied in the time between assessment of the EWS score and the occurrence of the adverse event. The intervals ranged from measurement of the EWS score 24 h before the adverse event to assessment of the EWS score at the time of hospital admission. Due to the design of the TRACE study, only EWS scores from POD1 and POD3 were available for our analysis and the timing of the occurrence of a postoperative complication was unrecorded, and could potentially have been on postoperative days five to seven, or even at postoperative day thirty. It is studied that the EWS score gradually increases the last 20 h before the adverse event.(Zografakis-Sfakianakis et al. 2018) Therefore, it is plausible that an increase in the time interval between the EWS score measurement and a postoperative complication worsened the performance of the EWS score in our study. Nevertheless, it is still valuable to examine the performance of routine assessment of EWS scores in the early postoperative period, as the goal is to predict patient deterioration as early as possible and most complications arise in the first days after surgery.(Preckel et al. 2020; Thompson, et al. 2003)

Overall, good calibration was observed among older and younger patients in low risk groups. High risk groups were underrepresented in our cohort, however, the model tended to slightly underestimate the risk of severe complications in older patients. In older patients the incidence of postoperative complications was higher compared to younger patients, but the number of severe complications was similar. This is probably because older patients underwent high-risk surgical procedures less frequently than younger patients. No differences were observed for in-hospital mortality, which was lower compared to those reported in the original TRACE study, due to the exclusion of study participants with missing information, as explained in the methods section.(The TRACE Study Investigators Group 2023) In addition, no differences in total EWS score between older and younger patients on both postoperative days were observed. However, a distinct difference in distribution of four vital parameters was detected between older and younger patients, of which RR and oxygen saturation deviated the most.

The performance of the EWS score is usually described in terms of discrimination, yet the practical usefulness of this measure is challenging.(Grooth et al. 2018) The terms missed event rate and nonevent rate have more clinical value because they illustrate how many actual adverse events are not detected by the EWS score, and how many EWS alarms are not followed by an adverse event. In this study the threshold score of ≥ 3 points showed unacceptable high rates of at least 69% and 75% respectively. The high missed event rates and nonevent rates are partly explained by the design of this study. When the EWS score is measured closer to the event's occurrence, as happens in practice on the ward according to standard care, these rates are expected to be lower. In addition, the high nonevent rates may be biased due to interventions in patients with high EWS scores early in the postoperative process, thereby preventing future complications. Nevertheless, the results of a review still show a wide range in missed events rates between 19% and 69% and nonevent rates ranged between 72% and 99% for the EWS score.(Grooth et al. 2018) This may be explained by intermittent measurements of the EWS score, which can result in deterioration of a patient being unnoticed. Continuous remote monitoring, whereby sensors measure vital signs more frequently, could prevent this.(Weenk et al. 2019) Recent studies on remote monitoring systems showed improved patient outcomes in postoperative patients.(Posthuma and Preckel 2023)

This study has some limitations. First, the unknown time between the EWS score and the occurrence of a severe postoperative complication may have negatively impacted the predictive performance of the EWS score. Second, to increase the power of this study, the intervention cohort of the TRACE study was included. The TRACE intervention did not reduce postoperative complications, except for a lower incidence of any renal complication in the intervention group (1% versus 1.7%, P = 0.014).(The TRACE Study Investigators Group 2023) We do not believe that this small difference biased our results, which is supported by a sensitivity analysis conducted in the control group and demonstrated equal discrimination compared to the total patient cohort. In this study, we defined older patients as ≥ 70 years old, given improvements in health among older patients and increased life expectancy, instead of the gerontological definition of elderly being ≥ 65 years old. Strengths of this study are its prospective design and heterogeneous patient group. Additionally, this study conducted a calibration analysis and focused on older surgical patients.

Conclusion

The predictive performance of the EWS score among older and younger patients was moderate. A shortcoming of the intermittent EWS score measurement in this study is the high missed event rate and nonevent rate. Future studies should demonstrate whether remote monitoring, guiding appropriate and timely interventions, improves postoperative patient outcome.