Online haemodiafiltration and all-cause mortality: how fragile are the results of the studies published so far? (2024)

Associated Data

Data Availability Statement

To the Editor,

Cardiovascular disease is a major cause of morbidity in patients on dialysis and accounts for almost 50% of deaths in these patients [1]. Although this high mortality is attributable to many reasons, the limited clearance of potentially harmful middle molecules through standard haemodialysis (HD) may be a contributor. Online haemodiafiltration (HDF) augments the clearance of middle molecular weight uraemic toxins through convective clearance while standard HD treatment is primarily based on diffusion [1].

In order to assess the potential superiority of online HDF versus HD on hard outcomes, such as all-cause mortality, we conducted a meta-analysis. Five randomized control trials (RCTs) were identified through a systematic review of three electronic databases (PubMed, Scopus, Web of Science) from inception to 1 October 2023 [2–6]. The main data of these RCTs are summarized in Table1. The five RCTs enrolled 4143 adult patients on HD. Of these, 2078 were randomly assigned to online HDF and 2065 to high- or low-flux HD. The mean follow-up period was 2.3years [standard deviation (SD) 0.6]. More than half of the patients were male (62.8%), 31.3% had diabetes mellitus and 83.7% had an arteriovenous fistula. An overall estimate of the effect size [expressed as the odds ratio (OR) given a similar length of follow-up across the included RCTs] was calculated using both a common effects model (Mantel–Haenszel method) [7] and a random effects model (DerSimonian–Laird method) [8] (Fig.1). Statistical heterogeneity was assessed by the I² statistic [9]. Given the small number of RCTs included in this meta-analysis, we did not evaluate publication bias using a funnel plot [7, 10]. All statistical tests were two sided and used a significance level of P<.05. We used Stata version 17.0 (StataCorp, College Station, TX, USA) and its meta-analysis package and R version 4.2.2 (R Foundation for Statistical Computing, Vienna, Austria) for all statistical analyses.

A statistically significant reduction in all-cause mortality of the HDF group when compared with the HD group was shown [random effects OR 0.76 (95% confidence interval 0.65–0.88); I²=0%] (Fig.1).

In 2014, Walsh etal. [11] introduced an ancillary tool that quantified the fragility (or robustness) of findings that report statistically significant P-values: they called this the Fragility Index (FI). The FI is a crucial statistical metric, easy to interpret, primarily employed to assess the robustness of research findings for dichotomous outcomes. It represents the minimum number of participants whose status needs to change from an ‘event’ to a ‘non-event’ (or vice versa), thus the results switch from being statistically significant to non-significant [11]. Although there is not an established cut-off, a higher FI score can be considered indicative of more robust results [12]. To mitigate the potential effect of the sample size on the FI, the Fragility Quotient (FQ) has been proposed and is calculated as follows: FQ=FI/sample size [13].

There is no consensus on a cut-off value to indicate whether a study is fragile or robust. However, empiric observation suggests that an FQ ≤0.03 raises a concern about the fragility of the findings [13]. Based on this background, we calculated the FI and FQ of the RCTs included in the meta-analysis. Table1 reports the FI and FQ for the outcome ‘all-cause mortality’ for the two RCTs in which the results were statistically significant [3, 6]. Both scores of the study by Maduell etal. were relatively low (FI=13, FQ=0.014) [3]. Those of the CONVINCE trial (NTR7138), recently published in the New England Journal of Medicine, were also low (FI=3, FQ=0.002) [6]. This means that, for CONVINCE, just three participants, i.e. 0.2% of the study population, would be required to change the results from statistically significant to non-significant. An FI was also calculated for the outcome ‘all-cause mortality’ for the five RCTs pooled together [14]: it was 44. This high value of FI underscores the robustness of the collective data of the RCTs. When normalized by the FQ (0.01), it appears to be influenced by the large sample size of the meta-analysis.

It must be underlined that the FI has some mathematical limitations [15, 16], including its dependence on the underlying risk, sample size, binary outcomes and use of Fisher’s exact test. These factors could partly influence accurate calculation of the FI for time-to-event data [17]. In order to address this limit, a survival-inferred fragility index (SIFI) was recently developed [17]. Specifically, the SIFI is defined as the minimum number of reassignments of the best survivors (defined as the patients with the longest follow-up time, regardless of having an event or being censored; the worst survivors were defined as the patients with the earliest event) from the experimental group to the control group, resulting in a loss of significance (in this specific case, defined as α=0.05 using the logrank test) [17]. Thus the SIFI can provide a measure that both accounts for events over time and retains all the advantages of the original FI. To calculate the SIFI, we first digitized the Kaplan–Meier curves in each eligible study using Digitizelt software (http://www.digitizeit.de/). Subsequently, with a specific code written for R software (https://github.com/davidbomze/SIFI), we calculated the SIFI. As reported in Table1, the SIFI in the CONVINCE trial was 18 [6], whereas the SIFI in the study by Maduell etal. [3] was 81. In other words, the statistical significance of the CONVINCE trial might be lost with a change in assignment of 18 patients, representing only 1.3% of the entire population included in the RCT.

It is worth noting that Kampman etal. [18] recently published a study aimed at determining whether the statistical fragility of RCTs in high-impact journals has improved over the last decade. The FI was calculated for all eligible RCTs published from 2014 to 2021 in the New England Journal of Medicine, The Lancet, the Journal of the American Medical Association, the British Medical Journal and the Annals of Internal Medicine. Of 2544 screened RCTs, 643 were eligible for the FI analysis. These RCTs had a median sample size of 625, a median FI of 12 and a median FQ of 0.015. This represents an improvement in robustness when compared with the median FI of 8 of RCTs published a decade earlier in the same five journals (P<.001) [11]. The study's conclusions indicate that over the last decade, the median statistical robustness of RCTs published in high-impact journals has improved. Yet the unchanged lower bound of the interquartile range reveals that statistical significance in 25% of trials is still dependent on three or fewer events [18]. Surprisingly, despite the popularity of the FI as a review subject, Kampman etal. [18] found only one trial in their database of 2544 RCTs in high-impact journals that mentioned this metric.

Regarding the SIFI, Bomze etal. [17] found that 45 phase 3 RCTs evaluating immune checkpoint inhibitors had a low SIFI, for overall survival, resulting in a loss of statistical significance with the reassignment of a small fraction of the entire sample size.

In conclusion, our study shows that the results of the RCTs and meta-analysis reporting a statistically significant reduction in the risk of all-cause mortality in patients undergoing online HDF compared with HD were not very robust [3, 6]. The FI (for binary outcomes) and the SIFI (for time-to-event data) provide additional means of assessing and communicating the strength of statistical conclusions. They should be employed as a valuable supplement to reported P-values in RCTs, helping readers look beyond merely reaching statistical significance.

ACKNOWLEDGEMENTS

The EuDial Working Group is a part of the European Renal Association.

Contributor Information

Yuri Battaglia, University of Verona, Department of Medicine, Verona, Italy. Pederzoli Hospital, Nephrology and Dialysis Unit, Peschiera del Garda, Italy.

Alessandro Mantovani, Section of Endocrinology, Diabetes and Metabolism, Department of Medicine, University and Azienda Ospedaliera Universitaria Integrata of Verona, Verona, Italy.

Rukshana Shroff, Renal Unit, UCL Great Ormond Street Hospital and Institute of Child Health, London, UK.

Gaetano Alfano, Nephrology Dialysis and Transplant Unit, University Hospital of Modena, Modena, Italy.

Björn Meijers, Nephrology Unit, University Hospitals Leuven and Department of Microbiology, Immunology and Organ Transplantation, KU Leuven, Leuven, Belgium.

Casper Franssen, Department of Nephrology, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.

Christian Combe, Department of Nephrology, CHU de Bordeaux and INSERM U1026, University of Bordeaux, Bordeaux, France.

Carlo Basile, Associazione Nefrologica Gabriella Sebastio, Martina Franca, Italy.

FUNDING

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

DATA AVAILABILITY STATEMENT

The data underlying this article will be shared upon reasonable request to the corresponding author.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.

REFERENCES