Predictive value of machine learning models for lymph node metastasis in gastric cancer: A two-center study

Tong Lu,Miao Lu,Dong Wu,Yuan-Yuan Ding,Hao-Nan Liu,Tao-Tao Li,Da-Qing Song

Abstract BACKGROUND Gastric cancer is one of the most common malignant tumors in the digestive system,ranking sixth in incidence and fourth in mortality worldwide.Since 42.5% of metastatic lymph nodes in gastric cancer belong to nodule type and peripheral type,the application of imaging diagnosis is restricted.AIM To establish models for predicting the risk of lymph node metastasis in gastric cancer patients using machine learning (ML) algorithms and to evaluate their predictive performance in clinical practice.METHODS Data of a total of 369 patients who underwent radical gastrectomy at the Department of General Surgery of Affiliated Hospital of Xuzhou Medical University (Xuzhou,China) from March 2016 to November 2019 were collected and retrospectively analyzed as the training group.In addition,data of 123 patients who underwent radical gastrectomy at the Department of General Surgery of Jining First People’s Hospital (Jining,China) were collected and analyzed as the verification group.Seven ML models,including decision tree,random forest,support vector machine (SVM),gradient boosting machine,naive Bayes,neural network,and logistic regression,were developed to evaluate the occurrence of lymph node metastasis in patients with gastric cancer.The ML models were established following ten cross-validation iterations using the training dataset,and subsequently,each model was assessed using the test dataset.The models’ performance was evaluated by comparing the area under the receiver operating characteristic curve of each model.RESULTS Among the seven ML models,except for SVM,the other ones exhibited higher accuracy and reliability,and the influences of various risk factors on the models are intuitive.CONCLUSION The ML models developed exhibit strong predictive capabilities for lymph node metastasis in gastric cancer,which can aid in personalized clinical diagnosis and treatment.

Key Words: Machine learning;Prediction model;Gastric cancer;Lymph node metastasis

lNTRODUCTlON

Gastric cancer is one of the most common malignant tumors in the digestive system,ranking sixth in the world in incidence and fourth in mortality[1].At present,gastric cancer typically is managed with comprehensive treatment that includes surgery.However,the overall 5-year survival rate remains below 50%[2].In the Tumor-Node-Metastasis staging system of the American Joint Committee on Cancer,N represents the number of lymph node metastases,which is itself an independent factor in predicting the overall survival rate of gastric cancer patients[3].However,there are some difficulties in the exploration of lymph nodes in patients with gastric cancer,such as multiple regional lymph nodes located in the abdominal cavity,which are not easy to explore preoperatively.In addition,42.5% of metastatic lymph nodes in gastric cancer belong to nodule type and peripheral type,restricting the application of imaging diagnosis[4,5].

Artificial intelligence refers to the ability of machines to independently replicate typical human intellectual processes[6].Artificial intelligence has various applications in the medical field,encompassing image processing,computer vision,machine learning (ML),artificial neural networks (ANNs),and convolutional neural networks (CNNs).ML can assist physicians in interpreting clinical data through the computer-aided diagnostic (CAD) system.The CAD can be categorized into three stages: Feature recognition,feature extraction,and clinical reasoning.It is feasible to feed variables related to gastric cancer lymph node metastasis into the system and develop a risk model for lymph node metastasis of gastric cancer using a more advanced ML-based algorithm[7,8].ML algorithms play crucial roles in assisting diagnosis and predicting prognosis by processing a large amount of complex medical data[9,10].A clinical prediction model can be proposed and optimized through the training dataset,and subsequently examined through the external validation dataset to determine its external validity and adaptability to other patients[11,12].The clinical utility of ML within the realm of artificial intelligence is increasingly attracting clinicians’ attention,and it is also applied to help diagnose and treat various clinical diseases,including gastric cancer.

MATERlALS AND METHODS

Study subjects

A total of 369 patients who underwent radical gastrectomy at the Department of General Surgery of the Affiliated Hospital of Xuzhou Medical University (Xuzhou,China) from March 2016 to November 2019 were enrolled as the training group,and 123 patients who underwent radical gastrectomy at the Department of General Surgery of Jining First People’s Hospital (Jining,China) were enrolled as the verification group.The inclusion criteria were as follows: (1) Newly diagnosed gastric cancer patients with complete medical records;(2) radical resection for primary gastric cancer was performed in either of the two hospitals,and lymph node metastasis was confirmed by imaging and pathology;and (3) no anti-tumor therapy,such as radiotherapy or chemotherapy,was performed preoperatively.The exclusion criteria were: (1) Combination with other malignant tumors;(2) preoperative complications of other infectious diseases,blood system diseases,autoimmune diseases,and other diseases that could affect inflammatory indicators;(3) recently or currently receiving anti-inflammatory or immunosuppressive therapy;(4) preoperative blood transfusion treatment;(5) severe liver and kidney dysfunction;and (6) incomplete clinical data (Figure 1).

Figure 1 Flow chart. GBM: Gradient boosting machine.

Observational indicators

Clinical data,such as patient name,age,gender,and other clinicopathological data,including routine blood parameters,tumor location,maximum tumor diameter,depth of invasion,and the presence or absence of lymph node metastasis,were collected from all patients.Blood samples were collected in the morning on an empty stomach on the day after admission to determine neutrophil count,platelet count,monocyte count,and lymphocyte count using the Sysmex XE-2100 Automatic Blood Analyzer.Carcinoembryonic antigen (CEA) level in the blood was also measured.The panimmune-inflammation value (PIV) and CEA level were utilized to establish clinical prediction models.PIV was calculated as (neutrophil count × platelet count × monocyte count)/lymphocyte count.

Statistical analysis

Continuous variables are expressed as the mean ± SD,and categorical variables are presented as percentages.LR was employed to identify the independent risk factors associated with lymph node metastasis in gastric cancer patients.This analysis allowed for the calculation of odds ratios (ORs) and their corresponding 95% confidence intervals.An OR greater than 1 indicated that the variable was a positive risk factor affecting the outcome,while an OR less than 1 suggested that the variable was a negative risk factor influencing the outcome.Statistical significance was defined as aPvalue of less than 0.05.The statistical analyses and modeling procedures were carried out using SPSS 20.0 software (IBM,Armonk,NY,United States) and R-Studio 25.0 software (R Foundation for Statistical Computing,Vienna,Austria).Several packages were utilized to train models and draw relevant graphs,with the caret package applied for training and validating ML models.In addition to the fundamental linear model (linear LR),seven ML models were fitted,including LR,random forest (RF),gradient boosting machine (GBM),decision tree (DT),support vector machine (SVM),naive Bayes (NB),and multi-layer perceptron (MLP),as illustrated in Figure 2.

Figure 2 Prediction performance evaluation of each model.SVM: Support vector machine;AUC: Area under the receiver operating characteristic curve;MLP: Multi-layer perceptron.

The training dataset was combined with the validation dataset,and seven ML algorithms were employed to establish prediction models.LR is a classification algorithm designed to establish a relationship between a feature and the probability of a specific outcome.Rather than using LR for estimating class probability,it employs S-shaped functions for modeling[13,14].DT is primarily utilized for classification tasks.It begins at the root node to split the dataset based on the most informative feature,creating decision points that segment the data into distinct classes[15].RF is an extension of the DT method and functions as an ensemble approach.It generates multiple DTs,with the majority vote from these trees determining the final class prediction of the model[16,17].MLP is an ML algorithm inspired by biological neural networks.ANNs consist of interconnected nodes that communicate through connections[18,19].SVM classifies data by defining boundaries that separate classes.The optimization process aims to maximize the margin between these class boundaries.While SVM generally outperforms LR,its computational complexity may lead to longer training time during model development[20,21].GBM is a boosting technique that serves as a numerical optimization algorithm for constructing additive models that minimize loss functions[22,23].NB is a straightforward classification algorithm that calculates the probability of each category’s occurrence given the item to be classified.The item is assigned to the category with the highest probability[24,25].

Performance evaluation of the models involved various metrics,including accuracy,recall,and other indicators.The primary indicator for predicting binary classification results was the area under the receiver operating characteristic curve (AUC).This metric varies from 0 to 1,with higher values signifying a superior performance.Additionally,for models with two outcomes,the area under the accuracy-recall curve was utilized,illustrating the trade-off between true accuracy and positive predictive value,and the F1 score,defined as the harmonic mean of recall and accuracy.The models underwent 10-fold cross-validation on the training dataset and then assessed for their performance on the test dataset.According to the optimal model,a network estimator was developed to facilitate disease prediction using patient data.This estimator enables surgeons to assess the risk of lymph node metastasis in gastric cancer patients.

RESULTS

Baseline clinical data in the training group and verification group

The comparison of clinical data between the two groups is presented in Table 1.Gender,age,tumor location,and surgical method exhibited no significant differences between the two groups (P＞ 0.05).In the training dataset,the proportion of patients with total gastrectomy,neurovascular invasion,and maximum tumor diameter ＞ 5 cm was significantly higher in patients with lymph node metastasis than in those without (P＜ 0.05).In the verification dataset,the number of patients who were aged ＞ 60 years old and had neurovascular invasion and maximum tumor diameter ＞ 5 cm was significantly greater in patients with lymph node metastasis than in those without (P＜ 0.05).

Table 1 Comparison of clinical data between the two groups

The results of Mann-WhitneyUtest revealed that there were no statistically significant differences in the depth of infiltration,PIV,or CEA level between the two groups (P＞ 0.05).It was found that the depth of infiltration and CEA level in patients with lymph node metastasis were significantly higher than those in patients without (P＜ 0.05).In the training dataset,the infiltration depth,PIV,and CEA level in patients with lymph node metastasis were significantly greater than those in patients without (P＜ 0.05).

Evaluation of predictive performance of each model

In order to compare the predictive performance of the seven ML-based models,this study employed ten-fold crossvalidation and utilized the AUC value,validated on the test dataset,as the primary metric for assessing their performance.As shown in Table 2 and Figure 2,the GBM model exhibited the best performance in predicting the occurrence of lymph node metastasis in gastric cancer patients,with an average AUC of 0.927.In this study,a web-based online estimator,along with feature importance (Figure 3) and Shapley Additive Explanations (SHAP) summary plot (Figure 4),was developed based on the GBM model.Feature importance enables the visualization of the model’s internal results,highlighting the significance of specific variables within the model.Utilizing the optimal GBM model,we have developed a web-based risk calculator (https://gastric.shinyapps.io/gbm4Lymph).By entering the clinical characteristics of patients with gastric cancer and lymph node metastasis,healthcare professionals can predict the risk of lymph node development in these patients (Figure 5).

Figure 3 Feature importance. SHAP: Shapley Additive Explanations;CEA: Carcinoembryonic antigen;PIV: Pan-immune-inflammation value.

Figure 4 Shapley Additive Explanations summary plot.SHAP: Shapley Additive Explanations;CEA: Carcinoembryonic antigen;PIV: Pan-immuneinflammation value.

Figure 5 Web risk calculator. SHAP: Shapley Additive Explanations;CEA: Carcinoembryonic antigen;PIV: Pan-immune-inflammation value.

Table 2 Prediction performance evaluation of each model

DlSCUSSlON

As a result of the limited early detection of gastric cancer,over 50% of patients are diagnosed at advanced stages or with metastasis.At present,surgery is the main method for the treatment of gastric cancer,and lymph node metastasis is regarded as the main factor affecting the stage,grade,and survival rate of gastric cancer[26,27].Therefore,early prediction of the occurrence of lymph node metastasis is vital.To date,several scholars have concentrated on lymph node metastasis in gastric cancer,while few studies have developed tools to provide accurate predictions.Therefore,the development of precise predictive models is essential to facilitate collaborative decision-making for clinicians and patients.The continuous advancement of artificial intelligence in the field of clinical research has led to the introduction of innovative approaches.

ML represents an evolving frontier in the field of medicine,drawing substantial resources to connect computer science and statistical analysis with medical challenges.ML has the capacity to effectively handle extensive,diverse,and intricate medical data.Consequently,the implementation of ML techniques in medicine is widely regarded as the cornerstone of future endeavors in biomedical research,personalized medicine,and computer-aided diagnosis[28,29].Specifically,the operational framework of ML involves development of algorithms to execute numerous tasks,refining the algorithms iteratively to optimize performance.Ultimately,this process yields a model that establishes connections between multiple variables and target outcomes.In the present study,clinical data were collected,and ML algorithms were employed to develop a model for assessing the risk of lymph node metastasis in gastric cancer.By leveraging multiple variables,clinicians can employ this AI-driven approach to select more efficacious treatment strategies[30-33].

In this study,in addition to some clinicopathological data,hematological indicators,namely,immunoinflammatory factors (PIV and CEA),were utilized to develop the prediction models.PIV is a novel blood-based biomarker that integrates different subsets of peripheral blood immune cells,neutrophils,platelets,monocytes,and lymphocytes.As PIV has the potential to comprehensively represent patients’ immunity and systemic inflammation,it may potentially serve as a robust predictor in advanced cancer patients undergoing cytotoxic chemotherapy,immunotherapy,and targeted therapy.It has been previously demonstrated that PIV is mainly dependent on neutrophil-to-lymphocyte ratio,platelet-tolymphocyte ratio,lymphocyte-to-monocyte ratio,and other indicators in predicting cancer prognosis[34,35].CEA is a widely used serum tumor marker in clinical practice,particularly in the early screening of various types of cancer,and elevation of its elevation is also regarded as an independent risk factor for poor prognosis of gastric cancer[36].Development of a model based on combination of clinicopathological data with hematological suggestions can better reflect the physiological and pathological changes of patients with gastric cancer during the disease,making the model more representative.

Using ML,seven models were established for comparative analysis,utilizing the AUC as the benchmark for assessment.The outcomes are summarized as follows: The AUC for the DT model was 0.824,the RF model yielded an AUC of 0.923,the AUC for SVM was 0.721,and the GBM model demonstrated an AUC of 0.927.The NB model’s AUC stood at 0.914,while the NNET model’s AUC reached 0.907.The results of the seven models indicated that the GBM model displayed the most reliable performance,while SVM exhibited the least promising results.Furthermore,a feature importance table was developed based on the highly effective GBM model,which highlighted that factors,such as nerve or vascular invasion,CEA level,maximum tumor diameter,PIV,age,and tumor site,were significant contributors to the occurrence of lymph node metastasis.

Using the best-performing GBM model,feature importance assessment was conducted.This analysis highlighted the significance of specific indicators within the model,providing new insights into the model’s structure.To understand the relationship between the direction of lymph node metastasis in gastric cancer and the importance of its main predictors,a SHAP summary plot was drawn.This method was utilized to explain the predictions of ML models.SHAP-Beeswarm diagrams,a common visualization tool in SHAP method,display the effect of each feature on the predicted results.The horizontal axis of the plot represents the SHAP value,indicating the contribution of each feature to the predicted result,while the vertical axis represents the feature name.Each data point in the diagram represents a sample,with its horizontal position indicating the sample’s influence on the prediction result.Data points closer to the left side of the graph negatively impact the result,while those closer to the right side positively impact the result.The vertical position of the data point represents the feature name,with each feature having a corresponding vertical position.

According to the optimal GBM model,a web-based risk calculator was developed.By inputting patients’ clinical characteristics,it can directly predict the probability of lymph node metastasis in patients with gastric cancer.This tool is user-friendly and straightforward,making it accessible for healthcare practitioners.It serves as a valuable resource in diagnosis and treatment,providing significant support for clinicians.

CONCLUSlON

In summary,based on the clinicopathological data of 492 gastric cancer patients in two centers,ML algorithms were utilized to establish clinical models and conduct cross-validation,and AUC values were finally compared to draw conclusions.In addition to SVM,other ML models have exhibited promising accuracy and reliability,as well as better predictive value for gastric cancer lymph node metastasis.Among them,GBM outperformed the others,with the highest predictive value and accuracy.This study demonstrated that ML could reveal the potential of clinical data to reflect disease conditions,thereby assisting clinicians in evaluating patients’ conditions and making more informed treatment decisions.

ARTlCLE HlGHLlGHTS

Research background

Gastric cancer is one of the most common malignant tumors of the digestive system,ranking sixth in incidence and fourth in mortality worldwide.Machine learning (ML) represents an evolving frontier in the field of medicine,drawing substantial resources to connect computer science and statistical analysis with medical challenges.ML has the capacity to effectively handle extensive,diverse,and intricate medical data.Consequently,the implementation of ML techniques in medicine is widely regarded as the cornerstone of future endeavors in biomedical research,personalized medicine,and computer-aided diagnosis.

Research motivation

Using machine learning-based models to predict lymph node metastasis of gastric cancer is helpful to individualized diagnosis and treatment of gastric cancer patients.

Research objectives

Based on the clinicopathological data of 492 gastric cancer patients in two centers,we used ML algorithms to establish clinical models and conduct cross-validation,and finally compared the area under the receiver operating characteristic curve to draw conclusions.In addition to support vector machine,other ML models have good accuracy and reliability,and have better predictive value for gastric cancer lymph node metastasis.Among them,gradient boosting machine (GBM) has the best performance and the highest predictive value and accuracy.Through this study,ML can dig out the ability of clinical data to reflect disease,which can help clinicians evaluate patients' conditions and make better treatment decisions.

Research methods

Seven machine algorithm models were built with data from two centers,and then their performance was evaluated.Based on GBM model,a web-based online estimator and Shapley Additive Explanations summary plot were established.

Research results

ML can tap into the ability of clinical data to reflect disease,which can help clinicians assess patients' conditions and make better treatment decisions.

Research conclusions

ML algorithms have been used to establish an optimal prediction model for lymph node metastasis in gastric cancer,which is helpful for clinical risk stratification and individualized diagnosis and treatment of gastric cancer patients.

Research perspectives

In the future,multi-center data are needed to verify the external applicability of our model.

FOOTNOTES

Co-first authors:Tong Lu and Miao Lu.

Author contributions:Lu T and Wu D designed the study and wrote the manuscript;Li TT and Ding YY analyzed the data;Liu HN collected the data;Lu M and Song DQ revised the manuscript;Lu T and Lu M designed the study and wrote the article;Wu D and Ding YY analyzed the data;Li T and Song DQ revised the article;Lu M is one of the co-first authors of this paper.Lu T and Lu M contributed equally to this work as co-first authors.The research was performed as a collaborative effort,and the designation of co-first authors authorship accurately reflects the distribution of responsibilities and burdens associated with the time and effort required to complete the study and the resultant paper.This also ensures effective communication and management of post-submission matters,ultimately enhancing the paper's quality and reliability.Lu T and Lu M contributed efforts of equal substance throughout the research process.The choice of these researchers as co-first authors acknowledges and respects this equal contribution,while recognizing the spirit of teamwork and collaboration of this study.

lnstitutional review board statement:This study was reviewed and approved by the Ethics Committee of the Affiliated Hospital of Xuzhou Medical University and Jining First People’s Hospital.

lnformed consent statement:All study participants or their legal guardian provided informed written consent about personal and medical data collection prior to study enrolment.

Conflict-of-interest statement:We have no financial relationships to disclose.

Data sharing statement:No additional data are available.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers.It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license,which permits others to distribute,remix,adapt,build upon this work non-commercially,and license their derivative works on different terms,provided the original work is properly cited and the use is non-commercial.See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:China

ORClD number:Tong Lu 0000-0002-7674-2065;Da-Qing Song 0009-0001-5677-1598.

S-Editor:Qu XL

L-Editor:Wang TQ

P-Editor:Xu ZH