Providing an Evaluation Model for Medical Machine Learning in the Case of Heart Disease
Authors
-
Fatemeh shahhosseiny
Master of Science, Department of Statistics, Allameh Tabatabaee University, Tehran, Iran.
https://orcid.org/0009-0004-6637-3472
-
kimia Zarooj Hosseini
Student Research Committee, PhD Student in Medical Informatics, Department of Information Technology and Health Management, Faculty of Management and Medical Informatics, Iran University of Medical Sciences, Tehran, Iran.
https://orcid.org/0000-0002-5783-5039
-
Fatemeh Ahouz
Department of Computer Engineering, Faculty of Energy and Data Sciences, Behbahan Khatam Alanbia University of Technology, Behbahan, Iran.
https://orcid.org/0000-0002-3533-8605
-
Amin Golabpour
Department of Health Informatics Technology, School of Allied Medical Sciences, Shahroud University of Medical Sciences, Shahroud, Iran.
https://orcid.org/0000-0001-7649-4033
DOI:
https://doi.org/10.22100/ijhs.v12i3.1351Keywords:
Heart disease, Decision tree, Artificial intelligenceAbstract
Background: Cardiovascular diseases, the global number one killer, require early diagnosis to reduce premature mortality and enhance quality of life. Decision tree algorithms, whose transparency and credibility are highly valued, were used in order to capture intelligible diagnostic rules for the prediction of heart disease. They were validated and tested by doctors as clinically acceptable.
Method: This study experimented on a heart disease data set with statistical tests, splitting it 80:20 into training and test set for distribution studies. A decision tree method generated diagnostic rules from the training set, and PPV or NPV and Support for each rule were calculated. Rules with value less than threshold were removed, and the remaining rules were tested, recalculating PPV/NPV and Support. Non-compliant rules were removed, and clinicians reviewed final rules for clinical usability.
Result: This study statistically analyzed a heart disease dataset, splitting it 80:20 into training and test sets, with distributions validated. A decision tree algorithm generated diagnostic rules from the training set, assessed for positive predictive value (PPV) or negative predictive value (NPV) and Support. Rules below thresholds were discarded, and non-compliant adjusted rules were eliminated. Physicians evaluated the final rules for clinical acceptability.
Conclusion: This article highlights the crucial role of expert-based qualitative evaluation in validating and optimizing decision tree-induced rules. Optimization rules are accepted and satisfy more than original rules, as shown through comparisons of expert ratings. The findings underscore the necessity of model accuracy, interpretability, and clinical acceptability for the implementation of AI systems in health care.
References
Oude Wolcherink MJ, Behr CM, Pouwels X, Doggen CJM, Koffijberg H. Health Economic Research Assessing the Value of Early Detection of Cardiovascular Disease: A Systematic Review. PharmacoEconomics. 2023;41(10):1183-203. doi: 10.1007/s40273-023-01287-2
Sahakyan M, Aung Z, Rahwan T. Explainable Artificial Intelligence for Tabular Datum: A Survey. IEEE Access. 2021; 9:135392-422. doi: 10.1109/ACCESS.2021.3116481
Josephson CB, Wiebe S. Precision Medicine: Academic dreaming or clinical reality? Epilepsia. 2021;62: S78-S89. doi: 10.1111/epi.16739
Schwalbe N, Wahl B. Artificial intelligence and the future of global health. The Lancet. 2020;395(10236):1579-86. doi: 10.1016/S0140-6736(20)30226-9
Peng JF, Zou KQ, Zhou M, Teng Y, Zhu XY, Zhang FF, et al. An Explainable Artificial Intelligence Framework for the Deterioration Risk Prediction of Hepatitis Patients. Journal of Medical Systems. 2021;45 (5). doi: 10.1007/s10916-021-01736-5
Parziale A, Senatore R, Della Cioppa A, Marcelli A. Cartesian genetic programming for diagnosis of Parkinson disease through handwriting analysis: Performance vs. interpretability issues. Artificial Intelligence in Medicine.2021;111. doi: 10.1016/j.artmed.2020.101984
Torshizi R, Karimani EG, Etminani K, Akbarin MM, Jamialahmadi K, Shirdel A, Rahimi H, Allahyari A, Golabpour A, Rafatpanah H. Altered expression of cell cycle regulators in adult T-cell leukemia/lymphoma patients. Reports of Biochemistry and Molecular Biology. 2017;6(1):88.
Golabpour A, Shirazi HM, Farahi A, Kootiani AZ, Beigi H. A fuzzy solution based on Memetic algorithms for timetabling. In 2008 International Conference on MultiMedia and Information Technology 2008 (pp. 108-110). IEEE. doi: 10.1109/MMIT.2008.193
Wilkinson J, Arnold KF, Murray EJ, van Smeden M, Carr K, Sippy R, et al. Time to reality check the promises of machine learning-powered precision medicine. The Lancet Digital Health. 2020;2(12): e677-e80. doi: 10.1016/S2589-7500(20)30200-4
Petch J, Di S, Nelson W. Opening the Black Box: The Promise and Limitations of Explainable Machine Learning in Cardiology. Canadian Journal of Cardiology. 2022;38(2):204-13. doi: 10.1016/j.cjca.2021.09.004
Lee CK, Samad M, Hofer I, Cannesson M, Baldi P. Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality. Npj Digital Medicine. 2021;4 (1). doi: 10.1038/s41746-020-00377-1
Welchowski T, Maloney KO, Mitchell R, Schmid M. Techniques to Improve Ecological Interpretability of Black-Box Machine Learning Models. Journal of Agricultural Biological and Environmental Statistics. 2022;27(1):175-97. doi: 10.1007/s13253-021-00479-7
Zihni E, Madai VI, Livne M, Galinovic I, Khalil AA, Fiebach JB, et al. Opening the black box of artificial intelligence for clinical decision support: A study predicting stroke outcome. PLoS ONE. 2020;15 (4). doi: 10.1371/journal.pone.0231166
Singh A, Misra SC, Kumar S. Smart healthcare: rough set theory in predicting heart disease. Advances in Computing, Informatics, Networking and Cybersecurity: A Book Honoring Professor Mohammad S Obaidat's Significant Scientific Contributions: Springer; 2022. p. 155-80. doi: 10.1007/978-3-030-87049-2_5
Suthaharan S, Decision Tree Learning. In: Machine Learning Models and Algorithms for Big Data Classification. Integrated Series in Information Systems. 2016:237-69. doi: 10.1007/978-1-4899-7641-3_10
Song YY, Lu Y. Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry. 2015;27 (2):13-5.
Kannan A, Fries JA, Kramer E, Chen JJ, Shah N, Amatriain X. The accuracy vs. coverage trade-off in patient-facing diagnosis models. AMIA Joint Summits on Translational Science proceedings AMIA Joint Summits on Translational Science. 2020; 2020:298 307.
Gundumogula M, Gundumogula MJIJoH, Science S. Importance of focus groups in qualitative research. 2020;8(11):299-302. doi: 10.24940/theijhss/2020/v8/i11/HS2011-082
Cuhls K. The Delphi method: an introduction. Delphi methods in the social and health sciences: concepts, applications and case studies: Springer; 2023. p. 3-27. doi: 10.1007/978-3-658-38862-1_1
Rai R, Sisodia DS, editors. Real-time data augmentation based transfer learning model for breast cancer diagnosis using histopathological images. Advances in Biomedical Engineering and Technology: Select Proceedings of ICBEST 2018; 2021: Springer. doi: 10.1007/978-981-15-6329-4_39
Das AK, Biswas SK, Mandal A, Bhattacharya A, Sanyal SJESwA. Machine Learning based Intelligent System for Breast Cancer Prediction (MLISBCP). 2024; 242:122673. doi: 10.1016/j.eswa.2023.122673
Alqudah A, Alqudah AMJIJoR. Sliding window based support vector machine system for classification of breast cancer using histopathological microscopic images. 2022;68(1):59-67. doi: 10.1080/03772063.2019.1583610
Agliata A, Giordano D, Bardozzo F, Bottiglieri S, Facchiano A, Tagliaferri RJIJoMS. Machine learning as a support for the diagnosis of type 2 diabetes. 2023;24(7):6775. doi: 10.3390/ijms24076775
Pal M, Parija S, Panda G, editors. Improved prediction of diabetes mellitus using machine learning based approach. 2021 2nd International Conference on Range Technology (ICORT); 2021: IEEE. doi: 10.1109/ICORT52730.2021.9581774
James DE, Vimina E, editors. Machine learning-based early diabetes prediction. Intelligent Sustainable Systems: Proceedings of ICISS 2021; 2022: Springer. doi: 10.1007/978-981-16-2422-3_52
Fan Z, Guo Y, Gu X, Huang R, Miao WJSR. Development and validation of an artificial neural network model for non-invasive gastric cancer screening and diagnosis. 2022;12(1):21795. doi: 10.1038/s41598-022-26477-4
Zahmatkesh Zakariaee A, Sadr H, Yamaghani MR. A New Hybrid Method to Detect Risk of Gastric Cancer using Machine Learning Techniques. Journal of AI and Data Mining. 2023;11 (4):505-15.
Li C, Liu S, Zhang Q, Wan D, Shen R, Wang Z, et al. Combining Raman spectroscopy and machine learning to assist early diagnosis of gastric cancer. 2023; 287:122049. doi: 10.1016/j.saa.2022.122049
Almasinejad P, Golabpour A, Mollakhalili Meybodi MR, Mirzaie K, Khosravi A. A dynamic model for inputting missing medical data. Journal of Healthcare Engineering. 2021;2021(1):1203726. doi: 10.1155/2021/1203726
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Publisher Correction: Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature medicine. 2022;28(10):2218. doi: 10.1038/s41591-022-01951-8
Hogg HDJ, Martindale APL, Liu X, Denniston AK. Clinical Evaluation of Artificial Intelligence-Enabled Interventions. Investigative Ophthalmology and Visual Science. 2024;65(10):10. doi: 10.1167/iovs.65.10.10
Bin-Jumah MN, Al-Abdan M, Al-Basher G, Alarifi S. Molecular Mechanism of Cytotoxicity, Genotoxicity, and Anticancer Potential of Green Gold Nanoparticles on Human Liver Normal and Cancerous Cells. Dose-response: a publication of International Hormesis Society. 2020;18(2):1559325820912154. doi: 10.1177/1559325820912154
Laureano-Phillips J, Robinson RD, Aryal S, Blair S, Wilson D, Boyd K, et al. HEART score risk stratification of low-risk chest pain patients in the emergency department: a systematic review and meta-analysis. 2019;74(2):187-203. doi: 10.1016/j.annemergmed.2018.12.010
Doi T, Langsted A, Nordestgaard BGJJotACoC. Elevated remnant cholesterol reclassifies risk of ischemic heart disease and myocardial infarction. 2022;79(24):2383-97. doi: 10.1016/j.jacc.2022.03.384
Hermiz C, Sedhai YR. Angina. 2020.
Nanchen DJH. Resting heart rate: what is normal? BMJ Publishing Group Ltd and British Cardiovascular Society; 2018. p. 1048-9. doi: 10.1136/heartjnl-2017-312731
Fitzgerald BT, Smith E, Scalia GM. What are the prognostic implications and factors relating to exercise induced electrocardiographic ST segment changes in the setting of a non-ischemic stress echocardiogram? 2022; 364:157-61. doi: 10.1016/j.ijcard.2022.06.031
Blumenthal RS, Becker DM, Moy TF, Coresh J, Wilder LB, Becker LCJC. Exercise thallium tomography predicts future clinically manifest coronary heart disease in a high-risk asymptomatic population. 1996;93(5):915-23. doi: 10.1161/01.CIR.93.5.915
Yazdani A, Varathan KD, Chiam YK, Malik AW, Wan Ahmad WAJBmi, making d. A novel approach for heart disease prediction using strength scores with significant predictors. 2021;21(1):194. doi: 10.1186/s12911-021-01527-5
Mokeddem SAJAI. A fuzzy classification model for myocardial infarction risk assessment. 2018;48(5):1233-50.
Published
Issue
Section
License
The Copyright Form should be downloaded and signed by corresponding author in the fourth step "upload supplementary files" during submission process.
After acceptance, copyright form should be downloaded and signed by all authors one by one ( "summery --> supp. file" part and click on "add a supplementary file" link).
