Predictive Models for Performance in High-Stakes Medical Assessments: A Systematic Review within Health Professions Education
Authors
-
Haniye Mastour
Department of Medical Education, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran.
https://orcid.org/0000-0003-4893-0998
-
Toktam Dehghani
School of Medical Education and Learning Technologies, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
https://orcid.org/0000-0003-3931-2880
-
Mahdie Jajroudi
Department of Medical Informatics, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Pharmaceutical Sciences Research Center, Institute of Pharmaceutical Technology, Mashhad University of Medical Sciences, Mashhad, Iran.
https://orcid.org/0000-0002-4605-5221
-
Mohammadreza Farrokhnia
Department of Learning, Data Analytics and Technology, Instructional Technology Section, University of Twente, Enschede, the Netherlands.
https://orcid.org/0000-0002-0150-5372
-
Mitra Zarei
Medical Librarian and Information Counselor, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran.
https://orcid.org/0000-0001-8309-7052
-
Saeid Eslami
Department of Medical Informatics, School of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran; Pharmaceutical Sciences Research Center, Institute of Pharmaceutical Technology, Mashhad University of Medical Sciences, Mashhad, Iran; Department of Medical Informatics, UMC-Location AMC, University of Amsterdam, Amsterdam, Netherlands.
https://orcid.org/0000-0003-3755-1212
DOI:
https://doi.org/10.22100/ijhs.v12i5.1446Keywords:
Systematic review, Machine learning, Assessment, Medicine, Medical educationAbstract
Background: High-stakes medical assessments play a critical role in certifying future healthcare professionals. Enhancing these assessments through predictive models offers a promising way to identify at-risk students and predict exam outcomes. The current study aims to address this need by providing a comprehensive overview of the application and effectiveness of predictive models, particularly in high-stakes medical assessments.
Methods: This systematic review searched four major databases (MEDLINE, PubMed, Scopus, and Web of Science) for studies using predictive models to evaluate performance in high-stakes medical assessments, following PRISMA guidelines and the Best Evidence in Medical Education framework.
Results: The review included 14 studies across various stages of professional training, including medical students, residents, chiropractors, and physician assistants. Sample sizes ranged from 147 to 5,825 participants, and most studies used local datasets. Regression models were the most commonly employed (78%), while random forests and artificial neural networks were frequently used for classification tasks. Although deep learning models were not observed, 14% of studies incorporated explainable AI (XAI) to improve interpretability. To provide a broader context, four systematic reviews of predictive modeling in nonmedical high-stakes assessments were also analyzed.
Conclusion: While predictive methods show significant promise for enhancing the predictive ability of high-stakes medical assessments, there is still ample room for improvement. Integrating comprehensive feature sets and longitudinal tracking is crucial for increasing accuracy. Expanding studies to more extensive, diverse populations can improve generalizability and pave the way for more precise evaluations and better educational outcomes.
References
Bhanji F, Naik V, Skoll A, Pittini R, Daniels VJ, Bacchus CM, et al. Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System. Perspectives on Medical Education. 2024;13(1):68-74. doi: 10.5334/pme.965
Mee J, Pandian R, Wolczynski J, et al. An experimental comparison of multiple-choice and short-answer questions on a high-stakes test for medical students. Advances in Health Sciences Education. 2024;29(3):783-801. doi: 10.1007/s10459-023-10266-3
Templeton K, Vanston P, Luciw-Dubas UA, et al. Navigating a High-Stakes Assessment in Medical School: Students' Lived Experiences During a Stressful Period of Preparation. Journal of Medical Regulation. 2022;108(2):7-18. doi: 10.30770/2572-1852-108.2.7
Gosnell PW. Bridging the Gap Between High-Stakes Testing and 21st Century Skills. Clemson University; 2023.
Thoma B, Monteiro S, Pardhan A, Waters H, Chan T. Replacing high-stakes summative examinations with graduated medical licensure in Canada. Canadian Medical Association journal. 2022;194(5):E168-E170. doi: 10.1503/cmaj.211816
Lucey CR, Hauer KE, Boatright D, Fernandez A. Medical education's wicked problem: Achieving equity in assessment for medical learners. Academic Medicine. 2020;95(12S):S98-S108. doi: 10.1097/ACM.0000000000003717
French S, Dickerson A, Mulder RA. A review of the benefits and drawbacks of high-stakes final examinations in higher education. Higher Education. 88, 893-918 (2024). doi: 10.1007/s10734-023-01148-z
Franco D'Souza R, Mathew M, Mishra V, Surapaneni KM. Twelve tips for addressing ethical concerns in the implementation of artificial intelligence in medical education. Medical Education Online. 2024;29(1):2330250. doi: 10.1080/10872981.2024.2330250
Lykourentzou I, Giannoukos I, Nikolopoulos V, Mpardis G, Loumos V. Dropout prediction in e-learning courses through the combination of machine learning techniques. Computers and Education. 2009;53(3):950-965. doi: 10.1016/j.compedu.2009.05.010
Asif R, Merceron A, Ali SA, Haider NG. Analyzing undergraduate students' performance using educational DM. Computers and Education. 2017;113:177-194. doi: 10.1016/j.compedu.2017.05.007
Ojajuni O, Ayeni F, Akodu O, et al. Predicting Student Academic Performance Using Machine Learning. Springer International Publishing; 2021:481-491. doi: 10.1007/978-3-030-87013-3_36
Sobiesuo EM, Edmond A, Issaka CA, Appiah SCY. Knowledge of AI use and prediction of AI adoption among selected residents in the greater Kumasi area of Ghana. Discover Artificial Intelligence. 2025;5(1):1-15. doi: 10.1007/s44163-025-00558-5
Alalawi K, Athauda R, Chiong R. Contextualizing the current state of research on the use of machine learning for student performance prediction: A systematic literature review. Engineering Reports. 2023;5(12):e12699. doi: 10.1002/eng2.12699
Fahd K, Venkatraman S, Miah SJ, Ahmed K. Application of machine learning in higher education to assess student academic performance, at-risk, and attrition: A meta-analysis of literature. Education and Information Technologies. 2022:1-33.
Sekeroglu B, Abiyev R, Ilhan A, Arslan M, Idoko JB. Systematic literature review on machine learning and student performance prediction: Critical gaps and possible remedies. Applied Sciences. 2021;11(22):10907. doi: 10.3390/app112210907
Zhai X, Yin Y, Pellegrino JW, Haudek KC, Shi L. Applying machine learning in science assessment: a systematic review. Studies in Science Education. 2020;56(1):111-151. doi: 10.1080/03057267.2020.1735757
Mastour H, Dehghani T, Moradi E, Eslami S. Early prediction of medical students' performance in high-stakes examinations using machine learning approaches. Heliyon. 2023;9(7):e18248. doi: 10.1016/j.heliyon.2023.e18248
Mastour H, Dehghani T, Moradi E, Eslami S. Explainable artificial intelligence for predicting medical students' performance in comprehensive assessments. Scientific Reports. 2025;15(1):23752. doi: 10.1038/s41598-025-07460-1
Mastour H, Dehghani T, Jajroudi M, Moradi E, Zarei M, Eslami S. Prediction of medical sciences students' performance on high-stakes examinations using machine learning models: a protocol for a systematic review. BMJ Open. 2023;13. doi: 10.1136/bmjopen-2022-064956
Amir-Behghadami M, Janati A. Population, Intervention, Comparison, Outcomes and Study (PICOS) design as a framework to formulate eligibility criteria in systematic reviews. Emergency Medicine Journal. 2020;37(6):387. doi: 10.1136/emermed-2020-209567
Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, et al. The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Medical teacher. 2009;31(4):282-298. doi: 10.1080/01421590902889897
Mortaz Hejri S, Jalili M, Shirazi M, Masoomi R, Nedjat S, Norcini J. The utility of mini-Clinical Evaluation Exercise (mini-CEX) in undergraduate and postgraduate medical education: protocol for a systematic review. Systematic Reviews. 2017;6(1):1-8. doi: 10.1186/s13643-017-0539-y
Li M, Gao Q, Yu T. Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters. BMC Cancer. 2023;23(1):799. doi: 10.1186/s12885-023-11325-z
Nichols TR, Wisner PM, Cripe G, Gulabchand L. Putting the kappa statistic to use. The Quality Assurance Journal. 2010;13(3-4):57-61. doi: 10.1002/qaj.481
Zhong Q, Wang H, Christensen P, McNeil K, Linton M, Payton M. Early prediction of the risk of scoring lower than 500 on the COMLEX 1. BMC Medical Education. 2021;21(1):70. doi: 10.1186/s12909-021-02501-5
Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Medical Education. 2016;16:128. doi: 10.1186/s12909-016-0652-y
Ghaffari-Rafi A, Lee RE, Fang R, Miles JD. Multivariable analysis of factors associated with USMLE scores across US medical schools. BMC Medical Education. 2019;19(1):154. doi: 10.1186/s12909-019-1605-z
Sieg M, Roselló Atanet I, Tomova MT, et al. Discovering unknown response patterns in progress test data to improve the estimation of student performance. BMC Medical Education. 2023;23(1):193. doi: 10.1186/s12909-023-04172-w
Bird JB, Olvet DM, Willey JM, Brenner JM. A Generalizable Approach to Predicting Performance on USMLE Step 2 CK. Advances in Medical Education and Practice. 2022;13:939-944. doi: 10.2147/AMEP.S373300
Althewini A, Al Baz N. Prediction of Admission Tests for Medical Students' Academic Performance. Advances in Medical Education and Practice. 2022;13:1287-1292. doi: 10.2147/AMEP.S355474
Lee MW, Johnson TR, Kibble J. Development of statistical models to predict medical student performance on the USMLE Step 1 as a catalyst for deployment of student services. Medical Science Educator. 2017;27:663-671. doi: 10.1007/s40670-017-0452-y
Gullo CA, McCarthy MJ, Shapiro JI, Miller BL. Predicting medical student success on licensure exams. Medical Science Educator. 2015;25:447-453. doi: 10.1007/s40670-015-0179-6
Amirhajlou L, Sohrabi Z, Alebouyeh MR, et al. Application of DM techniques for predicting residents' performance on pre-board examinations: A case study. Journal of Education and Health Promotion. 2019;8:108. doi: 10.4103/jehp.jehp_394_18
Himelfarb I, Shotts BL, Gow AR. Examining the validity of chiropractic grade point averages for predicting National Board of Chiropractic Examiners Part I exam scores. Journal of Chiropractic Education. 2022;36(1):1-12. doi: 10.7899/JCE-20-5
Kumar A, DiJohnson T, Edwards RA, Walker L. The Application of Adaptive Minimum Match k-Nearest Neighbors to Identify At-Risk Students in Health Professions Education. Journal of Physician Assistant Education. 2023;34(3):171-177. doi: 10.1097/JPA.0000000000000513
Barber C, Hammond R, Gula L, Tithecott G, Chahine S. In search of black swans: identifying students at risk of failing licensing examinations. Academic Medicine. 2018;93(3):478-485. doi: 10.1097/ACM.0000000000001938
Puri N, McCarthy M, Miller B. Validity and Reliability of Pre-matriculation and Institutional Assessments in Predicting USMLE STEP 1 Success: Lessons From a Traditional 2 x 2 Curricular Model. Frontiers in Medicine. 2022;8:798876. doi: 10.3389/fmed.2021.798876
Haupt F, Kanzow P. The Relation Between Students' Theoretical Knowledge and Practical Skills in Endodontics: Retrospective Analysis. Interactive Journal of Medical Research. 2023;12:e46305. doi: 10.2196/46305
Rayhan M, Alam MGR, Dewan MAA, Ahmed MHU. Appraisal of high-stake examinations during SARS-CoV-2 emergency with responsible and transparent AI: Evidence of fair and detrimental assessment. Computers and Education: Artificial Intelligence. 2022;3:100077. doi: 10.1016/j.caeai.2022.100077
Published
Issue
Section
License
The Copyright Form should be downloaded and signed by corresponding author in the fourth step "upload supplementary files" during submission process.
After acceptance, copyright form should be downloaded and signed by all authors one by one ( "summery --> supp. file" part and click on "add a supplementary file" link).
