ABSTRACT

Introduction

Large language models (LLMs) have recently shown strong potential in medical education, yet their performance compared with human learners in specialty-level examinations remains unclear. This study aimed to evaluate the performance of LLMs compared to human groups on a 50-question emergency medicine test from the Turkish Medical Specialty Examination.

Methods

A cross-sectional study was conducted at İstanbul Medipol University in 2024, involving 40 medical students and postgraduates and six LLMs (ChatGPT 4o, Claude Sonnet 3.5, Gemini Advanced, ChatGPT 4.0 Mini, Gemini Flash, Claude Haiku). Participants completed a 50-question test. Correct answers were analyzed using Welch’s one-way analysis of variance (ANOVA), Levene’s test for homogeneity of variances, and Games-Howell post-hoc tests.

Results

Claude Sonnet 3.5 achieved the highest mean correct answers (46.4±0.548), followed by ChatGPT 4o (44.6±1.14) and Gemini Advanced (43.6±1.67). Postgraduates with 5+ years of experience scored 43.5±3.03, while fifth-year medical students scored the lowest (29.1±3.73). Welch’s ANOVA indicated significant group differences [F(9, 20.8): 31.3, p<0.001]. Post-hoc tests revealed LLMs outperformed most human groups, with Claude Sonnet 3.5 significantly surpassing Claude Haiku (mean difference: 9.6, p=0.028).

Conclusion

LLMs demonstrated superior performance compared to most human groups, indicating their potential as educational tools in emergency medicine.

Introduction

Emergency medicine requires quick and precise decision-making and skills, both of which are rigorously evaluated by examinations such as the Turkish Medical Specialty Examination (TUS). The TUS assesses competency through complex, scenario-based questions that demand clinical reasoning and practical knowledge (1). Developments in artificial intelligence (AI), particularly large language models (LLMs), have introduced tools capable of addressing medical queries, thereby increasing interest in their performance on standardized tests (2). When trained on extensive datasets, LLMs produce responses that resemble expert knowledge, suggesting applications in medical education and clinical support (3, 4). However, their ability to address specialty-specific, time-sensitive questions in emergency medicine remains largely unexamined (5).

Few studies have compared LLMs with human learners in specialty examinations. Studies on general medical licensing exams, such as the United States Medical Licensing Examination (USMLE), have shown that LLMs, such as ChatGPT, outperform medical students (6, 7). Emergency medicine, however, presents unique challenges; it requires rapid synthesis of clinical information under pressure and tests both human and AI capabilities (8). The literature lacks comparisons among LLMs, medical students at different training levels, and experienced postgraduates in this field (9). This gap is significant, as LLMs can enhance training by offering accessible educational resources, but their effectiveness in high-pressure specialties requires validation (10).

Unlike prior studies primarily focused on USMLE-style examinations, this study evaluates LLM performance using the TUS, which reflects a different linguistic and curricular context. By directly comparing LLMs with emergency medicine trainees and experienced specialists, this study provides specialty-specific evidence that extends existing AI examination literature.

This study aims to evaluate the performance of LLMs compared with that of medical students and postgraduates on a 50-question emergency medicine test from the TUS, using a cross-sectional design.

Methods

This cross-sectional study was approved by the İstanbul Medipol University Non-Interventional Clinical Research Ethics Committee (decision number: 664, date: 04.07.2024). The research was conducted at a Medipol Mega University Hospital Emergency Medicine Department in 2024. All participants provided informed consent prior to participation in the study. No patient data were collected or used in this research.

The study included 40 human participants: 10 fifth-year medical students, 10 sixth-year medical students, 10 first-year postgraduates, and 10 postgraduates with 5 or more years of experience in emergency medicine. Postgraduate participants consisted of emergency medicine residents (PG1) and board-certified emergency medicine specialists with at least five years of clinical experience (PG5+).

The inclusion criterion was enrollment in or graduation from a medical program, including postgraduates specializing in emergency medicine. No exclusion criteria were applied, as all eligible participants completed the test. Six LLMs “ChatGPT 4o, Claude Sonnet 3.5, Gemini Advanced, ChatGPT 4o Mini, Gemini Flash, and Claude Haiku” were also tested, each of which was evaluated five times to account for response variability. The participants were selected consecutively from the university’s medical program and its emergency medicine department to ensure a representative sample.

Large Language Model Configuration and Prompting

The following LLMs were evaluated: ChatGPT-4o and ChatGPT-4o Mini (OpenAI; accessed May 2024), Claude Sonnet 3.5 and Claude Haiku (Anthropic; accessed June 2024), and Gemini Advanced and Gemini Flash (Google DeepMind; accessed June 2024).

All models were accessed through their official web-based interfaces using the default inference settings. Temperature, top-p, and related sampling parameters were not manually adjusted to reflect typical real-world user conditions and enhance reproducibility. A standardized prompt was used for all LLMs. Models were instructed to select the single best answer from the provided multiple-choice options without providing explanations or other commentary.

No chain-of-thought reasoning or clinical justification was explicitly requested. Prompt used for all LLMs: “You are answering a multiple-choice emergency medicine examination question. Select the single best answer (A, B, C, D, or E). Do not provide explanations or additional text.”

The primary outcome was the number of correct answers on a 50-question multiple-choice test derived from past TUS examinations that focused on emergency medicine topics. Potential confounders, such as test familiarity or LLM version updates, were minimized by standardizing test conditions and using the latest model versions available in 2024. The test was validated by faculty experts for relevance and difficulty, ensuring content validity. The human participants completed the test under proctored conditions in a controlled environment.

While human participants completed the test under time-limited examination conditions, LLMs were not subject to time constraints.

The 50-item test was derived from TUS questions previously administered between 2018 and 2023. Items covered the core domains of emergency medicine, including trauma, toxicology, cardiology, neurology, infectious diseases, and critical care. Content validity was assessed by two senior emergency medicine faculty members who independently reviewed each item for relevance, clarity, and curriculum alignment. Disagreements were resolved by consensus. No items were modified from their original wording.

Each correct answer was scored as 1 point, with a maximum score of 50. Two researchers independently verified LLM responses, achieving high interrater reliability (kappa=0.95). No specific training was provided for data collection, but the process was standardized to reduce variability.

Selection bias was minimized by including all eligible participants consecutively, whereas information bias was reduced through standardized question presentation and scoring protocols. The variability in the LLM responses was addressed by conducting five test iterations per model and averaging the results. For LLMs, the unit of analysis was the model itself rather than individual runs. Five repeated runs were performed to estimate response variability; model-level mean scores and standard deviations (SDs) were used in all primary analyses.

The sample size was calculated to detect a five-point mean difference in the number of correct answers between groups, with 80% power and an alpha of 0.05, requiring at least 10 participants in each human group and five iterations per LLM, based on prior studies of medical examination performance. The number of correct answers was treated as a continuous variable and summarized via means and SDs.

Statistical Analysis

Statistical analyses included descriptive statistics to report means and SDs. Normality was confirmed via Shapiro-Wilk tests. Owing to nonhomogeneous variances [Levene’s test, F (9, 60): 2.5, p=0.017], Welch’s one-way analysis of variance (ANOVA) was used to assess differences in correct answers across groups, followed by Games–Howell post-hoc tests to identify specific group differences. As a sensitivity analysis, results were re-evaluated using model-level mean scores without treating individual LLM runs as independent observations. This approach yielded consistent group-level conclusions. No missing data were observed, so no imputation was needed. Analyses were performed using SPSS version 27. A p value <0.05 was considered significant. The primary outcome was the number of correct answers on the 50-question test.

Results

Participants

Forty human participants (mean age 28.5±4.2 years; 60% male) and six LLMs were included in the study. No exclusions occurred, and no data were missing.

Descriptive Data

Figure 1 shows the mean number of correct answers and the corresponding SDs. Table 1 presents the descriptive performance characteristics of all human groups and LLMs, including measures of central tendency and dispersion, to provide an overall comparison prior to inferential analyses. Claude Sonnet 3.5 scored highest (46.400±0.548), followed by ChatGPT 4o (44.600±1.140), and Gemini Advanced (43.600±1.670). Among human participants, PG5+ physicians scored 43.5±3.03, whereas fifth-year medical students scored the lowest (29.1±3.73).

Main Results

Welch’s ANOVA revealed significant group differences [F (9, 20.8): 31.3, p<0.001)] (Table 2).

Games-Howell post-hoc tests (Table 3) indicated that Claude Sonnet 3.5 outperformed both Claude Haiku (mean difference: 9.6; p=0.028) and most human groups (e.g., MS5; mean difference: 17.3; p<0.001). The difference between Claude Sonnet 3.5 and fifth-year medical students was large (mean difference: 17.3 points; 95% confidence interval: 12.4–22.1; Cohen’s d: 3.9). ChatGPT 4o and Gemini Advanced surpassed MS5 and MS6 (p<0.001). No significant differences were observed between PG5+ postgraduate physicians and top-performing LLMs (e.g., Claude Sonnet 3.5; mean difference: 2.9, p=0.21). While multiple pairwise comparisons were performed, the descriptive summary highlighted clear performance stratification across groups, which guided the interpretation of inferential results presented in Table 3.

Analysis of Performance Trends

The most proficient LLMs, Claude, Sonnet 3.5, ChatGPT 4o, and Gemini Advanced, exhibited noteworthy consistency, as indicated by their minimal SDs (0.548, 1.14, and 1.67, respectively). In contrast, ChatGPT 4o Mini exhibited the highest variability among all the groups (SD: 6.8), indicating potential inconsistencies in its performance. This higher variability may reflect architectural characteristics of lightweight models, which can be more sensitive to question phrasing and may lack domain-specific fine-tuning for emergency medicine content.

Among human groups, the performance gap between experience levels is notable. MS5 students, with the lowest mean score (29.1±3.73), were significantly outperformed by all LLMs and more experienced human groups (p<0.001 in most post-hoc comparisons).

This gap highlights the steep learning curve in emergency medicine, where foundational knowledge alone is insufficient without clinical exposure. MS6 students, after an additional year of training, improved to 33.4±4.86, but still lagged behind postgraduates and LLMs. The post-hoc analysis revealed that the difference between MS5 and MS6 was not statistically significant (mean difference: 4.3, p=0.48), suggesting that an additional year of medical school training may not substantially enhance performance on specialty-specific exams without targeted emergency medicine experience. Postgraduates with 5+ years of experience (PG5+) achieved a mean score of 43.5±3.03 and closely approached the performance of the top LLMs. The absence of a significant difference between PG5+ and Claude Sonnet 3.5 (mean difference: 2.9, p=0.21) or between PG5+ and ChatGPT 4o (mean difference: 1.1, p=0.985) underscores the competitive performance of experienced clinicians in this domain. However, their SD (3.03) indicates greater variability than that of the top LLMs, possibly reflecting differences in individual expertise or test-taking strategies among the participants.

Levene’s test result [F(9, 60): 2.5, p=0.017] indicates heterogeneity of variances across groups, consistent with the observed differences in SDs. For example, Claude Sonnet 3.5’s exceptionally low SD (0.548) contrasts sharply with ChatGPT 4o Mini’s high SD (6.8). This heterogeneity justified the use of Welch’s ANOVA and Games–Howell post-hoc tests, which are robust to unequal variances.

The post-hoc analysis further reveals that the performance hierarchy among LLMs is not uniform. The significant outperformance of Claude Sonnet 3.5 relative to Claude Haiku (mean difference: 9.6, p=0.028) indicates substantial differences in capability even within the same family of models. Similarly, ChatGPT 4o outperforms Claude Haiku with a mean difference of 7.8 (p=0.052), although this result is slightly above the conventional significance threshold, indicating a trend rather than a statistically significant difference. A qualitative review suggested that both human participants and LLMs most frequently erred in toxicology and multi-step trauma questions, whereas cardiology and infectious disease questions were associated with higher accuracy across groups.

Discussion

This study demonstrated that LLMs, particularly Claude Sonnet 3.5, outperformed most human groups who took the 50-question emergency medicine test administered by the TUS. The top-performing LLMs reached scores comparable to or higher than those of postgraduates with more than five years of experience.

Emergency medicine plays a pivotal role in managing acute, life-threatening conditions, demanding rapid decision-making and extensive clinical knowledge (11). Examinations such as the TUS ensure practitioners are prepared for high-pressure scenarios, highlighting the importance of valid training programs (1, 12). The global shortage of emergency physicians highlights the need for innovative solutions to support training, especially in areas with limited access to experienced instructors (13). LLMs offer a potential solution to these gaps by providing accurate answers to complex questions. The performance of these methods in this study suggests that they could improve preparation for standardized tests, such as TUS. The frequent physiological changes during acute medical events highlight the need for accurate and easily accessible knowledge, which LLMs seem to provide effectively (14).

This study found that Claude Sonnet 3.5 outperformed most human groups, which is consistent with recent literature on AI in medical education. Roos et al. (15) reported that LLMs outperformed medical students on MCAT-style questions, suggesting their strength in knowledge-based assessments. A meta-analysis of 32 studies by Waldock et al. (16) confirmed that LLMs were consistently superior in general medical examinations, although specialty-specific research remains limited. A recent study from Japan by Akitomo et al. (17) revealed similar AI performance in dental board exams, supporting the current findings. However, Johri et al. (18) noted that LLMs sometimes struggle with context-specific reasoning in clinical scenarios; this challenge was not evident in this study, likely owing to the structured nature of the test questions. These results suggest that LLMs perform well in standardized settings but may face challenges in less structured environments (19, 20).

Although a large number of pairwise comparisons were conducted, the primary aim of this study was not to interpret each contrast in isolation but to identify overarching performance patterns across participant groups and model types.

When the results are examined collectively, three consistent trends emerge. First, a marked performance gap is observed between undergraduate medical students and both LLMs and experienced emergency physicians. Second, performance convergence is evident between senior emergency physicians (PG5+) and top-performing LLMs, suggesting comparable performance on examination-based assessments. Third, substantial variability among LLMs themselves highlights the importance of model architecture and design choices in determining examination performance.

Given their consistency and high accuracy, advanced LLMs have the potential to be reliable tools for emergency medicine training. However, the variability among models, such as ChatGPT 4.0 Mini, suggests that not all LLMs are equally suitable for such applications, necessitating careful model selection for educational purposes (21). For human learners, these data highlight the importance of clinical experience for improving performance, as evidenced by the progression from MS5 to PG5+ (22).

The findings are likely applicable to academic settings with standardized emergency medicine examinations, particularly in urban tertiary institutions. The strong performance of LLMs suggests that they could enhance training by providing scalable educational resources, especially where access to instructors is limited (23, 24). Their epidemiological and clinical importance lies in improving training efficiency, which could help increase the supply of qualified emergency medicine specialists.

The results indicate the potential of LLMs as educational tools in emergency medicine, based on their test performance. Their integration into medicine could support exam preparation and knowledge reinforcement. Further studies are needed to validate their clinical utility and address variability in responses.

Study Limitations

The study has several limitations that should be taken into account. The single-center design may limit its applicability to other educational settings. The controlled test environment does not replicate clinical pressures, potentially overestimating LLM performance. The variability in LLM responses across iterations indicates potential inconsistencies, which could affect reliability.

Limited qualitative analysis of response patterns restricts deeper insight into LLM reasoning processes. The absence of time pressure for LLMs may partially lead to an overestimation of their comparative performance. Given the large number of pairwise comparisons, the interpretation focused on contrasts with the greatest educational and clinical relevance rather than an exhaustive discussion of all individual differences.

Conclusion

This study demonstrates that LLMs outperform most human groups and perform comparably to experienced postgraduates on emergency medicine exam questions. These findings suggest that LLMs could be valuable educational tools for enhancing medical training in critical specialties. Integrating LLMs into educational programs may improve exam preparation and knowledge acquisition, particularly in resource-limited settings. Future research should investigate their clinical applications, address response variability, and validate findings across diverse contexts to ensure their effective integration into medical education.