توسعه اجتماعی

توسعه اجتماعی

بازطراحی آزمون های استاندارد در مواجهه با هوش مصنوعی: تحلیل مزایا و معایب مدل شفاهی

نوع مقاله : مقاله پژوهشی

نویسنده
گروه علوم تربیتی، دانشگاه پیام نور، تهران، ایران
چکیده
پژوهش حاضر با هدف بازطراحی آزمون‌ها در عصر هوش مصنوعی، به بررسی و مقایسه عملکرد سه مدل زبانی پیشرفته (GPT-4o، Google Gemini 1.5 Pro، Anthropic Claude 3.5 Sonnet) با دانشجویان انسانی در قالب آزمون‌های کتبی و شفاهی بازطراحی ‌شده است. روش تحقیق به صورت ترکیبی (کمی و کیفی) بوده و داده‌ها از طریق آزمون‌های استاندارد زبان و شناختی (IELTS، Raven’s Progressive Matrices، N-Back)، تحلیل خطا و پرسشنامه تجربه کاری گردآوری شد. جامعه آماری تحقیق شامل دانشجویان پزشکی دانشگاه علوم پزشکی آزاد تهران در سال تحصیلی 1404-1403 بود که معادل 585 نفر بوده و نمونه‌گیری به روش تصادفی طبقه‌بندی ‌شده انجام شد. حجم نمونه با استفاده از فرمول کوکران؛ معادل 232 نفر برآورد گردید.  نتایج نشان داد که مدل‌های هوش مصنوعی در اغلب شاخص‌ها (دقت، سرعت پاسخ‌گویی و میانگین نمرات) عملکردی بالاتر از انسان داشتند، هر چند نوع خطاهای آنان ماهیتاً با خطاهای انسانی متفاوت بود. همچنین، نسخه بازطراحی ‌شده آزمون شفاهی موجب بهبود تجربه کاربری، افزایش وضوح دستورالعمل‌ها، کاهش استرس و ارتقای ادراک عدالت در میان شرکت‌کنندگان شد. میان مؤلفه‌های پرسنامه تجربه کاری نیز همبستگی معناداری در  سطح 0.01 وجود داشت ( p<0.01). یافته‌ها بیانگر آن است که اگرچه هوش مصنوعی توانسته است در بازتولید دانش و پردازش سریع اطلاعات از انسان پیشی گیرد، اما آزمون‌های شفاهی و تعاملی همچنان ظرفیت بالاتری در سنجش مهارت‌های استدلالی، خلاقیت و تفکر انتقادی دارند.
کلیدواژه‌ها
موضوعات

عنوان مقاله English

Redesigning Standardized Assessments in the Era of Artificial Intelligence: A Critical Analysis of the Strengths and Weaknesses of Oral Examination Models

نویسنده English

Anita Homayounnia
Department of Educational Sciences, Payam Noor University, Tehran, Iran
چکیده English

The present study, aimed at redesigning examinations in the era of artificial intelligence, investigates and compares the performance of three advanced language models (GPT-4o, Google Gemini 1.5 Pro, Anthropic Claude 3.5 Sonnet) with human students within the framework of redesigned written and oral tests. This research employed a mixed-methods approach (quantitative and qualitative). Data were collected through standardized linguistic and cognitive assessments (IELTS, Raven’s Progressive Matrices, N-Back), error analysis, and user experience questionnaires. The statistical population comprised medical students at Islamic Azad University, Tehran Medical Sciences, during the 2024–2025 academic year (N=585). Stratified random sampling was applied, and the sample size was estimated at 232 participants using Cochran’s formula. Results indicated that AI models outperformed humans in most indicators (accuracy, response speed, and average scores), although the nature of their errors differed substantially from human errors. Furthermore, the redesigned oral examination improved user experience, enhanced clarity of instructions, reduced stress, and strengthened perceptions of fairness among participants. There was also a significant correlation between the components of the work experience questionnaire at the 0.01 level (p<0.01). Findings suggest that although artificial intelligence has surpassed humans in knowledge reproduction and rapid information processing, oral and interactive assessments still hold greater potential for evaluating reasoning, creativity, and critical thinking skills. Accordingly, redesigning examinations with a focus on interactive, context-based models not only safeguards the validity and fairness of assessment systems against AI-driven challenges but also creates a strategic opportunity for transforming the philosophy of education and evaluation.

کلیدواژه‌ها English

Artificial Intelligence
Human Students
Written and Oral Examinations
Error
Bazgir Jahantighi, Z.; Heydari Azad, Z.; Mir, S.; Shahraki, S. (2025). “Challenges of using artificial intelligence in designing standard tests by teachers”, Proceedings of the First International Conference of Leading Teachers in the Era of Transformative Research, Ahvaz. (Persian).
Celik, I.; Dindar, M.; Muukkonen, H.; Järvelä, S. (2022). “The promises and challenges of artificial intelligence for teachers: A systematic review of research”, TechTrends, 66(4), 616–630. https://doi.org/10.1007/s11528-022-00715-y.
Chang, L. (2024). “An Empirical Study on the Impact of Artificial Intelligence Assisted Vocational English Teaching on Students' Intercultural Communication Skills”, Conference: 2024 5th International Conference on Information Science and Education (ICISE-IE). DOI:10.1109/ICISE-IE64355.2024.11025429.
Dindar, M., Muukkonen, H., & Järvelä, S. (2022). “The promises and challenges of artificial intelligence for teachers: a systematic review of research”, Techtrends, 66(4), 616-630. https://doi.org/10.1007/s11528-022-00715-y. (Persian).
Dong, Y., Liu, Z., Wang, H., & Chen, Y. (2024). “Generalization or memorization: Data contamination and trustworthy evaluation for large language models”, arXiv preprint arXiv:2402.15938. doi: 10.18653/v1/2024.findings-acl.716.
Hamedi-Nasab, S., Rahimi, Sh. (2024). “Barriers and challenges of implementing artificial intelligence in the higher education system”, Quarterly Journal of Educational Planning Studies, 13(26), 57–73. doi: 10.22080/eps.2025.28149.2295. (Persian).
Hao, J.; von Davier, A.A.; Yaneva, V.; Lottridge, S.; von Davier, M.; Harris, D.J. (2024). “Transforming assessment: The impacts and implications of large language models and generative AI”, Educational Measurement: Issues and Practice, 43 (2), 16–29. doi: 10.1111/emip.12602. (Persian).
Hashemi, H.; Shahsavari, M.; Mahdi, F.; Ghaderi, D. (2025). “Investigating the application of artificial intelligence in designing intelligent tests and more accurate assessment of students’ abilities”, Proceedings of the First International Conference of Leading Teachers in the Era of Transformative Research, Ahvaz. (Persian).
Huawei, S.; Aryadoust, V. (2023). “A systematic review of automated writing evaluation systems”, Education and Information Technologies, 28 (1), 771–795. doi: 10.1007/s10639-022-11200-7.
Jiao, H.; Yadav, C.; Li, G. (2023). “Integrating psychometric analysis and machine learning to augment data for cheating detection in large-scale assessment”, Preprint from PsyArXiv . https://doi.org/10.31234/osf.io/fjz2c.
Jingshan, H. (2023). “Analysis of the Application of Artificial Intelligence in Education and Teaching”, Advances in Educational Technology and Psychology, 7(2), 63-67. doi: 10.23977/ aetp.2023.070210.
Kamalov, F.; Sulieman, H.; Calonge, D.S. (2021). “Machine learning based approach to exam cheating detection”, PLOS ONE, 16 (8), e0254340. doi: 10.1371/journal.pone.0254340. (Persian).
Karademir Coskun, T.; Alper, A. (2024). “Evaluating the evaluators: A comparative study of AI and teacher assessments in Higher Education”, Digital Education Review, 45, 124- 140. https://doi.org/ 10.1344/der.
Keykha, A.; Mohammadi, H.; Darabi, F.; Hosseini, S.S. (2025). “Identifying the Applications of Artificial Intelligence in the Assessment of Medical Students”, Strides Dev Med Educ, 22(1), e1512. doi:10.22062/sdme.2025.200833.1512. (Persian).
Khera, R.; Simon, M.A.; Ross, J. S. (2023). “Automation bias and assistive AI: Risk of harm from AI-driven clinical decision support”, JAMA, 330 (23), 2255–2257. doi: 10.1001/jama. 2023.22557.
Kitsara, I. (2022). “Artificial intelligence and the digital divide: From an innovation perspective”, In Platforms and artificial intelligence: The next generation of competences (pp. 245–265). Springer.
Kiss, G.; Moreira Bastos, S. (2025). “Experiences of online and face-to-face assessment of Artificial Intelligence subject”, Procedia Computer Science, 263, 385–390.
Kumar, Y.; Li, J.; Singh, R. (2023). “A testing framework for AI linguistic systems (testFAILS)”, Kean University Research Publications.
Lee, U.; Jung, H.; Jeon, Y.; Sohn, Y.; Hwang, W.; Moon, J.; Kim, H. (2024). “Few-shot is enough: Exploring ChatGPT prompt engineering method for automatic question generation in english education”, Education and Information Technologies, 29, 11483–11515. doi: 10.1007/s10639-023-12249-8.
Li, J.; Tang, T.; Zhao, W.X.; Nie, J.-Y.; Wen, J.-R. (2024). “Pre-trained language models for text generation: A survey”, ACM Computing Surveys, 56 (9), 1–39. doi: 10.1145/3649449.
Luan, H.; Géczy, P.; Lai, H.; Gobert, J.; Yang, S.; Ogata, H.; … & Tsai, C. (2020). “Challenges and future directions of big data and artificial intelligence in education”, Frontiers in Psychology, 11, 580820.  https://doi.org/10.3389/fpsyg.2020.580820.
Lund, B.D.; Wang, T. (2023). “Chatting about ChatGPT: How may AI and GPT impact academia and libraries?”, Library Hi Tech News, 40(3), 26–29. https://doi.org/10.1108/lhtn-01-2023-0009.
Novidi, A.; Gheysari, H. (2023). “Investigating the impact of artificial intelligence on improving organizational productivity”, Dynamic Management and Business Analysis, 2(3), 1–8. doi.org/10.22034/pmba.2023.710528. (Persian).
Nguyen, T. (2023). “Exploring the efficacy of chatgpt in language teaching”, Asiacall Online Journal, 14(2), 156-167. https://doi.org/10.54855/acoj.2314210.
Pack, A.; Barrett, A.; Escalante, J. (2024). “Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability”, Computers & Education: Artificial Intelligence, 6, 100234. https://doi.org/ 10.1016/j.caeai.2024.100234.
Parapadakis, D. (2020) “Can artificial intelligence help predict a learner’s needs? Lessons from predicting student satisfaction”, London Review of Education, 18 (2), 178–195. https://doi.org/ 10.14324/LRE.18.2.03.
Rahm, L. (2023). “Education, automation and AI: a genealogy of alternative futures”, Learning, Media and Technology, 48 (1), 6–24. doi:10.1080/17439884.2021.1977948.
Ranger, J.; Schmidt, N.; Wolgast, A. (2023). “Detecting cheating in large-scale assessment: The transfer of detectors to new tests”, Educational and Psychological Measurement, 83 (5), 1033–1058. doi: 10.1177/00131644221132723.
Reib, M. (2021). “The use of ai in education: practicalities and ethical considerations”, London Review of Education, 19 (1), 5, 1–14. https://doi.org/10.14324/lre.19.1.05.
Sayed, B., Younes, Z., Alkhayyat, A., Adhamova, I., Teferi, H. (2024). “To be with artificial intelligence in oral test or not to be: A probe into the traces of success in speaking skill, psychological well-being, autonomy, and academic buoyancy”, Language Testing in Asia, 14(1), 49. https://doi.org/ 10.1186/ s40468-024-00321-0. (Persian).
Sayin, A., & Gierl, M. (2024). “Using OpenAI GPT to generate reading comprehension items”, Educational Measurement: Issues and Practice, 43 (1), 5–18. doi: 10.1111/emip.12590.
Scarfe, P., Watcham, K., Clarke, A., & Roesch, E. (2024). “A real-world test of artificial intelligence infiltration of a university examinations system: a “Turing Test” case study”, PLOS ONE, 19(6): e0305354. https://doi.org/10.1371/journal.pone.0305354.
Selwyn, N., & Gašević, D. (2022). “Assessment in the age of artificial intelligence”, Computers and Education: Artificial Intelligence, 3, 100075. https://doi.org/10.1016/j.caeai.2022.100075.
Swiecki, Z.; Khosravi, H.; Chen, J.; Martinez-Maldonado, R.; Lodge, J. M., Milligan, S., 
Stanoyevitch, A. (2024). “Online assessment in the age of artificial intelligence”, Discover Education, 3(126). https://doi.org/10.1007/s44217-024-00212-9.
Tate T. P., Steis, J., Bailey, D., Steve, G., Moon, Y., Ritchie, D., Tseng, W., Warschauer, M. (2024). “Can AI provide useful holistic essay scoring?”, Computers & Education: Artificial Intelligence, 7, 100255. https://doi.org/10.1016/j.caeai.2024.100255.
Wang, Sh., Wang, F., Zhu, Zh., Wang, J., Tran, T., & Du, Zh. (2024). “Artificial intelligence in education: A systematic literature review”, Expert Systems with Applications, 252, 124167.
Wecks, J. O., Voshaar, J., Plate, B. J., Zimmermann, J. )2024(. “Generative AI Usage and Exam Performance”, SSRN Electronic Journal. DOI:10.2139/ssrn.4812513.
Williamson, B., Macgilchrist, F., & Potter, J. (2023). “Re-examining AI, automation and datafication in education”, Learning, Media and Technology, 48 (1), 1–5. doi: 10.1080/17439884.2023.2167830.
Wongvorachan, T., Lai, K. W., Bulut, O., Tsai, Y.-S., & Chen, G. (2022). “Artificial intelligence: Transforming the future of feedback in education”, Journal of Applied Testing Technology, 23(1), 95–116. Retrieved from. http://jattjournal.net/ index.php/ atp/article/view/170387.
Yu, L. & Yu, Z. (2023). “Qualitative and quantitative analyses of artificial intelligence ethics in education using vosviewer and citnetexplorer”, Frontiers in Psychology, 14, 1061778. https://doi.org/10.3389/fpsyg.2023.1061778.
Zaeri, F., Namvar, K., Zarei, M. A., & Ahmadi, F. (2025). “Examining the use of artificial intelligence in designing adaptive learning assessments”, Proceedings of the First International Conference on New Horizons in Educational Psychology and Teaching-Learning from the Teacher’s Perspective, Urmia. (Persian).
Zawacki‐Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). “Systematic review of research on artificial intelligence applications in higher education – where are the educators?”, International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0
Zhai, X., & Nehm, R. H. (2023). “AI and formative assessment: The train has left the station”, Journal of Research in Science Teaching, 60(6), 1390–1398. https://doi.org/10.1002/tea.21885.
Zhang, M., Ruan, C., & Johnson, M. S. (2024). “Explainable AI: Exploring subgroup differences in shortresponse scoring”, Paper presented at the annual meeting of the National Council for Measurement in Education, Philadelphia, PA.
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., … & Duan, N. (2024). “AGIEval: A human-centric benchmark for evaluating foundation models”, Findings of the Association for Computational Linguistics: NAACL 2024, 2299–2314. Doi: 10.18653/v1/2024.findings-naacl.149
Zhou, T., & Jiao, H. (2023). “Exploration of the stacking ensemble machine learning algorithm for cheating detection in large-scale assessment”, Educational and Psychological Measurement, 83 (4), 831–854. doi:10.1177/00131644221117193.