The Efficiency of Using ChatGPT in Designing EFL Reading Comprehension Tests

Abstract

Designing language tests manually and ensuring their quality is a complex process that needs a knowledgeable teacher with experience, especially for reading comprehension tests. Artificial Intelligence (AI) opens up new ways to assist in language education, generally, and the assessment process, particularly. This study compares EFL teacher-made and ChatGPT-designed reading comprehension tests by examining the performance of 51 second-year university students on the tests in order to examine the efficiency of using ChatGPT compared to humans in designing language tests. The tests consist of grammar, vocabulary, and reading comprehension skill items, and each test is on 20 marks. The comparison was done by giving both tests to the same group of students at the same time and place. The data was analyzed through the use of SPSS Software; both descriptive statistics and multivariate analysis using Pillai’s Trace for analyzing students’ responses indicated that there are significant differences in students’ performance across both tests and across the reading skills (vocabulary, grammar, and reading comprehension). Additionally, the test types and the skills affected students’ performance. The findings showed that the test created by ChatGPT is more effective and efficient in vocabulary and reading comprehension, but the teacher-made test is more effective in assessing students’ performance in grammar. When AI-generated tests are designed and reviewed properly, they can perform comparably to teacher-made tests in assessing reading skills. The study concludes that ChatGPT has a strong capability to assist teachers in designing reliable and valid test questions. The study highlights the value of a complementary approach that integrates AI-designed with teacher-designed tests to enhance the effectiveness and efficiency of the EFL language test design process.