77% vs 47%: At US university, students trump ChatGPT in accounting exam
Despite this, the researchers described ChatGPT's performance as ‘impressive’ and ‘game-changing.’
ChatGPT, the revolutionary AI-powered chatbot developed by OpenAI, was recently put to test at the Brigham Young University (BYU) in the US, with researchers, at the conclusion of the exercise, observing that BYU students fared better than ChatGPT in the accounting exam.
Despite this, the OpenAI tool's overall performance was found to be ‘impressive’ and a 'game changer that will change the way everyone teaches and learns - for the better.'
Why was the ‘exam’ conducted?
The exercise was conducted for one simple reason: researchers wanted to see how ChatGPT would fare in the accounting subject. They have published their findings in the journal ‘Issues in Accounting Education.’
How much did each side score?
While the students scored an overall average of 77% (76,7%), the AI bot scored 47% (47.4%). In 11.3% questions, it performed better than the humans, dorng particularly well in accounting information systems (AIS) and auditing. In tax, financial, and managerial assessments, on the other hand, its performance was ‘extremely poor.’
Other observations
The team made these key observations as well:
(1.) The chatbot did better in true/false questions (68.7% correct) and multiple-choice questions (59.5%).
(2.) It, however, struggled with short-answer questions (28.7% to 39.1%) and, in general, higher-order questions.
(3.) It also provided explanations for several answers, even if the answers were wrong; in other cases, it selected wrong multiple-choice answers, despite giving accurate descriptions.
(4.) Further, it was found to ‘make up’ facts, generating real-looking references that were, obviously, fabricated.
(5.) Similarly, the chatbot made errors for simple mathematical operations, like adding numbers in a subtraction problem, or dividing numbers incorrectly.
Who all participated in the study?
The study had total 327 co-authors from Brigham and 186 other educational institutes in 14 countries (including the US). Besides sitting for the ‘exam,’ BYU students fed text book questions to ChatGPT.
(With PTI inputs)
