As AI models race through the Mushroom Kingdom, some shine with lightning-fast reflexes while others stumble—raising big questions about the future of AI evaluation.
Super Mario Bros., the iconic game that once tested our reflexes and patience, is now pushing the limits of artificial intelligence. In a surprising twist, researchers at Hao AI Lab, University of California San Diego, are using the game as a battlefield for AI models, measuring how well they handle split-second decisions and unpredictable obstacles.
Who Came Out on Top?
In the ultimate test of AI agility, Claude 3.7 and Claude 3.5 raced through the pixelated chaos of Super Mario Bros. like seasoned speedrunners, dodging obstacles with quick reflexes and smart decision-making. This research sheds light on how AI models handle fast, action-based tasks rather than just text-based reasoning.
According to the research, these models didn’t just play the game, they mastered its rhythm and adapted in real time while rivals struggled to keep up.
Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o struggled, particularly due to latency issues, which hindered their ability to react in real time. The slowest model, OpenAI’s o1, performed the worst, as its decision-making delays made it nearly impossible to keep up with the game’s rapid pace.
How AI Actually Played Mario
Unlike traditional AI benchmarks, where models process static data, this experiment required AI to play the game using an emulator. Through the GamingAgent framework, the models analyzed in-game screenshots and generated Python-based commands to maneuver Mario, dodge obstacles, and tackle enemies.
This method challenges AI to interpret visual data and react instantly, a critical skill for real-world applications like robotics and autonomous systems. However, this approach has also sparked controversy!
While gaming benchmarks offer a dynamic way to test the capabilities of artificial intelligence, some experts question their effectiveness. AI researcher Andrej Karpathy pointed out that there is an "evaluation crisis" in AI metrics.
Traditional benchmarks like MMLU are becoming outdated and newer ones, such as Chatbot Arena, are potentially overfitting AI models. This raises doubts about whether performance in a video game truly reflects the potential of AI use cases in the real world?!
Is Super Mario the Benchmark Worth Trusting?
Despite skepticism, Super Mario Bros. has opened up an exciting new frontier for AI evaluation. Some AI models may dominate in the classic Mushroom Kingdom challenge, but does that really translate to real-world intelligence?
As AI keeps advancing, the debate over what truly defines smart technology is far from over!
