Beyond the Leaderboard
For years, the quality of an AI model was judged by its performance on standardised academic benchmarks. These tests, often involving massive multiple-choice questions or text prediction, created public leaderboards that ranked models from labs like Google,
OpenAI, and Anthropic. While useful, this approach has a growing problem: it’s like judging a master chef solely on their ability to name ingredients. These benchmarks don't effectively measure an AI's ability to perform complex, multi-step tasks, use software tools, or adapt to novel situations. Critics argue this has led to models that are good at passing exams but can be brittle or unreliable in the real world, a risk that grows as AI is integrated into critical sectors like finance, healthcare, and public safety.
A New Global Standard for Real-World Skills
In response, a new philosophy for AI testing is taking hold, one focused on practical capability. A prime example is the new Artificial Intelligence Technology Evaluation (AITE) initiative recently launched by the U.S. National Institute of Standards and Technology (NIST). Instead of public benchmarks, AITE provides a secure, sequestered environment where developers can voluntarily submit their models. The AI is then tested against private, 'blind' datasets on tasks that mimic real-world challenges. The goal is to provide a more rigorous and objective assessment of how an AI performs, preventing developers from simply training their models to ace a specific, known test. This move signals a major shift in how the world's most powerful AI systems will be judged, prioritising practical competence over theoretical knowledge.
India's Foundation for a Practical Future
This global pivot towards practical AI evaluation aligns perfectly with India's own strategic investments in the sector. The government's flagship IndiaAI Mission has been quietly building the infrastructure for a hands-on approach to artificial intelligence. As part of this push, it has established 27 data and AI labs across 21 states and union territories. While the global focus is on testing practical skills, India's initiative is focused on building them from the ground up. These labs are designed to provide students and professionals with hands-on training and experience in data science and AI, creating a talent pool that understands not just the theory but the application of AI. With nearly 200 more labs planned, India is creating a nationwide network to foster the exact kind of practical expertise the next generation of AI will demand.
What Does 'Practical' Actually Mean?
So, what does a 'practical' AI test look like? It's less about answering a trivia question and more about completing a project. For example, a test might ask an AI to act as an agent that needs to research a topic using a live web browser, synthesise the information into a report, create a slide deck summarising the findings, and then email it to a colleague. Other practical evaluations might involve an AI operating a drone to find an object, using code interpreters to solve a complex data analysis problem, or managing a workflow in a quality control lab. These tasks require reasoning, planning, and the ability to use other software tools—skills that are essential for AI to become a truly useful assistant rather than just a conversationalist.
Why This Matters for Indian Tech
For India's booming tech industry and aspiring developers, this shift is a massive opportunity. As the goalposts for 'good AI' move from benchmark scores to practical capabilities, the playing field changes. It's no longer just about having the biggest model or the most training data. Success will increasingly depend on building AI systems that are reliable, safe, and genuinely useful for specific, real-world jobs. India's investment in the 27 AI labs is a strategic move to equip its workforce for this new reality. By focusing on hands-on skills, the IndiaAI Mission is preparing a generation of developers who can build and fine-tune the kind of practical AI agents that businesses will be looking to hire and deploy. This ensures India isn't just a consumer of next-generation AI, but a key builder of it.
