New Delhi: Those who use apps to track food intake are aware of the challenges posed by traditional Indian meals. Understanding an Indian meal is far more
complex and challenging than analysing a burger or a salad. Researchers from the Centre for Visual Information Technology at IIIT Hyderabad are developing the AI tools necessary to understand a typical Indian thali, that often contains multiple dishes such as rice, dal, roti, chutney, curd and salad, with mixed textures and overlapping ingredients. The project began with a real-world healthcare need, focused on monitoring nutrition for pregnant women.
The Food Scanner system in action. (Image Credit: IIIT-H).
Lead of the project, CV Jawahar explains the challenge posed by Indian thalis, “If you are given a full plate of typical Indian food that not only has multiple dishes, but mixed ones like rice topped with dal, a roti hidden under a papad.… how do you understand what is there on a plate and eventually its nutritional value?” Existing food-scanner and nutrition apps assume a fixed menu and stable recipes. Indian food can vary every day even in the same place, and cafeteria menus can change overnight. Training a traditional supervised model repeatedly is impractical.
A zero-shot approach
The researchers have built a working prototype that can analyse an image of an Indian meal and estimate its contents. Instead of retraining models, the researchers build a zero-shot system, that first identifies food regions without recognising the item. Then, instead of rigid classification, the system uses retrieval-based prototype matching, which is scalable, flexible and realistic for places such as cafeterias and hospital messes. The current system works with an overhead camera, but the researchers plan to develop an app. To prepare for this transition, the team has captured data on Indian thalis from multiple angles, allowing for the food to me captured from a smartphone camera.
A sample from the Indian Thali Dataset on the left, and the Weight Estimation Dataset on the right. (Image Credit: IIIT-H).
One of the major challenges is the mixing of food. Rice, dal, gravies and vegetables can all get mixed up in an Indian thali, which makes calorie and nutrition estimation challenging. The research was presented at the 16th Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2025). Jawahar says, “We still have not addressed some of the fundamental issues.” The researchers aim to develop a system that combines food recognition, weight estimation and nutritional breakdown that is sufficiently accurate for everyday use.
