Graphcore and Arm Researchers Compress 11 Billion Parameter AI Model for Android Phones
A collaborative effort between Graphcore and Arm researchers has successfully compressed a large 11 billion parameter vision-language model (VLM) to fit within the storage constraints of an Android phone. This VLM, which originally required over 21 GB for its weights, was reduced to approximately 3.7 GB. The achievement involved developing a novel format called S3D8, co-designed with Arm CPUs, and implementing a quantization-aware training procedure that samples diverse prompts to maintain the original model's capabilities. The team also created a custom C++ engine for executing the Llama 3.2 Vision Instruct VLM, packaged into a native library, and an Android application to demonstrate the method. While the complete method is running on a phone, it is not yet ready for mainstream use due to the model's weight and the need for Vulkan kernels to improve prefill time. 
