Innovative Use of Microcontroller to Run Large Language Model with Limited Resources
A developer has successfully run a large language model (LLM) with 28.9 million parameters on an ESP32-S3 microcontroller, which has limited memory resources. This achievement was made possible by using an architectural technique called Per-Layer Embeddings, which stores most of the model's parameters in flash memory. This approach allows the model to operate within the microcontroller's constraints, generating text at a rate of 9.5 tokens per second. The project demonstrates the potential for running complex algorithms on low-cost hardware without relying on cloud connectivity. 
