OMLX Introduces LLM Inference Server with Advanced Caching for Apple Silicon
OMLX has launched an LLM inference server specifically optimized for Apple Silicon, featuring continuous batching and a tiered KV cache system. This server aims to enhance the performance and efficiency of running large language models (LLMs) on Mac devices. The tiered KV cache utilizes both RAM (hot tier) and SSD (cold tier) to store frequently accessed and less frequently accessed data, respectively. This design allows for faster retrieval of cached information, even across conversation context changes or server restarts, by offloading blocks to SSD in safetensors format when the hot cache is full. The system also supports multi-model serving, including text LLMs, vision-language models (VLMs), OCR models, embeddings, and rerankers, with features like LRU eviction, manual load/unload, model pinning, and per-model TTL to manage memory usage effectively. The server is compatible with macOS 15.0+ (Sequoia), Python 3.11–3.13, and Apple Silicon (M1/M2/M3/M4/M5). 
