In-Network Compute Accelerates AI Operations, Addressing Inference Workload Demands
The artificial intelligence industry is experiencing a significant shift as inference workloads surpass training workloads in volume, creating an urgent need for more efficient collective operations within network switches, known as In-Network Compute (INC). Traditional AllReduce algorithms, commonly used to orchestrate matrix multiplications across distributed AI models, involve a multi-step 'Ring' process that incurs repeated overheads. However, INC-enabled switches, such as the Scorpio™ X-Series 320 Lane Smart Fabric Switch, streamline this process. With INC, GPUs send data slices directly to the switch, which then reduces and distributes them while simultaneously streaming the next slice. This method effectively collapses the many-step ring behavior into a more direct operation, significantly reducing fixed overhead costs and improving data transfer efficiency. This advancement is particularly beneficial for small message sizes, which are characteristic of the decode phase in AI inference workloads.
