Cohere speeds model serving with a single 'megakernel.' Software optimization is intended to reduce inference cost

In a technical article, Cohere focuses on an aspect of AI that is often more important to corporate economics than another point in a quality benchmark: model-serving efficiency. Its 'megakernel' combines many decoding steps of the North Mini Code model into one large program running on the graphics processor. The goal is to reduce delays between individual kernels and make better use of memory bandwidth. The authors summarize the principle by saying that the right question is how efficiently memory bandwidth is used. In a test on one Nvidia H100 chip, Cohere compared its system with vLLM 0.24. With a batch of eight requests and a realistic distribution of experts, the megakernel was 1.32 times faster; with an artificially uniform distribution it was 1.14 times faster. The result shows that real operation of a mixture-of-experts model can benefit from the fact that requests often activate similar parts of the network. The test is narrowly limited to decoding and a specific model, however, so it cannot be translated directly into a general claim about all workloads. The vendor openly states the limitations as well. The current version does not support combined processing of the initial phase and decoding, the maximum batch size is eight, and initial input processing still uses conventional PyTorch kernels. This is an important detail for infrastructure buyers: a faster sub-stage does not necessarily mean the same speedup for the entire service. It depends on input length, output length, the number of concurrent users and the distribution of real workload. For enterprise AI operators, the news is interesting mainly in economic terms. If more requests can be served on the same hardware, the cost of one response falls and the need to buy additional accelerators is postponed. At high volume, software optimization can save as much as a cheaper chip. Companies should therefore compare not only the purchase price of hardware, but the total cost per million tokens under their specific workload, including energy, unused capacity and operating software. The broader impact is strategic. The AI infrastructure market will be shaped not only by competition among chip manufacturers, but also among serving layers, compilers and libraries that can make effective use of hardware. Optimization tightly bound to one model can deliver high performance while increasing the cost of switching models. Companies should therefore measure portability and maintenance costs alongside speed. The cheapest inference today may not be the cheapest architecture after two years of change. For financial management of AI, it is therefore useful to run an operational test on the company's own data. A vendor benchmark shows technological potential, but does not answer whether the workload is dominated by long inputs, short responses or peak concurrency. Only the profile of actual demand makes it possible to choose the appropriate kernel, batch size and hardware. Inference optimization is gradually becoming a separate discipline of cost management, much like cloud-infrastructure optimization. KEY TERMS Inference: The phase in which a trained model processes an input and generates an answer. Compute kernel: A program that performs a specific operation on a graphics processor. Memory bandwidth: The amount of data that can be moved between memory and the compute section of a chip in a given time. Batch: The number of requests processed by the system at the same time.

Cohere speeds model serving with a single 'megakernel.' Software optimization is intended to reduce inference cost

The full article is available to users registered at Hard Skills