Kog Inference Engine reaches up to 3.5x faster token generation on AMD Instinct MI300X GPUs
TL;DR
Kog Inference Engine hits up to 3.5× faster token generation than vLLM and TensorRT-LLM on AMD MI300X, across all tested model sizes (1B to 32B), with cross-GPU latency down to 4μs.
AMD covered Kog's inference results on MI300X in their engineering blog. Read AMD post ↗
Today, Kog AI launches a tech preview of the Kog Inference Engine (KIE): 3,000 output tokens/s per request on 8× AMD MI300X GPUs and 2,100 on 8× NVIDIA H200 (FP16, no speculative decoding). This preview runs a 2B model, with support for large third-party MoE models coming next at similar speeds.
We implemented the entire LLM decode pass in a single persistent kernel, no kernel launches, no interruptions, achieving 3,000+ tokens/s per request on AMD MI300X.
DTP is a new Transformer architecture that hides communication overhead behind computation and weight streaming, enabling significantly faster batch-size-one inference on AMD and NVIDIA GPUs.
