IBM and Red Hat deploy GLM-5.2 on 544 H100 GPUs with llm-d

IBM Research and Red Hat say they deployed the 753-billion-parameter GLM-5.2 model with llm-d on 544 NVIDIA H100 GPUs for an internal coding workload that ranged from hundreds to thousands of concurrent agents.

The companies used an existing H100 fleet rather than newer accelerators, placing the deployment alongside Cohere’s megakernel serving engine for North Mini Code as another H100-based inference design. In an AutomationBench run with 2,500 agents, IBM and Red Hat reported 7,612 requests per minute, peaks of 134.89 million input tokens and 6.05 million output tokens per minute, and no preemptions. At 3,000 agents, they measured 6.60 million output tokens per minute without preemptions or failures. Those results, like the other performance figures below, come from IBM and Red Hat and have not been independently reproduced.

Prefix-aware routing was one part of the system. The llm-d scheduler documentation says its precise mode tracks cache state from model-server events and directs a request toward a worker that already holds matching context. IBM and Red Hat reported that precise matching increased CyberGym throughput by 79% and cut time to first token by 67% compared with optimized approximate matching. In a 400-agent CyberGym run, they measured 967.7 requests per minute, a p90 time to first token of 17.59 seconds and a 73.18% local prefix-hit rate, versus 44.46% with approximate routing.

Caching also extended beyond GPU memory. The tiered-cache design moves evicted key-value, or KV, blocks into CPU memory and optionally storage, allowing later requests to restore attention state instead of recomputing a long prompt. The peer-to-peer cache path lets a worker retrieve blocks from another worker’s CPU tier over NIXL when the router identifies a useful copy.

Across their benchmark program, IBM and Red Hat said the CPU tier stored 2.53 PiB of prompt blocks and restored 2.16 PiB to GPUs in an average of 79.6 milliseconds. They reported that 85.2% of measured input tokens came from cache, leaving 14.8% for uncached prefill. They also said NIXL completed 6.2 million KV transfers averaging 2.71 GiB, sustained about 580 Gb/s at fleet p90, and had no observed transfer failures.

For model execution, wide expert parallelism spreads mixture-of-experts MLP layers across nodes while using data-parallel attention. llm-d’s documentation says that layout avoids the KV-cache replication associated with tensor parallelism for models that use multi-head latent attention.

The deployment separated prompt prefill from token decode, too. Prefill is compute-heavy, while decode is constrained more by memory bandwidth. Independently scalable pools let operators assign capacity to each phase separately, with NIXL moving KV blocks to decode workers over RDMA-capable networking.

The team also enabled GLM-5.2’s multi-token-prediction layer for speculative decoding. Z.ai’s model card documents that layer, the model’s 1-million-token context window and its MIT license. IBM and Red Hat said multi-token prediction raised output throughput at high concurrency.

The companies distinguished the internal workload from the structured benchmarks and said the tests used fresh prefixes and seeds so earlier cache state would not affect results. They reported no serving errors across the benchmarks. They also estimated that self-hosting on H100 GPUs cost five to ten times less per token than equivalent commercial APIs at then-current cloud rental rates. IBM’s account did not disclose the compared APIs, rental-rate inputs, token mix or full calculation needed to verify that estimate.