BOOST Runtime Accelerated LLM Inference Throughput
Researchers demonstrated a method to concurrently access host memory and HBM to reduce latency in AI model serving.
Updated on Sept. 21, 2026 in Semiconductors
Live Poll
Do you believe new research methods will make AI technology significantly more efficient for users?
In September 2026, researchers from Georgia Tech, Nvidia Research, and Stanford University published a paper detailing the BOOST runtime system. This research-stage architecture improves large language model (LLM) inference throughput by enabling concurrent access to both host memory and High Bandwidth Memory (HBM).
Why it matters
The system was developed to maximize bandwidth utilization by bridging the speed gap between high-speed GPU memory and slower host memory. This approach could improve the efficiency of deploying massive models on existing hardware configurations.
Testing on a Grace Hopper system showed that BOOST improved Time-per-Output-Token (TPOT) by 4.3 percent compared to HBM-only serving. While the system demonstrated a 31 percent gain in high-throughput scenarios, active prefetching techniques actually degraded TPOT by 6 percent.
The players
Nvidia Research
The research arm of the GPU architecture leader focused on advancing silicon performance and AI efficiency.
Georgia Tech
A public research university recognized for its advanced work in computer science and high-performance computing.
Stanford University
A global hub for academic research in artificial intelligence, chip architecture, and system software.
The details
BOOST employs modulo-based page placement to balance the distribution of static model weights, effectively eliminating access-ratio variance. The system also makes the free KV (key-value) page pool wave-aware, ensuring that memory management aligns with the compute kernels used for attention calculations. By leveraging these kernel access patterns, the software enables concurrent data movement from host memory and HBM without requiring manual modifications to existing kernels.
Timeline
September 2026: The BOOST technical paper was published.
The Tech Race
This development follows an industry-wide trend of using software to mitigate the memory bandwidth bottlenecks inherent in modern AI hardware. It extends research into heterogeneous memory management, providing a competitive alternative to naive prefetching strategies.
This research currently exists at the architecture level and does not yet affect consumer-facing AI applications. Developers and engineers in high-throughput data centers should watch for future implementations of these memory-tiering strategies in production inference frameworks.
The takeaway
The research highlights that memory hierarchy management is becoming as critical as raw compute power for AI inference efficiency. Watch for the integration of wave-aware memory techniques into open-source inference engines to see if these gains materialize in production environments.
Further reading
For broader trends in memory optimization, see our coverage of Semiconductors.
More information
Review the full technical paper on BOOST for detailed benchmark methodology and architecture diagrams.
Source note: This article includes information reported by Semiconductor Engineering.
Live Poll
Do you believe new research methods will make AI technology significantly more efficient for users?
