The Curious Engineer

All blogs / AI / ML

LMSYS Org

lmsys.org/blog

Read the latest from LMSYS Org — research updates, technical deep dives, and announcements on large models, inference systems, reinforcement learning, and evaluation.

AI / MLPerformance / optimization

’23’24’25’26
Posts
136 from 30 Mar 2023 to 29 Oct 2026
Posting
weekly (81 in the last year)
Kind
project news

2026 69

  1. Training SWE agents with Miles on MI355X: taming long-tail trajectories and complex sandbox coordination
  2. SGLang security update: improving request isolation
  3. miniTE Technical Report: Stable and Efficient Reinforcement Learning with Native NVFP4
  4. More Than DeepInfer: What SGLang Delivers
  5. Announcing the SGLang AWS Trainium Plugin Beta
  6. Elastic Expert Parallelism Scaling in SGLang: Growing a Live MoE Deployment with NIXL
  7. SGLang-TorchTPU: PyTorch-Native Production Serving on TPUs
  8. SGLang Low-Latency Mode: Kernels and Runtime
  9. SGLang Powers Beam in Production
  10. SGLang and Miles Add Day-0 Support for Muse Spark Open
  11. SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Super VL
  12. SGLang and Miles on NVIDIA Vera Rubin
  13. Scaling JEV-like Decision Models with SGLang
  14. Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache
  15. SGLang and Miles Add Day-0 Support for DeepSeek-V4.1
  16. Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack
  17. Infer-forge: Harness, Loop, and Graph Engineering Around SGLang
  18. MiniMax-H3 on 8×H200: 1.95× Lossless, Up to 6.24× at 0.76–0.91 SSIM
  19. Qwen3.8-Flash-Next: Day-0 Support in SGLang
  20. Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell
  21. Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon
  22. Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O
  23. Pushing the Limits of Serving DeepSeek-V4-Pro
  24. Miles v0.1: Production-level Post-training
  25. Advanced CUDA Graph Techniques in SGLang
  26. SGLang and Miles Add Day-0 Support for Qwen3.8
  27. SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Lightning
  28. Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
  29. SGLang Adds Day-0 Support for Muse Glimmer, a Multimodal Model Built for Local Agentic Workflows
  30. HPC-Ops × SGLang: High-Performance Attention, Router GEMM, and MoE Kernels from Tencent Hunyuan
  31. Full-Stack Performance Optimization of AR+DiT in SGL-Diffusion
  32. SpecForge v0.3.0: a Unified Disaggregated and Colocated Speculative Decoding Stack, and New Open SpecBundle Draft Models
  33. RadixArk Joins Forces with Google to Bring Full SGLang Features to TPUs
  34. Towards Blackwell-Native 8-bit and 4-bit RL: End-to-End MXFP8 and NVFP4 RL in Miles
  35. Toward a Cleaner Quantization Stack in SGLang
  36. SGLang and Miles Add Day-0 Support for Kimi K3
  37. OPD Support in Miles
  38. SGLang and Miles Add Day-0 Support for Inkling, a Frontier Multimodal Model
  39. Serving GLM5.2 NVFP4 Agentic Workload with SGLang: Reaching 500 TPS in 2 Weeks
  40. Bringing DeepSeek-V4 Flash RL Training to AMD Instinct MI355X GPUs with Miles
  41. Accelerating SGLang HiCache with Netpreme X-Mem™ MPU
  42. DSpark in SGLang: Speculative Decoding with Confidence-Driven, Variable-Length Verification
  43. Agent-Assisted SGLang Development: An Initial Exploration
  44. Improving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB
  45. Optimizing Ling-2.6-1T on TPU with SGLang-JAX: Hiding MoE Data Movement Behind Compute with One Pallas Kernel
  46. MOSS-TTS Local Transformer v1.5 on SGLang-Omni: Serving Native-Streaming 48 kHz Speech
  47. The next generation of speculative decoding: DFlash and Spec V2
  48. Announcing the Recipient of the 2026 LMSYS PhD Fellowship
  49. No Token Left Behind: Demystifying Token-In-Token-Out in Miles
  50. Higgs Audio v3 TTS on SGLang-Omni: Real-Time, Controllable Speech for Voice Agents
  51. SGLang and Miles Add Day-0 Support for NVIDIA Nemotron 3 Ultra for Long-Running Autonomous Agents
  52. Heterogeneous CPU + GPU EPD Disaggregation to Boost VLM Serving
  53. Win on TCO: How AMD Instinct™ MI355X Achieves Cost-Competitive Distributed Inference Through SGLang with MoRI
  54. Updating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL
  55. DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles
  56. HiSparse: Turbocharging Sparse Attention with Hierarchical Memory
  57. Highlights of SGLang at NVIDIA GTC 2026
  58. Elastic EP in SGLang: Achieving Partial Failure Tolerance for DeepSeek MoE Deployments
  59. ROCm Support for Miles: Large-Scale RL Post-Training on AMD Instinct™ GPUs
  60. SGLang Adds Day-0 Support for NVIDIA Nemotron 3 Super for building High-Efficiency Multi-Agent Systems
  61. Unlocking 25x Inference Performance with SGLang on NVIDIA GB300 NVL72
  62. Deploying DeepSeek on GB300 NVL72: Big Wins in Long-Context Inference
  63. SGLang-Diffusion: Advanced Optimizations for Production-Ready Video Generation
  64. Unleashing Computational Power: Ultimate Latency Optimization of Qwen3 and Qwen3-VL on AMD MI300X Series
  65. Squeezing 1TB Model Rollout into a Single H200: INT4 QAT RL End-to-End Practice
  66. Optimizing GLM4-MoE for Production: 65% Faster TTFT with SGLang
  67. SGLang-Diffusion: Two Months In
  68. Pipeline Parallelism in SGLang: Scaling to Million-Token Contexts and Beyond
  69. EPD Disaggregation: Elastic Encoder Scaling for Vision-Language Models in SGLang

2025 39

  1. SpecBundle & SpecForge v0.2: Production-Ready Speculative Decoding Models and Framework
  2. Power Up Diffusion LLMs: Day‑0 Support for LLaDA 2.0
  3. Mini-SGLang: Efficient Inference Engine in a Nutshell
  4. SGLang Day-0 Support for MiMo-V2-Flash Model
  5. SGLang Adds Day-0 Support for the Highly Efficient, Open Nemotron 3 Nano Hybrid MoE Model
  6. Let Tensors Fly — Accelerating Large Model Weight Loading with R-Fork
  7. Boost SGLang Inference: Native NVIDIA Model Optimizer Integration for Seamless Quantization and Deployment
  8. From research to production: Accelerate OSS LLM with EAGLE-3 on Vertex
  9. Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL
  10. LMSYS Fellowship Program
  11. Introducing Miles — RL Framework To Fire Up Large-Scale MoE Training
  12. 🚀 AutoRound Meets SGLang: Enabling Quantized Model Inference with AutoRound
  13. SGLang Diffusion: Accelerating Video and Image Generation
  14. "No Free Lunch": Deconstruct Efficient Attention with MiniMax M2
  15. Optimizing GPT-OSS on NVIDIA DGX Spark: Getting the Most Out of Your Spark
  16. SGLang-Jax: An Open-Source Solution for Native TPU Inference
  17. Accelerating Hybrid Inference in SGLang with KTransformers CPU Kernels
  18. SGLang and NVIDIA Accelerating SemiAnalysis InferenceMAX and GB200 Together
  19. NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
  20. SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention
  21. PD-Multiplexing: Unlocking High-Goodput LLM Serving with GreenContext
  22. Together with SGLang: Best Practices for Serving DeepSeek-R1 on H20-96G
  23. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput
  24. Towards Deterministic Inference in SGLang and Reproducible RL Training
  25. Optimizing FP4 Mixed-Precision Inference on AMD GPUs
  26. SGLang HiCache: Fast Hierarchical KV Caching with Your Favorite Storage Backends
  27. LongCat-Flash: Deploying Meituan's Agentic Model with SGLang
  28. Fine-tune and deploy gpt-oss MXFP4: ModelOpt + SGLang
  29. SGLang for gpt-oss: From Day 0 Support to Enhanced Performance
  30. GLM-4.5 Meets SGLang: Reasoning, Coding, and Agentic Abilities
  31. SpecForge: Accelerating Speculative Decoding Training for SGLang
  32. Deploying Kimi K2 with PD Disaggregation and Large-Scale Expert Parallelism on 128 H200 GPUs
  33. Accelerating SGLang with Multiple Token Prediction
  34. How to support new VLMs into SGLang: A Case Study with NVILA
  35. Cost Effective Deployment of DeepSeek R1 with Intel® Xeon® 6 CPU on SGLang
  36. slime: An SGLang-Native Post-Training Framework for RL Scaling
  37. OME: Revolutionizing LLM Infrastructure with Model-Driven Architecture
  38. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput
  39. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs

2024 15

  1. SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs
  2. Announcing a New Site for Chatbot Arena
  3. RedTeam Arena: An Open-Source, Community-driven Jailbreaking Platform
  4. SGLang v0.3 Release: 7x Faster DeepSeek MLA, 1.5x Faster torch.compile, Multi-Image/Video LLaVA-OneVision
  5. Does style matter? Disentangling style and substance in Chatbot Arena
  6. Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM)
  7. RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing
  8. The Multimodal Arena is Here!
  9. Introducing Hard Prompts Category in Chatbot Arena
  10. What’s up with Llama 3? Arena data analysis
  11. LMSYS Kaggle Competition – Predicting Human Preference with $100,000 in Prizes
  12. From Live Data to High-Quality Benchmarks: The Arena-Hard Pipeline
  13. LMSYS Chatbot Arena: Live and Community-Driven LLM Evaluation
  14. Fast JSON Decoding for Local LLMs with Compressed Finite State Machine
  15. Fast and Expressive LLM Inference with RadixAttention and SGLang

2023 13

  1. Chatbot Arena: New models & Elo system update
  2. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  3. Recipe for Serving Thousands of Concurrent LoRA Adapters
  4. Catch me if you can! How to beat GPT-4 with a 13B model
  5. ToxicChat: A Benchmark for Content Moderation in Real-world User-AI Interactions
  6. Chatbot Arena Conversation Dataset Release
  7. How Long Can Open-Source LLMs Truly Promise on Context Length?
  8. Chatbot Arena Leaderboard Week 8: Introducing MT-Bench and Vicuna-33B
  9. Building a Truly "Open" OpenAI API Server with Open Models Locally
  10. Chatbot Arena Leaderboard Updates (Week 4)
  11. Chatbot Arena Leaderboard Updates (Week 2)
  12. Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings
  13. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality