A 28nm Speculative-Decoding LLM Processor Achieving 105-to-685μs/Token Latency for Billion-Parameter Models

📋 论文概要

该论文提出了一种基于推测解码的LLM处理器，通过三个创新点（指数双重用MAC、草稿反向传播目标混合精度数据流、草稿提前启动并行计算）解决了LLM解码瓶颈，实现了10.29倍速度提升。

💡 主要创新点

• 指数双重用MAC：通过重用指数部分减少计算冗余，提高乘法累加效率。
• 草稿反向传播目标混合精度数据流：在草稿模型反向传播时采用混合精度，优化权重和KV缓存效率。
• 草稿提前启动并行计算：让草稿模型提前开始解码，与目标模型并行执行，减少硬件闲置。

🏷 关键词

📄 原文摘要

Abstract LLMs face decoding bottlenecks. Speculative Decoding (SD) reduces latency via a small draft model for serial decoding and a large target model to verify in parallel. Despite this advantage, it still suffers from exponent redundancy, inefficient weights/KV, and hardware underutilization. We propose an SD LLM processor with three innovations: an exponent dual-reuse MAC, draft back-propagation target mixed-precision dataflow, and draft early start parallel compute, achieving 10.29× speed increase. Large language models (LLMs) have achieved remarkable advancements in various AI tasks [1-4]. However, the autoregressive sequential decoding introduces a performance bottleneck [5-6]. Speculative Decoding (SD) has emerged to mitigate the latency of autoregressive decoding via alternating rapid drafting and parallel verification [7-14]. Generally, SD contains a small draft model and a large target model, where the draft model

👥 作者与机构

Yang Wang1, Huanyu Wang1, Jiaxin Yang1, Yutong Su1, Ruiqi Guo1, Zhiheng Yue1, Jiangyuan Gu1, Shaojun Wei1, Yang Hu1, Shouyi Yin1,2,3

Tsinghua University, Beijing, China, 2Shanghai Artificial Intelligence Laboratory, Shanghai, China, 3International Innovation Center of Tsinghua University, Shanghai, China