EdgeDiff: 418.4mJ/Inference Multi-Modal Few-Step Diffusion Model Accelerator with Mixed-Precision and Reordered Group Quantization

📋 论文概要

本文提出EdgeDiff，一种专为少步扩散模型设计的多模态加速器，通过混合精度和重排序组查询技术，在边缘设备上以418.4mJ/推理的低能耗实现高质量图像生成，解决了传统扩散模型计算和内存访问成本高的问题。

💡 主要创新点

• 针对少步扩散模型（FSDM）定制硬件架构，将去噪步骤减少至1-4步，大幅降低计算和内存访问开销。
• 采用混合精度计算和重排序组查询策略，在保持图像质量的同时进一步优化能效。

🏷 关键词

📄 原文摘要

need for high-performing image-generative models, including the diffusion model (DM) [2, 3]. A conventional DM requires numerous UNet-based denoising timesteps (~50), leading to high computation and external memory access (EMA) costs. Recently, the Few-Step Diffusion Model (FSDM) [4] was introduced, as shown in Fig. 23.3.1, to reduce the denoising timesteps to 1-4 through knowledge distillation, while maintaining high image quality, reducing computations and EMA by 22.0× and 42.3×, respectively. However, prior diffusionmodel architectures, which accelerated many steps of a DM [5, 6] through inter-timestep redundancy in the UNet, fail to speed up the few denoising steps of a FSDM due to the lack of redundancy between timesteps. Moreover, a multi-modal DM introduces additional

👥 作者与机构

Sangjin Kim, Jungjun Oh, Jeonggyu So, Yuseon Choi, Sangyeob Kim,

Dongseok Im, Gwangtae Park, Hoi-Jun Yoo KAIST, Daejeon, Korea The increasing demand for image generation on mobile devices [1] highlights the