A 28nm 0.22µJ/Token Memory-Compute-Intensity-Aware CNN-Transformer Accelerator with Hybrid-Attention-Based Layer-Fusion and Cascaded Pruning for Semantic-Segmentation

📋 论文概要

本文提出了一款28nm工艺的CNN-Transformer混合加速器，通过内存计算强度感知和混合注意力层融合技术，实现了0.22µJ/Token的高能效，解决了语义分割任务中计算与内存开销大的问题。

💡 主要创新点

• 提出内存计算强度感知架构，动态调整计算与内存资源分配以优化能效。
• 设计了基于混合注意力的层融合方法，减少了CNN与Transformer之间的数据搬运开销。

🏷 关键词

📄 原文摘要

Luhong Liang2, Yitong Zhou2, Di Pang2, Man-To Yung2, Dong Zhang1,2, Xijie Huang1,2, Shih-Yang Liu1,2, Yongkun Wu1,2, Fengshi Tian1,2, Chi-Ying Tsui1,2, Fengbin Tu1,2, Kwang-Ting Cheng1,2 The Hong Kong University of Science and Technology, Hong Kong, China 2 AI Chip Center for Emerging Smart System, Hong Kong, China 1 *Equally Credited Authors (ECAs) Recently, hybrid models integrating a CNN and a Transformer (ConvFormer), shown in Fig. 23.2.1, have achieved significant advancements in semantic segmentation tasks [1-4], which are critical for autonomous driving and embodied intelligence. The CNN enhances the multi-scale feature extraction ability of the Transformer to achieve pixel-level classification, but the large token length (TL) demand of semantic segmentation (>16K TL) incurs significant computation and memory overheads. Prior NN accelerators [5-12] demonstrate that sparse computing and pruning can effectively reduce computation and

👥 作者与机构

Pingcheng Dong*1,2, Yonghao Tan*1,2, Xuejiao Liu2, Peng Luo2, Yu Liu2,