A 915-1220 TOPS/W, 976-1301 GOPS Hybrid In-Memory Computing Based Always-On Image Processing for Neuromorphic Vision Sensors

Abstract

Neuromorphic vision sensors (NVSs) save energy and reduce data at the source by asynchronously recording changes in temporal contrast. Thus, NVS provides an opportunity to exploit temporal and spatial redundancy in video streams by enabling the following deep neural network (DNN) processor for object recognition to process only foreground object regions in valid frames. However, the NVS data inevitably contains noise leading to false frame generation. Moreover, objects may be fragmented due to a lack of events leading to wrong object region proposals (RPs). Hence, it is important to have an always ON image processor to perform image restoration (IR) and RP operations for NVS data. In this article, we propose a hybrid memory bitcell with collocated static random access memory (SRAM) and dynamic random access memory (DRAM) consist- ing of 11 transistors [11T-collocated SRAM and DRAM (CRAM)] to perform in-memory computing (IMC)-based IR and RP for event-based binary image (EBBI) frame from a stationary NVS. We propose IMC-based charge diffusion for IR (denoise and region filling) by enabling a 2-D interconnection of bitcells across the whole array for globally parallel computing. The proposed CRAM supports projection mode for IMC-based RP, which enables 1-D projection of objects on the horizontal and vertical axes and finds regions through a recently proposed iterative algorithm. We also proposed an RP update (RPU) algorithm and hardware to improve RP accuracy by 1.6 × over the pr