A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using

Abstract

This article presents a 0.56/0.8 V multimode vision sensor for temporal and spatial information extraction with processing in sensor (PIS) technique. The proposed six tran- sistors and one capacitor (6T1C) temporal contrast pixel (TCP) structure using the exposure compensation scheme (ECS) can realize in-pixel temporal contrast calculation and motion event reporting (ER) with global shutter exposure and seamless frame difference (FD) based on the pulsewidth modulation (PWM) operation. In addition, the column-parallel local binary pattern (LBP) extraction provides further spatial feature information without the power-consuming analog-to-digital conversion to meet the power-efficient requirement for always-on-edge devices. According to the temporal and spatial information extracted from FD and LBP modes, this work realizes the PIS operations for dynamic depth calculation of a stereo vision system, which filters out the static scene and provides the depth information of the dynamic objects only. To further alleviate the data transfer bandwidth, the region of interest (ROI) capability is also embedded in the sensor for data windowing and object locating. The prototyped vision sensor was fabricated in TSMC’s standard 0.18 µm CMOS process and verified. The vision sensor provides five operating modes that can adapt to different tasks, which include the raw image mode (IM), FD mode, ER mode, LBP mode, and ROI mode. The measurement results show a maximum frame rate of 540/819/540 fram