MetaVRain: A Mobile Neural 3-D Rendering Processor With Bundle-Frame-Familiarity-Based NeRF Acceleration and Hybrid DNN Computing

Abstract

This article presents MetaVRain, a low-power neu- ral 3-D rendering processor for metaverse realization on mobile devices. The MetaVRain mainly focused on solving a high opera- tional intensity problem that appeared during the neural radiance fields (NeRFs)-based rendering. It imitates brain-inspired visual perception processes and constructs a new NeRF acceleration architecture, bundle-frame-familiarity (BuFF). The built-in visual perception core (VPC) realizes BuFF architecture by acceler- ating three visual perception stages: 1) spatial attention (SA); 2) temporal familiarity (TF); and 3) top-down attention (TDA). Both TF unit (TFU) and SA unit (SAU) included in VPC help the processor not to suffer from external memory bandwidth. Moreover, unlike conventional GPU solutions, they remove useless operations that do not cause PSNR drop using an out-of-order task allocator. After removing useless operations, the remaining deep neural network (DNN) inference (INF) tasks are accelerated by a hybrid neural engine (HNE) which utilizes two different neural engines (NEs) 1-D and 2-D NEs. It uses 1-D NE for input activation (IA) sparsity exploitation and 2-D NE to maximize the data reusability. Thanks to the centrifugal sampling (CS)-based output sparsity prediction, it can dynamically allocate tasks to 1-D or 2-D NE. In addition, it accelerates the positional encoding part by adopting periodic polynomial-based sinusoidal function approximation. The MetaVRain suggests a modulo- based