Raúl Taranco

Efficient image processing is increasingly crucial in constrained embedded and real-time platforms, especially in emerging applications such as Autonomous Driving (AD) or Augmented/Virtual Reality (AR/VR). A commonality among most image processing operations is their reliance on primitives like convolutions and stencil operations, which typically utilize a sliding window dataflow. Many existing implementations are domain-specific, lacking generality, or are programmable at the cost of sacrificing performance and energy efficiency. Among the latter, the CPU-based platforms that typically rely on Vector Processing Units (VPUs) often miss critical optimization opportunities, particularly those arising from the overlapping nature of the mentioned windowed image processing operations. In response, we propose SLIDEX, a novel high-performance and energy-efficient vector ISA extension to exploit Sliding Window Processing (SWP) in conventional CPUs. SWP extends the conventional vector SIMD execution model, treating vector registers like variable-sized groups of overlapping pixel windows. SLIDEX-enabled VPU processes multiple windows simultaneously, maximizing the Data Level Parallelism (DLP) achievable per instruction while maintaining the same vector length. Furthermore, it significantly reduces the need for data access, movement, and alignment, decreasing memory and register file accesses compared to traditional SIMD designs. To support SLIDEX, we introduce a cost-effective microarchitecture designed for easy integration into existing VPUs with minimal modifications. We demonstrate the efficacy of SLIDEX by testing it on a state-of-the-art visual localization task critical in AD and AR/VR. The results are compelling: SLIDEX achieves significant speedups in vital tasks such as 2D convolutions for image filtering and stencil operations for feature extraction, leading to an overall speedup of ∼ 1.2 × and up to 19% energy reduction compared to traditional vector extensions.

Continuous Vision (CV) systems are essential for emerging applications like Autonomous Driving (AD) and Augmented/Virtual Reality (AR/VR). A standard CV System-on-a-Chip (SoC) pipeline includes a frontend for image capture and a backend for executing vision algorithms. The frontend typically captures successive similar images with gradual positional and orientational variations. As a result, many regions between consecutive frames yield nearly identical results when processed in the backend. Despite this, current systems process every image region at the camera’s sampling rate, overlooking the fact that the actual rate of change in these regions could be significantly lower. In this work, we introduce δLTA (\deltaont’t Look Twice, it’s Alright), a novel frontend that decouples camera frame sampling from backend processing by extending the camera with the ability to discard redundant image regions before they enter subsequent CV pipeline stages. δLTA informs the backend about the image regions that have notably changed, allowing it to focus solely on processing these distinctive areas and reusing previous results to approximate the outcome for similar ones. As a result, the backend processes each image region using different processing rates based on its temporal variation. δLTA features a new Image Signal Processing (ISP) design providing similarity filtering functionality, seamlessly integrated with other ISP stages to incur zero-latency overhead in the worst-case scenario. It also offers an interface for frontend-backend collaboration to fine-tune similarity filtering based on the application requirements. To illustrate the benefits of this novel approach, we apply it to a state-of-the-art CV localization application, typically employed in AD and AR/VR. We show that δLTA removes a significant fraction of unneeded frontend and backend memory accesses and redundant backend computations, which reduces the application latency by 15.22% and its energy consumption by 17%.