High-resolution Camera Embedded 360° Vision Vision System Design

Embedded 360° vision vision systems with multiple high-resolution cameras have entered various applications such as automotive sensor fusion, video surveillance, target detection, and motion analysis. In this type of system, multiple real-time camera video streams (up to 6) are aggregated and frame-by-frame processed for distortion and other image artifact corrections, exposure and white balance adjustments, and then dynamically tiled into a 360° panoramic view It is output at 4K resolution and 60 fps frame rate and finally projected onto a spherical coordinate space.

Current high-resolution fish-eye camera lenses used for this type of application typically have a wide-angle field of view (FOV) . One of the biggest bottlenecks in the look-a-view camera system is: The multi-camera input data is stored/read and accessed from the external memory in real time, and then processed as a single frame. The hardware needs to complete the processing operation between the incoming raw sensor data of the input camera and the spliced ​​output video within one frame delay.

High-performance computing platforms have been moving toward the use of FPGAs in conjunction with CPUs to provide dedicated hardware acceleration for real-time image processing tasks . This configuration allows the CPU to focus on particularly complex algorithms where they can quickly switch threads and contexts and assign repetitive tasks to an FPGA to act as a configurable hardware accelerator/coprocessor/offload engine. Even if the FPGA and the CPU are used as discrete devices, the system can improve overall efficiency because these technologies do not collide, but they fit together like a glove on the hand.

For example, an image obtained from a fisheye lens suffers from severe distortion, and thus a splicing operation generated based on multiple camera videos is a highly computationally intensive task because it is a point-pixel operation. This splicing requires a lot of real-time image processing and a highly parallelized architecture. However, this next-generation application exceeds the performance that FPGAs can continue to implement, primarily due to the latency of the chip's throughput data. This in turn affects the overall system's overall latency, throughput speed, and performance.

Add an eFPGA semiconductor intellectual property (IP) that can be embedded with the CPU in an SoC. Compared with an independent FPGA chip plus CPU solution, the embedded FPGA array structure has unique advantages, the main advantage is that the performance is stronger. An eFPGA can be directly connected to an ASIC (without I/O buffers) through a wide parallel interface, providing significantly improved throughput and latency counted in single-digit clock cycles. Low latency is the key to complex image real-time processing, such as correcting the distortion of the fisheye lens.

With Speedcore eFPGA IP, customers can define their logic, memory, and DSP resource requirements, and then Achronix can configure their IP to meet their needs. Look-up tables (LUTs), RAM cell blocks, and DSP64 cell blocks can be combined like blocks to create the best programmable structure for any given application.

In addition to standard logic, embedded memory, and DSP unit modules, customers can define their own function blocks within the Speedcore eFPGA structure. By integrating these custom function blocks with the traditional building blocks into the logic array structure, optimized functions can be added to reduce the area and improve the performance of the target application, which can greatly improve the performance of the eFPGA, especially for embedded applications. Visual and image processing algorithms are very effective.

Using a custom cell block to successfully solve high-performance image processing is a good example . You can achieve excellent performance by using only the one-time-only (YOLO), the most advanced, real-time object detection algorithm that uses neural networks. The early method greatly improved performance. The algorithm relies on a large number of matrix multipliers, and when implemented in an FPGA, these matrix multipliers need to be built using DSP and RAM blocks; the optimal configuration between DSP and RAM blocks required by YOLO, with a typical FPGA Problems with the mismatch found in the array structure can occur. For example, an FPGA array structure may provide 18x27 multiply/accumulate unit blocks and 32x128 RAM DSP blocks, and the best solution at this time may be a 16x8 DSP block with 48x1024 RAM. By creating custom blocks that implement the best DSP and RAM block configurations, the resulting Speedcore array architecture uses 40% less chip area to achieve the same functionality and achieve a higher level of system performance.

Embedding an FPGA array structure in an SoC provides two additional system-level benefits:

Lower power consumption - Programmable I / O circuits make up half of the total FPGA chip's total power consumption. An eFPGA can be connected directly to other modules in the host SoC without the need for a large programmable I / O buffer. .

Lower system cost - Since the eFPGA only needs to implement specific functions, the eFPGA die size is much smaller than the equivalent stand-alone FPGA chip because the eFPGA no longer requires a programmable I/O buffer and unnecessary interface logic.

With ultra-low latency and real-time processing capabilities, vision systems based on 360° views can be efficiently implemented. Speedcore eFPGA with custom cell blocks are used with a CPU in the same master SoC, making it ideal for implementing dedicated functions such as target detection And image recognition, distortion and distortion correction, and finally the final image stitching together. Embedding the FPGA array structure in the SoC is a natural development process of system integration in the ultra-deep submicron era.