Posts made by Kennico

Kennico

你好
1.stdio.h找不到的问题，可能是你的操作系统上一些c的标准库没有安装，导致gcc无法编译，这部分网上资料比较多，你可以去搜一搜如何安装
2.我们现在用的vitis版本是2021.2版本

Kennico

Raw Denoise

The RAWDNS module implements image denoising on the raw domain.The the NLMs(Non-local Means) algorithm based on block matching is used for denoising.

Parameter

typedef struct{
    uint6   sigma;    //range[0-63]
    uint1   eb;       //{0,1}
    uint7   Filterpara; //range[0-127]
    uint12  invksigma2; //1/ksigma2 <<12
}rawdns_register;

Param sigma and Filterpara jointly control the intensity of the filtering.Param invksigma2 is the quantized reciprocal of the product of sigma and Filterpara。

Input & Ouput

typedef stream_u12 Stream_t;//input raw data
typedef stream_u12 Stream_t;//output rgb data

Resources

The algorithm uses a 11x11 window to complete the denoising process, and the storage resources are listed as follows

uint12 rawdns_lines[10][8192];//10x8192 sram
uint12 rawdns_block[11][11];// 11x11 registers

Algorithm

Function rawdns_process describes the block matching based NLM algorithm, and the main calculation process is the following loop.

for(k=1;k<=9;k+=2){
    for(l=1;l<=9;l+=2){
        if((k!=5) || (l!=5))
        {
            eur_distance = Cal_Eur_Distance(rawdns_block,k,l);
            diff = rawdns_max(eur_distance, sigma2);
            weight = Cal_weight(diff,rawdns_reg ,ksigma2);
            if(weight > max_weight)
            {
                max_weight = weight;
            }
            total_weight += weight;
            total_value += weight * rawdns_block[k][l];
           }
        }
    }

The Cal_Eur_Distance function calculates the Euclidean distance between different blocks, and cal_weight calculates the Gaussian weights of different pixels.Finally, according to the calculation result, the pixel value after denoising(usually center pixel of the window) is output.

if(total_weight == 0)
    return rawdns_block[5][5];
else
    return rawdns_clip(total_value/total_weight);

Kennico

Demosaicking

In order to reflect the real color information of the real world, a single pixel needs to contain RGB tri-chromatic component information at the same time, and the obtained image is called true color image. The process of recovering the missing chromaticity information of each pixel is called Color Interpolation, also known as Demosaic.

Parameter

typedef struct {
    bool        eb; // 1 bit
}

Input & Ouput

typedef stream_u12 Stream_t;//input raw data
typedef stream_u36 Stream_3t;//output rgb data

Resources

The algorithm uses a 5x5 window to complete the interpolation process, and the storage resources are listed as follows

rawData_t rawWindow[5][5];//5x5 registers
rawData_t lineBuf[4][4096];//4x4096 sram

Algorithm

For different arrangements of Bayer arrays(determined by bayerPattern), the demosaicing algorithms are different.The basic idea is that the channel corresponding to the center pixel of the window remains unchanged, and the other channels are interpolated according to the channels of the surrounding pixels.

if (bayerPattern == 0)
    {
        pixTmp.r = rawWindow[2][2];
        pixTmp.g = 4 * rawWindow[2][2] - rawWindow[0][2] - rawWindow[2][0] - rawWindow[4][2] - rawWindow[2][4] \
                   + 2 * (rawWindow[3][2] + rawWindow[2][3] + rawWindow[1][2] + rawWindow[2][1]);
        pixTmp.b = 6 * rawWindow[2][2] - 3 * (rawWindow[0][2] + rawWindow[2][0] + rawWindow[4][2] + rawWindow[2][4]) / 2 \
                   + 2 * (rawWindow[1][1] + rawWindow[1][3] + rawWindow[3][1] + rawWindow[3][3]);
        pixTmp.g = pixTmp.g / 8;
        pixTmp.b = pixTmp.b / 8; 
    }
else if (bayerPattern == 1)
    {
        pixTmp.r = 5 * rawWindow[2][2] - rawWindow[2][0] - rawWindow[1][1] - rawWindow[3][1] - rawWindow[1][3] - rawWindow[3][3] - rawWindow[2][4] \
                   + (rawWindow[0][2] + rawWindow[4][2]) / 2 + 4 * (rawWindow[2][1] + rawWindow[2][3]);
        pixTmp.g = rawWindow[2][2];
        pixTmp.b = 5 * rawWindow[2][2] - rawWindow[0][2] - rawWindow[1][1] - rawWindow[1][3] - rawWindow[4][2] - rawWindow[3][1] - rawWindow[3][3] \
                   + (rawWindow[2][0] + rawWindow[2][4]) / 2 + 4 * (rawWindow[1][2] + rawWindow[3][2]);
        pixTmp.r = pixTmp.r / 8;
        pixTmp.b = pixTmp.b / 8;
    }
else if (bayerPattern == 2)
    {
        pixTmp.r = 5 * rawWindow[2][2] - rawWindow[0][2] - rawWindow[1][1] - rawWindow[1][3] - rawWindow[4][2] - rawWindow[3][1] - rawWindow[3][3] \
                   + (rawWindow[2][0] + rawWindow[2][4]) / 2 + 4 * (rawWindow[1][2] + rawWindow[3][2]);
        pixTmp.g = rawWindow[2][2];
        pixTmp.b = 5 * rawWindow[2][2] - rawWindow[2][0] - rawWindow[1][1] - rawWindow[3][1] - rawWindow[1][3] - rawWindow[3][3] - rawWindow[2][4] \
                   + (rawWindow[0][2] + rawWindow[4][2]) / 2 + 4 * (rawWindow[2][1] + rawWindow[2][3]);
        pixTmp.r = pixTmp.r / 8;
        pixTmp.b = pixTmp.b / 8;
    }
else
    {
        pixTmp.r = 6 * rawWindow[2][2] - 3 * (rawWindow[0][2] + rawWindow[2][0] + rawWindow[4][2] + rawWindow[2][4]) / 2 \
                   + 2 * (rawWindow[1][1] + rawWindow[1][3] + rawWindow[3][1] + rawWindow[3][3]);
        pixTmp.g = 4 * rawWindow[2][2] - rawWindow[0][2] - rawWindow[2][0] - rawWindow[4][2] - rawWindow[2][4] \
                   + 2 * (rawWindow[3][2] + rawWindow[2][3] + rawWindow[1][2] + rawWindow[2][1]);
        pixTmp.b = rawWindow[2][2];
        pixTmp.r = pixTmp.r / 8;
        pixTmp.g = pixTmp.g / 8;
    }

Kennico

Edge Enhancement

Edge enhancement is an image processing filter that enhances the edge contrast of an image or video in an attempt to improve its acutance (apparent sharpness).

ee.h

EE.h header file contains some basic information of ee module. 
The relevant parameters of the ee module are stored in the ee_register struct. The eb parameter controls the switch of the module, and the coeff parameter controls the intensity of sharpening.

typedef struct{
    bool eb;
    uint8 coeff;
}ee_register;

The RGB value of each pixel is stored in the ee_pixel_t struct.

typedef struct{
   uint12 r;
   uint12 g;
   uint12 b;
}ee_pixel_t;

The edgeenhancement function is used as the main function of the ee module, which will call the eeprocess function to perform edge enhancement operations. And we will talk about it later.

void edgeenhancement(top_register&, ee_register&, stream_u36&, stream_u36&);
uint36 eeprocess(uint36[5][5], ee_register&);

ee.cpp

edgeenhancement

As mentioned above, edgeenhancement function serves as the main function to perform the sharpening operation of the entire image and will be translated by the HLS compiler into the RTL description of the ee module.

EE module contains two storage unit ee_block and ee_line.
ee_block will be converted into registers for convolution operations, and ee_line will be translated into sram to improve throughput.

    uint36 ee_block[5][5];
    uint36 ee_lines[4][4096];

The following pragma instructs the compiler to translate the above two arrays into the corresponding memory.

set_directive_array_partition -type block -factor 5 -dim 1 "edgeenhancement" ee_block
set_directive_array_partition -type block -factor 4 -dim 1 "edgeenhancement" ee_lines

single_loop traverses each pixel to process one pixel in one cycle.

The following code implements the transfer of data between storage units.

block_refresh_loop_out:for(k = 0;k < 5;k++){
            block_refresh_loop_in:for(l = 0;l < 4;l++){
                ee_block[k][l] = ee_block[k][l+1];
            }
        }

        for(k = 0;k < 4;k++){
            ee_block[k][4] = ee_lines[k][j];
        }
        ee_block[4][4] = src_in;

        for(k = 0;k < 4;k++){
            ee_lines[k][j] = ee_block[k+1][4];
        }

When eb is 0, the input pixels are output directly without any processing.
Only when eb is 1, the module will perform the sharpening operation on the input data,that is, call the ee_process function.

if(ee_top.eb)
        {
            if((i > 3) && (j > 3))
            {
                dst_out = eeprocess(ee_block, ee_top);
            }
            else
            {
                dst_out = ee_block[2][2];
            }

            if((i > 2) || ((i == 2) && (j >= 2)))
            {
                dst.write(dst_out);
            }

padding_loop handles the case of pixels at the border of the image, which will be output directly from ee_line.

       padding_loop_1:for(k = 0;k < 2;k++){
           dst_out = ee_lines[1][isp_top.frameWidth - 2 + k];
           dst.write(dst_out);
       }

       padding_loop_2:for(k = 0;k < 2;k++){
           padding_loop_3:for(i = 0;i < isp_top.frameWidth;i++){
               dst.write(ee_lines[k+2][i]);
           }

ee_process

The function ee_process implements the unsharp masking algorithm.

First, the low-frequency part of the image low_feq_r,low_feq_g,low_feq_b is obtained by performing low-pass filtering twice on the image. 
The filtering process is represented by the following two loops.Gaussian filtering comes first.

for(k = 0;k < 5 ;k++)
    {
        for(l = 0;l < 5 ;l++)
        {
            rblock[k][l] = (int13)(ee_block[k][l] >> 24);
            gblock[k][l] = (int13)((ee_block[k][l] >> 12) & 0xfff);
            bblock[k][l] = (int13)(ee_block[k][l] & 0xfff);
            sharpen_threhold_r += guass_55[k][l] * rblock[k][l];
            sharpen_threhold_g += guass_55[k][l] * gblock[k][l];
            sharpen_threhold_b += guass_55[k][l] * bblock[k][l];
        }
    }
//guass_55[5][5] = {1,2,4,2,1,2,4,8,4,2,4,8,16,8,4,2,4,8,4,2,1,2,4,2,1};

The second one is average filtering.

 for(k = 0; k < 5; k++) {
        for (l = 0; l < 4; l++) {

            feq_l[l] = rblock[l][k]/2 + rblock[l + 1][k]/2;
            feq_h[l] = rblock[l][k]/2 - rblock[l + 1][k]/2;

            if(feq_h[l] > sharpen_threhold_r)
            {
                feq_h[l] -= sharpen_threhold_r;
            }
            else if(feq_h[l] < - sharpen_threhold_r)
            {
                feq_h[l] += sharpen_threhold_r;
            }
            else
            {
                feq_h[l] = 0;
            }
        }

        temp_r = feq_l[1]/2 + feq_h[1]/2 + feq_l[2]/2 + feq_h[2]/2;
        result_v[k] = ee_clip(temp_r);

The second step is to subtract the low-frequency image from the original image to obtain high-frequency details. 
Finally,The high-frequency part is multiplied by the set coeff parameter, and then superimposed on the original image to achieve image sharpening.

    high_feq_r = (int15)rblock[2][2] - low_feq_r;
    r_middle = high_feq_r * ee_top.coeff + 8;
    r_result = (r_middle >> 4) + low_feq_r;
    //red channnel as an example