YUV Denoise

Algorithm

Our YUV denoise module performs in YUV space.we choose the Non-local mean as our denosing algorithm in our hls implementation since it’s a hardware-friendly algorithm and can be realized by using several line buffers to support for the window serach operation required by each pixel.

The NLM algorithm¹ is a simple but effective denosing method. Given a discrete noisy image $v = {v(i) | i ∈ I}$, the estimated value $NLv$, for a pixel i, is computed as a weighted average of all the pixels in the image,
$$ NLv=\sum_{j \in L}\omega(i,j)v(j) $$

the weights $\omega(i,j)$ are defined as:

$$\omega(i,j)=\frac{1}{Z(i)}e^{-\frac{\Vert v(N_i)-v(N_j)\Vert^2,a}{h^2}}$$

$v(N_i)$ is a vector formed by the grey values of the pixels around a squre neighborhood of fixed size centoered at pixel i .

As shown in Fig 1, similar pixel neighborhoods will be given a large weight, $w(p,q1)$ and $w(p,q2)$, while much different neighborhoods give a small weight $w(p,q3)$ .

HLS Implementation

register parameters

    typedef struct{
    uint1 eb;//1 bit
    uint14 ysigma2;// sigma = [5,127]
    uint10 yinvsigma2;// invsigma = (1/sigma)<< 7
    uint14 uvsigma2;//[5,127]
    uint10 uvinvsigma2;//(1 / sigma) << 7
    uint4 yfilt;//[0.1~0.8] << 4
    uint4 uvfilt;//[0.1~0.8] << 4
    uint5 yinvfilt;//(1 / filt) << 1
    uint5 uvinvfilt;//(1 / filt) << 1
    uint14 yH2;
    uint18 yinvH2;
    uint14 uvH2;
    uint18 uvinvH2;
    }yuvdns_register;

The explanation of parameters in yuvdns_register are listed in the following table

parameters	explanation
eb	controls the yuvdns module is enable or not
ysigma2	controls the squared deviation of the gaussian kernel used for distance measure for in Y plane
uvsigma2	controls the squared deviation of the gaussian kernel used for distance measure for in UV plane
yH2	acts as a degree of filtering in Y plane
uvH2	acts as a degree of filtering in UV plane

functions

yuv444dns

void yuv444dns(top_register top_reg, yuvdns_register yuvdns_reg, stream_u10 &src_y, stream_u10 &src_u, stream_u10 &src_v, stream_u10 &dst_y, stream_u10 &dst_u, stream_u10 &dst_v)

input/output description

Params	description
top_reg	global configure register
yuvdns_reg	a configure register used only by yuvdns moudule
src_y	Y component of source image streams
src_u	U component of source image streams
src_v	V component of source image streams
dst_y	Y compoent of denoised image streams
dst_u	U compoent of denoised image streams
dst_v	V compoent of denoised image streams

Return value

None

function description

yuv444dns is the top denosing function,it performs NLM denoise for incoming src_y、src_u、src_v pixel streams and outputs denosied pixel streams dst_y、dst_u、dst_v seperately and simutaneosly.

in the original Non local mean algorithm, the denosied value of a pixel is computed as a weighted average of all the pixels in the image which has a high demand of line buffers to store neighbor pixels, thus we decrease the search area from the whole image to a 9x9 neigbor Window around the pixel.

in yuv444dns function, 8 line buffers are defined to store previous line data, and a 9x9 2D array to store the searching window around the center pixel.

uint10 yWindow[9][9];
uint10 uWindow[9][9];
uint10 vWindow[9][9];
uint10 ylineBuf[8][4096];
uint10 ulineBuf[8][4096];
uint10 vlineBuf[8][4096];

for each pixel prepared to be denosied, we get the 9x9 window formed by it’s arounding pixels

the window pixel preparation could be diveded into the following 3 steps

Step 1
datas in the 9x9 window shift left 1 pixel width

for(uint4 i = 0; i < 9; i++){
    for(uint4 j = 0; j < 8; j++){                           
        yWindow[i][j] = yWindow[i][j+1];
        uWindow[i][j] = uWindow[i][j+1];
        vWindow[i][j] = vWindow[i][j+1];
    }
}

Step 2

The 9x9 Window reads 8x1 pixel data from the 8 linebuffers and the input stream pixel y_t, u_t ,v_t, then the 9x1 new pixels will be stored in the rightest column of the Window

for(uint4 i = 0; i < 8; i++){
    yWindow[i][8] = ylineBuf[i][col];
    uWindow[i][8] = ulineBuf[i][col];
    vWindow[i][8] = vlineBuf[i][col];
}
yWindow[8][8] = y_t;
uWindow[8][8] = u_t;
vWindow[8][8] = v_t;

Step 3

Update the line buffer

for(uint4 i = 0; i < 7; i++){
    ylineBuf[i][col] = ylineBuf[i+1][col];
    ulineBuf[i][col] = ulineBuf[i+1][col];
    vlineBuf[i][col] = vlineBuf[i+1][col];
    }
ylineBuf[7][col] = y_t;
ulineBuf[7][col] = u_t;     
vlineBuf[7][col] = v_t;  

When the above 3 steps are finished, a yuvdns_nlm function(will be discussed later) will be called to perform the block serach operation.

Additionally, it's worth noting that for those border pixels which do not have enough neibor pixels to form a 9x9 search window, we directly ouput its’ origin value. finally every pixels of a frame will be denoised except the 4-pixel width border.

yuvdns_nlm

function declaration

uint10 yuvdns_nlm(uint10 Window[9][9],uint14 sigma2, uint14 H2,uint18 invH2){
uint8  weight_1[8]={255,226,200,176,156,138,122,108};
    uint8  weight_2[16]={88,72,59,48,39,32,26,21,17,14,11,9,7,5,2,0};

    uint21 diff;
    uint22 diff_1;
    uint22 diff_2;
    uint22 diff_3;
    uint28 diff_tmp; 
    uint8 weight = 0, maxweight = 0; 
    uint14 totalweight =0; 
    uint25 totalvalue = 0;
    nlm_row_loop:for (uint4 j = 1; j < 8; j++){
        nlm_col_loop:for (uint4 i = 1; i < 8; i++) {
            if (i != 4 || j != 4) {
                uint10 dis_1 = 0;
                uint20 dis_11 = 0;
                dis_1 = yuvdns_abs(Window[j-1][i-1],Window[3][3]);
                dis_11 =  dis_1 * dis_1;

                uint10 dis_2 = 0;
                uint20 dis_22 = 0;
                dis_2 = yuvdns_abs(Window[j-1][i],Window[3][4]);
                dis_22 =  dis_2 * dis_2;

                uint10 dis_3 = 0;
                uint20 dis_33 = 0;
                dis_3 = yuvdns_abs(Window[j-1][i+1],Window[3][5]);
                dis_33 = dis_3 * dis_3;

                uint10 dis_4 = 0;
                uint20 dis_44 = 0;
                dis_4 = yuvdns_abs(Window[j][i-1],Window[4][3]);
                dis_44 = dis_4 * dis_4;

                uint10 dis_5 = 0;
                uint20 dis_55 = 0;
                dis_5 = yuvdns_abs(Window[j][i],Window[4][4]);
                dis_55 = dis_5 * dis_5;

                uint10 dis_6 = 0;
                uint20 dis_66 = 0;
                dis_6 = yuvdns_abs(Window[j][i+1],Window[4][5]);
                dis_66 = dis_6 * dis_6;

                uint10 dis_7 = 0;
                uint20 dis_77 = 0;
                dis_7 = yuvdns_abs(Window[j+1][i-1],Window[5][3]);
                dis_77 = dis_7 * dis_7;

                uint10 dis_8 = 0;
                uint20 dis_88 = 0;
                dis_8 = yuvdns_abs(Window[j+1][i],Window[5][4]);
                dis_88 = dis_8 * dis_8;

                uint10 dis_9 = 0;
                uint20 dis_99 = 0;
                dis_9 = yuvdns_abs(Window[j+1][i+1],Window[5][5]);
                dis_99 = dis_9 * dis_9;

                diff_1 = dis_11 + dis_22 + dis_33;
                diff_2 = dis_44 + dis_55 + dis_66;
                diff_3 = dis_77 + dis_88 + dis_99;

                diff = (diff_1 + diff_2 + diff_3) >> 3;

                if(diff < 2 * sigma2){
                	diff = 0;
                }else{
                	diff = diff - 2 * sigma2;
                }

                uint32 count = 0;
                if(H2 == 0){
                    weight = 0;
                }                   
                else if(diff <= H2){
                    diff_tmp = 7 * diff;
                    count = (diff_tmp * invH2)>>14;
                    weight = weight_1[count];
                }
                else{
                    diff_tmp = 5 * diff;
                    count = (diff_tmp * invH2)>>14;
                    count = yuvdns_weight2_clip((count -5));
                    weight = weight_2[count];
                }

                if(weight > maxweight){
                    maxweight = weight;
                }
                totalweight = totalweight + weight;
                totalvalue = totalvalue + Window[j][i] * weight;
           }
        }
    }
    totalweight =  totalweight + maxweight;
    totalvalue = totalvalue + Window[4][4] * maxweight;

    if(totalweight==0)
        return Window[4][4];
    else
        return yuvdns_clip(totalvalue / totalweight, YUVDNS_Max_Value, 0);
}

parameters description

Params	description
Window[9][9]	search area of the pixel to be denoised
sigma2	squared deviation of gaussian kernel for distance measure
H2	$h^2$
invH2	$\frac{1}{h^2}$

Return

the denoised value of the center pixel in Window[9][9]

function description

yuvdns_nlm function performs the nlm search in the 9x9 Window which calculates the similarities between each neighbor pixel and the center pixel. Finally, yuvdns_nlm returns the denosied pixel value.

Reference

[1]Buades, A. , B. Coll , and J. M. Morel . "A non-local algorithm for image denoising." Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on IEEE, 2005.