Open Access
Add to BibSonomy
Abstract
Based on super-resolution processing, reconstruction refers to the technique of transforming several low-resolution images into a high-resolution image as a general tendency. The fractal super-resolution method is the most effective technique for obtaining a high-resolution image from a single low-resolution image. This technology performs expansion by partitioning the original image into block groups in the local region, which are called range blocks, and block groups in the global region, which are called domain blocks. Further, it replaces a range block with the domain block that resembles it the most. In this study, blocks containing image edges are subjected to fractal super-resolution processing using luminance variance, and areas with gradual changes in density are expanded by interpolation.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Presently, there is an ongoing active research for converting a low-resolution image into a high-resolution image, and it is motivated by the desire for higher television and display resolutions as well as improved printing techniques. Although there are various proposed expansion techniques, such as using interpolation and frequency, super-resolution processing is now very popular [1].
Additionally, the learning type [2] and the reorganization type [3] methods of super-resolution processing exist. However, the fractal super-resolution processing, a reorganization type [4]–[7], is well-known for obtaining high-definition pictures from a single picture without requiring mechanical learning and a database.
Fractal super-resolution is a super-resolution technique that focuses on the self-similarity of images. Luong et al. [8] proposed that images are self-similar, a property of fractal. Fractal super-resolution processing follows the outlined procedure: First, focus on the range blocks of a picture and search for a domain block, which is similar to the range block. Second, downsize the domain block using geometric transformation, and code it to record the approximation result as a parameter. Finally, estimate the high-resolution image by changing the recorded parameter depending on the amount of expansion ratio, then perform the decoding and converging of luminance dispersion by iterative processing. This method is unable to reproduce the image adequately when the matching block cannot be found. This is because the method expands the image by replacing the range block with the domain block.
Therefore, recently, various methods have been proposed to overcome this shortcoming. One of the methods ignores the feature of the result (i.e., the interpolation formula) and reduces time (i.e., convergence of luminance), using the result of the interpolation formula as the initial image at decoding [5]. The second method turns the target image to increase the number of domain blocks used for comparison [6]. Further, another method speeds up the processing, maintaining high similarity, by focusing the range of searching for the domain block to the surrounding range blocks [7]. These methods increase mismatching at decoding, due to an increase in the domain blocks, and this causes noises.
Therefore, we propose a new method, which expands the continuity of the edges and prevents mismatch by obtaining the pattern information of luminance dispersion and by comparing the blocks that only have the same pattern when the range includes the edge. Additionally, when the shade is in the gradation range, it is expressed as a contrast change of speed using interpolation.
2. Methods
2.1 Regional division
First, a source image is divided into replacing area (called range block) (M × N pixel) and the area used for replacing (called domain block) (P × Q pixel). The size of the range block used here is a 2 × 2 pixel, the domain block is 4 × 4 pixel, and the expansion ratio is 2. Examples of the regional division are shown in Fig. 1 and Fig. 2.
Furthermore, it becomes possible to replace a block with the highest similarity by rotating the source image by arbitrary degrees as shown in Fig. 3. This is because the rotation increases the number of patterns that refer to the domain block. The coordinate of the domain block in the rotated source image is obtained by Eq. (1), where θ is the rotational center of the image, (x0, y0) is the center of the rotating image, (k, l) is the coordinate before rotation, and (k’, l’) is the coordinate after rotation.
The range blocks including the edge is expand high-definitive by fractal coding, expand parameter, and fractal decoding. The range blocks that does not include the edge is expanded speedily by the conventional interpolated resolution.
2.2 Fractal coding
To perform fractal coding, a search of the domain block that has the highest similarity to the range block is conducted. The domain block is approximated to the range block by reducing and affine transforming it, and the domain block with the minimum square error is considered the optimal solution. Then the coordinate of domain block (k, l) and the parameter used for transformation are saved. The procedure of the fractal coding is shown in Fig. 4 and Eq. (2)-Eq. (5), where Rij is the range block which is attention, Dkl is the domain block which is used for comparing, Ravg is the mean value of pixel of Rij, Davg is the mean value of pixel of Dkl, s is the method of reducing, ɛ is the method of affine transforming, and α is the intensity transforming scaling factor. Further, the calculation of the sum in Eq. (4) and Eq. (5) is performed within the block size.
In Eq. (2), a new block R’ij is formed, which is obtained by calculating the difference between each pixel and Ravg and transforming it to compare Dkl and R’ij in Eq. (3). In this section, there is first a scale-down and matching of all block sizes, then the reduced domain block S (Dkl) is formed. Further, the difference between each pixel and Davg is calculated, rotated by 0°/90°/180°/270°, and flipped right oblique/left oblique/horizontal/vertical to obtain the result. The result obtained is D’kl. Therefore, D’kl has 8 patterns. The intensity transforming scaling factor α of each D’kl in Eq. (4) and the sum of squared error (SSE) in Eq. (5) are calculated. However, it takes significant time to compare a range block with all domain blocks. Thus, the searching area is restricted around the range block, the SSE within the restriction area of domain block is calculated, so as to increase speed and maintain accuracy.
Moreover, for the fractal coding, there is focus on the luminance dispersion patterns of blocks do not change by affine transformation, and we classify which of the groups shown in Fig. 5 is closest to the block for all blocks. It is possible to omit processing after the affine transformation for blocks that are predicted to have a large SSE by determining whether they are in the same group as the range block of interest. This enables us to increase the speed and to prevent false matching while maintaining edge continuity. After searching the domain block Dkl that has minimum SSE, it is taken as the optimal solution of R’ij, and the coordinate of Dkl, the method of down scaling s, the method of transforming ɛ, and brightness scaling α is saved for use in decoding. If the Dkl is the block made with rotation image, the angle θ are saved as well for use in decoding.
Fig. 5. Patterns of the brightness (bright pixel refers to being above average of brightness and dark pixel refers to being below average of brightness)
2.3 Parameter expansion
The size of the output area is determined by multiplying the expansion ratio, r, with the size of input image. Then, the sizes of the expansion range block (rM × rN pixel) and the expansion domain block (rP × rQ pixel) are determined. The expansion ratio is multiplied with the coordinate of domain block in the block that is saved by coding, to ensure that the domain block corresponds with the expansion image. In this section, the expansion range block is a 4 × 4 pixel and the expansion domain block is an 8 × 8 pixel.
2.4 Fractal decoding
Fractal decoding is performed by iterating the process that changed the expansion domain block into expansion range block. A sized output area got by parameter expansion is prepared, and the area is divided into the size of the expansion range block. Based on the coordinate that is saved at coding, the best expansion domain block for each expansion range block from output area is selected. Down-scaling, transforming, and brightness scaling are performed using the stored parameter. Further, the pixel is replaced by the pixel value of the transformed expansion domain block. The fractal decoding process is shown in Fig. 6 and Eq. (6), where NRij is the target of expansion range block, and s, ɛ, and α are down scaling, transforming, and luminance scaling, respectively, stored by the parameter at coding. Further, Ravg is the mean of Rij before the parameter is expanded, NDkl is the expansion domain block made from stored coordinate, and NDavg is the mean.
As shown in Fig. 6, the image is expanded by reconstructing all the range blocks. If there are pixels with overlapping other range blocks, use the average value. As luminance is not converged by only one decoding, the expansion image is acquired by iterating decoding.3. Results and discussion
The experiment was evaluated to examine the extent to which the original image can be reproduced using the proposed method, when the test image was reduced to 50%, and then expanded twice, compared to conventional method (Ref. [7]). In the experiment, Baboon, Gold Hill, and Lena were used as test images. They were inspected in the case of restricting the searching area of the domain block to 8 pixels from the range block. Additionally, Bicubic interpolation was used to expand to obtain a relatively high accuracy. Moreover, we assumed that there were various cases of the image roughness, and we prepared three resampling methods namely, area average, Lanczos3, and nearest neighbor. Then we experimented for 9 kinds of pictures. To examine the extent that can be reproduced by the source image, the result image was evaluated by PSNR and SSIM (Ref. [9]). PSNR indicates the maximum value of the signal that can be taken as a ratio to the noise by image quality deterioration. The higher the PSNR value, the higher the reproduced ratio. Similarly, SSIM is the evaluation indicator, but it is closer to the human subjective evaluation. As the value of SSIM approaches 100%, the source image is reproduced perfectly. The results used for these is shown in Table 1.
Table 1. Evaluation result
Further, as these numerical values of evaluation do not completely match the human subjective evaluation, the resulting image was also evaluated with the naked eye. Since the pictures reduced by the nearest neighbor were the most produced noticeable differences between the conventional and proposed methods, we made a comparison on that. The conventional method and the proposed method were compared to examine the age sequence and the noise degree. The results of partial comparison are shown in Fig. 7–Fig. 10, where picture (a) is the local image from the source image, (b) is the result of using the conventional method, and (c) is the result of using the proposed method. The top part shows the results of edge extraction using differential filter. In the result of Baboon shown in Fig. 7, neither method smoothly expanded the edge of the eye, and the precision results are nearly identical. In the result of Gold Hill shown in Fig. 8, both methods can reproduce the edge, but the proposed method reproduced more of the diagonal edge. This might be due to the reduction in the miss-match between blocks obtained by dividing the groups using luminance dispersion at coding. Finally, in the result of Lena shown in Fig. 9 and Fig. 10, it can be seen that the noise around the edge decreased in Fig. 9, and the edge over the bridge of the nose from the inner corner of the eye was more reproduced than the proposed method.
Finally, the processing time to output the enlarged image was measured. We measured only the time for enlarging the image, not the time for loading and outputting the image for each method. The specifications of the computers used in the experiments are shown in Table 2. n the experiments, the test image of Lena was used. The processing times for each method are shown in Table 3.
Table 2. Specifications of a computer
Table 3. Processing time
Table 3 shows that the results of the proposed method can be obtained in about half the time of the conventional method. In this study, we have applied the proposed method to still images. However, when it is used for real-time processing, it may be possible to reduce the time by using a high-performance computer or decentralized processing using a computer.
4. Conclusion
Depending on whether the range block has the edge or not, fractal super-resolution processing using luminance dispersion or the expansion method using interpolation should be selected. The better method would maintain the sequence of edges and reduce the noise at expansion. The block should be guarded from mismatching, and the speed increased by calculating the luminance dispersion pattern at fractal coding during fractal super-resolution.
Several test images were experimented, and the quality of the result images was checked by two kinds of evaluation indicators as well as by the naked eye. The proposed method was confirmed to better in maintaining the edge and reducing the noise than the conventional methods. Moreover, the proposed method can be obtained in about half the time of the conventional method.
As fractal super-resolution expands images by replacing the similar area in the source image, there are challenges that hinder the creation of the resulting image, in the case of not searching the similar areas due to the low quality of the image. In a future study, we intend to work on these problems. More accuracy can be achieved by combining fractal super-resolution with texture analysis, as in the method of Zhang et al. [10].
Disclosures
The authors declare no conflicts of interest.
References
1. H. Komori, “Super Resolution,” ITEJ 63(10), 1400–1402 (2009). [CrossRef]
2. T. Ono, Y. Taguchi, T. Mita, and T. Ida, “Example-Based Super-Resolution by Evaluating Compatibility of Neighboring Blocks with Increment Sign,” IEICE technical report. Image engineering 107(538), 307–312 (2008).
3. M. Tanaka and M. Okutomi, “Super-resolution: High-resolution Image Reconstruction from Multiple Low-resolution Images,” ITEJ 62(3), 337–342 (2008). [CrossRef]
4. Y. Tamura, Y. Takehisa, and K. Tanaka, “Basic Study on Image Enlargement Using Fractal Coding Concept,” Proceedings of the Image Media Processing Symposium, pp. 63–64 (2008).
5. M. Sato, “Resolution conversion method and resolution conversion apparatus,” Japan patent, JP 2006, 54899A (2009).
6. M. Komiya and Y. Bao, “Resolution Enhancement of the Image of the Fractal Using a Neighboring Pixel,” ADVANTY2009 Symposium, pp. 123–126 (Nov. 2009)
7. K. Wada and Y. Bao, “Resolution Enhancement of the Image of the Fractal Using a Neighboring Pixel,” The lecture proceedings of the Institute of Image Information and Television Engineers WINTER ANNUAL CONVENTION, No. 2011, pp. 1–5 (2011)
8. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]
9. H. Luong, A. Ledda, and W. Philips, “An im- age interpolation scheme for repetitive structures,” Proc. International Conference on Image Analysis and Recognition, pp.104–115 (2006).
10. Y. Zhang, Q. Fan, F. Bao, Y. Liu, and C. Zhang, “Single-Image Super-Resolution Based on Rational Fractal Interpolation,” IEEE Trans. on Image Process. 27(8), 3782–3797 (2018). [CrossRef]
