Abstract

Despite the advances in image sensors, mainstream RGB sensors are still struggling from low quantum efficiency due to the low sensitivity of the Bayer color filter array. To address this issue, a sparse color sensor uses mostly panchromatic white pixels and a smaller percentage of sparse color pixels to provide better low-light photography performance than a conventional Bayer RGB sensor. However, due to the lack of a proper color reconstruction method, sparse color sensors have not been developed thus far. This study proposes a deep-learning-based method for sparse color reconstruction that can realize such a sparse color sensor. The proposed color reconstruction method consists of a novel two-stage deep model followed by an adversarial training technique to reduce visual artifacts in the reconstructed color image. In simulations and experiments, visual results and quantitative comparisons demonstrate that the proposed color reconstruction method can outperform existing methods. In addition, a prototype system was developed using a hybrid color-plus-mono camera system. Experiments using the prototype system reveal the feasibility of a very sparse color sensor in different lighting conditions.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

1. Introduction

Standalone digital cameras have evolved significantly in the past decade. In general, mainstream red-green-blue (RGB) sensors incorporate an alternating mosaic pattern of a color filter array (CFA). The most widely used mosaic pattern is known as the Bayer pattern [1]. Typically, interpolation-based demosaicking techniques [2, 3] have been used to reconstruct the RGB channels of a color image from the mosaic pattern. Despite the advances in demosaicking techniques [4,5], Bayer RGB sensors have inevitable drawbacks. Notably, the physical properties of the Bayer CFA degrade the imaging performance in low-light conditions.

In digital image sensors, a CFA sits on top of the photodiodes. Unfortunately, the CFA absorbs and reflects some of the incoming photons. A Bayer sensor generally captures only one-third of the actual incoming light. Therefore, Bayer sensors exhibit a low level of quantum efficiency (QE, a sensor’s ability to turn light into an electrical signal) [6,7]. Hence, Bayer sensors yield a low signal-to-noise ratio (SNR) and capture less detailed information when photographing in low-light environments [8].

To overcome this disadvantage of Bayer sensors, panchromatic white pixels have been used along with RGB color pixels in an image sensor (e.g., a RGBW sensor [9–11]). In general, panchromatic pixels provide a better QE than RGB pixels. The fundamental concept of the new color filters has been adopted from the human visual system (HVS). Specifically, panchromatic pixels are the replica of rod cells, while RGB pixels try to mimic cone cells. Note that, in the retina, 95% of the photoreceptors are rod cells and only 5% are cone cells. Also, in scotopic vision (i.e., low-light conditions), rod cells are primarily responsible for capturing luminance information [12]. Thus, to achieve human-like vision performance particularly in low-light conditions, image sensors may have a very high ratio of white pixels [13] (e.g., at least 95% white pixels as with human vision). Throughout this paper, we refer to this type of image sensor as a sparse color sensor because it has only a small percentage of color pixels, similar to the percentage of cone cells in human eyes. The goal of the sparse color sensor is to improve low-light performance while providing the quality of well-lit images comparable to that of Bayer sensors [14]. There is a broad range of potential applications for the sparse color sensor, such as autonomous driving of moving vehicles, robotics, and security cameras.

Despite the wide range of potential applications for sparse color sensors, the reconstruction of a full RGB image from a sparse set of color information has been considered to be a challenging task. Particularly, a very sparse color pixel arrangement can produce severe visual artifacts in the final reconstructed image. Due to the difficulty of color reconstruction, none of the existing methods [10,11,15–22] have thus far achieved a sparse ratio in the CFA that can be comparable to the HVS (i.e., less than 5% color pixels). Considering the challenges, most of the recent works [11,17,18] focused on reconstructing RGB images from a limited amount of information; i.e., the ratio between color pixels and white pixels (e.g., up to 60% white pixels [11]). A recent study [15] proposed an optimization-based method to reconstruct an RGB image from 11% RGB pixels (their CFA pattern is known as CFZ-14; see Fig. 1(a)). Unfortunately, their reconstruction method [15] also falls short of expectations along with the generation of aliasing artifacts [19].

 

Fig. 1 Examples of sparse color filter patterns. (a) Existing CFZ-14 pattern with a spare ratio (89% panchromatic pixels and 11% RGB pixels). (b) Extended version of CFZ-14 with a very sparse ratio (98.86% panchromatic pixels and 1.13% RGB pixels), used in our experiments. In the figures, W denotes panchromatic white pixels.

Download Full Size | PPT Slide | PDF

Meanwhile, state-of-the-art colorization models for image editing tasks have demonstrated the capability of deep learning to recover full colors from sparse user hints (e.g., colorization of grayscale images [23]). Although this is not directly related to color reconstruction for a sparse color sensor, it could be a possible approach for color reconstruction from very sparse color filter patterns. However, the following question remains unanswered. Can the state-of-the-art colorization model be used for developing a very sparse color sensor (e.g., 1% RGB pixels and 99% white pixels)?

A preliminary experiment was conducted to study the feasibility of the existing methods with a very sparse pixel arrangement. To obtain approximately 1% of the RGB pixels, the CFZ-14 [15] CFA was extended by three times (see Fig. 1). Figure 2 shows the color reconstruction results for the existing and proposed methods (more examples and details of the experiments are given in Section 3).

 

Fig. 2 Visual artifacts generated by the existing methods for sparse color reconstruction. (a) Ground truth (the inset is a zoomed-in image of the marked area). (b) Input Image (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (Proposed). (g) sGAN (Proposed).

Download Full Size | PPT Slide | PDF

The preliminary experiment has revealed that the existing color reconstruction methods produce severe visual artifacts, including color bleeding, false-color artifacts, and, in some cases, blocking artifacts. Although the deep learning-based method (i.e., Zhang’s method [23]) performs better than the optimization-based methods (i.e., Chakrabarti-14 [15]), it also produces the visual artifacts. To realize a sparse color sensor, the first step is to develop an acceptable color reconstruction method that can produce visually plausible results.

This study aims to propose a deep learning-based sparse color reconstruction method. The proposed deep reconstruction model can mitigate visual artifacts while retaining the details of panchromatic pixels. The proposed model is comprised of a two-stage network. The first stage consists of a stacked convolutional neural network to recover a full-luminance image from the missing panchromatic pixels. In the first stage, low-level edge features are also extracted from the recovered luminance image. The second stage utilizes the outputs of the first stage along with sparse RGB pixels to reconstruct a complete RGB image. The architecture of the second stage network is designed as a U-net with dense blocks; this is followed by adversarial training [24]. In this study, we denote the proposed two-stage reconstruction model as sNet (structure network), while we denote the proposed model with an adversarial training as sGAN (sNet with Generative Adversarial Network). Intensive experiments were conducted to study the usefulness of the proposed model for sparse color reconstruction. The contributions of this study are as follows:

  • A two-stage deep reconstruction model is proposed to perform color reconstruction from a very sparse color filter pattern (e.g., fixed or random 1% color pixels). It can produce visually plausible color images in any lighting conditions, and it can retain the details from panchromatic pixels in low-light conditions.
  • Color bleeding artifacts are reduced by feeding edge information extracted in the first stage into the second stage network and propagating it through the whole layers in the second stage (i.e., the color reconstruction network (CRN)).
  • False-color artifacts are also reduced by using adversarial training in the second stage.
  • A sparse color sensor prototype (with 1% color pixels) was developed by using a hybrid color-plus-mono camera system to test the feasibility of the proposed reconstruction method in different lighting conditions.

This paper is organized into six sections. Section 2 describes the proposed sparse color reconstruction network. Section 3 demonstrates the experiments and comparisons with existing studies. In Section 4, a prototype system is constructed and tested. Finally, Sections 5 and 6 conclude this work.

2. Sparse color reconstruction

A typical sparse CFA has two different types of pixels. One is a panchromatic pixel (also known as a white pixel), and the other is an RGB color pixel. The ultimate goal is to reconstruct a three-channel RGB image that can be represented as F : {PS, CS} → IR, where PS and CS are the input panchromatic and sparse RGB pixels, respectively. In this study, a novel two-stage color reconstruction model (F) is proposed by utilizing the capability of deep learning to obtain the final three-channel color image (IR). As shown in Fig. 3, the two stages are performed as follows:

  • Stage I: A luminance recovery network (LRN) recovers the one-channel luminance image (PR) from the input panchromatic pixels (PS) with some missing pixel information originally located in the color pixel positions. Here, the low-level features for edge information (ER) are also extracted and used in the second stage for mitigating color bleeding artifacts.
  • Stage II: This stage is comprised of a color reconstruction network (CRN) followed by an adversarial training block. It reconstructs a final three-channel RGB image (IR) from the input set (𝕂). The input set (𝕂) includes the outputs of the first stage (i.e., PR, ER) and the sparse RGB pixels (CS). The adversarial block (discriminator) is set to guide the CRN (generator) to reconstruct images having color and realism comparable to those of a Bayer sensor.

 

Fig. 3 Overall framework of the proposed color reconstruction method. LRN:luminance recovery network. CRN:color reconstruction network.

Download Full Size | PPT Slide | PDF

2.1. Stage I:luminance recovery for missing panchromatic pixels

Stage I consists of a stacked convolutional neural network (CNN). The proposed luminance recovery network (LRN) learns to recover the luminance image (PR) from the input panchromatic pixels (PS). Note that the input data encompasses holes (missing pixels on the RGB positions). The final goal of LRN is to map the input PS as PR ∈ [0, 1]H×W×1, where PR is the reconstructed monochrome image with a dimension of H × W × 1. Here, H and W are the height and width of the image. Along with recovering the mono image PR, an edge map (ER) is extracted as a low-level feature. Here, Canny edge extraction [25] is used to extract the edge feature, where ER ∈ [0, 1]H×W×1. An optimal parameter setting (minVal = 100, maxVal = 200, and apertureSize = 3) has been adopted as suggested in a previous study [26]. Figures 4(a), 4(b), and 4(c) demonstrate the input, output, and extracted edge map in this stage, respectively.

 

Fig. 4 Input and output samples for Stage I. (a) Input luminance image with missing pixels. (b) Recovered luminance image. (c) Edge map extracted from the recovered image.

Download Full Size | PPT Slide | PDF

2.1.1. LRN architecture

The LRN model is a stacked 5-layer convolutional neural network that maps an input (PS) into the recovered luminance image (PR). As shown in Fig. 5, the first three layers of the LRN have a depth size of 64, followed by two convolutional layers with a depth size of 128. Each layer has a kernel size of 3 × 3 and it is activated with a ReLU [27] function. The final layer of the LRN has a dimension of H × W × 1, where H and W represent the input image dimension. The output layer is activated with hyperbolic tangent (tanh) [28] function. The output of the LRN along with the edge image is used as an input of the next stage for color reconstruction.

 

Fig. 5 Model architecture of luminance recovery network (LRN).

Download Full Size | PPT Slide | PDF

2.1.2. Optimization of stage I

Each layer of the LRN was normalized with a batch normalization function. Moreover, to achieve the optimal loss, a loss function must be adopted. Among the existing loss functions, L1, L2, and structural similarity (SSIM) are widely used for image synthesis [29]. It has been reported that, in many cases, L2 loss can produce blurry images [30]. L1 loss is explicit from the blur effect. However, it is known that the SSIM loss can prevent structural losses in the model output. To take advantage of both types of loss, recent studies suggested a combination of L1 + SSIM loss for image synthesis. In this work, an L1 + SSIM loss is applied for the LRN. The loss is given by:

LRN(PG,PR)=PGPR1+SSIM(PG,PR)
Note that, during the training, the loss for the LRN was optimized independently. Thus, it does not have any effect on the later stage (i.e., CRN).

2.2. Stage II:color reconstruction network (CRN)

The inputs of the second stage obtained from the first stage. Here, the edge image (ER) from the first stage aims to assist the color reconstruction network (CRN) in reducing color bleeding artifacts. Specifically, the CRN takes a set of input as 𝕂 = {PR, CS, ER}. The primary goal of the CRN is to map 𝕂 as IR ∈ [0, 1]H×W×3, where IR denotes the final reconstructed color image with RGB channels. Due to the advantages of the generative adversarial network (GAN) in terms of colorfulness in image synthesis tasks [30], the proposed CRN is also guided by an adversarial training process. Hence, the CRN can be considered as the generator (G) in combination with the adversarial block as a discriminator (D),as shown in Fig. 3.

2.2.1. Generator:CRN architecture

The CRN is designed as a dense U-Net [31–33]. Typically, a dense U-Net is a combination of Dense Convolution Network (aka DenseNet) [34] and U-Net [35]. The dense U-Net architecture utilizes the dense blocks (a building block of DenseNet [34]) for feature extraction and propagation throughout the densely connected layers while keeping the overall architecture shape as the original U-Net. This study extends the dense U-Net by concatenating a low-level edge feature map in an early stage along with the sparse RGB and panchromatic input. Moreover, the CRN uses repetitive dense blocks and skip connections to propagate the input edge map in the early stage into the later stages, as it helps to reduce color bleeding artifacts in the sparse color reconstruction.

Specifically, a dense block consists of a set of convolutional layers. Any layer of a dense block can have a dense connection to all subsequent layers. A dense connection within a dense block connects all layers with the collective knowledge as well as the direct access to the input. A dense block concatenates the outputs of each respective layer and feeds them into the next block through a transition layer [30]. The consecutive dense blocks have a hyperparameter known as growth rate. In addition, the growth rate determines how much information should be added to each corresponding layer. Here, the growth rate is denoted as g.

As shown in Fig. 6, the CRN takes a set 𝕂 = {PR, CS, ER} as an input. The sparse RGB (CS) and reconstructed luminance image (PR) are processed through the first convolutional layer (Conv1) with a kernel size of 3 × 3, depth size of 15, and a ReLU activation function. The edge map (PR) is then concatenated with the output of the first convolutional layer. Thus, the output dimension becomes H × W × 16, where H is the image height and W is the image width. The concatenated output of Conv1 is then fed into the consecutive dense blocks in a feed-forward manner, where each dense block is connected to the next dense block with a transition layer. In general, the dense blocks have several convolutional layers with a kernel size of 3 × 3 and stride size of one. Moreover, each of these layers is normalized with a batch normalization function and activated with the ReLU function. In the proposed model, a growth rate of g = 8 is used to increase the number of convolutional layers over each occurrence of a dense block. The Dense1 – 3 blocks have different output dimensions of 64, 128, and 256, respectively. The output of each dense block passes through a transition block. Moreover, each of the Tran1 – 3 blocks consist of a convolution layer and a downsampling layer. Each convolution layer of the transition blocks has a kernel size of 3 × 3, stride =1 and it is activated with the ReLU function. The downsampling layer also performs a convolution operation with a stride size of two at the same kernel size. The third transition block (i.e., Tran3) is followed by three more convolutional blocks. The Conv2 block has two more convolutional layers. The Conv3 block consists of five dilated convolution layers (dilation=2) for high-level feature extraction followed by the Conv4 block with a single convolution layer. Each layer of the Conv2 – 4 blocks has a kernel size of 3 × 3, stride size of one, and a depth of 512. Moreover, these layers are normalized with a batch normalization function and activated with the ReLU function. The Conv5 – 7 blocks are used as upsampling blocks. Each of these blocks is composed of a 2 × 2 upsampling convolution followed by another convolution layer with a kernel size of 3 × 3. Like previous convolution blocks, the Conv5 – 7 layers are also normalized with a batch normalization function and activated with the ReLU function. The Conv5 – 7 blocks have the output dimension of 256, 128, 64, respectively. Note that each upsampling layer of the Conv5 – 7 blocks has an individual skip connection with the Dense1 – 3 blocks, respectively. The skip connections allow the network to propagate the low-level features to contribute to the final image reconstruction. The Conv8 block is the final layer in the CRN architecture. It has an output dimension of H × W × 3. The final reconstructed image (IR) is activated with the tanh function.

 

Fig. 6 Model architecture of color reconstruction network (CRN).

Download Full Size | PPT Slide | PDF

2.2.2. Discriminator architecture

In the GAN concept, a discriminator takes the images generated by a generator. It then makes a distinction between the real and false images as a binary classifier [24]. This study uses a stacked CNN block as a discriminator. In the proposed model, the discriminator accepts input as 𝔻 = {CS, PR, ER, IR, IG}. As shown in Fig. 7, the first three layers have a depth of 64, followed by another convolution layer with a depth size of 128. Each convolution layer has a kernel size of 3 × 3 with a stride size of one. A batch normalization is performed in all of the convolutional layers and the output is activated with the ReLU function. The final layer of the discriminator is then flattened and activated with a sigmoid function [36]. Thus, it could make the distinction between real and false images generated by the CRN.

 

Fig. 7 Model architecture of the adversarial block (i.e., discriminator).

Download Full Size | PPT Slide | PDF

2.2.3. Optimization of stage II

The key idea of GAN training relies on the concept of game theory known as the minimax game [24].The generator (G) aims to produce a realistic output that cannot be distinguished by the discriminator (D). By keeping the main principle of the adversarial training, different variants of GAN have been proposed in recent years. Among them, conditional GAN (cGAN) [37] is well known for its capability of generating colorful images [30]. As mentioned in previous studies [30,38], the cGAN learns a mapping as G : {X, Z} → Y, where X is the observed image along with a noise vector Z. The generator tries to produce a realistic image Y. In addition, the noise vector Z helps the generator to produce a non-deterministic output [39]. The goal of the cGAN is then given by

cGAN(G,D)=𝔼X,Y[logD(X,Y)]+𝔼X,Z[log(1D(x,G(Z,Z))]

As with the LRN, the CRN also utilizes the advantage of L1+SSIM. As a consequence, the generator loss CRN is given by

CRN(G)=YG(X,Z)1+SSIM(Y,G(X,Z))
Here, the SSIM is calculated for each channel of a color image [29]. Note that a recent study [30] has reported that a generator can learn to ignore the noise vector (Z) while training. Subsequently, the dropout was used instead of Gaussian noise. This study follows the previous approach for stabilizing the GAN training. The final loss of the CRN as a generator is derived as
CRN(IG,IR)=IGIR1+SSIM(IG,IR)
where the IG is the ground-truth image and IR is the output image of the CRN. In general, the final goal of the optimization for Stage II is set as follows:
G*=argminGmaxDcGAN(G,D)+λCRN(G)
Here, the final GAN loss is optimized by maximizing the discriminator loss and minimizing the generator loss. In this study, the generator loss is multiplied with a reconstruction loss by λ = 100, as used in previous studies [30].

2.3. Hyperperemeters and model training

The different sub-networks of the proposed reconstruction model are optimized with their respective loss functions. However, the overall network architecture shares some common hyperparameters that allow the individual sub-networks to obtain optimal results. For a deep network, finding hyperparameters that best fit the model training is considered one of the most essential processes. Thus, before commencing final model training, hyperparameters are selected. To train the final model, a rate of 0.001 was found to be the best learning rate. Moreover, the Adam optimizer [40] is used to optimize the corresponding deep models, where the parameters are tuned as β1 = 0.5 and β2 = 0.9. The batch size is fixed at five images per step. The proposed model is trained with 100,000 training samples selected arbitrarily from the ImageNet dataset [41]. The real pixel data for a sparse RGBW sensor are necessary to train our reconstruction model. However, it is also true that it is difficult to obtain the real pixel data without a hardware setup. For this reason, in our simulation for the RGBW data, we used the W pixel values simply computed by the simple luminance conversion from the R, G, and B pixel values in a given RGB image (i.e., RGB to Lab conversion). Specifically, in our implementation, note that, for the color pixels (RGB three channels), we only kept a single channel value at each pixel position as per the given CFA pattern. The rest of the channel values were eliminated. For the model training, we considered an input image as four-channel data, where the channel 1–4 represents R, G, B, and W, respectively. For each pixel position, we only kept one channel information (i.e., R, G, B, or W). The rest of the channel information was replaced with “0” (no information). Note that the each training sample has been sampled into a fixed dimension of 256 × 256 pixels.

More specifically, the following training strategy was used:

  • First train the LRN until it achieves the optimal loss (100,000 steps to be exact).
  • As per suggestion of recent papers [30], the generator (G) has been trained to minimize log (1 − D(x, G(x, z)) in every training step, while the discriminator (D) has been trained to maximize log D(x, y) in every n mod 2 = 0 steps.

All training was carried out on a machine running on Ubuntu 16.04.4 with a hardware configuration of an Intel Core i7-7920HQ processor and a random-access memory of 24 GB. An Nvidia Titan XP GPU was used to accelerate the training process.

3. Comparison of results for sparse color reconstruction

Several experiments were conducted to study the feasibility of the proposed reconstruction method for sparse CFA. In particular, the proposed model is compared with several state-of-the-art methods. To do so, an extended version of the existing CFA is used (see Fig. 1(b)). This version of CFA allows us to obtain a very sparse ratio of 98.87% luminance and 1.13% of RGB color pixels. Moreover, a random sparse CFA is used to show the capability of color reconstruction for any sparse color patterns at random positions (see Fig. 12 and Section 3.1.4). Note that this study focuses on the color reconstruction method rather than the development of new CFA arrangements.

3.1. Visual and quantitative comparisons

The state-of-the-art methods used for performance comparison were:1) Chakrabarti-14 [15], 2) Zhang [23], and 3) Chakrabarti-16 [18]. Note that none of the existing works were proposed to handle a very sparse CFA (i.e., 1% color pixels). The original implementation of the Chakrabarti-14 method was obtained from a publicly available site at [42]. We only modified the input part in their code for testing the CFA pattern (18 ×18 grid) used in the experiments. Zhang’s method [23] was proposed for user hints-based image colorization. The original implementation of Zhang’s method was obtained from [43]. Charkrabarti-16 [18] mainly focused on proposing a learning-based sparse CFA. The original implementation of the Chakrabati-16 method was obtained from [44]. For the comparison with Charkrabarti-16 [18], only the reconstruction model was used and fed with full-sized images (in the original paper, image patches were used as an input [18]). Apart from that, the input layers of all the existing methods [15,18,23] were modified to investigate their feasibility for the sparse color reconstruction. However, to make the comparisons as fair as possible, the training samples and CFA pattern are kept the same as the ones used to train the proposed model. The learning-based models (i.e., Chakrabarti-16 [18] and Zhang [23]) were trained from scratch with their suggested hyperparameters. A public dataset [45] was used to test all models. The testing images are different from the training samples.

3.1.1. Color bleeding improvement

The presence of color bleeding artifacts is one of the most critical issues in colorization. As shown in Figs. 8(c)–8(e), some dominant color in a region can spill over to its neighbors. As mentioned earlier, the proposed color reconstruction method addresses this issue by feeding and propagating the edge information into the CRN. Without the adversarial training, the two-stage deep model itself can reduce the color bleeding. Here, the proposed two-stage deep model without the adversarial training is denoted as sNet. The proposed sGAN also includes sNet as a generator. Figure 8 and Fig. 9 demonstrate the color bleeding improvement through the proposed methods (both sNet without the adversarial block and sGAN with the adversarial block).

 

Fig. 8 Color bleeding improvement through sNet and sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).

Download Full Size | PPT Slide | PDF

 

Fig. 9 Color bleeding improvement through sNet and sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).

Download Full Size | PPT Slide | PDF

3.1.2. False color improvement

False-color artifacts are another challenge for the sparse color reconstruction. In most cases, when a target image contains small, complex object regions, and the color hints are very sparse, the reconstruction method is unable to recover the actual colors. We have observed that, in many cases, false-color artifacts could appear as desaturated colors or unnatural colors that differ greatly from the ground truth color. Figure 10 and Fig. 11 demonstrate the false-color reduction through the proposed sGAN. As shown in Fig. 10(g), the sGAN outperforms the previous models by improving colorfulness in the reconstructed images. These results reveal that the sGAN helps the sNet to learn precise features that can reduce false-color artifacts.

 

Fig. 10 False-color improvement through the sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).

Download Full Size | PPT Slide | PDF

 

Fig. 11 False-color improvement through the sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).

Download Full Size | PPT Slide | PDF

3.1.3. Quantitative comparisons

Besides the visual comparison results, quantitative comparisons has been conducted with the previous methods. As used in previous studies [11, 16, 20], color peak signal to noise ratio (CPSNR) and CIELAB color difference metrics (ΔE) have been used to evaluate the quantitative comparison. The CPSNR for an image has been calculated as follows:

CPSNR(IG,IR)=10log10255213HWc3iHjWIG(i,j,c)IR(i,j,c)22,
where H, W, IG, and IR represent the height of the image, the width of the image, the ground-truth image, and the reconstructed image, respectively. Note that c is an index for the three RGB color channels.

Table 1 and Table 2 show the mean CPSNR values (dB) and the mean CIELAB color difference (ΔE) of all models for two different public datasets known as Kodak [46] and IMAX [47]. The Kodak dataset contains 24 color images of the size of 768 × 512 pixels, while the IMAX dataset has a collection of 18 images with a resolution of 500 × 500 pixels. The quantitative results appear to demonstrate that the proposed reconstruction method can outperform the existing methods for the public datasets.

Tables Icon

Table 1. Mean CPSNR (dB) Calculated with Kodak and IMAX Datasets.

Tables Icon

Table 2. Mean CIELAB Color Difference (ΔE) Calculated with Kodak and IMAX Datasets.

3.1.4. Color reconstruction from a random sparse CFA

Furthermore, a random sparse CFA was used to show that the proposed method is capable of properly reconstructing full-color images from any sparse color patterns at random positions. Figure 12 shows the random sparse CFA used in this experiment. The random sparse pattern is incorporated with a grid size of 20 × 20 pixels, where each grid in an image contains only four arbitrary color pixels (one for red, one for blue, and two for green pixels at random positions per grid). Note that this allows us to obtain a very sparse ratio of 1% color pixels along with 99% panchromatic pixels in a random pixel arrangement.

 

Fig. 12 Example illustration of the random CFA pattern with 1% color pixels used in our experiments. W denotes panchromatic white pixels. R, G, and B denote red, green, and blue color pixels, respectively.

Download Full Size | PPT Slide | PDF

Figure 13 and Fig. 14 demonstrate the visual results obtained from the random sparse CFA. As shown in the figures, the proposed sGAN can produce visually compelling RGB images that are very similar to the ground-truth images without any severe visual artifacts. From the figures, it is observed that the proposed sGAN, even with a random 1% color pixels, can also produce colorful images comparable to images produced by a Bayer sensor.

 

Fig. 13 Example results reconstructed from random 1% sparse RGB pixels. (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image. (e) Zoomed versions of the ground truth (2000% magnification). (f) Zoomed versions of the reconstructed image (2000% magnification). (g) Zoomed versions of the ground truth. (h) Zoomed versions of the reconstructed image. (i) Zoomed versions of the ground truth (2000% magnification). (j) Zoomed versions of the reconstructed image (2000% magnification).

Download Full Size | PPT Slide | PDF

 

Fig. 14 Example results reconstructed from random 1% sparse RGB pixels. (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image

Download Full Size | PPT Slide | PDF

In our experiments, we tested our model with many different testing images captured with different image sensors. We found that, in most cases, our model is consistent and capable of producing correct and natural colors, as shown in Fig. 13 and Fig. 14. However, depending on the image content and the position of color hints, our model can produce false color artifacts in some extreme cases (e.g., a small blob). If we magnify the original version of Fig. 13 by a large factor (e.g., 2000% magnification in this example; see Fig. 13(f)), it can be visible that the proposed model can sometimes have false color artifacts for some small blobs. However, the adversarial guidance from the discriminator helps the proposed CRN model to produce natural colors in the reconstructed image although the color generated by the CRN are different from the ground-truth.

The proposed reconstruction method was also tested with a standard simple test image with high frequency patterns (i.e., a slanted bar image for MTF50 evaluation [48]). Figure 15 shows the reconstruction results obtained by the proposed model. The results also confirm that the proposed model can properly reconstruct the full image from a set of sparse color pixels without severe visual artifacts.

 

Fig. 15 Reconstruction results from random 1% sparse RGB pixels using a standard simple test image with high frequency patterns (a slanted bar image). (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image.

Download Full Size | PPT Slide | PDF

4. Prototype system using hybrid monochrome-plus-RGB cameras

The previous sections demonstrate the simulation results that were obtained using the test images previously captured by Bayer sensors. Note that in Section 3, the panchromatic pixels were obtained from the RGB to grayscale conversion. However, this is not the case in a real sparse color sensor. Particularly, panchromatic pixels and color pixels in an actual sparse color sensor can have different characteristics due to different color filters that may also require the use of different image processing pipelines for white pixels and color pixels, respectively. In this study, a prototype system was developed using a hybrid camera system to closely simulate the process of color reconstruction from sparse color pixels. The hybrid camera system consists of a pair of monochrome and RGB color cameras. With this configuration, the panchromatic pixel values are obtained from the monochrome sensor, while the RGB pixel values are obtained from the Bayer color sensor. Thus, the hybrid camera system can simulate a sparse color input with the desired pixel ratio that feeds into the proposed color reconstruction model.

4.1. Hardware setup

Table. 3 shows the specification for the two cameras used for the hybrid camera system. Note that the monochrome camera is identical to the color camera except for the absence of the Bayer color filter. Therefore, the quantum efficiency of the monochrome camera is higher than that of the color camera. An optical breadboard was used to mount both cameras into same surface, as shown in Fig. 16. Moreover, an optical beam splitter (50/50 reflection-transmission) was used to capture the same scene with the same amount of incoming light to both cameras. The beam splitter has a dimension of 80mm × 80mm × 1mm. It is mounted in a mirror holder that is again mounted to a steel post. In such a manner, both cameras could cover the same field of view (FOV). The steel posts used to mount the cameras and the splitter have a height of 50 mm and 35 mm, respectively. Figure 16 shows the prototype system along with an example pair of monochrome and color images captured by each camera.

 

Fig. 16 Hybrid camera system using two cameras and a beam splitter. (a) Side-view. (b) Top-view. (c) Example monochrome image captured by the monochrome camera. (d) Example RGB image with the same field of view captured by the color camera.

Download Full Size | PPT Slide | PDF

Tables Icon

Table 3. Specification of the Two Cameras Used for Our Prototype.

4.2. Experiment procedure

The captured image pair were sampled prior to inputting them to the color reconstruction method. In order to generate the sparse color input images using the hybrid camera system, the following steps were taken:

  • To capture the images with the monochrome and RGB cameras, FlyCapture SDK [49] was used with the default camera settings. The exposure time for each camera was automatically selected by the FleaCapture depending on the image capture environment. It allowed to test the proposed model with random exposure settings. A stereo image pair was captured while keeping the settings constant.
  • Despite the use of a beam splitter, there could be slight geometrical misalignment between a stereo pair due to the camera positions. Before capturing the image pair, a homography transform matrix was calculated in a camera calibration process using chess-board images [50]. This homography matrix was used to align the coordinate of the image pair.
  • A sparse color sensor should generate a single input image depending on the given CFA patterns. Thus, both images were sampled and merged into a single image. The sampling was carried out according to the sparse CFA pattern used for training the reconstruction model.

Note that due to the different physical property of the image sensors, intensity differences between the panchromatic and color pixels can occur [14], which can have an impact on the model output. In our initial setup with two cameras, we also found that the luminance levels can be different for the pixels at the same position captured by the two cameras. This luminance difference can be related to the exposure issue. Our initial model could not handle this difference and have some negative affect on reconstruction images such as visual artifacts. In our experiments, we addressed this exposure-related issue with a data augmentation technique in learning our model. To that end, we randomly tweaked the intensity values for both RGB and white pixels so that their intensity levels of the synthesized image were intentionally different. To obtain the intensity difference between RGB and white pixels in training samples (the same samples as we used for noise contamination), we multiplied the original intensity levels of each image with a random factor ranged from 0.5 to 1.5. Our model was retrained with the augmented data until the model converged to cope with the intensity difference and correctly reconstruct colors. With this data augmentation technique, our model learned the ability of intensity normalization between the color and mono pixels. The experimental results obtained by the hybrid camera have revealed that our model can properly reconstruct the full-color image from the different intensity input pixels. The authors believe that our technique with data augmentation can help to address the exposure issue in part.

In addition, to provide a diversity of noisy image data in training our model, the Gaussian noise has been added into the training images. The standard deviation ranged from 0.001 to 0.1 and its value was randomly selected to augment each training image. Here, all images from the training set has been used for the augmentation. After the noise contamination (i.e., data augmentation), the model has been re-trained only with the augmented images using the transfer learning technique [51]. It allows our model to jointly learn the denoising effect while reconstructing a full-color image.

4.3. Visual results from the prototype system

One of the benefits of the sparse color sensor is the performance improvement in low light conditions because the sparse color sensor uses many panchromatic pixels with better QE than that of color pixels. In addition, the sparse color sensor should be able to retain the color information as well as a conventional Bayer sensor in any lighting condition. To investigate the performance of the proposed color reconstruction method, different lighting conditions were considered as follows:

  • Well-lit condition: This can be considered as a normal indoor lighting condition. Figure 17 shows the comparison between the outputs obtained from the Bayer RGB sensor and sparse color reconstruction in a well-lit condition (86 lux). The visual results demonstrate that the proposed method can reconstruct full-color images without any visual artifacts comparable to the Bayer sensor.
  • Low-light Condition: In low-light conditions, an RGB Bayer sensor often struggles to capture the details in the scene. As mentioned earlier, panchromatic pixels can provide better picture quality than RGB pixels in terms of noise and sharpness. Figure 18 demonstrates the advantage of a large number of panchromatic pixels in terms of detail. As shown in Fig. 18, the proposed color reconstruction model is capable of reconstructing correct color information while preserving details from the panchromatic pixels.
  • Extreme low-light condition: An extreme low-light condition has also been considered to explore the advantage of a very sparse color sensor. Figure 19 shows the performance of the proposed reconstruction model in an extreme low-light condition (6 lux). The extreme low-light scene reveals that the sparse color sensor with around 99% panchromatic pixels can improve low-light performance. In addition, the proposed model can reconstruct full-color images from the very sparse color hints without any visual artifacts in this low-light condition.

 

Fig. 17 Well-lit comparison at 86 lux. The figures in the first row show the full versions of the images captured with an RGB sensor (left) and reconstructed with a sparse color sensor prototype (right). The figures in the second row show the zoomed regions. In every pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the color reconstruction method.

Download Full Size | PPT Slide | PDF

 

Fig. 18 Low-light comparison at 12 lux. In each pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the color reconstruction method.

Download Full Size | PPT Slide | PDF

 

Fig. 19 Extreme low-light comparison at 6 lux. In each pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the sparse color reconstruction method.

Download Full Size | PPT Slide | PDF

4.4. Denoising effect

As mentioned earlier, our model was retrained to jointly learn the denoising effect while reconstructing a full-color image. Figure 20 shows examples for the denoising effect of our model. To obtain this result, the RGB images captured with the hybrid camera system has been used. After that, the Gaussian noise has been added to the test images. As shown in Fig. 20, the proposed model can denoise the Gaussian noisy image at some level of noise, while reconstructing the full-color image from sparse color hints.

 

Fig. 20 Example results that demonstrate the denoising capability of the proposed model, while reconstructing the full-color image. (a) Left:input RGB image (without noise), Right:reconstructed image. (b) Left:input RGB image with low noise (standard deviation of 0.03), Right:denoised and reconstructed image. (c) Left:input RGB image with high noise (standard deviation of 0.08), Right:denoised and reconstructed image.

Download Full Size | PPT Slide | PDF

5. Discussion

In general, the computational complexity of a deep model is calculated by the number of learnable parameters and floating-point operations (FLOPs) [52]. With our current implementation, the proposed model comprises of 26.5M trainable parameters (= 296,579 for the stage I + 26,395,854 for the stage II) with 53.9M FLOPs per training step. However, our model consists of less amount of trainable parameters than that of the well-known deep models (e.g., Resnet (60.4M), VGG (138.4M), InceptionNet (55.9M), etc.). Particularly, the dense blocks in our CRN part help us to reduce the trainable parameters, as mentioned in previous studies [34]. In addition, our hardware setup requires 630 ms for a single step (5 images for each step). Note that the computational time can heavily depend on the hardware setup. In our current hardware setup, we had required training time of 30 hours for the full training. However, in testing, the inference time is significantly reduced. For a 1024 × 720 resolution image, our model required a reconstruction time of 530 ms and for a 2048 × 1920 image, our model required a reconstruction time of 930 ms, which are less than 1 sec.

The current model requires heavy computational resources. Therefore, it is required to further investigate the optimization of the current model, in order to embed the proposed network into real-time hardware devices, such as ISPs. The future direction of this work will be to investigate hardware-friendly efficient implementations of the proposed model. Moreover, this study does not cover all the limitations to make a new real image sensor. Clearly, there are several unmet limitations such as no modeling of pixel crosstalk, handling of different exposure settings between panchromatic and RGB pixels, exact modeling of sensor noises. In addit ion to the development of the color reconstruction method mainly focused in this study, these limitations remain as important future studies to realize such a sparse color sensor.

Apart from that, the sparsity of RGB pixels can depend on the spatial variations in the image content. The sparsity can also depend on the positions of the sparse RGB pixels (i.e., the positions of the color hints for reconstruction) that are related to the given CFA pattern. Recall that, in our experiments, we used two different CFA patterns (one is an 18 × 18 grid pattern, and another is a random pattern for each 20 × 20 grid). In the experiments using various image contents in our test dataset and the CFA patterns, we have observed that our model can handle up to 1% of sparsity.

It is worth further noting that designing an optimal sparse CFA pattern would also be challenging for future work. Without that, it is difficult to exactly quantify the sparsity of the RGB pixels that are required for correct color reconstruction. Apart from that, when the sparsity becomes inadequate, the model can produce false color artifacts in some cases. In the experiments, a small image blob (e.g., a small blob inside an 18 × 18 grid) can have a chance to produce false color artifacts. Because there are no color hints within the blob, in this case, our model cannot propagate appropriate color hints to colorize the blob (for example, see Figs. 13(i) and 13(j)). However, we would like to point out that the proposed model uses an adversarial block as a discriminator that guides the generator (i.e., CRN) to produce natural and well-matched color images. Based on the observations from the experimental results, the proposed model can naturally reconstruct small color blobs although there may be no color hints within the blob.

Furthermore, the LRN can be replaced with a simple interpolation to simplify the proposed model. In most cases, a simple interpolation method can fill the luminance holes correctly since the holes are very sparse. However, we would like to point out that, in some cases (e.g., complex objects in Fig. 4), the LRN can recover the image holes better than a simple interpolation-based method [53,54]. We found that the incorrect hole recovery could have an effect on the edge extraction. Hence, it can affect the color reconstruction (since the second stage of the proposed model uses the extracted edge). Moreover, compared to the stage II, the stage I with LRN comprises a very small number of parameters (296,579 trainable parameters, which is approximately 1% of the overall parameters). Thus, it is worth noting that the LRN does not significantly affect the model training and inference in terms of the computational complexity. Furthermore, the LRN has been implemented in a modular manner so that the CRN part can be easily implemented and tested without the existence of the LRN. Our code is available at the following site: https://github.com/gachoncvip/sGAN.

6. Conclusion

A two-stage color reconstruction method was proposed to reconstruct full-color images from a very sparse color pixel along with most panchromatic pixels. The comparison results have demonstrated that the proposed reconstruction model outperforms the state-of-the-art methods. Importantly, the simulation results have revealed that the proposed model can handle fixed or random sparse CFAs up to 1% of sparse color pixels. In particular, the proposed model can reduce color bleeding and false-color artifacts in the final reconstructed images, compared to the previous methods. In addition, a hybrid dual camera system was developed to investigate the feasibility of a new sparse color sensor. We believe that the proposed color reconstruction model can be used to realize such a sparse color sensor.

Funding

National Research Foundation of Korea (2016R1D1A1B03931087).

Acknowledgments

The authors thank H. W. Jang for his help in implementing the network architecture.

References

1. B. E. Bayer, “Color imaging array,” U.S. patent 3,971,065 (1976).

2. B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005). [CrossRef]  

3. L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005). [CrossRef]   [PubMed]  

4. M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

5. P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628. [CrossRef]  

6. H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014). [CrossRef]  

7. F. Sigernes, M. Dyrland, N. Peters, D. A. Lorentzen, T. Svenøe, K. Heia, S. Chernouss, C. S. Deehr, and M. Kosch, “The absolute sensitivity of digital colour cameras,” Opt. Express 17(22), 20211–20220 (2009). [CrossRef]   [PubMed]  

8. Y. J. Jung, “Enhancement of low light level images using color-plus-mono dual camera,” Opt. Express 25(10), 12029–12051 (2017). [CrossRef]   [PubMed]  

9. I. Hirota, “Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus,” U.S. patent 9,392,238 (2016).

10. T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton Jr, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

11. C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016). [CrossRef]   [PubMed]  

12. R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973). [CrossRef]   [PubMed]  

13. Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013). [CrossRef]  

14. D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011). [CrossRef]  

15. A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.

16. M. Singh and T. Singh, “Linear universal demosaicking of regular pattern color filter arrays,” in Proceedings on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (IEEE, 2012), pp. 1277–1280.

17. B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011). [CrossRef]   [PubMed]  

18. A. Chakrabarti, “Learning sensor multiplexing design through back-propagation,” in Advances in Neural Information Processing Systems, (2016), pp. 3081–3089.

19. J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017). [CrossRef]   [PubMed]  

20. H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017). [CrossRef]  

21. M. Parmar and B. A. Wandell, “Interleaved imaging: an imaging system design inspired by rod-cone vision,” in Digital Photography V, (International Society for Optics and Photonics, 2009), p. 725008. [CrossRef]  

22. Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K. [CrossRef]  

23. R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017). [CrossRef]  

24. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

25. P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005). [PubMed]  

26. J. Canny, “A computational approach to edge detection,” in Readings in Computer Vision, (Elsevier, 1987), pp. 184–203.

27. F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

28. B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).

29. H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

30. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

31. X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

32. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

33. H. W. Jang, College of Information Technology, Gachon University, Sujeong-Gu, Seongnam, 13120, South Korea and Y. J. Jung are preparing a manuscript to be called “Deep color transfer for color-plus-mono dual camera”.

34. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

35. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.

36. J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.

37. M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 (2014).

38. M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).

39. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

40. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

41. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

42. “Rethinnk color camera code,” http://vision.seas.harvard.edu/colorsensor/. Accessed: 2019-07-07.

43. “Automatic image colorization code,” https://richzhang.github.io/colorization/. Accessed: 2019-07-07.

44. “Learning sensor multiplexing design through back-propagation code,” https://github.com/ayanc/learncfa/. Accessed: 2019-07-07.

45. S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

46. X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.

47. L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

48. ISO-12233, “Imaging resolution, photography electronic still picture and responses, spatial frequency,” (2014).

49. “Flycapture sdk,” https://www.ptgrey.com/flycapture-sdk. Accessed: 2019-04-07.

50. Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

51. S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

52. S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

53. A. Telea, “An image inpainting technique based on the fast marching method,” J. Graph. Tools 9(1), 23–34 (2004).

54. M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

References

  • View by:
  • |
  • |
  • |

  1. B. E. Bayer, “Color imaging array,” U.S. patent 3,971,065 (1976).
  2. B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
    [Crossref]
  3. L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005).
    [Crossref] [PubMed]
  4. M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).
  5. P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
    [Crossref]
  6. H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
    [Crossref]
  7. F. Sigernes, M. Dyrland, N. Peters, D. A. Lorentzen, T. Svenøe, K. Heia, S. Chernouss, C. S. Deehr, and M. Kosch, “The absolute sensitivity of digital colour cameras,” Opt. Express 17(22), 20211–20220 (2009).
    [Crossref] [PubMed]
  8. Y. J. Jung, “Enhancement of low light level images using color-plus-mono dual camera,” Opt. Express 25(10), 12029–12051 (2017).
    [Crossref] [PubMed]
  9. I. Hirota, “Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus,” U.S. patent 9,392,238 (2016).
  10. T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).
  11. C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
    [Crossref] [PubMed]
  12. R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
    [Crossref] [PubMed]
  13. Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
    [Crossref]
  14. D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011).
    [Crossref]
  15. A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.
  16. M. Singh and T. Singh, “Linear universal demosaicking of regular pattern color filter arrays,” in Proceedings on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (IEEE, 2012), pp. 1277–1280.
  17. B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
    [Crossref] [PubMed]
  18. A. Chakrabarti, “Learning sensor multiplexing design through back-propagation,” in Advances in Neural Information Processing Systems, (2016), pp. 3081–3089.
  19. J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
    [Crossref] [PubMed]
  20. H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
    [Crossref]
  21. M. Parmar and B. A. Wandell, “Interleaved imaging: an imaging system design inspired by rod-cone vision,” in Digital Photography V, (International Society for Optics and Photonics, 2009), p. 725008.
    [Crossref]
  22. Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
    [Crossref]
  23. R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
    [Crossref]
  24. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.
  25. P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
    [PubMed]
  26. J. Canny, “A computational approach to edge detection,” in Readings in Computer Vision, (Elsevier, 1987), pp. 184–203.
  27. F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).
  28. B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).
  29. H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).
  30. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.
  31. X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).
  32. Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.
  33. H. W. Jang, College of Information Technology, Gachon University, Sujeong-Gu, Seongnam, 13120, South Korea and Y. J. Jung are preparing a manuscript to be called “Deep color transfer for color-plus-mono dual camera”.
  34. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.
  35. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.
  36. J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.
  37. M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 (2014).
  38. M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).
  39. D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).
  40. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).
  41. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.
  42. “Rethinnk color camera code,” http://vision.seas.harvard.edu/colorsensor/ . Accessed: 2019-07-07.
  43. “Automatic image colorization code,” https://richzhang.github.io/colorization/ . Accessed: 2019-07-07.
  44. “Learning sensor multiplexing design through back-propagation code,” https://github.com/ayanc/learncfa/ . Accessed: 2019-07-07.
  45. S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).
  46. X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.
  47. L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).
  48. ISO-12233, “Imaging resolution, photography electronic still picture and responses, spatial frequency,” (2014).
  49. “Flycapture sdk,” https://www.ptgrey.com/flycapture-sdk . Accessed: 2019-04-07.
  50. Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).
  51. S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).
  52. S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).
  53. A. Telea, “An image inpainting technique based on the fast marching method,” J. Graph. Tools 9(1), 23–34 (2004).
  54. M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

2018 (2)

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

2017 (5)

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

Y. J. Jung, “Enhancement of low light level images using color-plus-mono dual camera,” Opt. Express 25(10), 12029–12051 (2017).
[Crossref] [PubMed]

2016 (3)

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

2014 (1)

H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
[Crossref]

2013 (1)

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

2012 (1)

Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

2011 (4)

D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011).
[Crossref]

B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
[Crossref] [PubMed]

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).

2010 (1)

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

2009 (1)

2005 (3)

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005).
[Crossref] [PubMed]

P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
[PubMed]

2004 (1)

A. Telea, “An image inpainting technique based on the fast marching method,” J. Graph. Tools 9(1), 23–34 (2004).

1973 (1)

R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
[Crossref] [PubMed]

Abdulkadir, A.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

Adams, A.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Agostinelli, F.

F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

Altunbasak, Y.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

Ba, J.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

Bai, C.

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

Baldi, P.

F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

Bao, P.

P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
[PubMed]

Barron, J. T.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Bayer, B. E.

B. E. Bayer, “Color imaging array,” U.S. patent 3,971,065 (1976).

Bengio, Y.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Bertalmio, M.

M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

Bertozzi, A. L.

M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

Bianco, S.

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

Brox, T.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.

Buades, A.

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

Cadene, R.

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

Calvagno, G.

D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011).
[Crossref]

Canny, J.

J. Canny, “A computational approach to edge detection,” in Readings in Computer Vision, (Elsevier, 1987), pp. 184–203.

Celona, L.

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

Chakrabarti, A.

A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.

A. Chakrabarti, “Learning sensor multiplexing design through back-propagation,” in Advances in Neural Information Processing Systems, (2016), pp. 3081–3089.

Chaurasia, G.

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

Chen, H.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

Chen, J.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Chernouss, S.

Cho, B. H.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Çiçek, Ö.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

Compton, J. T.

T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

Couprie, C.

M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).

Courville, A.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Deehr, C. S.

Deng, J.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Dong, W.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Dou, Q.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

Dubois, E.

B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
[Crossref] [PubMed]

Durand, F.

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

Dyrland, M.

Efros, A. A.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

Farrell, J.

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

Farrell, J. E.

Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
[Crossref]

Fei-Fei, L.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Fischer, P.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.

Freeman, W. T.

A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.

Frosio, I.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

Fu, C.-W.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

Gallo, O.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

Ganguli, S.

J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.

Geiss, R.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Geng, X.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

Gharbi, M.

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

Glotzbach, J.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

Goodfellow, I.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Gunturk, B.

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.

Gunturk, B. K.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

Hamilton, J. F.

T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

Hao, P.

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

Hasinoff, S. W.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Heia, K.

Heng, P.-A.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

Hirota, I.

I. Hirota, “Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus,” U.S. patent 9,392,238 (2016).

Hoffman, M.

F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

Huang, G.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

Isola, P.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

Jang, H. W.

H. W. Jang, College of Information Technology, Gachon University, Sujeong-Gu, Seongnam, 13120, South Korea and Y. J. Jung are preparing a manuscript to be called “Deep color transfer for color-plus-mono dual camera”.

Jeon, G.

B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
[Crossref] [PubMed]

Jiang, H.

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

Jung, Y. J.

Y. J. Jung, “Enhancement of low light level images using color-plus-mono dual camera,” Opt. Express 25(10), 12029–12051 (2017).
[Crossref] [PubMed]

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

Kainz, F.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Kang, H. A.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Karlik, B.

B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).

Kautz, J.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

Kijima, T.

T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

Kim, H. Y.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Kingma, D. P.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

Kosch, M.

Lacy, P. L.

R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
[Crossref] [PubMed]

Lansel, S.

Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
[Crossref]

LeCun, Y.

M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).

Lee, H. G.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Lee, K.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Lee, S.-I.

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

Lee, W. J.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Lempitsky, V.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

Leung, B.

B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
[Crossref] [PubMed]

Levoy, M.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Li, J.

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

Li, K.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Li, L.-J.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Li, X.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.

Li, Y.

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

Lienkamp, S. S.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

Lin, A. S.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

Lin, Z.

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

Liu, H.

H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
[Crossref]

Liu, Z.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

Lorentzen, D. A.

Mathieu, M.

M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).

Menon, D.

D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011).
[Crossref]

Mersereau, R. M.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

Mirza, M.

M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 (2014).

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Nakamura, H.

T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

Napoletano, P.

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

Olgac, A. V.

B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).

Osindero, S.

M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 (2014).

Ozair, S.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Pan, S. J.

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

Paris, S.

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

Park, J. M.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Park, P. K.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Parmar, M.

M. Parmar and B. A. Wandell, “Interleaved imaging: an imaging system design inspired by rod-cone vision,” in Digital Photography V, (International Society for Optics and Photonics, 2009), p. 725008.
[Crossref]

Pennington, J.

J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.

Peters, N.

Pouget-Abadie, J.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Qi, X.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

Reid, M.

R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
[Crossref] [PubMed]

Ro, Y. M.

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

Roh, Y.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Ronneberger, O.

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.

Sadowski, P.

F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

Sapiro, G.

M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

Schafer, R. W.

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

Schoenholz, S.

J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.

Sharlet, D.

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

Sigernes, F.

Singh, M.

M. Singh and T. Singh, “Linear universal demosaicking of regular pattern color filter arrays,” in Proceedings on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (IEEE, 2012), pp. 1277–1280.

Singh, T.

M. Singh and T. Singh, “Linear universal demosaicking of regular pattern color filter arrays,” in Proceedings on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (IEEE, 2012), pp. 1277–1280.

Socher, R.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

Sohn, H.

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

Speranza, F.

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

Steinberg, R. H.

R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
[Crossref] [PubMed]

Svenøe, T.

Telea, A.

A. Telea, “An image inpainting technique based on the fast marching method,” J. Graph. Tools 9(1), 23–34 (2004).

Tian, Q.

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
[Crossref]

Ulyanov, D.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

Van Der Maaten, L.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

Vedaldi, A.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

Wandell, B. A.

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

M. Parmar and B. A. Wandell, “Interleaved imaging: an imaging system design inspired by rod-cone vision,” in Digital Photography V, (International Society for Optics and Photonics, 2009), p. 725008.
[Crossref]

Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
[Crossref]

Wang, J.

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

Wang, L.

H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
[Crossref]

Wang, Y.

H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
[Crossref]

Warde-Farley, D.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Weinberger, K. Q.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

Woo, J.

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

Wu, X.

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005).
[Crossref] [PubMed]

P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
[PubMed]

Xu, B.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

Yang, Q.

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

Yu, J.

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

Yu, T.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

Zhang, C.

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

Zhang, L.

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005).
[Crossref] [PubMed]

P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
[PubMed]

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.

Zhang, R.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

Zhao, H.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

Zhou, T.

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

Zhu, J.-Y.

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

Zickler, T.

A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.

ACM Trans. Graph. (2)

R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” ACM Trans. Graph. 36(4), 119 (2017).
[Crossref]

S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Trans. Graph. 35(6), 192 (2016).

ACM Trans.Graph. (1)

M. Gharbi, G. Chaurasia, S. Paris, and F. Durand, “Deep joint demosaicking and denoising,” ACM Trans.Graph. 35(6), 191 (2016).

IEEE Access (1)

S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” IEEE Access 6, 64270–64277 (2018).

IEEE J Transl Eng Heal. Med (1)

H. Liu, Y. Wang, and L. Wang, “The effect of light conditions on photoplethysmographic image acquisition using a commercial camera,” IEEE J Transl Eng Heal. Med 2, 1–11 (2014).
[Crossref]

IEEE Signal Process. Mag. (1)

B. K. Gunturk, J. Glotzbach, Y. Altunbasak, R. W. Schafer, and R. M. Mersereau, “Demosaicking: color filter array interpolation,” IEEE Signal Process. Mag. 22(1), 44–54 (2005).
[Crossref]

IEEE Trans. Circ. Syst. Video Tech. (1)

Y. J. Jung, H. Sohn, S.-I. Lee, F. Speranza, and Y. M. Ro, “Visual importance-and discomfort region-selective low-pass filtering for reducing visual discomfort in stereoscopic displays,” IEEE Trans. Circ. Syst. Video Tech. 23(8), 1408–1421 (2013).
[Crossref]

IEEE Trans. Comp. Imaging (1)

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for image restoration with neural networks,” IEEE Trans. Comp. Imaging 3(1), 47–57 (2017).

IEEE Trans. Consum. Electron. (1)

Y. J. Jung, H. Sohn, and Y. M. Ro, “Visual discomfort visualizer using stereo vision and time-of-flight depth cameras,” IEEE Trans. Consum. Electron. 58(2), 246–254 (2012).

IEEE Trans. Image Process. (5)

C. Zhang, Y. Li, J. Wang, and P. Hao, “Universal demosaicking of color filter arrays,” IEEE Trans. Image Process. 25(11), 5173–5186 (2016).
[Crossref] [PubMed]

B. Leung, G. Jeon, and E. Dubois, “Least-squares luma–chroma demultiplexing algorithm for bayer demosaicking,” IEEE Trans. Image Process. 20(7), 1885–1894 (2011).
[Crossref] [PubMed]

L. Zhang and X. Wu, “Color demosaicking via directional linear minimum mean square-error estimation,” IEEE Trans. Image Process. 14(12), 2167–2178 (2005).
[Crossref] [PubMed]

J. Li, C. Bai, Z. Lin, and J. Yu, “Automatic design of high-sensitivity color filter arrays with panchromatic pixels,” IEEE Trans. Image Process. 26(2), 870–883 (2017).
[Crossref] [PubMed]

H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell, “Learning the image processing pipeline,” IEEE Trans. Image Process. 26(10), 5032–5042 (2017).
[Crossref]

IEEE Trans. Knowl. Data Eng. (1)

S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

IEEE Trans. Med. Imag. (1)

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes,” IEEE Trans. Med. Imag. 37(12), 2663–2674 (2018).

IEEE Trans. Pattern Anal. Mach. Intell. (1)

P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005).
[PubMed]

Int. J. Artif. Intell. Expert. Syst. (1)

B. Karlik and A. V. Olgac, “Performance analysis of various activation functions in generalized mlp architectures of neural networks,” Int. J. Artif. Intell. Expert. Syst. 1(4), 111–122 (2011).

J. Comp. Neurol. (1)

R. H. Steinberg, M. Reid, and P. L. Lacy, “The distribution of rods and cones in the retina of the cat (felis domesticus),” J. Comp. Neurol. 148(2), 229–248 (1973).
[Crossref] [PubMed]

J. Electron. Imaging (1)

L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imaging 20(2), 023016 (2011).

J. Graph. Tools (1)

A. Telea, “An image inpainting technique based on the fast marching method,” J. Graph. Tools 9(1), 23–34 (2004).

Opt. Express (2)

Signal Process. Image Commun. (1)

D. Menon and G. Calvagno, “Color image demosaicking: An overview,” Signal Process. Image Commun. 26(8–9), 518–533 (2011).
[Crossref]

Other (30)

A. Chakrabarti, W. T. Freeman, and T. Zickler, “Rethinking color cameras,” in Proceedings of IEEE International Conference on Computational Photography (ICCP), (IEEE, 2014), pp. 1–8.

M. Singh and T. Singh, “Linear universal demosaicking of regular pattern color filter arrays,” in Proceedings on IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (IEEE, 2012), pp. 1277–1280.

A. Chakrabarti, “Learning sensor multiplexing design through back-propagation,” in Advances in Neural Information Processing Systems, (2016), pp. 3081–3089.

I. Hirota, “Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus,” U.S. patent 9,392,238 (2016).

T. Kijima, H. Nakamura, J. T. Compton, and J. F. Hamilton, “Image sensor with improved light sensitivity,” U.S. patent 7,688,368 (2010).

P. K. Park, B. H. Cho, J. M. Park, K. Lee, H. Y. Kim, H. A. Kang, H. G. Lee, J. Woo, Y. Roh, and W. J. Lee, “Performance improvement of deep learning based gesture recognition using spatiotemporal demosaicing technique,” in Proceedings of IEEE International Conference on Image Processing (ICIP), (IEEE, 2016), pp. 1624–1628.
[Crossref]

J. Canny, “A computational approach to edge detection,” in Readings in Computer Vision, (Elsevier, 1987), pp. 184–203.

F. Agostinelli, M. Hoffman, P. Sadowski, and P. Baldi, “Learning activation functions to improve deep neural networks,” arXiv preprint arXiv:1412.6830 (2014).

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems, (2014), pp. 2672–2680.

M. Parmar and B. A. Wandell, “Interleaved imaging: an imaging system design inspired by rod-cone vision,” in Digital Photography V, (International Society for Optics and Photonics, 2009), p. 725008.
[Crossref]

Q. Tian, S. Lansel, J. E. Farrell, and B. A. Wandell, “Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (l3) method,” in Digital Photography X, vol. 9023 (International Society for Optics and Photonics, 2014), p. 90230K.
[Crossref]

Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2016), pp. 424–432.

H. W. Jang, College of Information Technology, Gachon University, Sujeong-Gu, Seongnam, 13120, South Korea and Y. J. Jung are preparing a manuscript to be called “Deep color transfer for color-plus-mono dual camera”.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 4700–4708.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, (Springer, 2015), pp. 234–241.

J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” in Advances in Neural Information Processing Systems, (2017), pp. 4785–4795.

M. Mirza and S. Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784 (2014).

M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440 (2015).

D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” arXiv preprint arXiv:1607.08022 (2016).

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2009), pp. 248–255.

“Rethinnk color camera code,” http://vision.seas.harvard.edu/colorsensor/ . Accessed: 2019-07-07.

“Automatic image colorization code,” https://richzhang.github.io/colorization/ . Accessed: 2019-07-07.

“Learning sensor multiplexing design through back-propagation code,” https://github.com/ayanc/learncfa/ . Accessed: 2019-07-07.

M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2001), pp. 335–362.

ISO-12233, “Imaging resolution, photography electronic still picture and responses, spatial frequency,” (2014).

“Flycapture sdk,” https://www.ptgrey.com/flycapture-sdk . Accessed: 2019-04-07.

X. Li, B. Gunturk, and L. Zhang, “Image demosaicing: A systematic survey,” in Visual Communications and Image Processing 2008, (International Society for Optics and Photonics, 2008), p. 68221J.

B. E. Bayer, “Color imaging array,” U.S. patent 3,971,065 (1976).

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1125–1134.

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (20)

Fig. 1
Fig. 1 Examples of sparse color filter patterns. (a) Existing CFZ-14 pattern with a spare ratio (89% panchromatic pixels and 11% RGB pixels). (b) Extended version of CFZ-14 with a very sparse ratio (98.86% panchromatic pixels and 1.13% RGB pixels), used in our experiments. In the figures, W denotes panchromatic white pixels.
Fig. 2
Fig. 2 Visual artifacts generated by the existing methods for sparse color reconstruction. (a) Ground truth (the inset is a zoomed-in image of the marked area). (b) Input Image (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (Proposed). (g) sGAN (Proposed).
Fig. 3
Fig. 3 Overall framework of the proposed color reconstruction method. LRN:luminance recovery network. CRN:color reconstruction network.
Fig. 4
Fig. 4 Input and output samples for Stage I. (a) Input luminance image with missing pixels. (b) Recovered luminance image. (c) Edge map extracted from the recovered image.
Fig. 5
Fig. 5 Model architecture of luminance recovery network (LRN).
Fig. 6
Fig. 6 Model architecture of color reconstruction network (CRN).
Fig. 7
Fig. 7 Model architecture of the adversarial block (i.e., discriminator).
Fig. 8
Fig. 8 Color bleeding improvement through sNet and sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).
Fig. 9
Fig. 9 Color bleeding improvement through sNet and sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).
Fig. 10
Fig. 10 False-color improvement through the sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).
Fig. 11
Fig. 11 False-color improvement through the sGAN. (a) Ground truth (full image). (b) Ground truth (zoom). (c) Chakrabarti-14 [15]. (d) Zhang [23]. (e) Chakrabarti-16 [18]. (f) sNet (proposed). (g) sGAN (proposed).
Fig. 12
Fig. 12 Example illustration of the random CFA pattern with 1% color pixels used in our experiments. W denotes panchromatic white pixels. R, G, and B denote red, green, and blue color pixels, respectively.
Fig. 13
Fig. 13 Example results reconstructed from random 1% sparse RGB pixels. (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image. (e) Zoomed versions of the ground truth (2000% magnification). (f) Zoomed versions of the reconstructed image (2000% magnification). (g) Zoomed versions of the ground truth. (h) Zoomed versions of the reconstructed image. (i) Zoomed versions of the ground truth (2000% magnification). (j) Zoomed versions of the reconstructed image (2000% magnification).
Fig. 14
Fig. 14 Example results reconstructed from random 1% sparse RGB pixels. (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image
Fig. 15
Fig. 15 Reconstruction results from random 1% sparse RGB pixels using a standard simple test image with high frequency patterns (a slanted bar image). (a) Ground truth. (b) Reconstructed image. (c) Zoomed versions of the ground truth. (d) Zoomed versions of the reconstructed image.
Fig. 16
Fig. 16 Hybrid camera system using two cameras and a beam splitter. (a) Side-view. (b) Top-view. (c) Example monochrome image captured by the monochrome camera. (d) Example RGB image with the same field of view captured by the color camera.
Fig. 17
Fig. 17 Well-lit comparison at 86 lux. The figures in the first row show the full versions of the images captured with an RGB sensor (left) and reconstructed with a sparse color sensor prototype (right). The figures in the second row show the zoomed regions. In every pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the color reconstruction method.
Fig. 18
Fig. 18 Low-light comparison at 12 lux. In each pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the color reconstruction method.
Fig. 19
Fig. 19 Extreme low-light comparison at 6 lux. In each pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the sparse color reconstruction method.
Fig. 20
Fig. 20 Example results that demonstrate the denoising capability of the proposed model, while reconstructing the full-color image. (a) Left:input RGB image (without noise), Right:reconstructed image. (b) Left:input RGB image with low noise (standard deviation of 0.03), Right:denoised and reconstructed image. (c) Left:input RGB image with high noise (standard deviation of 0.08), Right:denoised and reconstructed image.

Tables (3)

Tables Icon

Table 1 Mean CPSNR (dB) Calculated with Kodak and IMAX Datasets.

Tables Icon

Table 2 Mean CIELAB Color Difference (ΔE) Calculated with Kodak and IMAX Datasets.

Tables Icon

Table 3 Specification of the Two Cameras Used for Our Prototype.

Equations (6)

Equations on this page are rendered with MathJax. Learn more.

LRN ( P G , P R ) = P G P R 1 + SSIM ( P G , P R )
cGAN ( G , D ) = 𝔼 X , Y [ log D ( X , Y ) ] + 𝔼 X , Z [ log ( 1 D ( x , G ( Z , Z ) ) ]
CRN ( G ) = Y G ( X , Z ) 1 + SSIM ( Y , G ( X , Z ) )
CRN ( I G , I R ) = I G I R 1 + SSIM ( I G , I R )
G * = argmin G max D cGAN ( G , D ) + λ CRN ( G )
CPSNR ( I G , I R ) = 10 log 10 255 2 1 3 HW c 3 i H j W I G ( i , j , c ) I R ( i , j , c ) 2 2 ,

Metrics