A quadrichromatic light-emitting diode (QLED) based visible light communication for mobile phone camera is proposed to improve data rate and enhance illumination effect at the same time. Different from color intensity modulation (CIM), we propose and use color ratio modulation (CRM) in CMOS image sensor based visible light communication to improve data rate. According to the spectral power distribution (SPD) of the QLED and the spectral response of the complementary-metal-oxide-semiconductor (CMOS) image sensor, color multiple-input multiple-output (CMIMO) channel model is set up first to obtain optimal 16-CRM constellation design. Taking full consideration of the high quality of color rendering index (CRI), tunable color temperature (CT), we design a specific data packet structure to realize illumination requirements. A decoding strategy is also addressed for demapping at the receiver. The experimental results demonstrate that the proposed scheme can realize a downlink data rate of 13.2kbit/s, meanwhile, the optical signal source is illumination compatible.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
As LED-based solid-state lighting (SSL) has become increasingly popular, more research has been focused on visible light communication (VLC) based on LEDs. Since light emitting diode (LED) has a short response time, coupled with its high-speed modulation characteristics, LED based VLC system can be a supplementary form of indoor wireless communication [1–3]. Currently many works have been performed in photodetector-based VLC systems [4,5]. Photodetector-based VLC systems utilize positive-intrinsic-negative (PIN) or avalanche photodiode (APD) to receive optical signal. However, in practical applications, using a photodetector as the wireless optical signal receiver is challenging, since a photodetector needs special circuit design. In addition of the photodetector, metal-oxide-semiconductor (CMOS) cameras and image sensors can also be used to record wireless optical signals due to pixel rows are activated sequentially which is also called the rolling shutter [6–8]. Since CMOS image sensor has the delay scanning characteristics, each line of the captured image is a one-dimensional time series which can be used to record the optical signal. Currently, most smartphones are equipped with CMOS camera, hence almost every smartphone is able to act as a wireless optical signal receiver to access the light information transmitted by LEDs. CMOS image sensor based VLC systems are found being used in many fields. In [9,10], geographic information identification code broadcast by LEDs can be detected by CMOS image sensor to realize vehicle navigation and communication. In [8–11], some CMOS image sensor based VLC systems are proposed to realize positioning in indoor and outdoor environment. In , LED is used as the data interface for a wearable hardware and a smart phone camera can be used to access the hardware.
The existing VLC systems using CMOS image sensor as the receiver face problems. Due to the low frame rate of commercial CMOS image sensor, data rate is limited. In order to improve data rate, multilevel modulation scheme using overlapping of two LEDs is proposed in  realizing date rate of 4.32kbit/s. Wavelength division multiplexing (WDM) with RGB LEDs is used in , improving data rate to 2.88kbit/s. However, these schemes fail to satisfy illumination requirements. Furthermore, the LED is not flicker-free when transmitting data.
We propose and focus on the use of color ratio modulation to improve downlink data rate and meet illumination requirements in CMOS image sensor based VLC by using a quadrichromatic LED (QLED). In this work, we first investigate and set up the color multiple-input multiple-output (CMIMO) channel model to obtain optimal 16-CRM constellation design in RGB ratio space of the CMOS image sensor. Both 16-CRM symbols and dimming symbols are encapsulated in data packet to realize the integration of communication and illumination. Since there are four primary color LEDs existing in the system, the proposed QLED device has the ability to tune color rendering index (CRI) and color temperature (CT) while transmitting data. Decoding algorithm for color rolling shutter pattern is also proposed in this paper. Finally a two-minute performance test has been carried out to confirm 13.2kb/s net data rata is realized with the premise of illumination constraints.
2. System principle
2.1 CMIMO channel model and 16-CRM modulation
In CMOS image sensor based VLC, on-off keying (OOK) is the most commonly used modulation, in which a white light LED is operating in fast switching state of “ON” and “OFF” to transmit binary data. Since there are only two LED states, one OOK symbol transmit only one bit. As shown in Fig. 1(a), due to the rolling shutter working mode, in the output image of CMOS image sensor, OOK modulation signal is recorded as black-and-white bars. However, CMOS image sensor is an essentially RGB color sensor, it can record not only the grayscale information but also the color information. Hence, the multicolor-LED can be utilized to improve data rate in CMOS image sensor based VLC.
In this work, a quadrichromatic LED is used as optical signal source, which contains four primary colors: red (R), green (G), blue (B) and amber (A). By varying luminous flux of the four LEDs during symbol durations, target color can be obtained, which is also called color shift keying (CSK). CSK is a RGB color intensity modulation (CIM) scheme that transmits data through the variation of color with fixed optical intensity. As shown in Fig. 1(b), a CSK symbol can also be recorded by the CMOS image in color bar format. In the raw image data of CMOS image sensor, color intensity information of R, G and B channel is stored in 0~255 level. Hence, color intensity vectors constitute 3D symbols in the signal space. However, in most cases, the rolling shutter pattern captured by a CMOS image sensor is not illumination-uniform due to the blooming effect .That means even if the CMOS image sensor is exposured to the same optical beam; absolute values of RGB information are different during different regions of the image. Hence, we proposed using RGB color ratio modulation (CRM) in CMOS image sensor based VLC to overcome this problem. The CRM symbol is obtained by normalizing a RGB color intensity vector, which is. Hence, CRM symbols are spread on a 2D plane in the signal space.
In this work, the spectral power distribution (SPD) of the quadrichromatic LED and the spectral response of the CMOS image sensor (IMX386) are investigated firstly in Fig. 2 (b). Let be the percentage of luminous flux vector of the quadrichromatic LED for a CRM symbol, be the SPD of the quadrichromatic LED and be the spectral response of the CMOS image sensor:16,17]. The optimal layout of 16-CRM constellations in Fig. 2(c) can be obtained by solving the following optimization problem:Table 1.
2.2 System configuration
Hardware configuration is shown in Fig. 3(a). Random binary sequence generator in personal computer (PC) provides binary bit stream to be sent. Micro Control Unit (MCU, STM32F407ZGT6) fetches data from the host PC via RS-485 bus. Since the transmission speed of RS-485 interface is >10Mbyte/s, transmission time of binary bit stream for an image frame can be ignored. The 4-channel Digital to Analog Converter (DAC, TLC5620) controlled by MCU provides driving voltage for the four analog LED dimmers (PT4115). Four analog dimmers produce continuous current for the quadrichromatic LED during a color bar duration. Between two adjacent color bar durations, an “OFF” time slot is added to provide clock synchronization signal by turning off all LEDs. At the receiver, mixed optical signal is detected by a mobile phone camera which is working in the resolution of 1080*1920 and frame rate of 60fps. Since there is a frame-to-frame processing time gap in CMOS image sensor, if the data is transmitted continuously, part of the data might be missing during this “blind” time . To prevent this phenomenon, each data frame lasts for three-image-frame time is a good choice to ensure a whole data frame can be recorded by the CMOS image sensor. The real-time video can be transmitted to PC via USB-Type C high speed interface, where the color rolling shutter pattern is decoded to original binary data. The data frame structure is shown in Fig. 3(b). Each data frame is consisted of the frame header, several data packets and the frame end. In the frame header, 16-CRM color samples are provided in order for decoding, which will be discussed in the rest of the paper. Since constellations are not considered to be equiprobable, the overall color of the quadrichromatic LED will not be stable when transmitting data. Hence, a specific data packet structure is designed to implement illumination requirements. In our design, nine symbols are encapsulated in a data packet, including five CRM symbols and four primary color symbols. The primary color symbol means the color bar generated by only one color LED. As shown in Fig. 4, we adopt a dynamic adjustment method to implement illumination requirements. Four primary color symbols are inserted to five CRM symbols to form a data packet. Assuming the total luminous flux of the quadrichromatic LED maintains unchanged during CRM symbol duration, by varying the luminous flux of the single color LED during a primary color symbol duration, the mixed optical signal during a data packet duration can be tuned to implement illumination requirements. Since we adopt 16-CRM in constellation design, data carried by a CRM symbol is four bits. Hence, in a data packet CRM symbols can transfer data of 20bits; on the other hand, four primary color symbols can appear in different orders to transfer four bits in a data packet. As a result, a data packet can totally bear three-byte (24bits) information.
2.3 illumination constraint
Since a quadrichromatic LED is used in the design, the device has the ability to tune color rendering index (CRI) under a given color temperature (CT) while transmitting data. As shown in Fig. 4(a), according to the SPD of the quadrichromatic LED, chromatic coordinate matrix of the QLED in CIE xyY color space can be obtained by a simple calculation :Table 1, chromaticity coordinates of each constellation in CIE xyY color space is given in Table 2. According to the additive color mixture rule, the SPD of a whole data packet is the time integral of the nine symbols, which is shown in Fig. 4(b). Assuming denotes the percentage of luminous flux vector of quadrichromatic LED for a CRM symbol and denotes the luminous flux of each CRM symbol contained in a data packet; luminous flux vector for a data packet can be expressed in the following vector form:Eq. (6),,,and are the undetermined parameters to implement illumination requirements. The constraint conditions of illumination can be described as:Eq. (6) is actually describing the following problem: there is a given quantity light of four colors, determining the luminous flux of the four primary color symbol to meet the CT and CRI constraints. Considering this is a quadrichromatic LED, the device is able to tune CRI under a specific CT. Since Eq. (7) is not linear, we are not able to give the analytic solutions; its numerical solution can be obtained by gradient descent or Newton method with an iterative strategy. In addition, the computational process of CRI is very complex . Thus it will be difficult to calculate the luminous flux of primary color symbols in real-time application. However, since the possible combinations of the five CRM symbols in a data packet is limited, once the color temperature and CRI constraint is given, the possible conditions for the luminous fluxes of primary color symbols can be calculated in advance. And a hash table can be built for quick query in practical applications.
2.4 Decoding algorithm
In 2.1, we design 16-CRM constellations in RGB ratio space according to spectral response of the CMOS image sensor. However, operating characteristic of CMOS image sensor is ignored. An operating characteristic of modern CMOS image sensor is that the captured raw RGB data are transformed by means of a complex sequence of operations applied by the sensor software, such as preprocessing, white balance adjustment, demosaicing, and color transformation . All these operations alter the numerical values of the RGB pixel data used in decoding, therefore affecting constellation classification. Since the layout of constellations is optimal under raw RGB data condition, decoding with raw RGB data can get the best bit error rate (BER) performance. In order to obtain raw RGB data, we need RGB data sets in the output image to determine the transformation equation. In frame header, 16-CRM constellations are sent in order to provide color samples. The coefficients of the polynomial transformation equations between the output RGB image and raw RGB are determined by these color samples. We may also regard this as a “color survey” procedure. The detailed decoding algorithm is shown in Fig. 5. The central row of the captured image is extracted first to obtain RGB color vector, which can be expressed as .Due to the “blooming effect” of CMOS image sensor, the image illumination is non-uniform. As non-uniform illumination of image can cause error in “color survey” procedure, Muti-Scale Retinex with Color Restore (MSRCR) algorithm with the filter parameter (0.1,0.1,0.1) and scale parameter (40,100,200) is used for illumination equalization at the beginning of decoding. The parameter values are determined by using an adaptive way in . The three values of the filter parameter and scale parameter are corresponding to the RGB channels in an image. Generally, a larger value of the filter parameter will result in a higher contrast degree of the processed image, and the scale parameter will determine the illumination uniformity of the processed image. Since an “OFF” time slot is added to two adjacent symbols, by checking the sum, the edge of each color bar can be obtained easily. Then the frame header is firstly sampled to obtain the three-order polynomial transformation equation. The “color survey” procedure is as follows: (1) Get the RGB valuesof a color sample. (2) Calculate the sum of RGB values . (3) Calculate the theoretical RGB values by, whereis the RGB ratio vector for the constellation point. (4) Get a training sample to. (5) Return to (1) until 16 training samples are all obtained. (6) Add three extra training samples: (255,0,0) to (255,0,0), (0,255,0) to (0,255,0) and (0,0,255) to (0,0,255). Since there are 19 undetermined coefficients in three order polynomial fitting, the purpose of step (6) is to obtain enough training samples. The transformation equation for a color channel can be expressed as:
3. Experiment and result
We investigate both illumination and communication performance of the proposed system. Figure 6 shows the experimental platform. Smart phone (XiaoMi-6) with the CMOS image sensor (IMX386) is used as a camera to record a video of two minutes. Video is transferred to the PC via USB type-C interface. An integrating sphere is used for measuring illumination parameters at the same time. In the experiment, mobile phone camera is working in manual setting mode with the resolution of 1080*1920 and the frame rate of 60fps. By tuning the frequency of drive signal, color bar can be set in different pixel width in output image when the distance between the smart phone camera and the LED lamp is fixed (2cm). However, during the rolling shutter operation of the CMOS image sensor, the pixel rows is activated without waiting for the adjacent pixel to finish scanning. Hence, there is overlapping time for each two adjacent pixel rows, which means unable to use one pixel to transmit a symbol. We confirm that 5 pixels/symbol is the limited resolution to balance both symbol rate and stability. Considering the “OFF” time slot between color bars, when the width of color bars is 5 pixels, each data frame is consisted of 204 symbols at least, including 16 color samples and 21 data packets. Since a whole data frame will last for three-image-frame time, the date rate is 224 bit/image-frame (28byte/image-frame). In order to simplify the calculation of CRI, we choose 5500K, 6500K and 7500K as the test color temperatures, which are the color temperatures of CIE standard illuminators, D55, D65 and D75 . The CRI constraint is set as >80, which can satisfy general indoor illumination requirements.
Figure 7(a) and (b) shows the illumination performance during the two-minute video time. It is obvious that both color temperature and CRI can roughly maintain at a stable level while the optical signal source is transmitting data. Small fluctuations in color temperature and CRI can be explained by the color samples transmitted in the data frame header.
The BER performance with different exposure time and color temperature is shown in Fig. 8 (a). It is indicated that as the increase of exposure time, the worse BER performance is obtained. The color rolling shutter pattern under 1/3000 s, 1/1500 s and 1/500 s exposure time is shown in the Fig. 8(b). From this figure, it is clear that long exposure time can lead to the severe “overlapping” of color bars. The reason is that, when exposure time is too long, the overflow charge will flow into the neighboring pixels making the “gap” between two color bars vanished. This may bring trouble to clock recovery. On the other hand, since the CMOS image sensor utilize digital registers to record the color intensity of R, G and B channels, when the LED flux luminous is too high, the digital registers will be in the “overflow” state. In this case, the color bar pattern will always be in “white” color, in other words, RGB ratio information is lost. The experimental results in Fig. 8 (a) also demonstrate that the MSRCR algorithm can reduce the bit error rate effectively, especially when the image is overexposured.
In addition, several CMOS image sensor based VLC systems are compared in Table 3. Compared with the schemes in Table 1, the advantage of our system is as follows: (1) the proposed modulation scheme can achieve the highest data rate; (2) while data transmitting, the optics signal is color temperature tunable with CRI constraint, which can meet modern illumination requirements; (3) using CRM to overcome “blooming effect”. On the other hand, the extra cost for our system is only a DAC and a time sequential controller.
In this paper, a quadrichromatic LED is used for optical signal source in CMOS image sensor based VLC to increase data rate and enhance illumination effect at the same time. We first explain the RGB color intensity signal space and RGB color ratio signal space in CSK modulation, and then we design optimal 16-CRM constellations in RGB color ratio signal space to overcome the problem of non-uniform image illumination caused by “blooming effect”. A specific data packet structure is also proposed to satisfy illumination requirement while transmitting data. At the receiver, MSRCR algorithm is used for illumination equalization and 16-CRM color samples are used to obtain three-order polynomial transformation equation between the raw RGB data and RGB data processed by sensor software. Experimental results show that when the CMOS image sensor is operating in the resolution of 1920*1080 and 60fps, our scheme can achieve a data rate of 13.2kbit/s, meanwhile, the optical signal source is color temperature tunable with CRI constraint, which can satisfy modern illumination requirements.
Foundation for Distinguished Young Talents in Higher Education in Guangdong, China (2014KQNCX154); Science and Technology Planning Project of Guangdong Province, China (2016B090918102); Educational Commission of Guangdong Province, China (2013KJCX0182) Technology Development Project of Guangdong Province, China (2017A010101034) Innovation Projects for Science supported by Department of Education of Guangdong Province (2016KTSCX141); Science Foundation for Young Teachers of Wuyi University (2018td01).
References and links
1. S. Rajagopal, R. D. Roberts, and S. K. Lim, “IEEE 802.15.7 visible light communication: modulation schemes and dimming support,” IEEE Commun. Mag. 50(3), 72–82 (2012). [CrossRef]
2. A. Jovicic, J. Li, and T. Richardson, “Visible light communication: opportunities, challenges and the path to market,” IEEE Commun. Mag. 51(12), 26–32 (2013). [CrossRef]
3. Y. Pei, S. Zhu, H. Yang, L. Zhao, X. Yi, J. Wang, and J. Li, “LED Modulation Characteristics in a Visible-Light Communication System,” Opt. Photonics J. 3(2), 139–142 (2017). [CrossRef]
4. J. H. Li, X. X. Huang, X. M. Ji, N. Chi, and J.-Y. Shi, “An integrated PIN-array receiver for visible light communication,” J. Opt. 17(10), 105805 (2015). [CrossRef]
5. S. Ray, M. M. Hella, M. M. Hossain, P. Zarkesh-Ha, and M. M. Hayat, “Speed optimized large area avalanche photodetector in standard CMOS technology for visible light communication,” in Proceeding of IEEE Sensors (IEEE, 2014), pp. 2147– 2150.
6. K. Liang, C. W. Chow, Y. Liu, and C. H. Yeh, “Thresholding schemes for visible light communications with CMOS camera using entropy-based algorithms,” Opt. Express 24(22), 25641–25646 (2016). [CrossRef] [PubMed]
8. M. S. Rahman, M. M. Haque, and K. D. Kim, “Indoor Positioning by LED Visible Light Communication and Image Sensors,” Iran. J. Electr. Comput. Eng. 1(2), 420–431 (2011).
9. T. H. Do and M. Yoo, “Visible light communication based vehicle positioning using LED street light and rolling shutter CMOS sensors,” Opt. Commun. 407, 112–126 (2018). [CrossRef]
10. P. Huynh, T. H. Do, and M. Yoo, “A Probability-Based Algorithm Using Image Sensors to Track the LED in a Vehicle Visible Light Communication System,” Sensors (Basel) 17(2), 347 (2017). [CrossRef] [PubMed]
11. M. G. Moon and S. I. Choi, “Indoor position estimation using image sensor based on VLC,” in proceeding of International Conference on Advanced Technologies for Communications. (IEEE, 2015), pp. 11–14.
12. V. P. Rachim, Y. Jiang, H. S. Lee, and W. Y. Chung, “Demonstration of long-distance hazard-free wearable EEG monitoring system using mobile phone visible light communication,” Opt. Express 25(2), 713–719 (2017). [CrossRef] [PubMed]
13. J. Shi, J. He, J. He, R. Deng, Y. Wei, F. Long, Y. Cheng, and L. Chen, “Multilevel modulation scheme using the overlapping of two light sources for visible light communication with mobile phone camera,” Opt. Express 25(14), 15905–15912 (2017). [CrossRef] [PubMed]
16. X. Liang, M. Yuan, J. Wang, Z. Ding, M. Jiang, and C. M. Zhao, “Constellation Design Enhancement for Color-Shift Keying Modulation of Quadrichromatic LEDs in Visible Light Communications,” J. Lightwave Technol. 35(17), 3650–3663 (2017). [CrossRef]
17. E. Monteiro and S. Hranilovic, “Design and Implementation of Color-Shift Keying for Visible Light Communications,” J. Lightwave Technol. 32(10), 2053–2060 (2014). [CrossRef]
18. C. G. Lee, D. K. Son, E. B. Cho, I. K. Moon, and Y. S. Park, “Development of an Illumination Measurement Device for Color Distribution Based on, a CIE 1931 XYZ Sensor,” J. Opt. Soc. Korea 15(1), 44–51 (2011). [CrossRef]
20. A. Molada-Tebar and J. L. Lerma, “Camera characterization for improving color archaeological documentation,” Color Res. Appl. 43(14), 47-57 (2017).
21. Y. Gao, L. Yun, and J. Shi, “Enhancement MSRCR algorithm of color fog image based on the adaptive scale,” Proc. SPIE 9159(3), 91591B (2014).
22. Y. Ohno, “Color Quality of White LEDs,” Top. Appl. Phys. 126, 349–371 (2013). [CrossRef]