Open Access
Add to BibSonomy
Abstract
In this paper, a high-accuracy motion detection (MD) scheme based on a look-up table (LUT) is proposed and experimentally demonstrated in an optical camera communication (OCC) system. The LUT consists of predefined motions and strings that represent the predefined motions. The predefined motions include straight lines, polylines, circles, and number shapes. At the transmitter, the data with on-off keying (OOK) format is modulated on an 8×8 light-emitting diode (LED) array. The motion is generated by the user’s finger in the free space link. At the receiver, the motion and data are captured by the mobile phone front camera. The captured motion is expressed as a string indicating directions of motion, then it is matched as a predefined motion in LUT by calculating the Levenshtein distance (LD) and modified Jaccard coefficient (MJC). Using the proposed scheme, four types of motions are recognized accurately and data transmission is achieved simultaneously. Also, 1760 motion samples from 4 users are investigated over the free space transmission. The experimental results show that the accuracy of the proposed MD scheme can reach 98% at the distance without the loss of finger centroids. In addition, as the transmitter is not blocked, the bit error rate (BER) is below 1e-6 at the distance of 80cm.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Visible light communication (VLC) has become more and more attractive because it enables both communication and illumination using light-emitting diode (LED) as transmitter [1]. Compared with radio frequency (RF), visible light has massive spectrum with license free. And VLC has the advantages of security and electromagnetic interference immunity [2,3]. Thus, VLC is a promising option for complementing RF communication in future wireless communication [4].
Nowadays, a complementary metal oxide semiconductor (CMOS) camera can be integrated in the smartphone. The indispensability of smart phones has made it popular to use CMOS cameras as VLC receivers, which is called optical camera communication (OCC). OCC systems can be applied for indoor positioning [5], intelligent transportation [6], environmental monitoring [7] and user identification [8]. In OCC systems, the receiver usually has two exposure modes: rolling shutter and global shutter. In rolling-shutter mode, the captured image consists of black and white stripes. To enhance the performance of OCC, different threshold schemes are proposed [9,10], and column matrix selection schemes are presented [11,12]. In global shutter, multiple information can be captured by the camera due to its spatial separation. Thus, in the case of LED array used as the transmitter, the global shutter is suitable for the demodulation with image recognition [13].
Recently, motion detection (MD) is introduced as an additional functionality in VLC systems, and the motions can be detected to control devices in a smart home or office [14]. In [14], the MD is based on intentional obstruction of PDs, but PDs are not flexible for MD. An OCC system with MD is presented, and a quadrant division based MD algorithm is proposed for detection of three motions as line, L-shape and circle [15]. However, it is difficult to distinguish some motions passing through the same quadrant, the number of detectable motion types and the detection accuracy are limited. To improve the accuracy of MD, trained neurons-based MD scheme in OCC is proposed, the processing time is 4s [16]. In [17], to reduce the processing time, the performance of neural network (NN) assisted MD schemes is evaluated, and the results indicate that resilient back propagation training algorithm based NN can identify linear, circular and curvature motion accurately and reduce the processing time effectively. But the use of NN will increase the cost of the system and requires a lot of training for improving the accuracy.
In this paper, a high-accuracy MD scheme based on look-up table (LUT) is proposed and experimentally demonstrated in OCC system. The LUT is established based on the directions of motions, which can be expressed as strings. These motions in LUT are predefined, including four types of motions such as straight line, polyline, circle, and number shape. After obtaining the directions of a captured motion, it will be matched as a predefined motion in LUT. The matching processing is based on Levenshtein Distance (LD) and modified Jaccard Coefficient (MJC). The proposed scheme can detect four types of motions in LUT. 1760 motion samples made by 4 users is used to verify the effectiveness of proposed scheme. Meanwhile, the data with OOK format is transmitted by an 8×8 LED array. These motions and data are captured simultaneously by a mobile phone front camera. In addition, the MD accuracy is evaluated, and the bit error rate (BER) performance is analyzed.
2. Principle
In the OCC system, an 8×8 LED array and the front CMOS camera of mobile phone act as the transmitter and receiver, respectively. The scene is shown is Fig. 1(a). For MD, the red shadow is the coverage of LED array, and the gray shadow is the field-of-view (FOV) of the camera. The finger is facing the screen of the mobile phone. It moves between LED array and the mobile phone to generate motions. Since the light from the LED array is reflected by the screen, it is shown by the red dotted line in Fig. 1(a), it will illuminate the front of the finger to make the outline of the finger noticeable. Meanwhile, the on-off keying (OOK) data is transmitted from the 8×8 LED array for communication. After free space transmission, the motions and data with OOK format are captured by camera in global shutter mode simultaneously.
Fig. 1. OCC system with MD. (a) system model, (b) division of communication and identification area.
Due to the spatial separation of CMOS cameras in global shutter mode, the LEDs and the finger can be separated for communication and MD, respectively. It is worth mentioning that as the finger covers the LED array to generate motions, some data transmitted from the LED array will be lost, and it results in the BER increasing. To reduce the block of LED array, a boundary line is set on mobile phone screen as shown in Fig. 1(b). The first 560 columns of pixels are used for communication, and the remaining 720 columns of pixels are used for MD. If the finger crosses the boundary in an image, the LED array will retransmit the data of the image.
2.1 The proposed MD scheme based on LUT
For the proposed MD scheme based on LUT, the motion can be identified by determining the direction of finger movement. The position of finger in each frame is represented by centroid coordinates of finger [15]. The directions of a motion can be obtained by a series of centroid coordinates. As shown in Fig. 2(a), the direction in motion plane is divided into 8 parts, which correspond to 8 directions (denoted as letters A-H). The orange dotted lines are boundary of 8 directions and orange numbers is the slope of boundaries. In addition, the LUT consists of the predefined motions and the corresponding strings. The maximum length of a string in LUT is 8. For a circle or number shape motion, there are some differences due to the motions of different users, thus, more than one string corresponds to it.
Fig. 2. (a) Defined 8 direction of motion, (b) four types of motion: i) straight line, ii) polyline, iii) circle, and iv) number shape.
In the paper, four types of motion can be detected using the proposed MD scheme based on LUT, it is depicted in Fig. 2(b). Since number ‘1’ and ‘0’ are repeated with straight line and circle, respectively, the number shape motion including number ‘2’ to ‘9’. These motions can be applied to control devices in smart home or office [14]. The number shape motions can be used to select different controlled devices, while the other three motions can control the behaviors of the selected device, such as turning ‘ON’ and ‘OFF’ or turning ‘UP’ and ‘DOWN’.
The flowchart of the proposed MD scheme based on LUT is illustrated in Fig. 3. It includes Step I as motion direction extraction and Step II as string matching in LUT.
At Step I, the finger centroid coordinates can be calculated after finger motion is detected. Then, the motion direction of each centroid can be determined by calculating the slope between this centroid coordinate and subsequent centroid coordinate. Assume that the centroid coordinates of the ith and (i+1)th frame are (${x_i}$, ${y_i}$) and (${x_{i + 1}}$, ${y_{i + 1}}$) in image coordinate system, respectively. Then the slope between the coordinates of the two centroid points is expressed as
where n is the total of centroids in a motion, and ${k_i}$ indicates motion direction of finger in the ith frame. However, the time interval between two consecutive frames is too short to reflect the trend of movement accurately. Thus, in this paper, the direction of finger in the ith frame is indicated by The value of q is related to the frame rate of camera. For estimating the motion direction accurately, the value of q should be proportional to the frame rate of camera. It is set to 3 in the paper.According to the slope range of 8 directions in the Fig. 2(a) and the value of ${y_{i + q}} - {y_i}$ or ${x_{i + q}} - {x_i}$, the direction of movement in the current frame can be determined. For instance, the ${k_i}$ is 2.000 and ${x_{i + q}} - {x_i} > 0$, thus, the motion direction of finger in ith frame is ‘B’. Then, a threshold σ is set to extract valid direction. If there are more than σ consecutive centroids moving in the same direction, the detected direction is valid, otherwise it is invalid and be removed as noise. Note that, according to movement speed and capturing frame rate, the value of σ needs to be adjusted. The value of σ is inversely proportional to the movement speed and proportional to the capturing frame rate. As a result, the directions of a motion can be detected from a series of centroid coordinates, and a string (${S_d}$) representing this motion.
For Step II, the purpose of string matching is to find the most similar string to ${S_d}$ in the LUT, the predefined motion corresponding to the string is considered to be detection result. The matching processing is conducted by calculating LDs and MJCs between ${S_d}$ and ${S_j}, j = 1,2, \ldots,m$, where m is the number of strings in LUT. LD indicates the minimum number of operations (as insertion, deletion or substitution) required to transform one string into another, and it is allowed that the length of two strings is different [18]. Therefore, LD is appropriate to measure the similarity of ${S_d}$ and ${S_j}$, and the smaller the LD, the higher the similarity. Jaccard Coefficient (JC) is used to measure the similarity of two sets without the same elements [19]. However, in this paper, a direction may appear more than once in a string representing motion, thus, we modified JC as MJC to measure the similarity of two multi-sets. Assume that there are two multi-sets ${\xi _a}$ and ${\xi _b}$, the MJC between them is defined as
where numerator and denominator represent the number of elements in the intersection and union of two multi-sets, respectively. In this paper, the string ${S_d}$ and a string in LUT can be also treated as two multi-sets, then $MJ{C_{Sd, Sj}},({j = 1,2, \ldots ,m} )$ is utilized to further measure the similarity of detected motion and a predefined motion in LUT, and the larger the value of $MJ{C_{Sd, Sj}}$, the higher the similarity.At Step II, $L{D_j}$ between ${S_d}$ and ${S_j}, ({j = 1,2, \ldots , m})$ is calculated firstly. Then the minimum ($L{D_{min}}$) and the number of it (${N_{LDmin}}$) is counted in $L{D_j}, ({j = 1,2, \ldots ,\; m} )$. As $L{D_{min}}$ exceeds a certain value, the detected motion is invalid. In this paper, the threshold is set to half of the maximum length of strings in the LUT. As ${N_{LDmin}}$ is 1, the predefined motion corresponding to ${N_{LDmin}}$ in LUT is detection result. Moreover, as ${N_{LDmin}}$ is greater than 1, the MJCs between these strings with $L{D_{min}}$ is further compared. $MJ{C_k}$ is calculated between ${S_d}$ and ${S_k},\; k = 1,2, \ldots ,\; {N_{LDmin}}$, where ${S_k}$ is the strings with $L{D_{min}}$ in LUT. Subsequently, the predefined motion with maximum MJC as $MJ{C_{max}}$ is the detection result.
2.2 Data transmission
For communication, the OOK data is transmitted by 8×8 LED array and is received by the front CMOS camera of mobile phone. The flicker frequency of LED array is 20Hz, and the frame rate of camera is 30fps, then 10 frames per second is redundant in the receiver. As shown in Fig. 4, it is the period of the LEDs and the camera. The duration of two packets transmitted by the LED array is equal to that of three frames captured by the camera. The first and the third frame capture the data of the former and the latter packet, respectively. But the middle frame captures the data of two packet interfering with each other, thus it will be discard at the receiver.
At the transmitter, for synchronization and positioning of the 8×8 LED array, a keyframe is inserted into the header of data packet, and 8×8 LED array in the keyframe are all turned on. Theoretically, the keyframe is only inserted at the beginning of the data one time, the influence on data rate caused by keyframe is negligible. In this paper, for BER analysis, the keyframe is inserted into the beginning of the data in a period of 1s. Regardless of data retransmission caused by blocking, the data rate can be given by
where ${N_D}$ and ${F_L}$ denote the length of transmitted data per frame and the flickering frequency of LEDs, respectively. T is the period of keyframe insertion. In the paper, ${R_d}$ is 1216bits/s (i.e. 64×20–64/1), and ${R_d}$ is closer to 1280bps as the increase of T.At the receiver, for demodulation, the sampling interval is obtained after positioning LED area. Then the coordinates of sampling points can be determined. A threshold is set for recovering data. In this paper, the threshold is set to 220, and it can be adjusted in different illumination intensity. If the grayscale value of pixels corresponding to the sampling points is greater than the threshold, then it is judged as ‘1’, otherwise ‘0’.
3. Experimental setup
Figure 5 shows the experiment setup of the OCC system for MD and communication. At the transmitter, the OOK data is generated from personal computer (PC). Then the OOK data is superimposed on LED array in the order of addresses of Arduino Uno Board. The LED driver is HT16K33 with voltage of 5V. The transmission link is free space, in which the finger of user moves. At the receiver, a mobile phone (Honor 10) with front camera of a resolution 1280×720 pixels is used to record the transmitted data and the motions in the form of video. The motion duration is about 1.5s∼4.5s, thus, for a camera with a frame rate of 30fps, about 45∼135 centroid coordinates are calculated for MD. After the captured video is converted to a series of frames, the image is separated two parts for motion detection and data demodulation.
To verify the proposed MD scheme, the experiments are performed by 4 users. When the distance between the transmitter and the receiver is 35cm and ambient light intensity is 0 lux, 4 users made motion with their fingers respectively, and about 20 times per motion (total is 4×20×22 = 1760). The detection accuracy of four kinds of motions is calculated for different users. During the movement, the finger of user is kept within about 10cm above the mobile phone screen, so that the light reflected by the screen is sufficient to illuminate the finger. The detection of fingers is closely related to the light reflected from the phone screen, and the light becomes weaker as the distance increases. Therefore, at the transmission distance from 20cm to 100cm, the detection accuracy is investigated and total motions is 100 (about 5 times per motion) at each distance. Meanwhile, the BER is measured in the case of without block of the LED array. And the duration time of communication is 3 minutes at each distance. The transmitted data is 218.88kbits (as 1216bits/s×180s=218.88kbits).
4. Experimental results and discussion
The experimental results based on the proposed MD scheme are shown in Fig. 6. The detection accuracy of four types of motion performed by 4 users is indicated in Fig. 6(a). For 4 users, the accuracy of straight motion reach 100%, and the lowest accuracy is 95%, which corresponds to the circular motion in user 2 and user 4. The average accuracy of four types of motions for 4 users is shown in Fig. 6(b). It is clear that the straight motion has the highest accuracy, it is 100%. This is because the straight motion with no direction change is the simplest among four types of motions. As the complexity of the motion increases, the accuracy decreases. Therefore, the circle and number shape motions are lower than that of straight and polyline motions. Furthermore, the overall detection accuracy of a total of 1760 motions is 98% in this experiment.
Fig. 6. Accuracy of MD for 4 users. (a) Comparison of four types of motions, (b) Average accuracy of four types of motions for 4 users, and overall accuracy is 98%.
Figure 7 shows part of motion trajectory detected correctly. The white points represent position of finger in frames. Due to the mirror effect of the front camera of mobile phone, the recorded trajectory is opposite horizontally to the actual motion. Thus, these trajectories have been horizontally flipped for easy viewing. During movement of user’s finger, there are two main interference factor. Firstly, if there is noise interference, the estimation of centroid point will be biased, it is resulting in the actual motion trajectory to deviate from the detected trajectory, as the red rectangle in the Fig. 7. Secondly, as shown by the blue rectangle in Fig. 7, due to the uniqueness of each finger motion, there are some unavoidable differences between the motions created by users and the predefined motions. The influence of the two factors can be effectively mitigated by matching in LUT, which demonstrated the robustness of the proposed MD scheme.
Fig. 7. MD results: straight, polyline, circle and number shape. The white dots are the detected centroids of finger. The gray arrows indicate the starting direction of motions. Note, due to the mirror effect of front camera of mobile phone, these picture have been horizontally flipped for easy viewing.
To investigate the influence of transmission distance on system performance, the MD accuracy, the loss rate of finger centroids and the BER performance are measured at the distance from 20 to 100cm, and the experimental results are illustrated in Fig. 8. For MD, the accuracy and the loss rate of centroids are almost 100% and 0% within 50cm distance, respectively. It demonstrated that the impact of transmission distance on detection accuracy is negligible when all finger centroids can be detected. With the increasing of transmission distance, the light intensity becomes weaker, thus, more centroids are lost. In this way, the accuracy decreases due to the loss of centroids. As the transmission distance is over 90cm, all centroids will be lost, the accuracy of MD is 0. In addition, for communication without finger blocking, the BER is below 1e-6 within 80cm. At the distance of 90cm and 100cm, the BER is 3.47e-5 and 3.73e-4, respectively. It is due to the spacing among LEDs become smaller as the increasing of transmission distance, the influence of blooming effect of camera is greater.
As shown in Fig. 9, it is grayscale histogram of LED area in image at different transmission distances. When the transmission distance is equal to or less than 80cm, the histograms have two clear peaks. Thus, the data can be recovered correctly. However, as the transmission distance increases, the left peak gradually disappears and the histograms are more and more concentrated on the right peak, which indicates the image has lower contrast caused by blooming effect. Subsequently, some data ‘0’ is misjudged as ‘1’, and the probability of misjudgment increases with transmission distances. Although the transmission distance can be longer, the MD is limited. It can be solved by increasing the power of the transmitter.
Furthermore, Table 1 shows the comparisons between the propose MD scheme and some related works [14,15,17]. In [14], 9 PDs are used as the receiver, it can achieve higher date rate, but the types of detected motion are straight lines and circles. The schemes using the camera can detect more motions [15,17]. Compared with the MD scheme based on quadrant division [15] and the NN assisted MD scheme [17] detecting 3 types of motions, in the paper, the proposed MD scheme based on LUT can detect four types of motion such as straight line, polyline, circular motion and number shape motion. For the accuracy of MD, the proposed scheme can reach 98% at the distance without loss of finger centroids, and a total of 1760 motion samples are from 4 users, which proves the stability of the proposed scheme.
Table 1. Comparison of several related works.
5. Conclusion
In the paper, a high-accuracy MD scheme based on LUT is proposed and experimentally demonstrated in OCC system. The proposed scheme can detect four types of motions, including straight line, polyline, circle, and number shape. Motions made by 4 users are sampled for measuring the accuracy of MD. Meanwhile, the OOK data is transmitted by an 8×8 LED array. The motions and data are captured simultaneously by a mobile phone front camera. The experimental results show that the accuracy can reach 98% at the distance without loss of finger centroids using the proposed MD scheme. In addition, it can reduce the influences from the noise and users effectively. Moreover, the transmission distance has little effect on the accuracy of MD when there is no centroid loss. Without finger blocking, the BER performance is below 1e-6 at the transmission distance of 80cm.
Funding
National Natural Science Foundation of China (61775054); Hunan Provincial Science and Technology Department (2016GK2011).
Disclosures
The authors declare no conflicts of interest.
References
1. P. H. Pathak, X. Feng, P. Hu, and P. Mohapatra, “Visible light communication, networking, and sensing: A survey, potential and challenges,” IEEE Commun. Surv. and Tutorials 17(4), 2047–2077 (2015). [CrossRef]
2. S. R. Teli, S. Zvanovec, and Z. Ghassemlooy, “Optical internet of things within 5G: Applications and challenges,” in 2018 IEEE International Conference on Internet of Things and Intelligence System (IOTAIS), pp. 40–45 (2018).
3. S. Zvanovec, P. Chvojka, P. A. Haigh, and Z. Ghassemlooy, “Visible light communications towards 5G,” Radioengineering 24(1), 1–9 (2015). [CrossRef]
4. J. He, Z. Li, J. He, and J. Shi, “Visible Laser Light Communication based on LDPC-Coded Multiband CAP and Adaptive Modulation,” J. Lightwave Technol. 37(4), 1207–1213 (2019). [CrossRef]
5. Y. Li, Z. Ghassemlooy, X. Tang, B. Lin, and Y. Zhang, “A VLC smartphone camera based indoor positioning system,” IEEE Photonics Technol. Lett. 30(13), 1171–1174 (2018). [CrossRef]
6. Y. Goto, I. Takai, T. Yamazato, H. Okada, T. Fujii, S. Kawahito, S. Arai, T. Yendo, and K. Kamakura, “A new automotive VLC system using optical communication image sensor,” IEEE Photonics J. 8(3), 1–17 (2016). [CrossRef]
7. Z. Ong, V. P. Rachim, and W.-Y. Chung, “Novel electromagnetic-interference-free indoor environment monitoring system by mobile camera-image-sensor-based VLC,” IEEE Photonics J. 9(5), 1–11 (2017). [CrossRef]
8. W. Ding, C. Wu, Y. Zi, H. Zou, J. Wang, J. Cheng, A. C. Wang, and Z. L. Wang, “Self-powered wireless optical transmission of mechanical agitation signals,” Nano Energy 47, 566–572 (2018). [CrossRef]
9. K. Liang, C.-W. Chow, Y. Liu, and C.-H. Yeh, “Thresholding schemes for visible light communications with CMOS camera using entropy-based algorithms,” Opt. Express 24(22), 25641–25646 (2016). [CrossRef]
10. J. He, Y. Zhou, R. Deng, J. Shi, J. He, Z. Jiang, and Q. Tang, “Efficient Sampling Scheme Based on Length Estimation for Optical Camera Communication,” IEEE Photonics Technol. Lett. 31(11), 841–844 (2019). [CrossRef]
11. J. Shi, J. He, J. He, Z. Jiang, Y. Zhou, and Y. Xiao, “Enabling user mobility for optical camera communication using mobile phone,” Opt. Express 26(17), 21762–21767 (2018). [CrossRef]
12. J. He, Z. Jiang, J. Shi, Y. Zhou, and J. He, “A novel column matrix selection scheme for VLC system with mobile phone camera,” IEEE Photonics Technol. Lett. 31(2), 149–152 (2019). [CrossRef]
13. W. A. Cahyadi, Y. H. Kim, Y. H. Chung, and C.-J. Ahn, “Mobile phone camera-based indoor visible light communications with rotation compensation,” IEEE Photonics J. 8(2), 1–8 (2016). [CrossRef]
14. A. Sewaiwar, S. V. Tiwari, and Y.-H. Chung, “Visible light communication based motion detection,” Opt. Express 23(14), 18769–18776 (2015). [CrossRef]
15. S. Teli, W. A. Cahyadi, and Y. H. Chung, “Optical camera communication: Motion over camera,” IEEE Commun. Mag. 55(8), 156–162 (2017). [CrossRef]
16. S. R. Teli, W. A. Cahyadi, and Y. H. Chung, “Trained neurons-based motion detection in optical camera communications,” Opt. Eng. 57(04), 1–4 (2018). [CrossRef]
17. S. R. Teli, S. Zvanovec, and Z. Ghassemlooy, “Performance evaluation of neural network assisted motion detection schemes implemented within indoor optical camera based communication,” Opt. Express 27(17), 24082–24092 (2019). [CrossRef]
18. J. Beernaerts, E. Debever, M. Lenoir, B. D. Baets, and N. V. Weghe, “A method based on the Levenshtein distance metric for the comparison of multiple movement patterns described by matrix sequences of different length,” Expert Systems With Applications 115, 373–385 (2019). [CrossRef]
19. H. Wang, Y. Du, J. Yi, Y. Sun, and F. Liang, “A new method for measuring topological structure similarity between complex trajectories,” IEEE Trans. Knowl. Data Eng. 31(10), 1836–1848 (2019). [CrossRef]
