## Abstract

We study the performance of various binary and nonbinary modulation methods applied to coherent laser communication through the turbulent atmosphere. We compare the spectral efficiencies and SNR efficiencies of complex modulations, and consider options for atmospheric compensation, including phase correction and diversity combining techniques. Our analysis shows that high communication rates require receivers with good sensitivity along with some technique to mitigate the effect of atmospheric fading.

© 2010 OSA

## 1. Introduction

Evaluating the performance of a heterodyne or homodyne receiver in the presence of atmospheric turbulence is generally difficult because of the complex ways turbulence affects the coherence of the received signal that is to be mixed with the local oscillator. The downconverted heterodyne or homodyne power is maximized when the spatial field of the received signal matches that of the local oscillator. Any mismatch of the amplitudes and phases of the two fields will result in a loss in downconverted power. Phase-compensated receivers offer the potential for overcoming atmospheric limitations by adaptive tracking of the beam wave-front and correction of atmospherically induced aberrations. Likewise, diversity receivers, which combine several replicas of the transmitted message, each corrupted independently by the atmosphere, can improve communication reliability because of the low probability of the simultaneous occurrence of deep fades in all the diversity channels. Here, we study in a unified framework the effects of wave-front distortion on the performance of synchronous (coherent) receivers utilizing phase compensation and diversity-combining techniques. Dynamic control of the phase-compensated receivers and electrical phase locking in diversity receivers are challenging techniques. The management of these techniques turns out to be the main limitation for the practical implementation of the advanced coherent receivers analyzed in this study.

The throughput of a transmission system can be increased by increasing the symbol rate, thus increasing the bandwidth utilized, or by increasing spectral efficiency. In many applications, bandwidth is constrained, and it is desirable to maximize spectral efficiency. Binary modulation encodes one bit per symbol, while nonbinary modulation encodes more than one bit per symbol, leading to higher spectral efficiency. In this study, we compare the spectral efficiencies and power efficiencies of several modulation formats using coherent detection in the presence of multiplicative noise (fading) from atmospheric turbulence and additive white Gaussian noise (AWGN). We assume that at the receiver, the dominant noise source is shot noise from the local oscillator laser, which can be modeled accurately as AWGN that is statistically independent of the turbulence fading.

We consider several *M*-ary modulation techniques with coherent detection, including phase-shift keying (PSK), quadrature-amplitude modulation (QAM), and pulse-amplitude modulation (PAM). The parameter *M* is the number of points in the signal constellation and, consequently, log_{2}
*M* describes the number of coded bits per symbol. We also consider binary differential phase-shift keying (2-DPSK or DBPSK) with differentially coherent detection. As we are primarily interested in the high-spectral-efficiency regime, we do not consider orthogonal modulation formats, such as pulse-position modulation (PPM) or frequency-shift keying (FSK).

The remainder of this paper is organized as follows. In Section 2, we review a mathematical model for a coherently received signal after propagation through the atmosphere. By noting that the downconverted signal current can be characterized as the sum of many contributions from different coherent regions within the aperture, we have shown that the probability density function (PDF) of this current can be well-approximated by a generalized Rice distribution. In our model, the parameters describing the PDF depend on the turbulence conditions and the degree of modal compensation applied at the receiver. We present the average BER probability for the various modulation formats considered in this analysis. In Section 3, we compare the spectral efficiencies and power efficiencies of various binary and nonbinary modulation formats. In Section 4, we provide concluding remarks.

## 2. Statistical model of optical heterodyne detection

When a signal experiences atmospheric turbulence during transmission, both its received envelope and its phase fluctuate over time. In the case of coherent detection, phase fluctuations can severely degrade performance unless measures are taken to compensate for them at the receiver. Here, we assume that after homodyne or heterodyne downconversion is used to obtain an electrical signal, the receiver is able to track any temporal phase fluctuations caused by turbulence (as well as those caused by laser phase noise), performing ideal coherent (synchronous) demodulation. Under this assumption, analyzing the receiver performance requires knowledge of only the envelope statistics of the downconverted electrical signal.

For a AWGN channel, with average power constraint *P* and noise power spectral density *N*
_{0}/2, *γ*
_{0}
*=P/N*
_{0}
*B* is the SNR per symbol per unit bandwidth *B*. The SNR per symbol *γ*
_{0} can be interpreted as the detected number of photons (photocounts) per symbol when 1/*B* is the symbol period. Coherently detected signals are modeled as narrowband RF signals with additive white Gaussian noise. In free-space optical communication through the turbulent atmosphere, we must consider fading channels, which are a class of channels with multiplicative noise. In the fading AWGN channel, we let *α*
^{2} denote the atmospheric channel power fading and (*P/N*
_{0}
*B*)*α*
^{2}=*γ*
_{0}
*α*
^{2} denote the instantaneous received SNR per symbol. Now, the SNR can be taken as the number of signal photons detected on the receiver aperture *γ*
_{0} multiplied by a heterodyne mixing efficiency *α*
^{2}. When, due to the impact of atmospheric turbulence-induced phase and amplitude fluctuations, the spatial field of the received signal does not properly matches that of the local oscillator, the contributions to the current signal from different parts of the receiver aperture can interfere destructively and result in a reduced instantaneous mixing efficiency, causing fading.

The statistical properties of the atmospheric random channel fading *α*
^{2} are related to the characteristics of atmospheric amplitude and phase fluctuations. Atmospheric log-amplitude fluctuations (scintillation) and phase variations (aberrations), can be characterized by their respective statistical variances, *σ _{χ}*

^{2}and

*σ*

_{f}^{2},

*σ*

_{β}^{2}is often referred to as the scintillation index [1]. In Eq. (1), the statistics of phase aberrations caused by atmospheric turbulence are assumed to be characterized by a Kolmogorov spectrum of turbulence [2]. In the analysis of [2], classical results for the phase variance

*σ*

_{f}^{2}were extended to consider modal compensation of atmospheric phase distortion. In such modal compensation, Zernike polynomials are widely used as basis functions because of their simple analytical expressions and their correspondence to classical aberrations [3]. The coefficient

*C*depends on

_{J}*J*[2]. A coefficient 1.0299 in the phase variance

*σ*

_{f}^{2}assumes that no terms are corrected by a receiver employing active modal compensation. For example, aberrations up to tilt, astigmatism, coma and fifth-order correspond to

*J*= 3, 6, 10 and 20, respectively. Ideally, it is desirable to choose

*J*large enough that the residual variance

*σ*

_{f}^{2}becomes negligible. In Eq. (1), the receiver aperture diameter

*D*is normalized by the wavefront coherence diameter

*r*

_{0}, which describes the spatial correlation of phase fluctuations in the receiver plane [4].

We have studied the effect of various parameters, including turbulence level, signal strength, receive aperture size, and the extent of wavefront compensation, in the statistical properties of the atmospheric random channel fading *α*
^{2} [5, 6]. We have separately quantified the effects of amplitude fluctuations and phase distortion, and have identified their impact on the maximal rate at which the information may be transferred. In most situations considered, amplitude fluctuations effects become negligible, and phase distortion become the dominant effect. Without excessive loss of generality, amplitude fluctuations can be neglected by assuming *σ _{β}*

^{2}=0 in all results presented in the following paragraphs.

Conditional on a realization of the atmospheric channel described by α, the atmospheric system can be modeled as an AWGN channel with instantaneous received SNR per symbol *γ=γ*
_{0}
*α*
^{2}. The statistical properties of the atmospheric random channel fading *α*
^{2} provide a statistical description of the SNR *γ*. We have previously modeled the impact of atmospheric turbulence-induced phase and amplitude fluctuations on free-space optical links using synchronous detection and found that the SNR γ at the output of a perfect *L*-element diversity coherent combiner in the atmosphere would be described by a noncentral chi-square distribution with 2*L* degrees of freedom [5, 6]:

*I*

_{L}_{−1}(∙) is the (

*L*–1)-order modified Bessel function of the first kind. The average SNR (or average detected photocounts) per symbol $\overline{\gamma}$ and the parameter 1

*/r*describe turbulence effects. Both $\overline{\gamma}$ and 1

*/r*are described in terms of the amplitude and phase variances

*σ*

_{χ}^{2}and

*σ*

_{f}^{2}in Eq. (1) [5]. The model leading to the PDF in Eq. (2) is based on the observation that the downconverted signal current can be characterized as the sum of many contributions from

*N*different coherent regions within one aperture. In this model, the signal is characterized as the sum of a constant (coherent) term and a random (incoherent) residual halo. The contrast parameter 1

*/r*is a measure of the strength of the residual halo relative to the coherent component. The parameter

*r*ranges between 0 and ∞. It can be shown that when the constant term is very weak (

*r*→0), turbulence fading causes the SNR to become gamma-distributed, just as in a speckle pattern [7]. Likewise, when the dominant term is very strong (

*r*→∞), the density function becomes highly peaked around the mean value $\overline{\gamma}$, and there is no fading to be considered.

The model behind Eq. (2) can also be applied to diversity schemes based on maximal-ratio combining (MRC) of received signals. MRC schemes assume perfect knowledge of the branch amplitudes and phases, require independent processing of each branch and require that the individual signals from each branch be weighted by their signal amplitude-to-noise variance ratios, then summed coherently [8]. Assuming *L* independent branch signals and equal average branch SNR per symbol $\overline{\gamma}$, a receiver using MRC will weight the *L* diversity branches by the complex conjugates of their respective fading gains and sum them. By setting *L* = 1, the PDF in Eq. (2) describes the SNR *γ* for a single receiving branch and corresponds to a noncentral chi-square probability with two degrees of freedom [1].

The moment generating function (MGF) is a useful tool for analyzing the average error probability in systems with fading, either with or without diversity [1]. The PDF in Eq. (2) can equivalently be expressed in terms of the MGF associated with SNR per symbol *γ*, which it is defined by as the expected value of$\mathrm{exp}\left(s\text{\hspace{0.17em}}\gamma \right)$. After some algebra, we obtain the MGF

*p*(

_{γ}*γ*) with the argument

*γ*reversed in sign.

## 3. Spectral and power efficiencies of coherent modulation formats

Before discussing the specific details of the performance evaluation in the atmospheric fading channel, we consider the error probability for ideal coherent detection on the AWGN channel. The symbol-error ratio (SER) for uncoded coherent modulation conditioned on the number of photons per symbol (SNR) *γ* may be approximated by [9]:

*a*and

*b*depend on the modulation type. In this approach,

*a*describes the number of nearest neighbors to a constellation point at the minimum distance, and

*b*is a constant that relates minimum distance to average symbol energy. In Table 1 , we summarize the specific values of

*a*and

*b*for the

*M*-ary coherent modulations PSK, QAM, and PAM considered in this analysis. A slightly different formulation should be used with differential PSK [9].

We can apply these AWGN error probability results to determine the performance of these same communication systems over the atmospheric fading channel. When fading is present, the received SNR *γ* varies randomly and, consequently, the SER *p _{s}*(

*E|γ*) conditioned on the SNR

*γ*is also random. The performance metrics depends on the rate of change of the fading and on the average SER. The unconditional SER

*p*(

_{s}*E*) of an ideal coherent receiver in the presence of fading must be obtained by averaging the AWGN conditional SER

*p*(

_{s}*E|γ*) in Eq. (4)

*p*(

_{γ}*γ*) is the PDF of the instantaneous fading SNR in Eq. (2). Our goal is to evaluate the various coherent modulation schemes described earlier for our atmospheric fading channel. An MGF-based approach is quite useful in simplifying the analysis [10]. By substituting Eq. (4) into Eq. (5), we obtain

*M*(

*s*) is defined by Eq. (3). Although this result cannot be put in a closed form, we are able to carry out the integration in Eq. (6) using a simple Gaussian-Legendre quadrature formula, which yields high accuracy. The MGF approach can also be applied to

*M*-ary differential PSK through a somewhat more complicated formulation [9, 11]. Note that performance specifications are generally more concerned with the bit-error ratio (BER)

*p*(

_{s}*E*)/log

_{2}

*M*as a function of the photons-per-bit (SNR per bit) γ/log

_{2}

*M*. Here, for

*M*-ary signaling, we make the typical assumption that the symbol energy is divided equally among all bits, and that Gray encoding is used so that, at reasonable SNRs, one symbol error corresponds to exactly one bit error.

It is interesting to compare BER of the different modulation schemes in AWGN and fading. For example, in Fig. 1
we plot the error probability of binary PSK in AWGN and inatmospheric fading. For *M*=2, PAM and PSK are identical modulation formats. As it should be expected, in AWGN the BER decreases exponentially with increasing photons per bit. However, under atmospheric fading, when no compensation techniques are considered, the BER decreases more gradually with increasing average photons per bit. We see that it requires approximately 4 photons per bit (6 dB SNR) to maintain a 10^{−3} bit error rate in AWGN while it requires more than 1000 photons per bit (larger than 30 dB SNR) to maintain the same error rate in fading when no atmospheric compensation techniques are considered. It is clear from these plots that to maintain good receiver sensitivity requires some technique to mitigate the effect of atmospheric fading. In these plots, turbulence is characterized by a moderate phase coherence length *r*
_{0} such that *D*/*r*
_{0}=4. When phase correction is applied in Fig. 1(a), the compensation of just 6 modes (phase aberrations are compensated up to astigmatism) brings the number of photons required to maintain the 10^{−3} BER down to 9 (9.5 dB SNR). When we consider MRC diversity combining of the received signal, using 8 independent branches (Fig. 1(b)) also reduces the power to acceptable levels (about 12 photons per bit, less than 11 dB SNR).

Likewise, in Fig. 2
we plot error probabilities and compare modulation formats PSK, QAM, PAM, and DPSK performances using the MGF-based approach described by Eq. (6). Coherent M-PSK has been analyzed for different constellation sizes (*M*=2, 4, 8, 16; note that theoretically 2- and 4-PSK have similar BER performance). For modulations QAM PAM, and DPSK, just the high-order 16-ary formats have been considered in the figure. Also, note that 4-QAM is equivalent to 4-PSK. In all modulations analyzed, we have assumed that either turbulence-induced phase aberrations have been compensated up to *J*=20 (5th-order aberrations) (Fig. 2(a)), or a 12-branch MRC combining has been used (Fig. 2b). The plots illustrate the photon efficiency for the various binary and nonbinary modulation formats using coherent detection.

If the channel bandwidth in Hz is *B*, and the number of points in the signal constellation is *M*, the spectral efficiency is defined as

*R*is the bit rate in bits/s,

_{b}*R*is the symbol rate in symbols/s, and unitless parameter

_{s}*R*≤1 is the rate of an error-correction encoder that is used to add redundancy to the signal in order to improve the photon efficiency. The uncoded modulations considered in this analysis correspond to

_{c}*R*=1. In all situations, prevention of intersymbol interference requires

_{c}*R*≤

_{s}*B*. Without a loss of generality, we assume the ideal case where

*R*=

_{s}*B*(1/

*B*is the symbol period), and use the number of coded bits per symbol log

_{2}

*M*as the figure of merit for spectral efficiency. Figure 3 compares spectral efficiency log

_{2}

*M*and photon per bit requirements for 10

^{−9}BER. It clearly illustrates the tradeoff between spectral efficiency and photon efficiency of various binary and nonbinary modulation formats applied to coherent laser communication through the turbulent atmosphere. A similar spectral efficiency analysis was done for fiber communications in DWDM transmission systems [12].

Based on the plots, at spectral efficiencies *S*=log_{2}
*M* below 1 bit/s/Hz, 2-PAM and 2-DPSK are attractive techniques. They are simple to implement and lead to the best photon efficiencies in terms of photons per bit required for 10^{−9} BER. Between 1 and 2 bits/s/Hz, 4-DPSK and 4-PSK are perhaps the most interesting modulation formats. At spectral efficiencies above 2 bits/s/Hz, 8-PSK and 16-QAM becomes the most appealing modulations. In general, the performance of 8-QAM and 8-PSK are very similar because the mean energy of the constellation is just slightly different in both modulations. The complicating factor is that the 8-QAM points are no longer all the same amplitude and so the demodulator must now correctly detect both phase and amplitude, rather than just phase, and consequently 8-PSK is a better choice. For data-rates beyond those of 8-PSK and 8-QAM, it is better to consider QAM since it attains a greater distance between contiguous points in the I-Q plane by distributing the points more squarely. In Fig. 3(a), the 150 photon-per-bit required for 16-QAM with phase compensating up to astigmatism (*J*=6) is better than the near 400-photons-per-bit requirement for the equivalent 16-PSK, and is even better than the near 200-photons-per-bit required for 16-PSK with a higher phase compensation (*J*=10). With *J*=6, 16-QAM offers a benefit of more than 4 dB over 16-PSK.

Figure 4
discuss the application of coherent modulations on hypothetical free-space optical communication links and incorporates many of the concepts and results discussed earlier in the paper. In Fig. 4, we present a summary of binary PSK sensitivity at 10^{−9} BER as a function of data-rate for various turbulence conditions and several degrees of atmospheric compensation. Heterodyne detection of BPSK provides among the best theoretical receiving sensitivity, it is spectrally efficient, and it is easily demodulated by using a balanced receiver. In Fig. 4, lines of constant power, representing notional laser communications link budgets, have been estimated for a 1550-nm working wavelength.

Throughout this study, we discuss methods of optimizing receiver sensitivity through use of atmospheric compensation techniques, including phase correction and diversity combining. As it can be seen in Fig. 4, high rates require both receivers with good sensitivity and some technique to mitigate the effect of atmospheric fading. For example, under turbulence conditions such that *D*/*r*
_{0}=2, and for a nominal –50 dBm power budget, a high-rate 1-Gbit/s link with ~3 dB margin can be achieved using an 8-branch (*L*=8) MRC combiner. Receiver sensitivity, just 3 dB over the AWGN limit of 18-photons-per-bit at 10^{−9} BER, is high. The excess margin could be used to deal with the stronger turbulence condition considered in the figure, where *D*/*r*
_{0} =4, or used to increase the data rate up to 2 Gbit/s with less than 1 dB margin. Under the same initial turbulence conditions *D*/*r*
_{0} =2, a receiver phase-compensated up to astigmatism (a moderate *J*=6) can achieve a 1-Gbit/s link with ~5 dB margin. A high receiver sensitivity just ~1.5 dB over the AWGN limit is expected. Now, the excess margin can be traded either for stronger turbulence conditions *D*/*r*
_{0}=4, while retaining a ~2 dB margin, or for an increase of the data rate up to 3 Gbit/s.

Mitigation techniques are essential for reaching the high data rates shown in Fig. 4. Note that if diversity branches of the BPSK combiner are reduced from *L*=8 to *L*=4, or phase compensation is limited to a modal expansion up to tilt (*J* =3, instead of the previous *J*=6), only low-data-rate links are possible. This can be easily understood in Fig. 4, as increasing the number of combiner branches *L* from 4 to 8 grants a diversity gain of ~9 dB. Similarly, increasing the order *J* of the phase compensation from 3 to 6 provides a net modal gain of ~7 dB. Now, for the same nominal –50 dB power link budget, and under turbulence conditions such as *D*/*r*
_{0} =2, the MRC 4-branch combiner can only support a medium-rate 200-Mbit/s link with ~1 dB margin. (If stronger turbulence conditions, such as those described by *D*/*r*
_{0} =4, are considered, the BPSK 4-branches MRC combiner suffers and only a low-rate 30-Mbit/s link with similar ~1 dB margin is supported). Note that these small 1-dB excess margins may become necessary to compensate for possible scintillation fading: Although in most situations considered, phase distortion becomes the dominant effect on the coherent performance of MRC combiners, the penalty due to amplitude fluctuations may still be relevant. For example, as seen in Fig. 4, for the BPSK 4-branches MRC combiner, atmospheric conditions leading to a scintillation index σ_{β}
^{2} = 1 yield a performance scintillation penalty of roughly 1.3 dB.

## 4. Conclusion

The noncentral chi-square distribution with 2*L* degrees of freedom, which we introduced recently as a model for atmospheric fading in a single coherent receiver affected by amplitude and phase fluctuations, is used to study the performance of various binary and nonbinary modulation methods for coherent laser communication through the turbulent atmosphere. We have compared the spectral efficiencies and SNR requirements of complex modulations in the presence of fading noise from atmospheric turbulence and local oscillator shot noise.

We have discussed methods of optimizing receiver sensitivity by using atmospheric compensation techniques, including phase correction and diversity combining. We have identified the impact of the combiner number of branches and amount of phase compensation on the performance of coherent modulation formats. For typical turbulence conditions, sizeable gains in spectral efficiency and SNR requirements are realizable by allowing for a rather small number of apertures or by compensating a modest number of modes. We have separately quantified the effects of amplitude fluctuations and phase distortion, and found that, in most situations considered, phase distortion becomes the dominant effect on the coherent performance and amplitude fluctuations are of less importance.

It has been demonstrated that at spectral efficiencies below 1 bit/s/Hz, 2-PAM and 2-DPSK are attractive techniques. Between 1 and 2 bits/s/Hz, 4-DPSK and 4-PSK are interesting options. At spectral efficiencies above 2 bits/s/Hz, 8-PSK and 16-QAM becomes the most appealing modulations. We have shown that high-order *M*-ary QAM is not just spectrally efficient, but also can significantly improve photon efficiency in atmospheric fading as compared to high-order PSK modulation.

## Acknowledgments

The research of Aniceto Belmonte was partially funded by the Spanish Department of Science and Innovation MICINN Grant No. TEC 2009-10025.

## References and links

**1. **J. W. Strohbehn, T. Wang, and J. P. Speck, “On the probability distribution of line-of-sight fluctuations of optical signals,” Radio Sci. **10**(1), 59–70 (1975). [CrossRef]

**2. **R. J. Noll, “Zernike polynomials and atmospheric turbulence,” J. Opt. Soc. Am. **66**(3), 207–211 (1976). [CrossRef]

**3. **M. Born, and E. Wolf, *Principles of Optics* (Cambridge University Press, 1999).

**4. **D. L. Fried, “Atmospheric modulation noise in an optical heterodyne receiver,” IEEE J. Quantum Electron. **3**(6), 213–221 (1967). [CrossRef]

**5. **A. Belmonte and J. Khan, “Performance of synchronous optical receivers using atmospheric compensation techniques,” Opt. Express **16**(18), 14151–14162 (2008). [CrossRef] [PubMed]

**6. **A. Belmonte and J. M. Kahn, “Capacity of coherent free-space optical links using diversity-combining techniques,” Opt. Express **17**(15), 12601–12611 (2009). [CrossRef] [PubMed]

**7. **J. W. Goodman, *Speckle Phenomena in Optics. Theory and Applications* (Ben Roberts & Company, 2007).

**8. **J. D. Parsons, “Diversity techniques in communications receivers,” in *Advanced Signal Processing*, D. A. Creasey, ed. (Peregrinus, 1985), Chap. 6.

**9. **J. G. Proakis, and M. Salehi, *Digital Communications*, (Mc Graw-Hill, 2007).

**10. **M. K. Simon and M.-S. Alouini, “A unified approach to the performance analysis of digital communications over generalized fading channels,” IEEE Proc. **86**(9), 1860–1877 (1998). [CrossRef]

**11. **A. Goldsmith, *Wireless Communications* (Cambridge University Press, 2005)

**12. **J. Kahn and K.-P. Ho, “Spectral efficiency limits and modulation/detection techniques for DWDM systems,” IEEE J. Sel. Top. Quantum Electron. **10**(2), 259–272 (2004). [CrossRef]