## Abstract

We theoretically study optical transmission characteristics of wavelength-division multiplexed (WDM) and polarization-multiplexed (POLMUX) signals using high-order optical quadrature-amplitude-modulation (QAM) formats up to 256. First, we conduct intensive computer simulations on bit-error rates (BERs) in WDM POLMUX QAM transmission systems and find maximum transmission distances under the influence of nonlinear impairments. Next, to elucidate the physics behind such nonlinear transmission characteristics, we calculate the distribution of constellation points for QAM signals as functions of the the launched power, the transmission distance, and the symbol rate. These results lead to a closed-form formula for BER of any QAM formats. From such formula, we derive simple laws that determine the maximum transmission distance and the optimum power as functions of the QAM order and the symbol rate. These laws can well explain the simulation results.

© 2011 OSA

## 1. Introduction

High-order optical quadrature-amplitude-modulation (QAM) formats, such as 16, 64, and 256QAM, have attracted significant attention because of their spectrally-efficient transmission characteristics in dense wavelength-division multiplexing (WDM) as well as polarization multiplexing (POLMUX) environments [1–3]. The recent development of digital coherent optical receivers may enable the use of such sophisticated modulation formats in the near future [4,5]. We have theoretically shown that 16QAM signals at 12.5 Gsymbol/s can be placed on the 25-GHz-spaced grid, and ultra long-haul transmission of 2,000 km with the spectral efficiency as high as 4 bit/s/Hz is possible even under the influence of self-phase modulation (SPM) and cross-phase modulation (XPM) between WDM and POLMUX channels [6]. Although the spectral efficiency can potentially be improved with higher-order QAM formats such as 64QAM and 256QAM, their transmission characteristics may suffer more seriously from nonlinear impairments due to SPM and XPM; however, neither experiments nor theoretical works have been reported so far.

This paper aims at analyzing the performance of WDM POLMUX high-order QAM transmission in a systematic manner. We assume that the dispersion-unmanaged link consists only of large-core single-mode fibers (SMFs), whereas the accumulated group-velocity dispersion (GVD) is fully compensated for at the digital coherent receiver. First, we conduct intensive computer simulations on bit-error rates (BERs) in WDM POLMUX QAM transmission systems and find maximum transmission distances under the influence of nonlinear impairments. Next, to elucidate the physics behind such nonlinear transmission characteristics, we calculate the distribution of constellation points for a 4QAM signal as functions of the the launched power, the transmission distance, and the symbol rate. These calculation results clearly show that waveform distortion stemming from the interplay between GVD and nonlinearity of fibers generates the Gaussian distribution for the constellation points. The variance of such Gaussian distribution is proportional to the transmission distance and the cube of the launched power, while it is independent of the symbol rate. The extension of these results to higher-order QAM formats leads to a closed-form formula for BER. From this BER formula, we derive simple laws that express the maximum transmission distance and the optimum power as functions of the QAM order and the symbol rate. The validity of these laws are confirmed by the simulation results.

The organization of the paper is as follows: In Sec. 2, we describe the simulation model of the transmission system and the digital coherent receiver. Section 3 presents simulation results on WDM POLMUX 4QAM, 16QAM, 64QAM and 256QAM transmission characteristics. Section 4 discusses how the distribution of constellation points of a 4QAM signal spreads out through GVD and nonlinearity of fibers for transmission. In Sec. 5, using the results obtained in Sec. 4, we derive closed-form formula for BER of QAM signals in WDM and POLMUX environments, which yields simple laws for high-order QAM transmission. Section 6 is the conclusion of the paper.

## 2. Simulation model

The model of the transmission system and the digital coherent receiver is shown in Fig. 1. The dispersion-unmanaged link has 80-km-long SMF spans. We assume large-core SMFs, which have the GVD value *D* of 19 ps/nm/km (*β*
_{2} = −24 ps^{2}/km), the nonlinearity coefficient *γ* of 1.1 /W/km, and the loss coefficient *α* of 0.17 dB/km at the wavelength of 1550 nm [8].

We use the 50% return-to-zero (RZ) waveform for the envelope of the complex amplitude of the signal electric field.

Differentially-encoded optical QAM signals are filtered out by root Nyquist filters with the roll-off parameter of 0.3 before transmission. An erbium-doped fiber amplifier (EDFA) with the noise figure of 4 dB compensates for the loss of each span. Linewidths of transmitter lasers are assumed to be negligible in our calculations to investigate only the nonlinear effect. The WDM channel spacing is twice as large as the symbol rate, and the maximum number of WDM channels is five.

WDM POLMUX 4, 16, 64 and 256QAM transmission characteristics are numerically analyzed based on the two-mode nonlinear Schrödinger equations given as

*E*and

_{x}*E*denote electric fields of the WDM signal with

_{y}*x*- and

*y*-polarization components, respectively. In these equations, randomly-distributed linear birefringence is assumed along the link [7].

At the receiver, the incoming signal of each polarization is detected with a phase-diversity homodyne receiver where the linewidth of a local laser is negligible. The received complex amplitude of each polarization tributary is filtered out by a root Nyquist filter with the roll-off parameter of 0.3 to select the center WDM channel. The accumulated GVD is compensated for by a fixed equalizer. After the data are resampled so as to keep one sample per symbol, the carrier phase is estimated by the 4-th power algorithm [9]. Finally, a nine-tap FIR filter equalizes the signal in an adaptive manner based on the decision-directed least-mean-square (DD-LMS) algorithm. After the equalizing process is converged, we differentially decode the symbol and count the number of bit errors. The total number of symbols is 2^{15} per channel. We have not introduced nonlinearity compensation based on the back-propagation method [11], because the real-time implementation of this method is very difficult due to huge computational complexity especially in WDM systems.

## 3. Simulation results

We calculate QAM transmission characteristics of a single channel, co-polarized three
WDM channels, co-polarized five WDM channels, and POLMUX five WDM channels. We
select the center WDM channel for BER estimation, which is most seriously affected
by XPM between WDM channels. In the case of POLMUX transmission, we evaluate BERs of
one of the two polarization tributaries. The number of spans *n* for
each QAM format is determined such that the BER of the polarization tributary of the
center WDM channels becomes lower than 10^{−3} at the optimum
launched power, when POLMUX five WDM channels are transmitted.

Figures 2(a)–2(d) show BERs of 4QAM,
16QAM, 64QAM, and 256QAM signals, respectively, calculated as a function of the
launched average power *P _{ave}*. Powers of all channels are
changed simultaneously. Green, red, black, and blue curves represent BERs when we
transmit a single channel, co-polarized three WDM channels, co-polarized five WDM
channels, and POLMUX five WDM channels, respectively. The symbol rate is 12.5
Gsymbol/s and the WDM channel spacing 25 GHz. Numbers of spans

*n*corresponding to (a)–(d) are 160, 37, 10, and 3, respectively, resulting in total transmission distances of 12,800 km, 2,960 km, 800 km, and 240 km. Note that the optimum power is about −9 dBm ~ −8 dBm, which is almost common in all of the QAM orders. Above this value, the BER performance is degraded by nonlinear impairments stemming from SPM and XPM between WDM channels and POLMUX channels.

Dots in Fig. 3 show the maximum number of
spans *n* as a function of the order of QAM *m*, when
POLMUX five WDM channels are transmitted. We find that *n* is
inversely proportional to *m* as seen from the solid line and that
transmission distances of 64QAM and 256QAM systems are severely limited below 1,000
km.

Next, changing the symbol rate, we perform similar simulations. Figure 4 shows maximum numbers of spans *n*
for 4, 16, 64, and 256QAM transmission as a function of the symbol rate
*B*. Solid lines represent the slope of *n*
∝ *B*
^{−2/3}, which is in good agreement with
simulation results in the range above 25 Gsymbol/s. On the other hand, the optimum
launched power *P _{opt}* to obtain the minimum BER for 4QAM
is plotted as a function of the symbol rate

*B*in Fig. 5. In higher-order QAM transmission systems, we have similar

*B*-versus-

*P*characteristics within the error range. The solid line represents the slope of

_{opt}*P*∝

_{opt}*B*

^{1/3}. Simulation results differ from this line significantly in the range below 25 Gsymbol/s. The theoretical background for these dependencies on

*B*will be discussed in Sec. 5 in detail.

## 4. Distribution of constellation points spread by nonlinear effects

In this section, to understand the physics behind the simulation results in Sec. 3, we discuss how constellation points of a 4QAM signal spread out by nonlinear effects. We consider transmission of five-channel WDM and POLMUX signals. The WDM channel spacing is twice as large as the symbol rate. The received complex amplitude is normalized such that the mean square of its absolute value is unity. In such a case, when the carrier-to-noise ratio (CNR) is high enough, the distance between constellation points is 2 as shown in Fig. 6, where ${\sigma}_{\mathit{\text{nor}}}^{2}$ stands for the variance of the distribution.

Let the symbol rate be 12.5 Gsymbol/s, the WDM channel spacing 25 GHz, and the
number of spans 160. We consider the real part of the complex amplitude ranging from
−2 to +2 in Fig. 6. This
range is divided into 1,000 sections, and the number of data falling in each section
is shown as a histogram for 2^{15} sampled data. Black curves in Figs. 7(a), 7(b), and 7(c) show distributions
of the real part of the received complex amplitude for launched average powers
*P _{ave}* of −14 dBm, −9 dBm, and
−4 dBm, respectively. On the other hand, red curves are Gaussian
distributions fitted to black ones.

When *P _{ave}* =−14 dBm, the BER performance
is determined by amplified spontaneous emission (ASE) from EDFAs as shown in Fig. 2(a). On the other hand, when

*P*=−4 dBm, the BER performance is degraded by nonlinear effects. At an intermediate power such as

_{ave}*P*=−9 dBm, both of ASE and nonlinear effects spread out the distribution of constellation points. Note that distributions are Gaussian in all of the three cases.

_{ave}Next, we calculate the variance ${\sigma}_{\mathit{\text{nor}}}^{2}$ of the Gaussian distribution as a function of the
launched average power *P _{ave}*. In Fig. 8, the blue curve is obtained when

*γ*= 0 and ASE is included,

*i.e.*, the linear case, whereas the red curve are obtained when ASE is neglected and fiber nonlinearity is included

*i.e.*, the nonlinear case. The black curve is calculated when both of ASE and fiber nonlinearity are taken into account. Since the black curve is simply equal to the sum of the red and blue curves, we find that the nonlinear impairment stems from waveform distortion due to GVD and fiber nonlinearity rather than ASE noise enhanced by fiber nonlinearity,

*i.e.*, Gordon-Mollenauer phase noise [10]. In the linear case, ASE determines the distribution of constellation points, and it follows that ${\sigma}_{\mathit{\text{nor}}}^{2}$ is inversely proportional to

*P*. On the other hand, in the nonlinear case, we find that ${\sigma}_{\mathit{\text{nor}}}^{2}$ is proportional to ${P}_{\mathit{\text{ave}}}^{2}$. This fact is understood as follows: First, inter-symbol interference (ISI) due to GVD generates the power fluctuation

_{ave}*δ*

*P*proportional to

*P*in all of the channels. Then, SPM and XPM induces the phase fluctuation

_{ave}*γδ*

*P*per unit length in the channel under consideration, which in turn spreads out the distribution of constellation points through GVD; thus, ${\sigma}_{\mathit{\text{nor}}}^{2}$ is proportional to (

*γ*

*P*)

_{ave}^{2}.

We move on to the dependence of ${\sigma}_{\mathit{\text{nor}}}^{2}$ on the transmission distance. In this calculation,
ASE is neglected and only fiber nonlinearity is taken into account. We demodulate
the signal at the end of each span and obtain the distribution of constellation
points, fixing the launched power *P _{ave}* at −7
dBm. Figure 9 shows ${\sigma}_{\mathit{\text{nor}}}^{2}$ as a function of the transmission distance. The red
curve and the black curve are those obtained when symbol rates are 100 Gsymbol/s and
12.5 Gsymbol/s, respectively. We find that ${\sigma}_{\mathit{\text{nor}}}^{2}$ is proportional to the number of spans

*n*, as shown by blue fitted lines. This result shows that evolution of the complex amplitude along the link obeys the two-dimensional random-walk model: from one span to the next, the signal sequence takes a random step from its last position, which results in the linear increase in the variance as a function of the number of spans.

Finally, we discuss the symbol-rate-dependence of ${\sigma}_{\mathit{\text{nor}}}^{2}$ in the nonlinear case. Figure 10 shows ${\sigma}_{\mathit{\text{nor}}}^{2}$ as a function of the symbol rate, when the launched
power *P _{ave}* is −7 dBm, and the number of spans
100. The variance is almost constant when the symbol rate is larger than 25
Gsymbol/s; however, it increases significantly when the symbol rate is lower than 25
Gsymbol/s. This tendency is explained in terms of modulation instability
[12]. The bandwidth of
modulation-instability gain is about 1 GHz in our case. Therefore, when

*B*< 25 Gsymbol/s, the modulation-instability effect inside the signal bandwidth cannot be ignored, and the variance of the Gaussian distribution is enhanced.

In conclusion of Sec. 4, we have the following equation for ${\sigma}_{\mathit{\text{nor}}}^{2}$:

*L*is the span length and

*C*is a constant. In the five-channel WDM and POLMUX environments,

_{p}*C*= 13.4 [km] above 50 Gsymbol/s; however, we need some corrections for

_{p}*C*at lower symbol rates as shown in Fig. 10.

_{p}In [13] and [14], closed-form expressions of
nonlinear ISI are derived. The dependence on *n* and
*P _{ave}* expressed by Eq. (3) is the same as that given in
[13] and [14]; however, the dependence on

*B*given by Eq. (3) is different from those derived in [13] and [14]. This may be because modulation instability is not taken into consideration in these references.

## 5. BER formula for high-order QAM transmission systems

Based on Eq. (3), we derive a
closed-form expression of BER. Let the minimum distance between QAM constellation
points be 2*δ* and the variance of the distribution
*σ*
^{2}. In such a case, the bit-error rate for
QAM signals is approximately given as

*D*=1, 3/4, 7/12, and 15/32 for differentially encoded 4QAM, 16QAM, 64QAM, and 256QAM, respectively. The average power of the

_{e}*m*-th order QAM signal

*P*(

_{ave}*m*) is given as

*f*denote the frequency of the carrier,

*G*the amplifier gain, and

*n*the spontaneous emission factor of amplifiers. On the other hand, the variance in the nonlinear region is given from Eq. (3) as

_{sp}From Eq. (8), we find that when

*at a certain value, for example 10*

_{min}^{−3}, the following relation must be satisfied for any values of

*m*,

*n*, and

*B*: where

*C*is a constant.

The analyses mentioned above lead to the following simple laws for QAM transmission in fixed WDM and POLMUX environments:

These laws can well explain simulation results given in Figs. 3, 4, and
5. Figure
3 shows that *n* ∝
*m*
^{−1} when *B* is fixed. On the
other hand, for a fixed *m*, Fig.
4 and Fig. 5 lead to relations of
*n* ∝ *B*
^{−2/3} and
*P _{opt}* ∝

*B*

^{1/3}, respectively, when

*B*is larger than 50 Gsymbol/s. Deviations of simulation results from these relations at lower symbol rates are due to the fact that

*C*in Eq. (3) is enhanced by modulation instability. It should be noted here that the larger GVD of fibers generally provides us with the better transmission performance because of the smaller bandwidth of modulation-instability gain.

_{p}Equation (8) is a closed-form BER
formula applicable to WDM and POLMUX QAM transmission. Once
*C _{p}* in Eq.
(10) is determined from computer simulations in fixed WDM and POLMUX
environments, we can calculate BER of any QAM formats using signal parameters
(

*n*,

*m*,

*B*, and

*P*) and link parameters (

_{ave}*L*,

*G*,

*n*, and

_{sp}*γ*). For five-channel WDM and POLMUX transmission at 12.5 Gsymbol/s, which is discussed in Sec. 3, we need to use

*C*=56.1 [km] read from Fig. 10. Then, BER curves calculated for 4QAM, 16QAM, 64QAM, and 256QAM are respectively shown by the black, red, blue, and green curves in Fig. 11. Numbers of spans are the same as those used in Sec. 3. Even using such a simple formula, we can obtain BER curves for any QAM formats which are in reasonable agreement with full simulation results given by Fig. 2.

_{p}## 6. Conclusions

We have analyzed the performance of WDM POLMUX high-order QAM transmission in a systematic manner. First, we conduct intensive computer simulations on BERs and find maximum transmission distances under the influence of nonlinear impairments. The maximum number of spans is inversely proportional to the order of QAM and transmission distances of 64QAM and 256QAM systems are severely limited below 1,000 km at 12.5 Gsymbol/s.

Next, to elucidate the physics behind such nonlinear transmission characteristics, we calculate the distribution of constellation points for QAM signals and find that waveform distortion stemming from the interplay between GVD and nonlinearity of fibers generates the Gaussian distribution for the constellation points. Using the dependence of the variance of such Gaussian distribution on the transmission distance, the launched power, and the symbol rate, we derive a closed-form expression of BER. This BER formula leads to simple laws that determine the maximum transmission distance and the optimum power for each QAM format as a function of the symbol rate.

## Acknowledgments

This work was supported in part by Grant-in-Aid for Scientific Research (A) ( 22246046), the Ministry of Education, Science, Sports and Culture, Japan.

## References and links

**1. **A. H. Gnauck, P. J. Winzer, S. Chandrasekhar, X. Liu, B. Zhu, and D. W. Peckham, “10×224-Gb/s WDM transmission of 28-Gbaud PDM 16-QAM on a 50-GHz grid transmission over 1,200 km of fiber,” in 2010 OSA Technical Digest of *Optical Fiber Communication Conference* (Optical Society of America, 2010), PDPB8.

**2. **A. Sano, T. Kobayashi, K. Ishihara, H. Masuda, S. Yamamoto, K. Mori, E. Yamazaki, E. Yoshida, Y. Miyamoto, T. Yamada, and H. Yamazaki, “240-Gb/s polarization-multiplexed 64-QAM modulation and blind detection using PLC-LN hybrid integrated modulator and digital coherent receiver,” in Proceedings of European Conference on Optical Communication (Sept. 2009), PD2.2.

**3. **M. Nakazawa, S. Okamoto, T. Omiya, K. Kasai, and M. Yoshida, “256 QAM (64 Gbit/s) coherent optical transmission over 160 km with an optical bandwidth of 5.4 GHz,” in *2010 OSA Technical Digest of Optical Fiber Communication Conference*
(Optical Society of America, 2010), OMJ5.

**4. **D.-S. Ly-Gagnon, S. Tsukamoto, K. Katoh, and K. Kikuchi, “Coherent detection of optical quadrature phase-shift keying signals with carrier phase estimation,” J. Lightwave Technol. **24**, 12–21 (2006). [CrossRef]

**5. **H. Sun, K.-T. Wu, and K. Roberts, “Real-time measurements of a 40 Gb/s coherent system,” Opt. Express **16**, 873–879 (2008). [CrossRef] [PubMed]

**6. **K. Kikuchi, “Ultra long-haul optical transmission characteristics of wavelength-division multiplexed dual-polarisation 16-quadrature-amplitude-modulation signals,” Electron. Lett. **46**, 433–434 (2010). [CrossRef]

**7. **S. G. Evangelides Jr., L. F. Mollenauer, J. P. Gordon, and N. S. Bergano, “Polarization multiplexing with solitons,” J. Lightwave Technol. **10**, 28–35 (1992). [CrossRef]

**8. **T. Kato, M. Hirano, M. Onishi, and M. Nishimura, “Ultra-low nonlinearity low-loss pure silica core fibre for long-haul WDM transmission,” Electron. Lett. **35**, 1615–1617 (1999). [CrossRef]

**9. **M. Seimetz, “Laser linewidth limitations for optical systems with high order modulation employing feed forward digital carrier phase estimation,” in *2008 OSA Technical Digest of Optical Fiber Communication Conference
* (Optical Society of America, 2008), OTuM2.

**10. **J. P. Gordon and L. F. Mollenauer, “Phase noise in photonic communications systems using linear amplifiers,” Opt. Lett. **15**, 1351–1353 (1990). [CrossRef] [PubMed]

**11. **E. Ip and J. M. Kahn, “Compensation of dispersion and nonlinear impairments using digital backpropagation,” J. Lightwave Technol. **26**, 3416–3425 (2008). [CrossRef]

**12. **G. P. Agrawal, *Nonlinear Fiber Optics* (Academic Press, 1989), Chap. 5.

**13. **P. Poggiolini, A. Carena, V. Curri, G. Bosco, and F. Forghieri, “Analytical modeling of nonlinear propagation in uncompensated optical transmission links,” IEEE Photon. Technol. Lett. **23**, 742–744 (2011). [CrossRef]

**14. **X. Chen and W. Shieh, “Closed-form expressions for nonlinear transmission performance of densely spaced coherent OFDM systems,” Opt. Express **18**, 19039–19054 (2010). [CrossRef] [PubMed]