## Abstract

Limitations in the performance of coherent transmission systems employing digital back-propagation due to four-wave mixing impairments are reported for the first time. A significant performance constraint is identified, originating from four-wave mixing between signals and amplified spontaneous emission noise which induces a linear increase in the standard deviation of the received field with signal power, and linear dependence on transmission distance.

© 2011 OSA

## 1. Introduction

The insatiable demand for information capacity is ultimately limited by the Kerr effect in the transmission fibre and amplified spontaneous emission (ASE) noise from the optical amplifiers. In a conventional system the signal power is optimized to give the highest possible optical signal-to-noise ratio (OSNR), whilst avoiding any significant nonlinear distortion. The recent introduction of coherent detection with the availability of high speed digital signal processing (DSP) technologies has to some extent alleviated such limitations by electronic mitigation of linear and nonlinear fibre impairments [1,2]. In particular, near perfect compensation of linear and nonlinear fibre impairments can be achieved by employing electronic signal processing using digital back-propagation (DBP) [3–6]. In principle this would allow arbitrarily high signal-to-noise-ratios and a system performance limited only by power handling capabilities of individual components. However, conventional back-propagation can only compensate for deterministic intra-channel nonlinear effects [3] since inter-channel effects may only be compensated with parallel processing [4,6]. In this work, we consider arbitrary DBP complexity in order to establish the maximum potential of this method, irrespective of the DBP bandwidth [6].

Nevertheless, when the ASE noise co-propagates with a strong signal, the transmission characteristics change significantly due to the nonlinear interaction between signal and noise through Kerr effect. For dispersion managed systems such interactions were classified as Gordon-Haus [7] and Gordon-Mollenauer [8] effects, modulation instability [9], and as four-wave mixing (FWM) [10,11]. The need to minimize the latter resulted in the widespread employment of dispersion maps with high local dispersions. This approach has, until recently been effective in reducing the impact of nonlinear signal-ASE interactions. However, the sensitivity of DSP enhanced coherent receiver, coupled with the reaches possible when DBP is employed would result in a significant increase in the impact of such impairments. The contemporary opinion is that for systems without optical dispersion compensation signal-ASE interactions are dominated by cross-phase modulation (XPM) effects [12] implying a linear dependence of standard deviation (normalized to the amplitude of transmitted pattern) of the received field on noise level and reach, but without an explicit power dependence. This leads to the assumption that DBP will enable the increase in signal launch power required to facilitate higher order modulation formats.

In this paper, we report the nature and extent of nonlinear interaction between signal and ASE in a 112 Gb/s polarization multiplexed quadrature phase-shifted keying (PM-QPSK) transmission system employing DBP. In common with other studies, we identified that an optimum launch power exists even with ideal DBP [5]. Examining the commonly accepted mechanisms enabling this effect, we identify that the statistic variations in the received constellations are consistent with the parametric amplification of ASE by the signal. By considering the impact of this four-wave mixing process between the signal and the ASE we find an excellent agreement between our numerical results and simple analytical predictions, which we report here for the first time. Also, we confirm that this effect limits the performance of long-haul transmission systems.

## 2. Simulation setup

The simulation setup is shown in Fig. 1
. The 112 Gb/s PM-QPSK transmitter was operated at 1550 nm and consisted of a continuous wave laser followed by two nested Mach-Zehnder modulator (MZM) structures for the x- and y-polarization states. Two independent pseudo-random bit sequences (PRBS) were encoded, one for each polarization, each de-multiplexed separately into two sampled two-level output symbol streams which were used to modulate an in-phase and a quadrature phase carrier, respectively. The number of transmitted symbols was 2^{13} per polarization per channel, and the sample-rate was set to be 16 samples/symbol. The 112 Gb/s PM-QPSK signal was propagated over a non dispersion-managed link using single stage erbium doped fibre amplifier (EDFA). The link comprised *60* × 80 km spans of single mode fibre (SMF) for transmission. The EDFA was modelled with a 4.5 dB noise figure and the total amplification gain was equal to the total loss in each span. The fibre had a loss (α) of 0.2 dB/km and a nonlinear coefficient (γ) of 1.4/W/km, and no inline optical dispersion compensation was used. Note that, we expect the findings of this report would remain valid, although at a higher transmission distance, given a fibre with lower loss, nonlinear coefficient and larger core area [13].

After fibre transmission, the received signal was pre-amplified (constant power of 0 dBm) and coherently detected to give the baseband electrical signal at a sampling rate of 2 samples/symbol. Transmission impairments were digitally compensated using DBP, which was numerically implemented by up-sampling the received signal to 16 samples/symbol and reconstructing the optical field from the in-phase and quadrature samples, followed by split-step Fourier method based solution of nonlinear Schrödinger equation. Note that the field was up-sampled to enable high precision DBP. Also the step-size was chosen adaptively such that in each step the change in phase of the optical field was more than 0.05 degrees, in order to establish the maximum potential performance of DBP. The signal was then re-sampled to 1 sample/symbol and passed through an FIR filter (fractionally-spaced taps) to optimise the electrical filter response and compensate any residual dispersion. Note that in all the cases, the received field was normalized to the amplitude of transmitted pattern in the filtering stage. Finally, the symbol decisions were made after polarization de-multiplexing and constellation de-mapping. The performance was assessed by direct error counting. All the numerical simulations were carried out using VPItransmissionMaker® v.8.3 and MATLAB® v.7.9.

## 3. Results and discussions

Typical results of our simulations are shown in Fig. 2 . The fibre dispersion is varied from 5 ps/nm/km to 20 ps/nm/km, and for each dispersion map the degradation in BER with launch power is shown after DBP. As it can be seen in Fig. 2a, varying the dispersion has a considerable impact on the system performance and associated BER. At low launch powers, the BER curves for various local dispersion values overlap since transmission is noise-limited. As the launch power is increased it is clear that the BER is low for a range of launch powers irrespective of local dispersion confirming that at such powers, the OSNR is adequate, and the nonlinear effects may be adequately compensated by DBP, suggesting that they are fully deterministic. However, the BER is severely degraded as the launch power is increased above 8 dBm. The existence of a signal power above which no further performance increase is possible suggests that not only a signal-ASE interaction exists which prevents further increase in the effective signal to noise ratio [5], but also degrades it with increasing signal power. The origin of such signal-ASE interaction is likely to be either XPM [12] or FWM [10,11]. Figure 2a (inset) plots the performance for local dispersion of 5 ps/nm/km with noise added after each amplifier, and for a noise free transmission with an equivalent noise loading at the receiver. It can be seen that the two cases overlap in noise-limited regime irrespective of the dispersion, however as the launch power is increased, the BER degrades significantly if in-line amplifier noise is applied, confirming that the signal-ASE interaction is a distributed effect.

Figure 2b illustrates the received constellation diagram for one of the polarizations after DBP at 4dBm. It is qualitatively shown that DBP enables clear identification of the mean location of individual symbols and the quality of the constellation map is found to be excellent due to the compensation of deterministic nonlinear fibre impairments. Note that one would expect to get similar distribution in a WDM transmission, if ideal full bandwidth DBP were employed. Figure 2b also shows the definition of angular and radial jitter in the constellations, used later in the text to evaluate the standard deviation of the received field.

Figure 3
plots the normalized standard deviation (σ_{norm}) of the received field after DBP for radial (Fig. 3a) and angular jitter (Fig. 3b) for different dispersion coefficients. It can be seen that at lower signal power σ_{norm} increases with decreasing OSNR for all the dispersion maps, as expected for a noise-limited system. For higher launch power levels, where the performance is affected by nonlinear effects, σ_{norm} of the received field shows a linear increase with signal power, thus we may conclude that the actual constellation variance varies quadratically with signal launch power. This is in contrast to previously reported mechanisms for the nonlinear interaction between signal and noise, which tend to demonstrate a linear dependence on the signal power, is solely dependent on the noise power [8,12] or explicitly neglects the impact of nonlinear effects on the noise [11]. However, a quadratic dependence of variance, or a linear increase of σ_{norm} with power would be expected for a FWM based signal-ASE interaction where the parametric amplification of ASE increases quadratically with signal power. The resonant enhancement or quasi phase matching of FWM along an amplified transmission line is well-known [14,15], as given below,

*E*is the nonlinearly generated field,

_{FWM}*w*is the angular frequency,

*E*and

_{p}, E_{q}*E*are signal components,

_{r}**n*is the refractive index,

*L*is the span length,

*χ*is the nonlinearity coefficient, degeneracy factor

^{3}*D*is either three or six for degenerate and non-degenerate FWM and

*Δβ*is the effective propagation constant difference. For a modulated signal whose total bandwidth exceeds the phase matching bandwidth of a single fibre span (~3GHz for standard single mode fibre) integration of Eq. (1) over the continuous power spectral density (PSD).is required, accounting for contributions from both strongly and weakly phase matched contributions.

Recently closed-from expressions have been derived for the nonlinear interaction between spectral components of signals with continuous spectra where the effects of strongly and weakly phase matched conditions are treated separately [16]. Following the same approach to the derivation, but replacing one of the signal fields by the noise field, results in a solution for the parametric amplification of ASE from the *N ^{th}* amplifier along the remainder of the link,

*I*is the nonlinear noise power spectral density,

_{SN-FWM}*I*is the signal power spectral density,

_{signal}*I*is the noise power spectral density from a single amplifier.

_{noise}*C*and

_{1}*C*represent weakly and strongly phase matched regimes, respectively.

_{2}*N*is the number of spans after a given amplifier,

*B*is signal bandwidth and

*β*the group velocity dispersion (D = −2πcβ

_{2}_{2}/λ

^{2}). From Eq. (2), treating the contribution from each amplifier as an independent random variable, with Gaussian statistics, the total nonlinear noise at the output of an

*M*-span system is,

The solid lines in Fig. 3 show the result of Eq. (3) for our simulation conditions, and show an excellent match. In contrast to the four-wave mixing experienced by an ultra wide-band signal [16], for signal-ASE FWM the contributions from both strongly and weakly phase matched terms are significant for the given 28 Gbaud system with DCF free transmission.

Summing the independent contribution from each amplifier gives a length variation with significant terms up to M^{3/2}. Figure 4a
shows the evolution of the noise σ_{norm} with length for a signal launch power of 5 dBm, close to the optimum shown in Fig. 3, and again shows an excellent agreement with analytical prediction of (2). Figure 4a also shows the expected evolution in σ_{norm} in the absence of nonlinearity, corresponding to noise loading at the receiver. Comparing the two curve fits reveals that the signal-ASE FWM process becomes dominant in this system after ~5,000 km for PM-QPSK.

In order to indicate the relative impact of the transmission limit originating from the signal-ASE FWM constraint, Fig. 4b plots the nonlinearity limited information spectral density (ISD) versus transmitted power density, following a similar analysis to [17]. The figure depicts that for the example reported here the effect of nonlinearities at high powers prevents indefinite growth in ISD. It can be seen that the effect of XPM becomes prominent at transmitted power densities beyond 0.01 W/THz, and a maximum ISD of 3 b/s/Hz/pol is predicted [18]. However, for a point to point transmission system such effects are compensable [6]. However, even with such compensation scheme, the transmission capacity remains limited, but by signal-ASE FWM. As can be seen in the figure, for this system configuration the maximum transmittable power spectral density may be only increased to 0.1 W/THz and the maximum ISD may only be increased by a factor of 2.

## 4. Conclusion

We have demonstrated the influence of four-wave mixing between signal and ASE on the transmission performance of a long-haul coherently-detected 112 Gb/s PM-QPSK system employing DBP. Our results show excellent agreement with analytical theory and suggest that the BER and the associated maximum reach remain ultimately limited even after digital back-propagation due to the signal-noise interactions, and that an optimum signal power exists even with DBP. Further investigation reveals that the evolution of the excess noise with signal power, dispersion and transmission length is fully consistent with the four-wave mixing between the signal and ASE, whereby the noise is parametrically amplified by the signal.

## Acknowledgments

This work is supported by Science Foundation Ireland under Grant 06/IN/I969.

## References and links

**1. **S. J. Savory, “Compensation of fibre impairments in digital coherent systems,” *in Proc. ECOC 2008*, Mo.3.D.1 (2008).

**2. **D. Rafique, M. Forzati, and J. Mårtensson, “Impact of Nonlinear Fibre Impairments in 112 Gb/s PM-QPSK Transmission with 43 Gb/s and 10.7 Gb/s Neighbours,” *in Proc. ICTON 2010*, paper We.D1.6 (2010).

**3. **C. Weber, C.-A. Bunge, and K. Petermann, “Fiber Nonlinearities in Systems Using Electronic Predistortion of Dispersion at 10 and 40 Gbit/s,” J. Lightwave Technol. **27**(16), 3654–3661 (2009). [CrossRef]

**4. **F. Yaman and G. Li, “Nonlinear Impairment Compensation for Polarization-Division Multiplexed WDM Transmission Using Digital Backward Propagation,” IEEE Photonics Technol. Lett. **1**, 144–152 (2009).

**5. **E. Ip and J. M. Kahn, “Compensation of Dispersion and Nonlinear Impairments Using Digital Backpropagation,” J. Lightwave Technol. **26**(20), 3416–3425 (2008). [CrossRef]

**6. **D. Rafique, J. Zhao, and A. D. Ellis, “Impact of Dispersion Map Management on the Performance of Back-Propagation for Nonlinear WDM Transmissions,” OECC’2010 , 00107 (2010).

**7. **J. P. Gordon and H. A. Haus, “‘’Random walk of coherently amplified solitons in optical fiber transmission,” Opt. Lett. **11**(10), 665–667 (1986). [CrossRef] [PubMed]

**8. **J. P. Gordon and L. F. Mollenauer, “Phase noise in photonic communication systems using linear amplifiers,” Opt. Lett. **15**(23), 1351–1353 (1990). [CrossRef] [PubMed]

**9. **R. Hui, M. O’Sullivan, A. Robinson, and M. Taylor, “Modulation instability and its impact in multispan optical amplified IMDD systems: Theory and experiments,” J. Lightwave Technol. **15**(7), 1071–1082 (1997). [CrossRef]

**10. **D. Marcuse, “Bit-error rate of lightwave systems at the zero dispersion wavelength,” J. Lightwave Technol. **9**(10), 1330–1334 (1991). [CrossRef]

**11. **J. Tang, “The Shannon channel capacity of dispersion-free nonlinear optical fiber transmission,” J. Lightwave Technol. **19**, pp1104 (2000).

**12. **A. Bononi, P. Serena, and N. Rossi, “Nonlinear signal-noise interactions in dispersion-managed links with various modulation formats,” Opt. Fiber Technol. **16**(2), 73–85 (2010). [CrossRef]

**13. **J. Cai, Y. Cai, C. Davidson, D. Foursa, A. Lucero, O. Sinkin, W. Patterson, A. Pilipetskii, G. Mohs, and N. Bergano, “Transmission of 96x100G Pre‐Filtered PDM‐RZ‐QPSK Channels with 300% Spectral Efficiency over 10,608km and 400% Spectral Efficiency over 4,368km,” *in Proc. OFC 2010*, PDPB10 (2010).

**14. **A. D. Ellis, and W. A. Stallard, “Four wave mixing in ultra long transmission systems incorporating linear amplifiers,” in *Non-Linear Effects in Fibre Communications, IEE Colloquium on*, 6/1–6/4, (1990).

**15. **K. Inoue and H. Toba, “Fiber four-wave mixing in multi-amplifier systems with nonuniform chromatic dispersion,” J. Lightwave Technol. **13**(1), 88–93 (1995). [CrossRef]

**16. **X. Chen and W. Shieh, “Closed-form expressions for nonlinear transmission performance of densely spaced coherent optical OFDM systems,” Opt. Express **18**(18), 19039–19054 (2010). [CrossRef] [PubMed]

**17. **A. D. Ellis, J. Zhao, and D. Cotter, “Approaching the Non-Linear Shannon Limit,” J. Lightwave Technol. **28**(4), 423–433 (2010). [CrossRef]

**18. **P. P. Mitra and J. B. Stark, “Nonlinear limits to the information capacity of optical fibre communications,” Nature **411**(6841), 1027–1030 (2001). [CrossRef] [PubMed]