## Abstract

We address the issue of intra-channel nonlinear compensation using a Volterra series nonlinear equalizer based on an analytical closed-form solution for the 3rd order Volterra kernel in frequency-domain. The performance of the method is investigated through numerical simulations for a single-channel optical system using a 20 Gbaud NRZ-QPSK test signal propagated over 1600 km of both standard single-mode fiber and non-zero dispersion shifted fiber. We carry on performance and computational effort comparisons with the well-known backward propagation split-step Fourier (BP-SSF) method. The alias-free frequency-domain implementation of the Volterra series nonlinear equalizer makes it an attractive approach to work at low sampling rates, enabling to surpass the maximum performance of BP-SSF at 2× oversampling. Linear and nonlinear equalization can be treated independently, providing more flexibility to the equalization subsystem. The parallel structure of the algorithm is also a key advantage in terms of real-time implementation.

© 2011 OSA

## 1. Introduction

The need for higher bandwidth in optical core networks is currently causing a race on the optimization of spectral efficiency. High-order advanced modulation formats require enhanced signal-to-noise ratio, obtained at the expense of increased power per channel. Bandwidth compatibility with the installed amplifiers imposes to reduce the inter-channel spacing. These requirements are clearly pushing signal transmission to the nonlinear regime. Therefore, digital post-compensation of nonlinear impairments in high-speed optical networks is currently attracting an increasingly attention.

In 2008, universal digital post-compensation of fiber impairments was proposed for the first time, using a backward propagation split-step Fourier (BP-SSF) method [1, 2]. This technique is based on an iterative procedure that separately evaluates the linear and nonlinear operators governing signal backward propagation over a certain spatial step. Despite of providing a remarkable improvement over linear equalization, this iterative technique still requires high computational power and imposes to jointly compensate for linear and nonlinear impairments.

The Volterra series can be used as a mathematical tool for the identification, analysis and equalization of dynamic and time-invariant nonlinear systems [3]. In optical communications, nonlinear electrical equalizers (NLEE) based on Volterra filters have been firstly applied for direct-detection systems [4] as an extension for common feed-forward equalizers (FFE) and decision feedback equalizers (DFE). More recently, Volterra filtering has been extended for coherent systems [5–7] in order to partially mitigate the Kerr effect. A direct performance comparison in a pseudo-linear coherent quadrature phase-shift keying (QPSK) link between a 23 taps long Volterra-based NLEE and a 1 step per span BP-SSF method reveals approximately 1 dB penalty in terms of required optical signal-to-noise ratio (OSNR) [5]. Still, this time-domain approach has been referred to require one order of magnitude more computations than a single-step BP-SSF [5]. Besides, the absence of an analytical description requires Volterra kernels to be determined adaptively using either data-aided or decision-directed strategies [6].

Back in 1997, a Volterra series transfer function (VSTF) was proposed to solve the nonlinear Schrödinger (NLS) equation for single-mode fibers, in frequency-domain [8]. This numerical approach has been recently reassessed within the scope of coherent optical systems, proving that a third-order truncated VSTF is an accurate model for the analysis of nonlinear effects [9]. In [10] an inverse modified VSTF has been applied to describe digital backward propagation in order to compensate for fiber impairments, showing a 2 dB improvement on the nonlinear tolerance over BP-SSF when only 2 samples per symbol are used. Retaining only its third-order nonlinear transfer function, a frequency-domain Volterra series nonlinear equalizer (FD-VSNE) can be obtained from the backward propagation VSTF (BP-VSTF), providing a more flexible way of compensating nonlinearities.

In this paper, we provide a detailed analysis of the FD-VSNE technique including performance assessment via numerical results and computational effort comparisons with other compensation techniques.

## 2. Frequency-domain Volterra series equalizer - theoretical formulation

Before proceeding with the FD-VSNE theoretical formulation we shall first identify the adopted analytical description for the NLS equation that governs signal propagation in fiber,

*A*is an abbreviation of

*A*(

*t,z*) describing the slowly varying complex envelope of the optical field at time

*t*and position

*z*,

*α*is the attenuation coefficient of the fiber,

*β*

_{2}accounts for the group velocity dispersion and

*γ*is the nonlinear coefficient accounting for the Kerr effect.

For each fiber span of length *L _{span}*, the third-order truncated BP-VSTF gives an estimative of the input field spectrum,

*Ã*(

_{in}*ω*), at the expense of the output field spectrum,

*Ã*(

_{out}*ω*) [10],

*H*′

_{1}(

*ω*) and

*H*′

_{3}(

*ω*

_{1},

*ω*

_{2},

*ω*–

*ω*

_{1}+

*ω*

_{2}) are the inverse linear kernel and the inverse third-order nonlinear kernel, respectively given by

Retaining only the third-order nonlinear term in Eq. (2) we obtain the discrete FD-VSNE. Consider a block of *N _{FFT}* signal samples, taken with sampling rate

*F*=

_{s}*N*/

_{FFT}*T*, where

*T*is the time window. This block of samples is transformed into frequency domain, employing a discrete Fourier transform (DFT), which has to be consistent with the previously defined NLS equation,

*ω*is the

_{n}*n*-th sample of the angular frequency vector,

*ω*. Since the DFT definition is based on a discretization of the continuous Fourier transform using a left Riemann sum, we may directly substitute the double integral in Eq. (2) by two algebraic sums over the entire integration range. In a span-by-span basis, each sample of the nonlinearly equalized input field spectrum,

*Ã*

_{eqNL}(

*ω*), is then obtained at the expense of an

_{n}*N*-length fast-Fourier transform (FFT) of the output field, by applying the transfer function

_{FFT}*n*

_{1}and

*n*

_{2}are auxiliary indices used to evaluate the double sum for each block of frequency samples.

In Fig. 1, we present an implementation schematic of the FD-VSNE. The received samples are stored in a *N _{FFT}*-length buffer and transformed into frequency domain using an FFT. Then, three delay lines are used to evaluate the double summation in Eq. (6) and the obtained value is sent to the accumulation block. Each time Eq. (6) is completed, a sample is released to the output frequency buffer. Finally, when the output frequency buffer is full, the time domain nonlinearly equalized signal is obtained by applying an inverse FFT (IFFT).

The original VSTF method, as it is presented in [8], may suffer from energy divergence problems when the input power is relatively high, limiting its practical application. In order to solve this issue, a modified VSTF method is presented in [11], based on a phase correction of the output optical field. A similar strategy can be employed to enhance the applicability of the FD-VSNE, extending the power operating range for equalization. Thereby, after applying the FD-VSNE as in Eq. (6), we may obtain the equalized optical field, *A _{eq}*, as follows,

*A*

_{eqLI}is the linearly equalized field, obtained using any linear equalization method of choice, and

*A*

_{eqNL}is the nonlinearly equalized field, as given by Eq. (6).

## 3. Performance assessment

#### 3.1. System model

A general model of an optical coherent system with single polarization transmission is shown in Fig. 2.

We have used the symmetric SSF method with a step-size of 10 m and a sampling rate of 640 GHz to solve the NLS equation, thereby emulating the propagation of a 20 Gbaud NRZ-QPSK signal over 20 × 80 km of both standard single-mode fiber (SSMF) and non-zero dispersion shifted fiber (NZDSF). No in-line dispersion compensation is used in this work. Optical amplification with ideal gain coefficient and 5 dB noise figure is applied at the end of each fiber span. Laser phase noise has been neglected. The 90 degree optical hybrid and the pair of balanced photodiodes are assumed to perform optical-to-electrical down-conversion without distorting the received signal. The SSMF was set with an attenuation of *α* = 0.2 dB/km, group velocity dispersion of *β*_{2} = −20.4 ps^{2}/km and Kerr coefficient of *γ* = 1.3 W^{−1}km^{−1}. In turn, for the NZDSF we have *α* = 0.2 dB/km, *β*_{2} = −6.0 ps^{2}/km and *γ* = 1.5 W^{−1}km^{−1}. As a figure of merit for compensation performance we use the error vector magnitude (EVM) percentage relatively to the optimal constellation, defined as
$\text{EVM}={\left(\sum {\left|{A}_{eq}-{A}_{tx}\right|}^{2}/\sum {\left|{A}_{tx}\right|}^{2}\right)}^{1/2}$, where *A _{tx}* and

*A*are the transmitted and equalized optical fields, respectively.

_{eq}After coherent detection, the baseband signal is passed through a third-order low-pass Butterworth filter with cutoff frequency at 80% of the symbol rate, in order to filter the out-of-band ASE noise and reduce the aliasing effect due to downsampling. Finally, the digital equalization stage follows. Linear equalization is performed with a frequency-domain chromatic dispersion equalizer (FD-CDE). Full compensation of linear and nonlinear impairments is achieved by BP-SSF and by FD-CDE in conjunction with FD-VSNE. We denote BP-SSF with *N _{steps}* steps per fiber span as BP-SSF

_{Nsteps}.

#### 3.2. Performance comparison with BP-SSF

In order to assess the performance of the third-order truncated FD-VSNE we have performed a set of numerical simulations exploiting the evolution of EVM as a function of the input power in the fiber for both SSMF and NZDSF links. We also provide a direct comparison of performance with the well-known BP-SSF method.

In Fig. 3(a), we present the obtained results for a 20×80 km SSMF link, taking 3 samples per symbol (*N _{sp}* = 3) to perform digital equalization. We may see that the three best performance curves corresponding to BP-SSF

_{8}, BP-SSF

_{64}and CDE+VSNE are completely overlaid. This fact enables us to draw two main conclusions. First, it becomes clear that BP-SSF reaches its performance limit at approximately 8 steps per span. This happens because the low temporal resolution sets an upper limit for compensation performance, above which it becomes useless to increase the spatial resolution. Secondly, we may also observe that the third-order approximation used to derive the FD-VSNE is sufficient to reach the best performance possible at this sampling rate. In turn, using only 2 samples per symbol (see Fig. 3(b)) we observe a significative degradation of performance in BP-SSF. This is due to the generation of high frequencies when the nonlinear operator is applied in time-domain, giving place to aliasing components when the signal is subsequently transposed to frequency-domain, in order to apply the linear step. In fact, this limitation has been already identified in [12], where a BP-SSF modified version is proposed to partially overcome this issue by low-pass filtering the intensity waveform that phase modulates the signal in each nonlinear step.

On its turn, the FD-VSNE avoids this aliasing enhancement phenomenon since it is entirely implemented in frequency-domain. It is important to clarify, though, that aliasing still occurs at the sampling stage, and that the FD-VSNE only avoids extra aliasing generation in the equalization stage. Taking advantage of this fact, FD-VSNE shows a ∼2 dB improvement on the non-linear tolerance over BP-SSF_{64} (considering the 10% EVM reference). Replacing the SSMF link by a NZDSF link (see Figs. 3(c) and 3(d)) we are able to draw similar conclusions, despite of a degradation of performance due to stronger nonlinear effects, which is well visible in the signal constellations taken at 6 dBm. At 2× oversampling, we may see that BP-SSF performance becomes now limited to 1 step per span due to stronger nonlinearities that generate stronger aliasing. In contrast, the FD-VSNE is still able to largely surpass BP-SSF proving to be an effective nonlinear equalization method even under extreme conditions.

#### 3.3. Required bandwidth for nonlinear equalization

As we have seen in the previous subsection, the FD-VSNE algorithm is able to operate with only 2 samples per symbol maintaining high performance. In terms of bandwidth this corresponds to use the information contained in 40 GHz to equalize a 20 Gbaud signal. In this section we aim to perform a more thorough assessment of the effective required bandwidth to apply the FD-VSNE method.

In Fig. 4 we present the evolution of EVM after nonlinear equalization as a function of the 3 dB cutoff frequency in the 3rd order Butterworth LPF. The results obtained at 3× oversampling (see Fig. 4(a)) show similar accuracies between the FD-VSNE and BP-SSF_{8} methods and also a similar evolution with respect to the LPF cutoff frequency. The maximum accuracy is attained when the LPF cutoff frequency is around 18 GHz, corresponding to 90% of the symbol rate. For higher cutoff frequencies the equalization performance tends to degrade due to aliasing enhancement in the sampling stage. On the other hand, narrower filtering degrades performance due to the attenuation of relevant spectral components. The optical spectra of the propagated signal at different filtering and sampling stages are shown in Fig. 5. In Fig. 4(b), we present a similar analysis for 2× oversampling. As expected, since the aliasing effect tends to become stronger for lower sampling rates, the optimum cutoff frequency is now reduced to ≈ 16 GHz, corresponding to 80% of the symbol rate. The only exception is the BP-SSF_{8} curve for an NZDSF transmission link, where the optimum LPF cutoff frequency is found at approximately 60% of the symbol rate. The strong effect of nonlinearities in the NZDSF link exposes the BP-SSF limitation in terms of internal aliasing generation. Although a narrower LPF before equalization can residually counteract this effect, it does not avoid a severe performance degradation relatively to the FD-VSNE method.

Consider now that 3 samples per symbol are available for digital equalization. The FD-VSNE method can then be applied over a total bandwidth of 60 GHz, as shown in Fig. 5. However, the double summation indices in Eq. (6) can be redefined in order to apply the method over a narrower bandwidth, thereby avoiding excessive computational effort. In Fig. 6, we can see more clearly how the spectral support extent used in the FD-VSNE evaluation impacts on its performance. We have gradually reduced the spectral region used for nonlinear equalization from 60 GHz (full spectrum at 3 samples per symbol) down to 25 GHz by adjusting the double summation range of the FD-VSNE.

Although the effect of the LPF cutoff frequency is minor, we may observe that a wider LPF tends to benefit equalization accuracy when FD-VSNE is evaluated over a broader spectral support, but it prejudices performance when a narrower spectral support is used. The EVM degradation between 60 GHz and 40 GHz is barely visible, confirming the FD-VSNE capability to operate at 2 samples per symbol with negligible loss of performance. However, as expected, a severe penalty quickly arises when the spectral support is further reduced, cutting higher power spectral components. In fact, we observe that at least 80–90% of two times the symbol rate is required for an accurate nonlinear equalization.

## 4. Computational effort

Computational effort is a key measure for digital equalization of fiber impairments since real-time implementation is limited by currently available processing speeds. Most of the computations required by the FD-VSNE arise from the double summation in Eq. (6), which can be viewed as dot product of square matrices containing
${N}_{FFT}^{2}$ elements. Although some operations are redundant and therefore can be suppressed, the overall numerical complexity is always proportional to
$O\left({N}_{FFT}^{2}\right)$ per sample. In contrast, the number of complex multiplies per sample required by BP-SSF evolves logarithmically with the FFT block-size, as *O*(*log*_{2}(*N _{FFT}*)). In Fig. 7, we show how the computational effort of both methods evolves with

*N*, where it becomes clear that reduced FFT block-sizes must be considered in order to keep FD-VSNE in a tolerable region of complexity. However, it is well known that there is a limit for reducing the FFT block-size without incurring inter-block interference.

_{FFT}We have used the overlap-save (OS) method to split the received samples into small FFT blocks. In our first try we have applied the OS method only at the link ends, i.e., the received samples are split into blocks and then rejoined when fully compensated (after *N _{span}* fiber spans). However, as we can see in Fig. 8 the accumulated dispersion along the entire link requires large FFT block-sizes (128/256 for the NZDSF link and 512 for the SSMF link). Alternatively, the OS method can also be applied in a span-by-span basis, reducing the accumulated dispersion that needs to be inverted at each step. This way, we are able to reduce the penalty-free minimum FFT block-size to 32 (for the NZDSF link) and 64 (for the SSMF link). In fact, these values can be further reduced to 16 and 32 at the expense of some inter-block interference, maintaining the system bellow the 10

^{−3}BER floor.

This analysis suggests that a span-by-span implementation of the FD-VSNE can be preferable in terms of computational requirements. A sub-span approach can also be considered if a further reduction of FFT block-size is worth the required additional iterations. The high performance of the current version of the method gives margin for further computational savings by neglecting less significant elements in the Volterra kernel. However, this topic is still under investigation.

A key advantage of the FD-VSNE approach over BP-SSF, which is not depicted in Fig. 7, lies in its parallel structure. Most operations within the FD-VSNE method are completely independent between themselves, which eases the application of parallel processing strategies in order to enable real-time implementation. Besides, the FD-VSNE is also independent from linear equalization, enabling to apply nonlinear compensation as an add-on for the equalization subsystem, bringing more flexibility and ease of implementation.

## 5. Conclusion

We have presented an intra-channel nonlinear equalizer based on an analytical closed-form expression for the third-order Volterra nonlinear kernel in frequency domain. Frequency-domain implementation avoids the generation of aliasing phenomena in the equalization stage, rendering nonlinear equalization feasible at 2 samples per symbol. The simulation results for a 20 Gbaud NRZ-QPSK signal propagated over 20 × 80 km of SSMF and NZDSF links show an improvement of approximately 2 dB on the nonlinear tolerance, relatively to an highly iterative symmetric BP-SSF method. With numerical complexity proportional to ${N}_{FFT}^{2}$ per sample, low computational effort requires the use of reduced FFT block-sizes. The independence between operations enhances parallel processing in order to make FD-VSNE feasible in real-time. The FD-VSNE has proved to be a very effective nonlinear equalization algorithm, simultaneously providing a more flexible way of mitigating nonlinearities, since nonlinear equalization can be implemented as an add-on for the equalization subsystem.

## Acknowledgments

The authors would like to thank Marco Forzati and Hou-Man Chin for the fruitful discussions. This work was supported in part by PT Inovação, SA through the projects “PosDig” and “Adapt-Dig”, and by the European Union within the framework of the EURO-FOS project, a Network of Excellence funded by the EU 7th ICT-Framework Programme. Fernando P. Guiomar also acknowledges his PhD grant from FCT (SFRH/BD/74049/2010, “Fundação para a Ciência e a Tecnologia”).

## References and links

**1. **E. Ip and J. M. Kahn, “Compensation of dispersion and nonlinear impairments using digital backpropagation,” J. Lightwave Technol. **26**(20), 3416–3425 (2008). http://www.opticsinfobase.org/jlt/abstract.cfm?URI=jlt-26-20-3416 [CrossRef]

**2. **X. Li, X. Chen, G. Goldfarb, E. Mateo, I. Kim, F. Yaman, and G. Li, “Electronic post-compensation of WDM transmission impairments using coherent detection and digital signal processing,” Opt. Express **16**(2), 880–888 (2008). http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-16-2-880 [CrossRef] [PubMed]

**3. **M. Schetzen, *The Volterra and Wiener Theories of Nonlinear Systems* (Wiley, 1989).

**4. **C. Xia and W. Rosenkranz, “Nonlinear electrical equalization for different modulation formats with optical filtering,” J. Lightwave Technol. **25**(4), 996–1001 (2007). http://www.opticsinfobase.org/jlt/abstract.cfm?URI=jlt-25-4-996 [CrossRef]

**5. **Y. Gao, F. Zgang, L. Dou, Z. Chen, and A. Xu, “Intra-channel nonlinearities mitigation in pseudo-linear coherent QPSK transmission systems via nonlinear electrical equalizer,” Opt. Commun. **282**(12), 2421–2425 (2009). [CrossRef]

**6. **Z. Pan, B. Châtelain, M. Chagnon, and D. V. Plant, “Volterra filtering for nonlinearity impairment mitigation in DP-16QAM and DP-QPSK fiber optic communication systems,” in Optical Fiber Communication Conference, OSA Technical Digest (CD) (Optical Society of America, 2011), paper JThA040. http://www.opticsinfobase.org/abstract.cfm?URI=OFC-2011-JThA040

**7. **R. Weidenfeld, M. Nazarathy, R. Noe, and I. Shpantzer, “Volterra nonlinear compensation of 100G coherent OFDM with baud-rate ADC, tolerable complexity and low intra-channel FWM/XPM error propagation,” in Optical Fiber Communication Conference, OSA Technical Digest (CD) (Optical Society of America, 2010), paper OTuE3. http://www.opticsinfobase.org/abstract.cfm?URI=OFC-2010-OTuE3

**8. **K. Peddanarappagari and M. Brandt-Pearce, “Volterra series transfer function of single-mode fibers,” J. Light-wave Technol. **15**(12), 2232–2241 (1997). [CrossRef]

**9. **J. D. Reis and A. L. Teixeira, “Unveiling nonlinear effects in dense coherent optical WDM systems with Volterra series,” Opt. Express **18**(8), 8660–8670 (2010). http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-18-8-8660 [CrossRef] [PubMed]

**10. **F. P. Guiomar, J. D. Reis, A. L. Teixeira, and A. N. Pinto, “Digital postcompensation using Volterra series transfer function,” IEEE Photon. Technol. Lett. **23**(19), 1412–1414 (2011). [CrossRef]

**11. **B. Xu and M. Brandt-Pearce, “Modified Volterra series transfer function method,” IEEE Photon. Technol. Lett. **14**(1), 47–49 (2002). [CrossRef]

**12. **L. B. Du and A. J. Lowery, “Improved single channel backpropagation for intra-channel fiber nonlinearity compensation in long-haul optical communication systems,” Opt. Express **18**(16), 17075–17088 (2010). http://www.opticsinfobase.org/oe/abstract.cfm?URI=oe-18-16-17075 [CrossRef] [PubMed]