## Abstract

We present a comparative analysis of three popular digital filters for chromatic dispersion compensation: a time-domain least mean square adaptive filter, a time-domain fiber dispersion finite impulse response filter, and a frequency-domain blind look-up filter. The filters are applied to equalize the chromatic dispersion in a 112-Gbit/s non-return-to-zero polarization division multiplexed quadrature phase shift keying transmission system. The characteristics of these filters are compared by evaluating their applicability for different fiber lengths, their usability for dispersion perturbations, and their computational complexity. In addition, the phase noise tolerance of these filters is also analyzed.

©2010 Optical Society of America

## 1. Introduction

The performance of high speed optical fiber transmission systems is severely affected by chromatic dispersion (CD) and polarization mode dispersion (PMD) as well as phase noise (PN) [1,2]. Coherent optical receivers employing digital filters allow significant equalization of chromatic dispersion in the electrical domain, instead of compensation by dispersion compensating fibers (DCFs) or dispersion compensating modules (DCMs) in the optical domain [3–10]. Several digital filters have been applied to compensate the CD in the time and the frequency domain [6–13]. H. Bülow and A. Färbert et al. have reported their CD equalization work using the maximum likelihood sequence estimation (MLSE) method [4,8]. S. J. Savory used a time-domain fiber dispersion finite impulse response (FD-FIR) filter to compensate the CD in 1000 km and 4000 km transmission fibers without using dispersion compensation fibers [10,11]. M. Kuschnerov and F. N. Hauske et al. have used the frequency domain equalizers (FDEs) to compensate the CD in coherent communication systems [12,13]. With the advent of these digital signal processing (DSP) techniques for CD compensation, the dream of optical transmission without any DCFs appears to become true. However, these reported digital filters have their own characteristics, and it is necessary to clearly understand the limitations of their utilizations.

In this paper, the time-domain least mean square (LMS) adaptive filter is demonstrated to compensate CD in a 112-Gbit/s non-return-to-zero polarization division multiplexed quadrature phase shift keying (NRZ-PDM-QPSK) coherent optical transmission system, which is realized in the VPI simulation platform [14]. Comparatively, the time-domain FD-FIR filter and the frequency-domain blind look-up (BLU) filter are also applied for CD equalization in this transmission system [10–13]. The principle of the LMS algorithm and the influence of the step size on the tap weights convergence in the adaptive filter are analyzed, and the structure of the FD-FIR filter and the BLU filter are also studied. Meanwhile, the effects of tap number in the LMS and the FD-FIR filters, as well as the FFT-size and overlap samples in the BLU filter are investigated for equalizing the fiber dispersion. We also present the detailed investigation in CD compensation using the three digital filters for different fiber lengths and illustrate their performance in the existence of dispersion perturbation due to the temperature variation. It is found that the time-domain FD-FIR filter does not work well when the fiber length is shorter than a certain distance. The corresponding reason and an improvement method are explained and developed. The optimum selection of tap number in FD-FIR filter are analyzed and discussed. The computational complexity of the three algorithms is also compared. Furthermore, the phase noise compensation (PNC) after the CD equalization using the three filters is also analyzed. Based on this work, the characteristics of the three digital filters are analyzed and illustrated, and we can find the optimum approach and effective usage for each digital filter in CD equalization.

The characteristics of a normalized LMS (NLMS) adaptive filter were preliminarily analyzed by comparing with the FD-FIR filter and the blind look-up filter in our previous work [15]. However, the NLMS adaptive filter, acting as a band-limit filter, can meliorate the back-to-back result in the coherent transmission system, and make it converge to a back-to-back result with a 3-tap NLMS post-filter. In order to be comparable to the behavior of NLMS filter, the back-to-back result and the CD equalization performance of FD-FIR filter and BLU filter are processed by a post-added 3-tap NLMS equalizer. Hence, some peculiar characteristics of the time-domain adaptive filter and fixed FD-FIR filter as well as the frequency-domain BLU filter are concealed and obscured. While in this paper, the optical and electrical filters in the coherent transmission system have been designated with the optimal bandwidths. Meanwhile, the LMS adaptive filter which we employ here can guarantee the back-to-back results to be identical with and without a 3-tap LMS post-equalizer. Therefore, the true features of the LMS adaptive filter and FD-FIR filter as well as the blind look-up filter can be investigated and compared directly.

## 2. Digital signal processing modules in coherent receiver

The schematic of the DSP modules in digital coherent receiver for linear fiber impairments is illustrated in Fig. 1 . The bit-error-rate (BER) is calculated through digital signal processing in five steps: CD compensation using FIR filters, adaptive PMD and polarization rotation (PR) equalization, carrier phase estimation (PE) for phase noise compensation, adaptive equalization for residual dispersion and symbol identification.

Generally, the LMS-based adaptive equalizer can compensate the carrier phase noise and the residual dispersion simultaneously. However, it only works satisfactorily when the phase fluctuation is small enough. The phase correlation between symbols with a long delay interval will fade out due to the large phase fluctuations occurring within the filter delay time. When we employ a single high-order adaptive filter for equalizing both PN and residual CD, the performance of the adaptive filter will degrade with the increment of the filter delay. Therefore, the function is realized by two steps, a one-tap NLMS filter for phase estimation and a multiple-tap LMS filter for residual dispersion equalization [16].

Our work focuses on analyzing and comparing the three DSP techniques for CD equalization. Meanwhile, the adaptive mitigation for PMD and polarization rotation is theoretically analyzed, and the phase noise compensation using an NLMS filter is numerically simulated. The necessity of the adaptive filter for residual dispersion is also discussed, when the fixed CD equalizer is used.

The current investigation does not consider the fiber nonlinear effects, since the DSP compensation for nonlinear effects represents a separate and highly involved study area which is beyond the scope of this paper.

## 3. Principle and structures of three digital filters for CD equalization

#### 3.1 Least mean square adaptive filter

The LMS filter employs an iterative algorithm that incorporates successive corrections to weights vector in the negative direction of the gradient vector which eventually leads to a minimum mean square error [17]. The principle of LMS filter is given by the following equations:

*n*represents the number of sample sequence, $\overrightarrow{w}\left(n\right)$ is the complex tap weights vector, ${\overrightarrow{w}}^{H}\left(n\right)$ is the Hermitian transform of $\overrightarrow{w}\left(n\right)$, $d\left(n\right)$ is the desired symbol, which corresponds to one case of the vector $\left[\begin{array}{cccc}1+i& 1-i& -1-i& -1+i\end{array}\right]$ for the QPSK coherent transmission system, $e\left(n\right)$ represents the estimation error between the output signal and the desired symbol, ${e}^{\ast}\left(n\right)$ is the conjugation of $e\left(n\right)$, and

*μ*is a key real coefficient called step size. The tap weights vector $\overrightarrow{w}\left(n\right)$ is updated in a symbol-by-symbol iterative manner, and achieves their convergence when $e\left(n\right)$ approaches zero.

In order to guarantee the convergence of tap weights vector $\overrightarrow{w}\left(n\right)$, the step size *μ* needs to satisfy the condition of $0<\mu <1/{\lambda}_{\mathrm{max}}$, where ${\lambda}_{\mathrm{max}}$ is the largest eigenvalue of the correlation matrix $R=\overrightarrow{x}\left(n\right){\overrightarrow{x}}^{H}\left(n\right)$ [17]. If the step size *μ* is chosen to be very small, then the algorithm converges very slowly. A large value of *μ* could lead to a faster convergence, but the algorithm will be less stable and safe, because sometimes the step size *μ* may be larger than $1/{\lambda}_{\mathrm{max}}$. An efficient method to designate the step size *μ* is to make it change with the time-dependent largest eigenvalue ${\lambda}_{\mathrm{max}}$, and this will lead to a fast and stable convergence, which refers to the variable-step-size LMS algorithm [17]. However, it is required to check the value of ${\lambda}_{\mathrm{max}}$ in each step, and so it will be cumbersome and computationally inefficient. In our simulation, we have chosen the step size to be a fixed small value, which can guarantee the convergence of the LMS filter.

The tap weights in LMS adaptive equalizer for 20 km fiber CD compensation is shown in Fig. 2 . The convergence for 9 tap weights in the LMS filter with step size equal to 0.1 is shown in Fig. 2(a), and we can find the tap weights obtain their convergence after about 5000 iterations. Note that the convergence speed of the LMS algorithm will slightly slow down with the increment of the tap number for the same fiber dispersion. About 8000 iterations are required to obtain the guaranteed convergence when 21 taps are used in the LMS filter for equalizing 20 km fiber dispersion. The magnitudes of converged tap weights are shown in Fig. 2(b), and it can be found that the central tap weights take more dominant roles. For a fixed fiber dispersion, the tap weights in LMS adaptive filter approach zero, when the corresponding taps order exceeds a certain value, and this value indicates the least required taps number for compensating the CD effectively, which is 9 taps for 20 km fiber shown in this figure. This illustrates the optimization characteristic of the LMS adaptive algorithm.

#### 3.2 Fiber dispersion FIR filter

Compared with the iteratively updated LMS filter, the tap weights in FD-FIR filter have a relatively simple specification [10,11], the tap weight ${a}_{k}$ in FD-FIR filter is given by the following equations:

*D*is the fiber chromatic dispersion coefficient,

*λ*is the central wavelength of the transmitted optical wave,

*z*is the fiber length in the transmission channel,

*T*is the sampling period, ${N}^{A}$ is the required maximum tap number for compensating the fiber dispersion, and $\lfloor x\rfloor $ denotes the nearest integer less than

*x*.

The tap weights of FD-FIR filter according to Eq. (4) for 20 km fiber (D = 16 ps/nm/km) are shown in Fig. 3 , which is the same with the one illustrated in our previous work [15]. For a fixed fiber dispersion, the magnitudes of tap weights in FD-FIR filter are constant, whereas the real and the imaginary parts vary periodically. The FD-FIR filter does not show the optimization characteristic, because excessive tap weights do not approach zero without window truncation. It is noted that the tap weights in Fig. 3 should be truncated properly for practical application. To avoid aliasing, the Nyquist frequency is chosen as the boundary of the frequency window in FD-FIR filter, namely the above tap weights are convolved with a sinc function in the time domain, and the redundant tap weights can be eliminated effectively.

To avoid the aliasing, the pass-band of the filter needs to be lower than the Nyquist frequency [10,11], and so the continuous time window of the FD-FIR filter is limited by the following inequality:

where the variables description is the same as the description in Eq. (4) and Eq. (5).So the continuous time window ${T}_{W}^{A}$ in the FD-FIR filter can be obtained from Eq. (6) as

The required tap weights number ${N}^{A}$ is calculated from the continuous time window quantized by the sampling period *T* as

Once the time window is determined, we can employ a Kaiser window to truncate the FD-FIR filter in the time domain for practical application [10,11].

The above analysis of window width in FD-FIR is indeed rational and correct by using the anti-aliasing approach [10,11]. From a physical aspect, it will be also reasonable to determine the filter length based on the broadening of an pulse propagating in the dispersive fiber channel [2,18]. Assuming the signal is a Gaussian pulse, the width of the filter can be calculated according to the broadened pulse duration [2,18], which is given by

where the parameter descriptions are the same with Eq. (4) and Eq. (5). The corresponding tap number is obtained as*x*.

The tap numbers for different fiber length calculated according to pulse broadening are illustrated in Table 1 , and compared with the tap numbers calculated from the analysis of anti-aliasing.

We find that most of the tap numbers calculated from pulse broadening are around 60% of the value calculated from the anti-aliasing analysis, which is consistent with the empirical factor in the reported results [10]. This number could be considered as a lower bound on the required number of taps for an effective CD compensation.

#### 3.3 Blind look-up filter

The function of the blind look-up filter comprises two steps [12,13]. The first step is to utilize a blind adaptive FDE to estimate the accumulated chromatic dispersion in the transmission system, which has already been introduced and analyzed previously [15]. The second step is to achieve the CD equalization using an overlap-save fast Fourier transform (FFT) method [12,13,19–22], assuming the amount of dispersion has been estimated precisely. The structure of the overlap-save BLU equalizer is illustrated in Fig. 4 . The time domain received digital signals are firstly divided into several sequence blocks with an overlap, where the block length is called the FFT-size. Then the sequence in each block are transformed into frequency domain data by the FFT operation, and afterwards multiplied by the inverse transfer function of the dispersive channel in the frequency domain. Furthermore, the data sequences are transformed into time domain signals by the inverse FFT (IFFT) operation, and finally the processed sequence blocks are combined together by the superposition with overlap samples. The recomposed data are the equalized output signal sequence. The zero-padded overlap-add method, which is another FFT convolution algorithm, can also be used for equalizing the chromatic dispersion in frequency domain [19,20].

It should be noted that the value of overlap is the pivotal parameter determined by the dispersion to be equalized [22], and the FFT-size is configurable provided it is larger than the overlap. For a proper overlap value, a large FFT-size may be more efficient, however, it will cost more memory resource of the hardware. In our work, the FFT-size is designated as the double of the overlap, so that BLU filter can be applied conveniently for equalizing different fiber dispersion only by determining the required FFT-size.

The transfer function of the frequency domain equalizer is given by

The pass-band of the BLU filter is determined by the Nyquist frequency, which is also given by

where ${\omega}_{N}$ is the Nyquist angular frequency of the system.The tap weights specification in the blind look-up filter can be expressed as

*N*is the FFT-size of the frequency domain equalizer.

_{FFT}In terms of the above analysis, the angular-frequency pass-band (AFPB) of BLU filter ${\omega}_{BLU}^{PB}=2{\omega}_{N}$ is a fixed value, whereas the AFPB of FD-FIR filter ${\omega}_{FD-FIR}^{PB}=2\cdot \frac{\pi c}{D{\lambda}^{2}z}\cdot {N}^{A}\cdot T$ varies with the tap number ${N}^{A}$. This is the fundamental reason for the different CD equalization performance between the FD-FIR filter and the BLU filter.

## 4. Principle of equalization for polarization dependent impairments and phase noise

#### 4.1 Adaptive PMD and polarization rotation equalization

The influence of PMD and polarization fluctuation can be compensated adaptively by the decision-directed LMS filter [17,23], which is expressed as the following equations:

#### 4.2 Normalized LMS filter for phase estimation

The one-tap NLMS filter can be employed for phase estimation [16,17], of which the tap weight is expressed as

## 5. Simulation investigation of PDM-QPSK transmission system

The setup of the 112-Gbit/s NRZ-PDM-QPSK coherent transmission system established in the VPI simulation platform is illustrated in Fig. 5
. The data sequence output from the four 28-Gbit/s pseudo random bit sequence (PRBS) generators are modulated into two orthogonally polarized NRZ-QPSK optical signals by the two Mach-Zehnder modulators. Then the orthogonally polarized signals are integrated into one fiber channel by a polarization beam combiner (PBC) to form the 112-Gbit/s NRZ-PDM-QPSK optical signal. Using a local oscillator (LO) in the coherent receiver, the received optical signals are mixed with the LO laser to be transformed into four electrical signals by the photodiodes. Then they are digitalized by the 8-bit analog-to-digital convertors (ADCs) at twice the symbol rate [24]. The sampled signals are processed by the digital equalizer, and the BER is then estimated from the data sequence of 2^{16} bits. The central wavelength of the transmitter laser and the LO laser are both 1553.6 nm. The standard single mode fibers (SSMFs) with the CD coefficient equal to 16 ps/nm/km are employed in all the simulation work.

Here we mainly concentrate our work on the CD compensation methods in DSP techniques, and so we neglected the influences of fiber attenuation, polarization mode dispersion and nonlinear effects in the comparison of the three filters. The PMD and polarization rotation equalization could be realized by employing the adaptive LMS filter as we mentioned above. The carrier phase estimation can be implemented by using a one-tap NLMS adaptive filter [16], which will be analyzed in our simulations afterwards.

With the increment of launched optical power, the fiber nonlinearities such as self-phase modulation (SPM), cross-phase modulation (XPM) and four-wave mixing (FWM) need to be considered in the long-haul wavelength-division multiplexing (WDM) transmission systems [25]. The fiber nonlinear impairments can be mitigated and compensated by using the digital backward propagation methods based on solving the nonlinear Schrodinger (NLS) equation and the Manakov equation [26,27].

## 6. Simulation results

#### 6.1 Static chromatic dispersion equalization

The CD compensation results using three digital filters are illustrated in Fig. 6
. Figure 6(a) indicates the CD equalization with 9 taps for 20 km fiber and 243 taps for 600 km fiber using the LMS and the FD-FIR filters, as well as 16 FFT-size (8 overlap) for 20 km fiber and 512 FFT-size (256 overlap) for 600 km fiber using the BLU filter. Obviously, the FD-FIR filter is not able to compensate the CD in 20 km fiber entirely. About 3 dB optical signal-to-noise ratio (OSNR) penalty from the back-to-back result at BER equal to 10^{−3} can be observed. This phenomenon contradicts with our previous result [15], because the additional 3-taps NLMS equalizer in previous simulation has mitigated the CD equalization penalty of the FD-FIR filter for short fibers. Here we can observe and analyze the true features of the three digital filters without any interference. Then we investigate the CD compensation for different fiber lengths using the three filters, which are shown in Fig. 6(b). It can be found that the LMS filter and the BLU filter show the same acceptable performance for different fiber lengths, while the FD-FIR filter will not behave satisfactorily until the fiber length exceeds 320 km.

The CD equalization for 20 km and 600 km fibers using the LMS filter and the FD-FIR filter with different number of taps are shown in Fig. 7
. Due to the optimum characteristic of LMS algorithm, the LMS filter has a slight improvement with the increment of tap number. However, the performance of the FD-FIR filter will degrade, when the tap number increases and exceeds the required tap number in Eq. (5). It is because the redundant taps will lead to the pass-band of the filter exceeding the Nyquist frequency, which will further result in the aliasing phenomenon. Here we should note that the redundant taps refers to no consideration of window truncation. We also find in Fig. 7(a) that the FD-FIR filter does not achieve a satisfactory CD equalization performance for 20 km fiber by using either the tap number (N^{A} = 9) analyzed by anti-aliasing or the tap number (N^{P} = 7) determined by pulse broadening.

From the above description, the FD-FIR filter does not achieve an acceptable CD equalization performance for short distance fibers, but it can work well for long fibers. When we use a series of delayed taps to approximate the filter time window ${T}_{W}^{A}$, the digitalized discrete time window ${T}_{N}^{A}={N}^{A}\cdot T$ could not attain exactly the same value as the continuous time window ${T}_{W}^{A}$, which is illustrated in Fig. 8 . The malfunction of FD-FIR filter for short fibers arises from this reason, and now we provide a more detailed explanation.

We calculate the relative error *p* of time window to evaluate the precision of time window approximation, which is given by

According to Eq. (7), a short fiber will have a relative small time window to keep the signal bandwidth to be lower than Nyquist frequency to avoid the aliasing phenomenon. However, such a small time window is not easy to be digitalized accurately with a fixed sampling period *T*. In order to broaden the time window, we need to raise the Nyquist frequency correspondingly. The Nyquist frequency is defined as half of the sampling frequency of the system, and this means we need to increase the sampling rate in the ADC modules. With sampling rate being increased, the Nyquist frequency are also raised, meanwhile, the sample period *T* is reduced, which allows the broadened continuous time window to be digitalized more precisely.

The relative errors of time window for different fiber length with different sampling rate are shown in Table 2 , where the positive time error means the aliasing occurring. We could find the relative error of time window for 20 km is reduced obviously with the sampling rate changing from 2 samples per symbol (Sa/Sy) to 8 Sa/Sy. Furthermore, the time error for 20 km fiber with 8 Sa/Sy is equal to the time error for 320 km fiber with 2 Sa/Sy, which is the acceptable fiber length limitation shown in Fig. 6(b). So we consider this method could have significant improving effects on the FD-FIR filter equalization performance for short fibers.

The improved method for 20 km fiber CD compensation using FD-FIR filter with different sampling rate is shown in Fig. 9
. We find in Fig. 9(a) that the CD equalization performance shows an obvious improvement with the increment of the sampling rate, and the FD-FIR filter can equalize the CD in 20 km fiber entirely with 8 sampling points per symbol. The performance of BER with normalized time window (T_{N}
^{A}/T_{W}
^{A}) using FD-FIR filter is shown in Fig. 9(b), where a significant improvement can also be found. Meanwhile, we find that the FD-FIR filter performs better when the value of T_{N}
^{A}/T_{W}
^{A} is around 1.0, which is consistent with our preceding analysis.

It can also be found both in Fig. 7 and Fig. 9(b) that the FD-FIR filter does not necessarily obtain the best CD equalization performance when using the maximum tap number calculated in Eq. (5). In Fig. 7, the calculated tap numbers from Eq. (5) are 9 taps for 20 km fiber and 243 taps for 600 km fiber. For a fixed sampling rate, two main aspects need to be taken into consideration in selection of the practical tap number in FD-FIR filter. One is to keep the system bandwidth lower than Nyquist frequency to avoid aliasing, which is the smaller the better, and the other is to equalize the CD influence in the received signal, which is the larger the better. To obtain the best BER performance, we need to incorporate the two factors, and the optimum tap number is between the maximum number calculated from Eq. (5) and the minimum number calculated from Eq. (10).

Although this improved method increases the necessary tap number in the FD-FIR filter and the required sampling rate in the coherent transmission system, which may not be suitable for very long distance fibers and high speed communication systems, we could improve the FD-FIR filter to compensate the CD in short distance fibers significantly by increasing the ADC sampling rate. Meanwhile, we could also put an adaptive post-filter after the FD-FIR filter to compensate the penalty in CD equalization for short distance fibers. However, here we mainly concentrate on analyzing and comparing the inherent characteristics of the three digital filters in CD compensation. Therefore, we hope to find the reason and improvement method in terms of the intrinsic properties of the FD-FIR filter. The fiber lengths are usually no less than hundreds of kilometers in practical transmission systems, therefore, the FD-FIR filter can be applied reasonably for the CD equalization in the systems with 2 Sa/Sy ADC sampling rate.

The performance of CD equalization using blind look-up filter with different FFT-size and overlap are illustrated in Fig. 10 . With the increment of the FFT-size, the BLU filter can show stable and converged acceptable performance for a certain fiber length. Note that the FFT-size is designated as twice of the overlap in Fig. 10(a). The critical FFT-size values (16 FFT-size for 20 km fiber and 32 FFT-size for 40 km fiber), actually indicate the required minimum overlap value which are 8 overlap samples for 20 km fiber and 16 overlap samples for 40 km fiber. It is also found that the performance of the BLU filter reveals dramatic degradation as the overlap size decreases. We have demonstrated that the overlap is the pivotal parameter in the BLU filter, and the FFT-size is not necessarily designated as double of the overlap. Figure 10(b) illustrates that with a fixed FFT-size (4096 samples) the BLU filter could work well for 4000 km fiber, provided the overlap is larger than 1152 samples (1152 = 0.28125 × 4096), which indicates the required minimum overlap for 4000 km fiber.

Actually, the minimum overlap in BLU filter can be calculated from the pulse width broadening [18,22], as illustrated in Table 3 . We observe that the calculated overlap values are in accord with the values determined by the CD equalization performance in our work.

#### 6.2 Dynamic chromatic dispersion equalization

The performance of the three filters for 2000 km fiber CD compensation in a dynamic dispersive system with an approximate $\pm 0.2\%$ chromatic dispersion variation (due to the change of temperature) are shown in Fig. 11 . We can see that except the LMS adaptive filter, the fixed FD-FIR and BLU filters will degrade due to the existence of the small dispersion variation. So it is necessary to add a post LMS adaptive filter (with 5 ~7 taps) after the fixed CD equalizer to compensate the residual dispersion in the practical coherent optical transmission systems. We have verified that the equalization penalty of FD-FIR and BLU filters in this 2000 km fiber transmission system with $\pm 0.2\%$ CD perturbation can be entirely compensated by a 5-tap LMS adaptive filter.

#### 6.3 Computational complexity of the three filters

We have also investigated the complexity of the three digital filters, and we evaluated the computational complexity in terms of complex multiplications per transmitted bit (Mults/bit) [28,29], which could be calculated as the following equations:

*C*,

_{LMS}*C*and

_{FD-FIR}*C*are the computational complexity of the three filters respectively,

_{BLU}*L*is the filter length for CD equalization,

_{CD}*M*is the number of points in signal constellation,

*n*is the oversampling ratio in samples per symbol,

_{SC}*N*is the length of FFT operation in the BLU filter. In the 112-Gbit/s NRZ-PDM-QPSK transmission system with a 2 Sa/Sy sampling rate at the ADC modules, we have

_{FFT}*M*= 4 and

*n*= 2 in the above formulas.

_{SC}The complexity of the three filters for different fiber dispersion is illustrated in Table 4 . We find that frequency-domain BLU filter is much more efficient than the time-domain LMS and FD-FIR filters when the accumulated chromatic dispersion is large (usually larger than 3000 ps/nm). Moreover, the LMS filter needs higher computational complexity than the FD-FIR filter because it consumes the additional multiplications operation to update the tap weights per symbol duration.

#### 6.4 Phase noise compensation

An example of phase noise compensation using the one-tap NLMS filter is illustrated in Fig. 12
. The simulation setup is the same as Fig. 5, where the linewidths of transmitter and LO lasers are both set to be 500 kHz. The figure illustrates the BER characteristics in phase estimation after the 600 km fiber CD equalization. It can be found that the LMS filter is more tolerant of phase noise than the FD-FIR and the BLU filters. Without phase noise compensation, the BER floor of LMS filter is around 10^{−3}, which is much better than FD-FIR and BLU filters (above 10^{−2}). Meanwhile, we find the phase noise compensation using the one-tap NLMS equalizer can achieve a satisfactory performance with a small penalty from the back-to-back result for all the three filters. More detailed investigation of the phase noise influence is in progress and will be presented in a separate publication.

## 7. Conclusions

To demonstrate the fundamental features of the time-domain adaptive and fixed filters as well as the frequency-domain equalizer, the LMS filter, the FD-FIR filter and the blind look-up filter are applied to compensate the CD in a 112-Gbit/s NRZ-PDM-QPSK coherent optical transmission system. Compared to our previous work, they are analyzed by evaluating their dynamic range and applicability in CD equalization without using an adaptive 3-tap post-equalizer, and some peculiar characteristics of these filters are consequently revealed. In terms of safety and stability, the LMS adaptive filter shows the best performance in CD equalization. However, it requires slow iteration for guaranteed convergence, and also the tap weights update increases the computational complexity of this algorithm. The LMS filter is especially appropriate for the practical communication systems in which the undefined factors and sudden perturbations probably exist. With respect of equalizer specification, the FD-FIR filter affords the simplest analytical tap weights specification. However, it does not show acceptable performance for short distance fibers. This can be rectified and improved by multiplying the ADC sampling rate or adding an adaptive LMS post-filter. In a practical system, the FD-FIR filter concatenate with an LMS filter (few taps) is usually used due to the lower complexity than a single LMS dispersion equalizer. From the aspect of speed and efficiency, the blind look-up filter will be faster and much more computationally efficient, especial for large fiber dispersion. However, its performance will degrade dramatically if the overlap in the equalizer does not reach the required minimum overlap size. It has been also demonstrated in our investigation that the LMS filter is more tolerant to the chromatic dispersion perturbation and the carrier phase noise than the other two filters.

Besides the chromatic dispersion equalization, our analysis and comparison does not take into account the influences of PMD and fiber nonlinearities, only the carrier phase estimation is investigated. Future efforts should incorporate comparing the CD equalization performance of the three methods in the return-to-zero (RZ) and the NRZ polarization division multiplexed QPSK transmission systems, as well as compensating the PMD, phase noise and the fiber nonlinearities in such coherent systems. Furthermore, these three methods should be studied for the utilization in the quadrature amplitude modulation (QAM) coherent systems.

## References and links

**1. **P. S. Henry, “Lightwave primer,” IEEE J. Quantum Electron. **21**(12), 1862–1879 (1985). [CrossRef]

**2. **G. P. Agrawal, *Fiber-optic communication systems 3rd Edition* (John Wiley & Sons, Inc., 2002), Chap. 2.

**3. **J. G. Proakis, *Digital communications 5th Edition* (McGraw-Hill Companies, Inc., 2008), Chap.10.

**4. **H. Bulow, F. Buchali, and A. Klekamp, “Electronic dispersion compensation,” J. Lightwave Technol. **26**(1), 158–167 (2008). [CrossRef]

**5. **M. G. Taylor, “Coherent detection method using DSP for demodulation of signal and subsequent equalization of propagation impairments,” IEEE Photon. Technol. Lett. **16**(2), 674–676 (2004). [CrossRef]

**6. **Y. Han and G. Li, “Coherent optical communication using polarization multiple-input-multiple-output,” Opt. Express **13**(19), 7527–7534 (2005). [CrossRef] [PubMed]

**7. **E. Ip and J. M. Kahn, “Digital equalization of chromatic dispersion and polarization mode dispersion,” J. Lightwave Technol. **25**(8), 2033–2043 (2007). [CrossRef]

**8. **A. Färbert, S. Langenbach, N. Stojanovic, C. Dorschky, T. Kupfer, C. Schulien, J. P. Elbers, H. Wernz, H. Griesser, and C. Glingener, “Performance of a 10.7 Gb/s receiver with digital equaliser using maximum likelihood sequence estimation,” in *Proceeding of IEEE European Conference on Optical Communication* (Stockholm, Sweden, 2004), paper Th4.1.5.

**9. **G. Goldfarb and G. Li, “Chromatic dispersion compensation using digital IIR filtering with coherent detection,” IEEE Photon. Technol. Lett. **19**(13), 969–971 (2007). [CrossRef]

**10. **S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Express **16**(2), 804–817 (2008). [CrossRef] [PubMed]

**11. **S. J. Savory, “Compensation of fibre impairments in digital coherent systems,” in *Proceeding of IEEE European Conference on Optical Communication* (Brussels, Belgium, 2008), paper Mo.3.D.1.

**12. **M. Kuschnerov, F. N. Hauske, K. Piyawanno, B. Spinnler, A. Napoli, and B. Lankl, “Adaptive chromatic dispersion equalization for non-dispersion managed coherent systems,” in *Proceeding of IEEE Conference on Optical Fiber Communication* (San Diego, California, 2009), paper OMT1.

**13. **M. Kuschnerov, F. N. Hauske, K. Piyawanno, B. Spinnler, M. S. Alfiad, A. Napoli, and B. Lankl, “DSP for coherent single-carrier receivers,” J. Lightwave Technol. **27**(16), 3614–3622 (2009). [CrossRef]

**15. **T. Xu, G. Jacobsen, S. Popov, J. Li, K. Wang, and A. T. Friberg, “Normalized LMS digital filter for chromatic dispersion equalization in 112-Gbit/s PDM-QPSK coherent optical transmission system,” Opt. Commun. **283**(6), 963–967 (2010). [CrossRef]

**16. **Y. Mori, C. Zhang, K. Igarashi, K. Katoh, and K. Kikuchi, “Unrepeated 200-km transmission of 40-Gbit/s 16-QAM signals using digital coherent receiver,” Opt. Express **17**(3), 1435–1441 (2009). [CrossRef] [PubMed]

**17. **S. Haykin, *Adaptive filter theory 4th Edition* (Prentice Hall, 2001).

**18. **M. Khafaji, H. Gustat, F. Ellinger, and C. Scheytt, “General time-domain representation of chromatic dispersion in single-mode fibers,” IEEE Photon. Technol. Lett. **22**, 314–316 (2010). [CrossRef]

**19. **A. V. Oppenheim, R. W. Schafer, and R. John, Buck, *Discrete-time signal processing 2nd Edition* (Prentice Hall, 1999).

**20. **J. G. Proakis, and D. G. Manolakis, *Digital signal processing 4th Edition* (Prentice Hall, 2006).

**21. **R. Kudo, T. Kobayashi, K. Ishihara, Y. Takatori, A. Sano, E. Yamada, H. Masuda, Y. Miyamoto, and M. Mizoguchi, “Two-stage overlap frequency domain equalization for long-haul optical systems,” in *Proceeding of IEEE Conference on Optical Fiber Communication* (San Diego, California, 2009), paper OMT3.

**22. **R. Kudo, T. Kobayashi, K. Ishihara, Y. Takatori, A. Sano, and Y. Miyamoto, “Coherent optical single carrier transmission using overlap frequency domain equalization for long-haul optical systems,” J. Lightwave Technol. **27**(16), 3721–3728 (2009). [CrossRef]

**23. **C. R. S. Fludger, T. Duthel, D. van den Borne, C. Schulien, E. D. Schmidt, T. Wuth, J. Geyer, E. De Man, G. D. Khoe, and H. de Waardt, “Coherent equalization and POLMUX-RZ-DQPSK for robust 100-GE transmission,” J. Lightwave Technol. **26**(1), 64–72 (2008). [CrossRef]

**24. **S. J. Savory, “Digital signal processing options in long haul transmission,” in *Proceeding of IEEE Conference on Optical Fiber Communication* (San Diego, California, 2008), paper OTuO3.

**25. **K. Kikuchi, and S. Y. Kim, “Investigation of nonlinear impairment effects on optical quadrature phase-shift keying signals transmitted through a long-haul system,” in *Proceedings of IEEE Laser and Electro-Optics Society Summer Topical Meetings* (Acapulco, Mexico, 2008), pp. 131–132.

**26. **G. Goldfarb, M. G. Taylor, and G. Li, “Experimental demonstration of fiber impairment compensation using the split-step finite-impulse-response filtering method,” IEEE Photon. Technol. Lett. **20**(22), 1887–1889 (2008). [CrossRef]

**27. **F. Yaman and G. Li, “Nonlinear impairment compensation for polarization-division multiplexed WDM transmission using digital backward propagation,” IEEE Photonics J. **1**(2), 144–152 (2009). [CrossRef]

**28. **B. Spinnler, F. N. Hauske, and M. Kuschnerov, “Adaptive equalizer complexity in coherent optical receivers,” in *Proceeding of IEEE European Conference on Optical Communication* (Brussels, Belgium, 2008), paper We.2.E.4.

**29. **B. Spinnler, “Complexity of algorithms for digital coherent receivers,” in *Proceeding of IEEE European Conference on Optical Communication* (Vienna, Austria, 2009), paper 7.3.6.