## Abstract

We experimentally characterize a maximum likelihood sequence estimation (MSLE) based receiver’s tolerance to first- and all-order polarization mode dispersion (PMD). We show that the response of the MLSE receiver to first-order PMD can be characterized in two ways depending on the differential group delay (DGD). In addition we show that first-order PMD-induced system penalties dominate those from high-order PMD. High-order PMD induces a large system penalty only when the first-order penalty is small, or the DGD exceeds a bit period.

© 2007 Optical Society of America

## 1. Introduction

The application of maximum likelihood sequence estimation (MLSE) to optical communication networks has received a lot of attention over the past three years [1–4]. A number of studies have looked at the specific application of MLSE-based receivers to systems with polarization mode dispersion (PMD) [1, 5–8]. The critical outcome of PMD is inter-symbol interference (ISI) that induces a system penalty [9]. MLSE receivers reduce the effect of PMD-induced ISI by maximizing the probability of correctly receiving a bit sequence [2].

Most previous studies of MLSE receiver tolerance to PMD have focused on the effects of first order PMD for the worst launch state, given by equal power splitting ratio between the two principal states of polarization (PSP) [1,5,7,9]. Studies that have investigated different launch states of polarization (SOP) have reported results from only a limited range of differential group delay (DGD) values [6,8]. In this paper, we completely characterize an MLSE receiver’s tolerance to first-order PMD over all launch SOPs and for DGD values up to 1.65 bit periods. We show that the first-order approximation for the PMD-induced penalty holds over a range of DGD values less than the bit period. Beyond this range we find that the MLSE receiver has a different response to PMD. To account for this we suggest different ways of characterizing MLSE receiver tolerance to PMD.

The full characterization of the MLSE receiver’s tolerance to first-order PMD enabled us to investigate the all-order PMD response in detail. Our results show that when the first-order PMD-induced penalty is small, high-order PMD causes an additional penalty. As the first-order penalty increases the effects of high-order PMD become less pronounced.

## 2. Experimental procedure

To quantify the PMD tolerance of our MLSE receiver, we measured the optical signal-to-noise ratio (OSNR) penalty, using the experimental setup shown in Fig.1. Using a commercial transponder, we encoded a 2^{31}-1 pseudo-random bit sequence (PRBS) payload with G.709 framing and forward error correction (FEC) into a single non-return to zero (NRZ) channel modulated at 11.003 Gbit/s. The channel was noise-loaded with amplified spontaneous emission (ASE), shaped by a 0.6 nm filter centered on the channel wavelength. The degree of polarization of the ASE was <5% across the C band. We used a 10% tap to measure the OSNR of the channel using an optical spectrum analyzer (OSA), avoiding any polarization sensitivity from the 3dB coupler. A polarization controller (PC) set the launch state of polarization (SOP) into the PMD source, and the polarization properties of the channel at the output of the PMD source were used to extract the power splitting ratio. The pre-FEC Bit-Error Ratio (BER) was measured for various ASE levels to determine the OSNR penalty at a pre-FEC BER=10^{-3}.

We investigated 41 optical channels with 100 GHz spacing from 192 THz to 196 THz, and used three different PMD sources. A variable first order PMD emulator (JDS Fitel PE4) generated controllable DGD, and in this paper we report values up to 150 ps. The second source was a 12km fiber, compensated for chromatic dispersion, which exhibited all orders of PMD. The third source consisted of the 12km fiber concatenated to a 60ps polarization maintaining fiber, to increase the mean DGD. The PMD of the all-order sources was characterized using the polarization controller, polarimeter, and CW probe laser shown in Fig.1. The probe laser was scanned across a 40 GHz band centered on the channel frequency, and Müller Matrix Method was used to calculate the DGD and PSP across each channel [10]. Table 1 summarizes the mean instantaneous first and second order PMD components, calculated using PMD data that was measured at the center frequency of the 41 channels investigated.

When the variable emulator was used as a PMD source, the launch SOP was held constant as the DGD was increased in 5 ps steps from 0 ps to 150 ps. When all-order sources were used, the channel’s launch SOP was stepped through a number of pre-programmed states that gave full coverage of the Poincaré sphere. For each launch SOP or DGD setting, ASE was increased in 0.5 dB steps and the BER and OSNR were recorded. The OSNR penalty at a pre-FEC BER=10^{-3} was calculated as the difference between the OSNR with PMD and a back-to-back (b2b) measure. We measured the back-to-back OSNR value to be 11.5±0.5 dB, where the variation was due to wavelength sensitivity of the receiver.

## 3. First-order PMD

For a standard receiver, the first-order PMD-induced system penalty *ε* is approximated by [9]:

where *A* is a system-specific constant, Δ*τ* is the DGD, *T* is the bit period and *γ* is the magnitude of the power splitting ratio between the two PSPs. The predicted relationship between PMD and the induced penalty in Eq.1 is quadratic, and the worst launch SOP is for equal power splitting between the PSPs (*γ*=0.5).

To characterize the MLSE receiver, we began by measuring the surface described by Eq.1, using the first order PMD source and an optical frequency of 194 THz. We investigated 141 different launch SOPs, corresponding to different *γ* values, and DGD values up to 1.65 bit periods. The results in Fig. 2(a) plot the surface described by Eq.1. Figure 2(b) shows an outage map of the same data [11]. The contours of the outage map represent lines of constant penalty, separated by 0.25 dB intervals. The outage map completely characterizes the tolerance of the MLSE receiver to first-order PMD, and highlights some interesting results. Firstly, for DGD values less than 100ps we observed a slight asymmetry in the dependence of penalty on launch SOP, in agreement with [11]. This is demonstrated by the symmetry of the outage map about *γ*=0.5, and for this data set the worst launch states were in the range of *γ*=0.45 to γ=0.50. Secondly, for DGD values greater than 120 ps we observed that the worst launch SOPs were for *γ*=0.30 and 0.74. These *γ* values corresponded to the transition between two different operating regions. In the first region, given by 0.3<*γ*<0.74, the penalty increases with DGD until a local maximum is reached, after which there is a slight improvement [1,6,7]. The second region, given by *γ*<0.3 and *γ*>0.74, is different: here the penalty increases monotonically over all DGD values investigated. We also observed a few anomalous sweeps, which have been omitted from Fig. 2, where the MLSE decoder became locked in a metastable state. This has been reported elsewhere [12].

To further analyze the first order data we recast Eq.1 as [13]:

where τ_{⊥} is a component of the PMD vector, which is perpendicular to the launched SOP [9]. Figure 3(a) shows the penalty plotted against τ_{⊥} for the entire dataset shown in Fig.2. Substantial deviations from the predictions of Eq. (2) are seen. Figure 3(b) shows a subset of the data, for DGD values up to 90ps. Using a least square quadratic fit to this dataset we found *A*=16.4 from Eq.2. The experimental data was modeled by this fit to within 0.6 dB up to 90 ps. In spite of the sophisticated correction and decision process in this MLSE receiver, its first-order PMD performance can be characterized by the single parameter *A* from Eq.2 for DGD values up to a bit period.

When operated at DGD values beyond one bit period, the first-order PMD penalty is sensitive to details of clock recovery and MLSE algorithm stability, and has a complex dependence on DGD and launch SOP. In this region outage maps provide the best means to characterize the PMD tolerance of the receiver. We suggest that the computation of *A* and use of outage maps are adopted by industry to characterize electronic equalizers’ PMD tolerance.

## 4. All-order PMD

Once we had fully characterized the MLSE receiver’s tolerance to first-order PMD, we were able to assess the tolerance of the receiver to all-order PMD. To achieve this we used the all-order PMD sources described in Section 2, and summarized in Table 1. To sample the all-order PMD space sufficiently, results were collected from 50 pre-programmed launch SOPs, measured over 41 channels with optical frequencies ranging from 192 THz to 196 THz at 100 GHz spacing. Figure 4 plots the penalty from all-order PMD against the perpendicular component of the PMD vector τ_{⊥}, computed using the instantaneous values of DGD and γ at the center of the selected ITU channel. For comparison, the first order fit given in Eq.2 is also plotted in the figure, using *A*=16.4. The quadratic trend of the dataset shows that penalties from all-order PMD states are dominated by the first-order PMD-induced penalty, over the all-order PMD states we sampled. The effects of high-order PMD induce a vertical spread in the penalty data compared to the first-order PMD, in agreement with the results for a standard receiver presented in [13].

To investigate the effects of high-order PMD we compared the all-order PMD-induced penalty results to the equivalent first order PMD-induced penalty, found by interpolating Fig. 2 at a given *γ* and DGD value. The results in Fig. 5 have been split into two data sets. The blue triangles represent data with DGD values less than a bit period (90.9 ps), while the red diamonds represent DGD values greater than 90 ps. Figure 5(a) plots the all-order penalty against the first-order penalty. For the lower DGD group, the penalty induced by high-order PMD is, on average, larger than the first-order penalty with values <1 dB. This is in agreement with results presented in [13], for a standard receiver. However, as the first order PMD-induced penalty increases beyond 1 dB, the introduction of high-order PMD can aid the mitigation provided by the MLSE receiver, in some circumstances, by reducing the effect of first order PMD. This result is in agreement with the simulation results presented in [14]. For the larger DGD group, high-order PMD causes an additional penalty over the first-order PMD-induced penalty of up to 2.09 dB.

To further investigate this data we plotted the difference between the all- and first-order penalty results against the power splitting ratio, *γ*. The results in Fig. 5(b) show that when the DGD is less than a bit period the high-order PMD-induced penalty exceeds the first-order PMD penalty when the launch SOP is close to the PSP. For DGD values greater than a bit period, the largest deviations of high-order penalty from first-order penalty occur at *γ*~0.25 and *γ*~0.75. This indicates that the effect of high-order PMD in this group is associated with the algorithms and clock recovery mechanisms specific to this MLSE receiver.

In summary, our investigations of MLSE receiver tolerance to all-order PMD have shown that first-order penalty PMD is the dominant cause of PMD-induced penalty. High-order PMD is problematic when the first-order penalty is small. As the first-order penalty increases the effect of high-order PMD can, in some cases, reduce the PMD-induced impairment. Beyond one bit period the MLSE receiver tolerance to all-order PMD is reduced, and large differences between first and all-order penalties are observed.

## 5. Conclusions

We characterized an MSLE receiver’s tolerance to first-order PMD, for DGD values up to 1.65 bit periods. We showed that for DGD values less than a bit period the MLSE receiver’s tolerance to PMD was characterized by the single parameter ‘*A*’, from the first-order PMD-induced penalty approximation. Once the DGD exceeded a bit period, the relationship between the PMD-induced penalty, DGD and launch SOP was more complex. In this case, the MLSE receiver’s tolerance to PMD was best charterised using an outage map. We propose that these methods of characterizing receivers’ tolerance to PMD be adopted by industry.

The study of first order PMD-induced penalties enabled us to characterize the MLSE receiver’s tolerance to all-order PMD. We showed that first-order penalty PMD is the dominant cause of PMD-induced penalty. High-order PMD induced a large additional penalty when the first-order penalty was small, or when the DGD exceeded a bit period. When the first-order penalty was appreciable, and the DGD was less than a bit period, we showed that high-order PMD can, in some cases, reduce the effects of first-order PMD.

## Acknowledgements

The authors would like to acknowledge Siemens, CoreOptics and OFS for the loan of equipment used in this experiment. P. Magill helped guide this work with insightful discussions.

## References and links

**1. **H.F. Haunstein*et al*, “Principles for Electronic Equalization of Polarization-Mode Dispersion,” IEEE J. Lightwave Technol. **22**, 1169 (2004). [CrossRef]

**2. **T. Nielsen*et al*, “OFC 2004 Workshop on Optical and Electronic Mitigation of Impariments,” IEEE J. Lightwave Technol. **23**, 131 (2005). [CrossRef]

**3. **C. Fludger*et al*, “ Core Optics: Enabling Open Tolerant Networks,” ECOC Workshop (2005).

**4. **A. Färbert*et al*, “Performance of a 10.7 Gb/s Receiver with Digital Equaliser using Maximum Likelihood Sequence Estimation,” Proc. ECOC 04, Cannes, France, Paper PD-Th4.1.5 (2004)

**5. **H.F. Haunstein*et al*, “Performance Comparison of MLSE and Iterative Equalization in FEC Systems for PMD Channels With Respect to Implementation Complexity,” IEEE J. Lightwave Technol. **24**, 4047 (2006). [CrossRef]

**6. **J.M Gene et al, “Joint PMD and Chromatic Dispersion Compensation Using an MLSE,” Proc. ECOC 06, Cannes, France, Paper We2.5.2 (2006).

**7. **I. Lobato Polo*et al*, “Comparison of Maximum Likelihood Sequence Estimation equalizer with OOK and DPSK at 10.7 Gb/s,” Proc. ECOC 06, Cannes, France, Paper We2.5.3 (2006)

**8. **J.M Gene*et al*, “Simultaneous Compensation of Polarization Mode Dispersion and Chromatic Dispersion Using Electronic Signal Processing,” IEEE J. Lightwave Techol, (to be published).

**9. **H. Kogelnik*et al*, “Polarization mode dispersion,” in Optical Fiber Communications, vol. IVb,
I. Kaminow, Ed. 725 (San Diego: CA: Academic, 2002).

**10. **R.M. Jopson*et al*, “Measurement of Second-Order Polarization-Mode Dispersion Vectors in Optical Fibers,” IEEE Photon. Technol. Lett. **11**, 1153 (1999). [CrossRef]

**11. **P.J. Winzer*et al*, “Precise Outage Specifications for First-Order PMD,” IEEE Photon. Technol. Lett. **16**, 449 (2004). [CrossRef]

**12. **M.D. Feuer*et al*, “Metastable States of MLSE Receiver Induced by Extreme PMD Conditions,” paper TuA1 3, LEOS Summer Topicals (2007).

**13. **M. Boroditsky*et al*, “Comparison of system penalties from first- and multiorder polarization-mode dispersion,” IEEE Photon. Technol. Lett. **17**, 1650 (2005). [CrossRef]

**14. **M. Shtaif and M. Boroditsky, “The effect of the frequency dependence of PMD on the performance of optical communications systems,” IEEE Photon. Technol. Lett. **15**, 1369 (2003). [CrossRef]