## Abstract

A maximum likelihood sequence detection (MLSD) receiver is used to detect data sequences in single-carrier coherent optical systems in the presence of laser phase noise. It requires no explicit phase estimation and involves only linear operations. It consistently shows improvement in the OSNR penalty (e.g., 1.1 dB at BER = 10^{−4} with memory length *L* =3) and the laser linewidth tolerance (e.g., around 4 times that of DAML at 1dB OSNR penalty at BER = 10^{−4} with memory length *L* =3) over the well-known DAML and *M*th power approaches in laser phase noise (LPN)-impaired coherent optical systems.

© 2011 OSA

## 1. Introduction

Based on the detection approach used, optical systems can be broadly classified into three categories, namely, noncoherent detection systems (e.g. on-off-keying (OOK)), differentially coherent detection systems (e.g. differential phase-shift keying (DPSK)), and coherent detection systems (e.g. phase-shift keying (PSK)) [1]. Compared to the former two approaches, a coherent detection system allows the receiver to process the signal based on the recovery of the full electric field. In this case, the spectral efficiency can be increased by log_{2}(*M*) when advanced modulation formats such as *M*-ary PSK (MPSK) is used. Nowadays, coherent receivers employing high-speed analog-to-digital converters (ADCs) and high-speed baseband digital signal processing (DSP) have become increasingly attractive, because they recover the carrier phase digitally and thus avoid the use of expensive optical phase lock loops. To allow a free-running local oscillator (LO) laser, it is crucial to recover the carrier phase in the presence of laser phase noise (LPN) due to the mismatch between the transmitter laser and the laser at the LO, which keeps rotating the phase of the received signals and causes a power penalty to the receiver sensitivity.

In the literature, although various phase estimation algorithms, such as normalized least-mean-square (NLMS) phase estimation [2], Wiener filter-based phase estimation [3], Kalman filter-based phase estimation [4], have been proposed to recover the carrier phase, they require either preset parameters (e.g. step size) or the statistics of the system noises. However, for coherent optical systems with unknown carrier phase noise, the two well-known approaches for designing a symbol-by-symbol (SBS) coherent detection receiver are *M*th-power [5] and decision-aided maximum likelihood (DAML) [6,7]. They have demonstrated comparable performances in the context of LPN-impaired, single-carrier coherent optical systems [6]. Both approaches are subject to the block length effect because of the trade-off between averaging over the LPN and averaging over the additive receiver noise [8]. However, unlike the *M*th power scheme’s heavy reliance on nonlinear operations, DAML is totally linear and free from phase-unwrapping [7]. Extending on the DAML phase estimator first proposed in [9] and on the maximum likelihood sequence detection (MLSD) result first derived in [10], we have proposed in [11] an efficient, Viterbi-type trellis-search algorithm for sequence detection of MPSK signals over the additive white Gaussian noise (AWGN) channel with unknown carrier phase.

In this paper, the MLSD receiver of [11] is used to detect an uncoded MPSK data sequence at the receiver end in a coherent optical system that is impaired by LPN, which is the major optical impairment to combat in a back-to-back coherent optical system. The use of a sequence detector rather than a SBS detector is more advantageous in a real-life communication environment, where data are sent in packets. We show by simulation that our MLSD receiver outperforms both the *M*th power and the DAML in the LPN-impaired coherent optical system. This MLSD receiver can be extended to the coded case, where good performance in the presence of LPN can also be obtained.

## 2. System setup

Figure 1
shows the system setup with an optical transmitter and a DSP-incorporated homodyne coherent receiver. Here, an example back-to-back QPSK system is used to demonstrate the effectiveness of our MLSD receiver; however, this single polarization coherent receiver structure is universal for any MPSK. Similar to the *M*th power and the DAML approaches, the signal is assumed to be differentially pre-coded at the transmitter to avoid ‘runaway’ or error propagation due to cycle slips in the phase tracking at the receiver [5, 7]. Alternatively, periodic pilot sequences may be inserted to prevent the ‘runaway’ of the tracked phase [5, 12].

The received optical field *E _{r}*(

*t*) beats with the optical field generated by the LO

*E*(

_{LO}*t*) in a standard 2x4 optical hybrid, which has four 3-dB couplers and an additional 90°-phase shifter [13]. The output signals from the optical hybrid are subsequently detected by two dual-photo diode based balanced detectors. The output signals of the two balanced detectors correspond to the in-phase (

*i*(

_{I}*t*)) and quadrature-phase (

*i*(

_{Q}*t*)) signals. These signals are sampled by using high-speed ADCs and then processed by the MLSD receiver for laser phase noise-aware sequence detection. Note that the MLSD receiver is optimum from the standpoint of eliminating laser phase noise, which is the dominant optical impairment in back-to-back coherent optical systems. Given our experimental setup as shown in Fig. 1 and our assumption that all the other impairments such as intersymbol interference due to chromatic dispersion, polarization mode dispersion and attenuation can be handled by using optical means and polarization is fully matched by using a polarization controller, the design is also globally optimum since the DSP unit now only needs to combat the laser phase noise.

## 3. The MLSD receiver

Let $s=\left[s\left(0\right)s\left(1\right)\mathrm{...}s\left(K-1\right)\right]$ denote the transmitted signal sequence, $r=[r\left(0\right)\text{r}\left(1\right)\mathrm{...}$ $\text{r}\left(K-1\right)]$ denote the received signal sequence and $\theta =\left[\theta \left(0\right)\theta \left(1\right)\mathrm{...}\theta \left(K-1\right)\right]$ denote the carrier phase process of the sequence of length *K* symbols, where *K* >> 1. Laser phase noise (LPN) is modeled as a Wiener process $\theta \left(k\right)={\displaystyle {\sum}_{m=-\infty}^{k}f(m)}$, where $\theta (k)$ is the phase in the *k*th symbol interval [*kT,* (*k +* 1)*T*), *T* is the symbol duration, and {$f\left(m\right)$} is a sequence of independent, identically distributed (i.i.d) Gaussian random variables, each with mean zero and variance ${\sigma}_{p}^{2}=2\pi \Delta fT$. Here, $\Delta f$is the total 3dB laser linewidth of the transmitter and LO, and it is generally assumed that the 3dB laser linewidth of both the lasers at the transmitter and LO are of the same value [14]. The received digital signal $r\left(k\right)$ in the *k*th symbol interval can be represented as

*k*th symbol interval, and {

*n*(

*k*)} is the sequence of i.i.d complex Gaussian variables due to the shot noise [6]. Specifically for QPSK, we have$s\left(k\right)\in \{{C}_{i}=\sqrt{{E}_{s}}{e}^{j{\varphi}_{i}\left(k\right)}:{C}_{0}=-{C}_{2},{C}_{1}=-{C}_{3}\}$, where $\varphi (k)$is the signal phase and

*E*is the symbol energy. The sequence

_{s}**s**is equivalent to a path through a trellis, whether it is coded or uncoded [11].

Figure 2
shows the trellis of an uncoded QPSK sequence. In this case, the trellis states at each time *t* = *kT* correspond to the possible values assumed by signal phase $\varphi \left(k\right)$. The branch entering each state is labeled with the two bits assigned to that particular value of the signal phase $\varphi \left(k\right)$ by using a Gray code. We assume the carrier phase changes slowly as in [6] so that it can be considered constant over a memory length of *L* symbol intervals, where $1<L<<K$. Our previous work in [15] provided a very detailed derivation to obtain the implementable branch metrics for a Viterbi-type trellis-search algorithm. Specifically for QPSK, at each time *t = kT*, when the receiver has to decide on the survivor at each state of the trellis, it chooses the path with the maximum metric ${\mu}^{QPSK}\left(k,s\left(k\right)\right)$ which can be computed recursively by using the reference phasor derived in [7] in the phase estimation portion of Eq. (29) in [15]. The metric obtained is as shown below.

The firm symbol decisions are made at each time by tracing back along all the survivor paths to detect for mergers of their tails, i.e., the firm decisions correspond to the symbols in the merged tails. The metric above is totally linear and can be calculated additively.

## 4. Performance evaluation

Table 1
compares the computational complexities of the three detection approaches, namely MLSD, DAML and *M*th power. First, as we can see from Eq. (2), MLSD requires no explicit phase estimation and thus has no associated nonlinear argument operation or phase unwrapping as required in the *M*th power scheme [5]. On the other hand, for the trellis-search based MLSD receiver, at each time point, (*M*−1) comparisons are performed at each state to choose one survivor form the *M* paths that enter that state. Therefore, a total of (*M*−1)*M* comparisons are performed at any time with *M* states. Finally, DAML and *M*th power are both SBS detection approaches, hence the decision of a symbol occurs right after the receipt of the symbol. However, our MLSD is a sequence detection receiver that implements trellis search, where the decision on a data symbol is made only when the tails of all the survivors merge. Hence, the detection delay of the symbols is random. In order to resolve this issue, here we divide a long sequence of *K* symbols into a train of subsequences each of length *D* (usually *D*<<*K*), where a termination symbol corresponding to a known modulation phase of zero radian is inserted at the end of each subsequence to flush the trellis back to state ‘0’. As a result, decisions on all data symbols prior to that termination symbol can be made at the latest upon the receipt of the termination symbol. Thus, the average detection delay per symbol, *t _{d}*, is not longer than (

*D*+1)/2.

In the rest of this section, Monte Carlo simulations are used to evaluate the performance of our MLSD receiver. The system under investigation is a differentially pre-coded QPSK coherent transmission system, as shown in Fig. 1, which has a transmission speed of 50Gbps. A long sequence of at least *K* = 10^{7} symbols is simulated. Furthermore, a preamble of length equal to the memory length *L* for the initial computation of the metric ${\mu}^{QPSK}\left(k,s\left(k\right)\right)$ is used to start the process. Since *L* is usually much shorter than the subsequence length *D* and only used once for transmitting the entire long sequence of *K* symbols, the overhead is negligible. Note that SNR/bit denotes signal-to-noise ratio per bit, which is obtained by dividing the SNR of the received symbols by the number of bits per symbol, i.e., 2 in this case.

Figure 3
shows the normalized constellation of the received signals when SNR/bit = 11dB, laser linewidth (LLW) = 20MHz, and *L* = 3. For the MLSD, *D* = 100 is used here. Figure 3(a) shows the actual received digitized signal before the high-speed DSP units for signal detection, where the laser phase noise (LPN) keeps rotating the phase of the carrier signal. Figure 3(b) shows the received signal if LPN is absent. From Fig. 3(c) and 3(d), we find that compared to DAML, the majority of the received signals after MLSD are spaced out more obviously. Hence the decision errors after MLSD are mainly due to the signals that are very near to the decision boundaries.

Figures 4 through 6 demonstrate the performance evaluation of MLSD receiver in detail. Figure 4 and Fig. 5 plot the bit error rate (BER) performance versus the SNR/bit.

Figure 4 illustrates the comparison of BER performance among the three receivers, where LLW = 20MHz. Figure 4(a) shows the results for the case of memory length *L* = 1, and Fig. 4(b) for *L* = 3. As expected, DAML and *M*th power demonstrate similar BER performances for both cases. On the other hand, MLSD shows consistent improvements, in terms of lower BER and better receiver sensitivity, over these two SBS receivers for both cases throughout the entire SNR region. For example, at SNR/bit = 11dB, when *L* = 3, the BER obtained is 4x10^{−5} for MLSD, and 2x10^{−4} for both DAML and *M*th power. Furthermore, the improvement of receiver sensitivity is more obvious at lower BER. For example, when *L* = 3 and BER = 10^{−4}, MLSD has an SNR/bit improvement of 1.1 dB over both DAML and *M*th power. For the same memory length, when BER is reduced to 10^{−5}, the improvement increases further to 1.4dB.

Figure 5 illustrates BER results for different subsequence lengths *D*. Here LLW = 10MHz and three different subsequence lengths are investigated. The results are compared with the DAML results only. We observe that the BER performances are very similar for different subsequence lengths. Similar findings are observed with different memory lengths (*L* = 3 and *L* = 5). However, in the case of *L* = 1, a shorter subsequence length achieves lower BER under the same SNR/bit. Note that a memory space is required at the MLSD receiver to hold a copy of the unmerged paths of the survivor sequences before a firm decision is reached, and in the worst case, the mergers only occur through the enforcement of the termination symbol. Hence, the worst case memory required at the receiver will be proportional to *D*. Thus, a shorter subsequence length means smaller receiver buffer sizes and shorter detection delays. For all cases, apparent improvements over the DAML are observed, e.g., about 0.5 dB improvements of SNR/bit at BER = 10^{−4} are shown for both *L* = 3 and *L* = 5.

Figure 6 shows the OSNR penalty curves at BER = 10^{−4} for MLSD, DAML and *M*th power as compared to ideal coherent detection. It shows that MLSD is able to achieve a lower OSNR penalty or higher laser linewidth (LLW) tolerance than both DAML and *M*th Power. For example, for 1dB OSNR penalty, when *L* = 3, MLSD is able to achieve a LLW tolerance (9MHz, in this case) around 4 times that of DAML (2.3MHz, in this case), where *M*th Power cannot achieve 1dB penalty for all laser linewidths.

## 5. Conclusions

We have presented a MLSD receiver that is suitable for coherent optical transmission systems that are impaired by laser phase noise (LPN). The simulation results showed that our receiver outperformed both DAML and *M*th power schemes in terms of lower BER at given receiver SNR and higher laser linewidth tolerance for a given OSNR penalty. More importantly, the larger the LPN is, the shorter the required memory length *L*, and the more obvious the improvement of our MLSD over the DAML and the *M*th power schemes. This lowers the receiver complexity, leading to less stringent requirements and lower costs for the transmitter and LO lasers.

## Acknowledgments

The authors would like to thank the support of A*STAR SERC PSF 092 101 0054.

## References and links

**1. **E. Ip, A. P. Lau, D. J. F. Barros, and J. M. Kahn, “Coherent detection in optical fiber systems,” Opt. Express **16**(2), 753–791 (2008). [CrossRef]

**2. **Y. Mori, C. Zhang, K. Igarashi, K. Katoh, and K. Kikuchi, “Unrepeated 200-km transmission of 40-Gbit/s 16-QAM signals using digital coherent receiver,” Opt. Express **17**(3), 1435–1441 (2009). [CrossRef]

**3. **E. Ip and J. M. Kahn, “Feedforward carrier recovery for coherent optical communications,” J. Lightwave Technol. **25**(9), 2675–2692 (2007). [CrossRef]

**4. **L. M. Pessoa, H. M. Salgado, and I. Darwazeh, “Joint mitigation of optical impairments and phase estimation in coherent optical systems,” *in Proceedings of IEEE LEOS Summer Topical Meetings**2008*, pp. 169–170.

**5. **D.-S. Ly-Gagnon, S. Tsukamoto, K. Katoh, and K. Kikuchi, “Coherent detection of optical quadrature phase-shift keying signals with carrier phase estimation,” J. Lightwave Technol. **24**(1), 12–21 (2006). [CrossRef]

**6. **S. Zhang, P. Y. Kam, J. Chen, and C. Yu, “Decision-aided maximum likelihood detection in coherent optical phase-shift-keying system,” Opt. Express **17**(2), 703–715 (2009). [CrossRef]

**7. **S. Zhang, P. Y. Kam, C. Yu, and J. Chen, “Decision-aided carrier phase estimation for coherent optical communications,” J. Lightwave Technol. **28**(11), 1597–1607 (2010). [CrossRef]

**8. **S. Zhang, P. Y. Kam, and C. Yu, “Block length effect of decision-aided maximum likelihood phase estimation in coherent optical communication systems,” *in Proceedings of OSA/CLEO/QELS**2009*, pp. 1–2, 2–4.

**9. **P. Y. Kam, “Maximum likelihood carrier phase recovery for linear suppressed-carrier digital data modulations,” IEEE Trans. Commun. **34**(6), 522–527 (1986). [CrossRef]

**10. **P. Y. Kam, “Maximum-likelihood digital data sequence estimation over the Gaussian channel with unknown carrier phase,” IEEE Trans. Commun. **35**(7), 764–767 (1987). [CrossRef]

**11. **P. Y. Kam and P. Sinha, “A Viterbi-type algorithm for efficient estimation of M-PSK sequences over the Gaussian channel with unknown carrier phase,” IEEE Trans. Commun. **43**(9), 2429–2433 (1995). [CrossRef]

**12. **S. Zhang, X. Li, P. Y. Kam, C. Yu, and J. Chen, “Pilot-assisted decision-aided maximum-likelihood phase estimation in coherent optical phase-modulated systems with nonlinear phase noise,” IEEE Photon. Technol. Lett. **22**(6), 380–382 (2010). [CrossRef]

**13. **M. Seimetz and C.-M. Weinert, “Options, feasibility, and availability of 2 x4 90° hybrids for coherent optical systems,” J. Lightwave Technol. **24**(3), 1317–1322 (2006). [CrossRef]

**14. **K. Kikuchi, T. Okoshi, M. Nagamatsu, and N. Henmi, “Degradation of bit-error rate in coherent optical communications due to spectral spread of the transmitter and the local oscillator,” J. Lightwave Technol. **2**(6), 1024–1033 (1984). [CrossRef]

**15. **Y. Li, P. Y. Kam, and C. C. Chui, “Adaptive sequence detection for MPSK/MQAM with unknown carrier phase characteristics,” *in Proceedings of WCNC**2009*, pp.1–6