## Abstract

We propose a novel recursive-algorithm based maximum a posteriori probability (MAP) detector in spectrally-efficient coherent wavelength division multiplexing (CoWDM) systems, and investigate its performance in a 1-bit/s/Hz on-off keyed (OOK) system limited by optical-signal-to-noise ratio. The proposed method decodes each sub-channel using the signal levels not only of the particular sub-channel but also of its adjacent sub-channels, and therefore can effectively compensate deterministic inter-sub-channel crosstalk as well as inter-symbol interference arising from narrow-band filtering and chromatic dispersion (CD). Numerical simulation of a five-channel OOK-based CoWDM system with 10Gbit/s per channel using either direct or coherent detection shows that the MAP decoder can eliminate the need for phase control of each optical carrier (which is necessarily required in a conventional CoWDM system), and greatly relaxes the spectral design of the demultiplexing filter at the receiver. It also significantly improves back-to-back sensitivity and CD tolerance of the system.

© 2009 OSA

## 1. Introduction

Coherent wavelength division multiplexing (CoWDM) and orthogonal frequency division multiplexing (OFDM) have attracted much interest recently due to their increased spectral efficiency and better tolerance to transmission impairments such as chromatic dispersion (CD) in optical networks [1–4]. Although in both techniques, the carrier spacing equals the symbol rate of each sub-channel, CoWDM and OFDM differ in their implementations. The OFDM technique performs sub-channel multiplexing and demultiplexing in the electrical domain using sophisticated digital signal processing (DSP) such as fast Fourier transform (FFT) and inverse FFT (IFFT), and transmits and detects the multiplexed signals in series in the optical domain [1,2]. However, the complexity of FFT/IFFT and the operating speed of the optical modulator and receiver scale with the total symbol rate of the OFDM signal, which limits application of this technique for ultra-wideband optical transmission. In contrast, the sub-channels in a CoWDM system are modulated, multiplexed, demultiplexed, and detected in parallel in the optical domain, such that the operating speed of the transmitters and receivers scales with the capacity of a single sub-channel rather than the total capacity. Consequently, the speed bottleneck at the electronic interfaces is eliminated. In a previous paper, 600Gbit/s on-off keying (OOK) CoWDM signal transmission with 43Gbit/s per channel at 2 bit/s/Hz was experimentally demonstrated [3].

Whereas practical CoWDM, which uses a combination of optical and electrical components, has the capability of operating at ultra-high data rate, the electrical DSP-based OFDM technique generally achieves superior orthogonality between sub-channels. This difference places more stringent limits on the design of CoWDM systems, including, for example, careful control of the phases of adjacent WDM sub-channels and tight specifications of the spectral design of the demultiplexing filter [3,4]. In addition, residual inter sub-channel crosstalk arising from imperfect system response results in performance penalties in back-to-back sensitivity and degrades CD tolerance by a factor of four [4].

Recently, a novel maximum a posteriori probability (MAP) technique was proposed for direct-detection (DD) CoWDM systems [5]. Such a MAP detector decodes each sub-channel using the signal levels not only of the particular sub-channel but also of its adjacent sub-channels. In this paper, we will extend our work by proposing a novel recursive algorithm having reduced complexity in a MAP detector using either direct or coherent detection, and investigate its performance in a 1-bit/s/Hz OOK system. It will be shown that such a MAP detector can eliminate the need for phase control of each optical carrier, and greatly relaxes the spectral specification of the demultiplexing filter. It also significantly improves back-to-back sensitivity and CD tolerance. In particular, coherent-detection based MAP enables optically-uncompensated transmission up to 300km at 10Gbit/s. Whilst this approach is also based on DSP, each MAP circuit only relies on the information from adjacent sub-channels, and so does not restrict the total data rate. It can therefore be viewed as a compromise between high-speed CoWDM using channel-by-channel hard decision and OFDM with joint electronic processing of all sub-channels.

## 2. Principle and simulation model

A conventional MAP detector, employed to compensate inter-symbol interference (ISI), calculates the probability of obtaining each possible value of a given bit based on the received signal amplitudes and selects the bit value which has the maximum probability [6]. In this paper, we extend the conventional concept to two dimensions to mitigate deterministic inter sub-channel crosstalk as well as ISI, as shown in Fig. 1. Defining *a*
_{i,k}, *b*
_{i,k}, and *r*
_{i,k} as the *k*
^{th} logical data, estimated data, and received signal amplitude for channel *i* respectively, a posteriori probability can be mathematically written as:

where *R*
_{i,k}={*r*
_{p,q}}, *p*∈{*i*-1 *i *i+1}, *q*∈{1, …, *k*+1} represents the received samples, and argmax *f* represents the value of the argument *x* for which the expression *f* attains its maximum value. The memory length used in Eq. (1) is 2 and can be easily extended to larger values at the expense of more computation complexity. Note that Eq. (1) estimates ai,k based on the observation of all previously received signal samples Ri,k, which give more information compared to those used in our previous MAP detector [5]. From Eq. (1), we can derive:

where *S _{i,k}*={

*a*

^{p,q}},

*p*∈{

*i*-1

*i i*+1},

*q*∈{1, …,

*k*+ 2} represents the transmitted signal. To establish the recursive algorithm for CoWDM MAP detection, we derive the following equation based on Eq. (2):

where *A*
_{i,k}={*a*
_{p,q}}, *p*∈{*i*-1 *i i*+1}, *q*∈{*k k*+1 *k*+2}. The summation of *P*
_{i,k-1} on the right-hand side of Eq. (4) can be obtained during the estimation of the previous bit *a*
_{i,k-1}. The joint conditional probability *P*(*r*
_{i-1,k}+1,*r*
_{i,k+1},*r*
_{i+1,k+1}|*A*
_{i,k}) is further simplified to the multiplication of individual probabilities which are obtained from a lookup table. The lookup table can be established using non-parametric histogram method. Note that direct detection is assumed in Eq. (4). In coherent detection where both in-phase and quadrature components are available, Eq. (4) is modified to:

where *r ^{o}_{i,k}* and

*r*are the in-phase and quadrature components of the received signal and assumed to be statistically independent in this paper.

^{q}_{i,k}The recursive feature of Eqs. (4) and (5) reduces the computation complexity by at least a factor of two compared to that without a recursive algorithm [5]. The complexity of a MAP detector in CoWDM systems is higher than that of a conventional MAP or maximum likelihood sequence estimation (MLSE) [6] but the overall complexity scales approximately linearly with the channel number N for large *N*. Note that the overall complexity of FFT/IFFT in the OFDM technique is also approximately proportional to N. However, the MAP detector performs parallel processing of each sub-channel, whereas OFDM requires joint signal processing of all sub-channels with associated synchronization, serial/parallel conversion etc, so that the total capacity is limited by the speed of the electronic circuitry. Therefore, CoWDM based on MAP processing would be a promising solution for ultra high-capacity spectrally-efficient optical transmission. Also note that joint MLSE, proposed to mitigate impairments for two tributaries of multi-level modulation formats [7,8], may also be used in multi-carrier systems. However, its complexity scales exponentially with *N*.

Figure 2 shows the simulation model used in this paper. Five coherent optical carriers with equal intensities and phases were produced using a comb generator. The sub-channel spacing was 10GHz, equal to the data rate of each sub-channel. The sub-channels were intensity modulated by uncorrelated 10Gbit/s OOK data trains using Mach-Zehnder modulators (MZMs), with temporally aligned eye crossings. The data trains consisted of 2^{11}-1 pseudo-random binary sequences (PRBS) repeated 5 times (10,235 bits) and different delays were applied to the five channels so that their bit sequences were uncorrelated. Ten ‘0’ bits and eleven ‘0’ bits were added before and after each data train to simplify the boundary conditions. The electrical ‘1’ bits were raised-cosine shaped with a roll-off coefficient of 0.4 and were simulated with 40 samples per bit. The extinction ratio of the modulated OOK signal was set to be 20dB. The modulated sub-channels were phase shifted by *ϕ*
_{i} (*i*=1,…,5) before they were combined and launched into a piece of single-mode fiber (SMF) with CD of 16ps/km/nm. In the fiber link, the signal power was assumed to be split equally between the fast and slow orthogonal polarization modes.

The noise of the optical preamplifier was modelled as additive white Gaussian noise with equal noise spectral power density for each polarization. The five sub-channels were demultiplexed by an asymmetric Mach-Zehnder interferometer (AMZI) with 20GHz free spectral range (FSR) and two 2nd-order Gaussian-shaped array waveguide gratings (AWGs). The signals after the AWGs had a power of -3dBm per channel and were detected either directly or coherently. In coherent detection, as shown in Fig. 2, the signals and local oscillators were separated into two linear polarization components with polarization beam splitters (PBSs), mixed by 90° optical hybrids, and detected by balanced detectors to extract the in-phase and quadrature components. The output powers of the local oscillators were 10dBm and their phases were assumed not to vary during the simulation so that electrical phase equalization was not required. The equivalent thermal noise spectral power density of the detectors was 18pA/Hz^{1/2}. After optical-to-electrical conversion, the signals were electrically amplified, filtered by 7GHz 4th-order Bessel electrical filters (EFs), sampled at one sample per bit, and analogue-to-digital converted with 4-bit resolution.

The simulation was iterated 15 times with different random number seeds to give a total of 153,525 simulated bits. The performance was evaluated in terms of the required normalized optical-signal-to-noise ratio (OSNR) to achieve a bit error rate (BER) of 5×10^{-4} by direct error counting. The normalized OSNR was defined by:

## 3. Results and discussions

Figure 3(a) shows the optical spectral power density of the CoWDM signal. Figures 3(b) and (c) depict the eye diagrams of channel 3 using DD for *ϕ*=*π*/2 and *π* respectively, assuming that at the transmitter, the five channels have fixed relative phases of (0 *ϕ* 0 *ϕ* 0). It can be clearly seen from the figure that the detected signal was degraded by residual inter sub-channel crosstalk from imperfect system response at 10GHz sub-channel spacing, and its performance depended on the phases of adjacent channels. Therefore, in practice, it is essential to optimize the phases to minimize the residual crosstalk at the eye center (sampling point) [3,4]. For a system with no dispersion, the optimum phase between adjacent channels is *±* π/2 [3]. This result was further verified by the dotted lines in Fig. 4, which shows the required normalized OSNR versus phase *ϕ* for Ch 1–3 (Ch 1: circles; Ch 2: triangles; Ch 3: squares) when the relative phases of the channels are: (a) (0 *ϕ* 0 *ϕ* 0); (b) (0 *ϕ* 2*ϕ* 3*ϕ* 4*ϕ*). Dotted, dashed, and solid lines represent the cases using conventional hard-decision, DD-based MAP, and coherent-detection based MAP respectively. The 3dB bandwidth of the AWG was 12.8GHz. As expected, the performance variations of Ch 4 and 5 were approximately the same as those of Ch 2 and 1, and were neglected in the figures. Figure 4 shows that when using hard decision, the performance was optimal when the phase difference between adjacent channels was *±*π/2 where the crosstalk was outside the eye as shown in Fig. 3(b). When the crosstalk was at the center of the eye (Fig. 3(c)), an additional 3~4dB OSNR penalty was induced. However, for a static interference pattern of arbitrary phase, MAP detection canceled the inter sub-channel crosstalk. Consequently, the required OSNR was significantly reduced, and the fluctuations as a function of the phase were less than 0.8dB. Coherent-detection based MAP detector, compared to that using DD, exhibited an additional 0.5~1dB performance improvement by using the recovered phase information.

For conventional hard decision, the bandwidth of the AWGs for channel demultiplexing was critical to balance ISI and inter sub-channel crosstalk [9]. Figure 5(a) shows the required normalized OSNR as a function of 3dB bandwidth of the AWGs for Ch 1–3. The relative phases of the sub-channels were (0 *π*/2 *π* 3π/2 2*π*). It can be seen that a CoWDM system using hard decision was particularly sensitive to narrow-band filtering, with 4~10dB OSNR penalty for a 3dB bandwidth of 10GHz. The MAP detector greatly reduced such sensitivity, and the OSNR penalty caused by narrow-band filtering was limited to less than 1.5dB for a filter bandwidth as small as 8GHz. We attribute this benefit to the inherent ISI mitigation capability of MAP detection. Consequently, the design of AWG was greatly relaxed.

The capability of CoWDM MAP detection to compensate ISI and inter sub-channel crosstalk was further verified by investigating dispersion limited optical transmission. Figure 5(b) shows the required normalized OSNR versus transmission distance by using hard decision (dotted), DD-based MAP (dashed), and coherent-detection based MAP (solid). The relative phases were set to be (0 *π*/2 *π* 3*π*/2 2*π*) for all data in the figure. The AWGs had 3dB bandwidth of 12.8GHz. From the figure, it is seen that by using hard decision, Ch 2 and Ch 3 exhibited poorer back-to-back sensitivity and less CD tolerance than Ch 1 due to larger inter sub-channel interference. At an OSNR of 15dB, the transmission distance was limited to between 40 and 60km (Ch 3 and Ch 1, respectively). By using DD-based MAP, knowledge of the adjacent sub-channels allowed for the compensation of deterministic inter sub-channel crosstalk, which led to 1.5~2.5dB back-to-back sensitivity improvement and a significant enhancement of the CD tolerance to give an optically-uncompensated transmission distance exceeding 125km at 15dB OSNR for all sub-channels. Such performance was further improved using coherent-detection based MAP making use of the recovered phase information, with an additional 0.6dB back-to-back sensitivity improvement and transmission reach extended to 300km at 15dB OSNR. Note that in the presence of CD, optimal performance using hard-decision CoWDM should be obtained by adjusting sub-channel phase relationship for each transmission distance [4]. However, given the reduction in phase sensitivity afforded by MAP detection, such adjustment was not necessary and the data in Fig. 5 was obtained with the phases set at their optimal back-to-back values.

Finally, the effect of quantization resolution of the received samples on the system performance was investigated, as shown in Fig. 6. For CoWDM MAP detection, the OSNR penalties for reducing the resolution were less than 2dB and 1dB for 3-bit and 4-bit resolution respectively. However, coherent-detection based MAP was more robust to smaller resolutions with a 5dB OSNR penalty for 2-bit resolution. At this resolution, the signal could not be recovered by the DD-based MAP.

## 4. Conclusions

We have proposed a recursive-algorithm based MAP detector in spectrally-efficient CoWDM systems using either direct or coherent detection, and investigated its performance in a 1-bit/s/Hz OOK-based system limited by OSNR. The proposed method features reduced complexity compared to previous work [5] and can effectively compensate deterministic inter-sub-channel crosstalk as well as ISI arising from narrow-band filtering and CD. Numerical simulation of a five-channel OOK based CoWDM system with 10Gbit/s per channel shows that the MAP detector can eliminate the need for phase control of each optical carrier, which is necessarily required in conventional CoWDM systems, and also greatly relax the spectral design of the demultiplexing filter. It also significantly improves the system performance including back-to-back sensitivity and CD tolerance. In particular, coherent-detection based MAP enables an additional 0.5~1dB back-to-back performance improvement compared to DD-based MAP and achieves optically-uncompensated transmission up to 300km. Therefore, MAP processing further increases the potential of CoWDM systems for ultra high-capacity spectrally-efficient optical transmission.

## Acknowledgements:

This work was supported by Science Foundation Ireland under grant number 06/IN/I969.

## References and links

**1. **W. Shieh, H. Bao, and Y. Tang, “Coherent optical OFDM: theory and design,” Opt. Express **16**(2), 841–859 (2008). [CrossRef]

**2. **A. Lowery and J. Armstrong, “Adaptation of orthogonal frequency division multiplexing to compensate impairments in optical transmission systems,” European Conference on Optical Communication, paper 4.2.1 (2008).

**3. **F. C. G. Gunning, T. Healy, X. Yang, and A. D. Ellis, “0.6Tbit/s capacity and 2bit/s/Hz spectral efficiency at 42.6Gsymbol/s using a single DFB laser with NRZ coherent WDM and polarization multiplexing,” European Conference on Lasers and Electro-Optics (E-CLEO), paper C18-5-FRI (2007).

**4. **F. C. G. Gunning, T. Healy, and A. D. Ellis, “Dispersion tolerance of coherent WDM,” IEEE Photon. Technol. Lett. **18**(12), 1338–1340 (2006). [CrossRef]

**5. **J. Zhao and A. D. Ellis, “Performance improvement using a novel MAP detector in coherent WDM systems,” European Conference on Optical Communication (2008), paper Tu.1.D.2.

**6. **J. Proakis, *Digital Communication*, 4th Edition, McGraw-Hill, 2001.

**7. **M. Cavallari, C. R. S. Fludger, and P. J. Anslow, “Electronic signal processing for differential phase modulation formats,” in Proc. Optical Fiber Communication Conference (2004), paper TuG2.

**8. **J. Zhao, L. K. Chen, and C. K. Chan, “Joint maximum likelihood sequence estimation for chromatic dispersion compensation in ASK-DPSK modulation format,” IEEE Photon. Technol. Lett. **19**(1), 73–75 (2007). [CrossRef]

**9. **A. D. Ellis and F. C. G. Gunning, “Spectral density enhancement using coherent WDM,” IEEE Photon. Technol. Lett. **17**(2), 504–506 (2005). [CrossRef]