## Abstract

We demonstrate a marked performance improvement in a 512 QAM transmission by employing frequency-domain equalization (FDE) instead of an FIR filter. FDE enables us to compensate for distortions due to hardware imperfections in the transmitter with higher precision, which successfully reduced the power penalty by 4 dB in a 54 Gbit/s (3 Gsymbol/s)-160 km transmission. FDE also allows the transmission distance to be extended up to 240 km.

© 2012 OSA

## 1. Introduction

Digital coherent technology with a multi-level modulation format has played a critical role in increasing spectral efficiency and expanding fiber capacity toward the 100 Tbit/s regime [1]. Advances in DSP technologies have enabled not only the coherent detection of high-speed multi-level optical signals with high precision, but also compensation for linear and nonlinear transmission impairments in the electrical domain, both in a static and adaptive way [2, 3]. These equalizers are also very useful for dealing with distortions caused by hardware imperfections in individual components, such as a non-ideal frequency response or skew in optical or electrical devices.

Time-domain equalization techniques using a finite impulse-response (FIR) filter are already commonly used in digital coherent receivers. However, the frequency resolution of FIR filters is typically limited to around a few tens of MHz, which is determined mainly by the finite number of filter taps in order to avoid complex calculations. Insufficient resolution in the equalizer becomes disadvantageous for signals with higher-order multiplicity levels such as 256 and 512 QAM [4–6], as these formats typically contain non-negligible frequency components even below 10 MHz.

Recently, a frequency-domain equalization (FDE) scheme, which was originally developed for wireless communication [7], has received a lot of attention in relation to optical communication [8]. FDE has been successfully applied to ultrahigh capacity WDM transmission with 16 QAM [9]. It features an equalization capability with less computational complexity than FIR filters by virtue of the FFT operation. FDE is also expected to improve the frequency resolution significantly compared with that obtained with FIR filters without the expense of calculation complexity.

In this paper, an FDE technique is applied to extremely high-order optical QAM for the first time, and a marked improvement is demonstrated in 512 QAM, 54 Gbit/s (3 Gsymbol/s) transmission performance over 160 km. The power penalty was greatly reduced as a result of the ability to compensate for a non-ideal frequency response especially in a lower frequency regime.

## 2. Capability of FDE for higher-order QAM

We first evaluated the capability of FDE for higher-order QAM and compared it with an FIR filter using a 256 QAM, 4 Gsymbol/s signal [1]. Figure 1(a)
and 1(b), respectively, show the back-to-back error vector magnitude (EVM) of the 256 QAM signal for various frequency resolution values when a digital FIR filter and FDE, respectively, were adopted as equalizers. In Fig. 1(a), the resolution of the FIR filter Δ*f* was varied by changing the number of FIR taps, *N*_{FIR}, which are related as Δ*f* = 4 Gsymbol/s / *N*_{FIR}. The maximum *N*_{FIR} was 99, which corresponds to the resolution Δ*f* = 40 MHz, where the minimum EVM was 1.38%. A larger *N*_{FIR} results in a significant increase in computational complexity and poor convergence of the tap coefficient calculation. Specifically, the number of real-valued multiplications per symbol, *n*_{FIR}, is given by *n*_{FIR} = 4*N*_{FIR}, i.e., it increases in proportion to the number of taps [10]. The relationship between *n*_{FIR} and *N*_{FIR} is shown in Fig. 2(a)
. When *N*_{FIR} = 99, *n*_{FIR} becomes as large as 396. In addition, the least mean square (LMS) algorithm, which we used to determine the tap coefficient, also requires additional multiplications, whose number also increases in proportion to *N*_{FIR}. We set the maximum *N*_{FIR} at 99 to avoid the slow convergence of the tap coefficient calculation with the LMS for larger *N*_{FIR} values.

In FDE, the data were converted to the frequency domain with FFT, and the transfer function of the equalizer was determined in order to compensate for the non-ideal frequency response of individual components. The resolution of FDE is related to the FFT size, *N*_{FFT}, through the relationship Δ*f* = 4 Gsymbol/s × (2 sample/symbol) / *N*_{FFT}. With an FFT size of *N*_{FFT} = 8192, Δ*f* can be greatly reduced to 1 MHz without too much computational complexity. In this case, EVM was improved to 0.94% as shown in Fig. 1(b), which is a consequence of the higher resolution. Figure 1(c) shows the same data given in Fig. 1(a) and 1(b) and they are plotted as a function of the frequency resolution Δ*f*. The two equalization schemes yield almost the same EVM improvement for a lower Δ*f*, while the EVM degrades faster for a larger Δ*f* with FIR. This may be attributed to a less accurate estimation of the filter coefficient with the LMS algorithm.

Figure 2 shows a comparison of the computation complexity for time-domain equalization with an FIR filter and FDE. With FDE, the number of real-valued multiplications per symbol, *n*_{FDE}, is estimated as follows. It is known that an FFT involves *n*_{FFT} = 4*N*_{FFT} log_{2}(*N*_{FFT}) real-valued multiplications [10]. Since FDE employs both FFT and IFFT, it includes 2*n*_{FFT} multiplications. Furthermore, since one symbol is represented by two samples, an FFT with a size of *N*_{FFT} accounts for *N*_{FFT}/2 symbols. Therefore, the number of multiplications per symbol is estimated as *n*_{FDE} = 2*n*_{FFT} / (*N*_{FFT}/2) = 8log_{2}(*N*_{FFT}), i.e., it increases only logarithmically. The relationship between *n*_{FDE} and *N*_{FFT} is shown in Fig. 2(b). Figure 2(c) shows a comparison of *n*_{FIR} and *n*_{FDE} as a function of Δ*f*. This clearly shows the advantage of FDE in terms of the lower computation complexity especially for Δ*f* values as low as 1 MHz and below, whereas with FIR such a low Δ*f* is very difficult to realize due to the rapid increase in the computation complexity. For example, if we set Δ*f* = 1 MHz, FDE requires *N*_{FFT} = 8192 and thus *n*_{FDE} = 104. On the other hand, the required number of taps with FIR is *N*_{FIR} = 4000, which corresponds to *n*_{FIR} = 16000.

## 3. Experimental setup

We applied the FDE technique to 512 QAM transmission. The experimental setup is shown in Fig. 3 . At the transmitter, coherent light emitted from an acetylene frequency-stabilized CW fiber laser at 1538.8 nm with a 4 kHz linewidth was modulated by an optical IQ modulator driven with a 3 Gsymbol/s, 512 QAM baseband signal generated by an arbitrary waveform generator (AWG) with a pattern length of 4096. The AWG was operated at 12 Gsample/s with a 10-bit resolution. The bandwidth of the 512 QAM signal was reduced to 4.05 GHz by employing a Nyquist filter with a roll-off factor of 0.35 in the AWG.

The optical QAM was polarization-multiplexed with a polarization beam combiner and launched into a transmission link. In parallel, part of the laser output was divided in front of the IQ modulator, and its frequency was downshifted by 2.03 GHz against the carrier frequency. This signal was combined with the QAM data, and used as a pilot tone in the receiver for the OPLL. The transmission link comprised two spans of 80 km SSMF, whose loss (0.2 dB/km) was compensated with EDFAs and Raman amplifiers. The Raman amplifier was backward pumped and provided a 13~14 dB gain out of the total gain of 17.0 dB at each span. Figure 4 shows the bit error rate (BER) after a 160 km transmission for various powers launched into each span. From this result, the launched power was optimally set at –2 dBm. The OSNRs before and after transmission were 44.0 and 36.0 dB, respectively, measured with a 0.1 nm resolution.

On the receiver side, after passing through a 0.7 nm optical filter and an EDFA for preamplification, the transmitted QAM data were combined with a local oscillator (LO) and received by a polarization-diversity coherent receiver. The LO was a frequency-tunable fiber laser, whose phase was locked to the transmitted pilot tone via an OPLL. The detected signals were digitized at 40 Gsample/s and processed with an offline DSP. Here, the polarization demultiplexing of the X- and Y-polarization signals was carried out by using a polarization controller in front of the polarization diversity coherent receiver. Specifically, a transmitted tone signal was maximized or minimized along the two polarization principal axes of the polarization diversity coherent receiver. We employed polarization demultiplexing in the optical domain instead of MIMO processing, because the DSP for polarization demultiplexing a 512 QAM signal is very complex due to its ultra-high multiplicity.

In the DSP, we first compensated for linear and nonlinear impairments with a back-propagation method [11] by solving the Manakov equation in the reverse direction. This enabled us to compensate for the XPM between two orthogonal polarizations, which causes severe impairments during transmission. We then converted the data to the frequency domain with FFT to apply FDE. The FFT size was set at 8192 as in the previous section. The FFT operation was carried out with a double-precision floating-point format. The non-ideal frequency response of the individual components was mainly caused by the LiNbO_{3} (LN) IQ modulator in the present experiment. Specifically, surface acoustic waves generated by the piezoelectric effect in the LN crystal degraded the low-frequency response of the modulator [12]. After converting the data back to the time domain with IFFT, the equalized QAM signal was demodulated into binary data, and finally the BER was evaluated.

## 4. Experimental result

Figure 5 shows the constellation diagrams of the 512 QAM signal before and after a 160 km transmission. The corresponding BER characteristics are shown in Fig. 6 . In Fig. 6, the gray and pink curves are the results obtained with FIR. As shown in Fig. 5(a), the back-to-back constellation had error vector magnitudes (EVMs) of 1.0 and 0.85% when equalized with FIR and FDE, respectively. This improvement is a consequence of the ability of FDE to eliminate distortions caused by hardware imperfections in the transmitter and receiver with a resolution better than FIR. This improvement is also apparent in the back-to-back BER performance, as shown by the gray and black curves in Fig. 6. After a 160 km transmission, the EVM was improved from 1.50 to 1.24% by employing FDE as shown in Fig. 5(b), corresponding to the pink and red curves in Fig. 6. As can be seen in Fig. 6, the power penalty was greatly reduced from 7.0 to 3.0 dB as a result of FDE.

We also evaluated the possibility of extending the transmission distance from 160 km, and calculated the maximum transmission reach. The relationship between the transmission distance and BER is shown in Fig. 7 . As shown by the red symbols, a BER below the FEC threshold is still obtained after 240 km with FDE, which is difficult to achieve with FIR.

## 5. Conclusion

We have successfully demonstrated the excellent potential of FDE for reducing impairments in QAM transmissions with multiplicity levels as high as 512. Because of its high frequency resolution, FDE enables us to realize frequency response compensation down to a low frequency range, which is responsible for impairments in such an extremely high-order QAM. As a result, the power penalty was greatly reduced from 7.0 to 3.0 dB in a 512 QAM, 54 Gbit/s (3 Gsymbol/s)-160 km transmission. We also showed that the maximum distance of a 512 QAM transmission could be extended to 240 km with FDE.

## References and links

**1. **M. Nakazawa, K. Kikuchi, and T. Miyazaki, *High Spectral Density Optical Communication Technologies* (Springer 2010).

**2. **E. Ip and J. M. Kahn, “Digital equalization of chromatic dispersion and polarization mode dispersion,” J. Lightwave Technol. **25**(8), 2033–2043 (2007). [CrossRef]

**3. **S. J. Savory, “Digital filters for coherent optical receivers,” Opt. Express **16**(2), 804–817 (2008). [CrossRef] [PubMed]

**4. **M. Nakazawa, S. Okamoto, T. Omiya, K. Kasai, and M. Yoshida, “256-QAM (64 Gb/s) coherent optical transmission over 160 km with an optical bandwidth of 5.4 GHz,” IEEE Photon. Technol. Lett. **22**(3), 185–187 (2010). [CrossRef]

**5. **S. Okamoto, K. Toyoda, T. Omiya, K. Kasai, M. Yoshida, and M. Nakazawa, “512 QAM (54 Gbit/s) coherent optical transmission over 150 km with an optical bandwidth of 4.1 GHz,” ECOC’10, PD2.3.

**6. **R. Schmogrow, D. Hillerkuss, S. Wolf, B. Bäuerle, M. Winter, P. Kleinow, B. Nebendahl, T. Dippon, P. C. Schindler, C. Koos, W. Freude, and J. Leuthold, “512QAM Nyquist sinc-pulse transmission at 54 Gbit/s in an optical bandwidth of 3 GHz,” Opt. Express **20**(6), 6439–6447 (2012). [CrossRef] [PubMed]

**7. **D. Falconer, S. L. Ariyavisitakul, A. Benyamin-Seeyar, and B. Eidson, “Frequency domain equalization for single-carrier broadband wireless systems,” IEEE Commun. Mag. **40**(4), 58–66 (2002). [CrossRef]

**8. **K. Ishihara, T. Kobayashi, R. Kudo, Y. Takatori, A. Sano, E. Yamada, H. Masuda, and Y. Miyamoto, “Coherent optical transmission with frequency-domain equalization,” ECOC’08, We.2.E.3.

**9. **A. Sano, H. Masuda, T. Kobayashi, M. Fujiwara, K. Horikoshi, E. Yoshida, Y. Miyamoto, M. Matsui, M. Mizoguchi, H. Yamazaki, Y. Sakamaki, and H. Ishii, “69.1-Tb/s (432 x 171-Gb/s) C- and extended L-Band transmission over 240 Km using PDM-16-QAM modulation and digital coherent detection,” OFC’10, PDPB7.

**10. **J. C. Geyer, C. R. S. Fluger, T. Duthel, C. Schulien, and B. Schumauss, “Efficient frequency domain chromatic dispersion compensation in a coherent polmux QPSK-receiver,” OFC’10, OWV5.

**11. **C. Paré, A. Villeneuve, P.-A. Bélanger, and N. J. Doran, “Compensating for dispersion and the nonlinear Kerr effect without phase conjugation,” Opt. Lett. **21**(7), 459–461 (1996). [CrossRef] [PubMed]

**12. **R. L. Jungerman and C. A. Flory, “Low-frequency acoustic anomalies in lithium niobate Mach-Zehnder interferometers,” Appl. Phys. Lett. **53**(16), 1477–1479 (1988). [CrossRef]