## Abstract

We describe an experimental and numerical comparison of a probabilistically shaped (PS) 4096 QAM signal and a uniformly shaped 1024 QAM signal. Both modulation formats have the same transmission rate and a spectral efficiency of 15.3 bit/s/Hz. In the computational simulation, we compared the generalized mutual information (GMI) of both modulation formats with bit-wise soft decision decoding and bit-wise hard decision decoding. For bit-wise hard decision decoding with an overhead of 7%, a shaping gain of 1.8 dB was attained. Then we successfully transmitted a single-channel PS-4096 QAM signal for the first time in an all-Raman amplified 160-km link, in which the transmission performance was improved compared with a uniformly shaped 1024 QAM with the same transmission rate. Transmissions with a high QAM multiplicity were achieved by using an optical phase-locked loop (OPLL) and a frequency stabilized fiber laser locked to an acetylene absorption line. Thanks to a shaping gain based on a bit-wise hard decision decoder, the 1.9-dB power margin, which agreed with the simulation result to within 0.1 dB, was increased after transmission.

© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

Coherent optical transmission with a higher order modulation format enables us to increase spectral efficiency (SE) and reduce the cost per bit with limited wavelength resources [1]. Many transmission experiments with SEs exceeding 10 bits/s/Hz have already been reported [2–7]. In Ref [2], 512 quadrature amplitude modulation (QAM) transmission with a potential SE of 12.4 bit/s/Hz and a capacity of 54 Gbit/s was realized by adopting an optical phase-locked loop (OPLL) technique, and also the optical signal-to-noise ratio (OSNR) was improved by using Raman amplifiers. A 512 QAM transmission at 54 Gbit/s was also reported, where the SE reached as high as 15 bit/s/Hz thanks to the Nyquist pulse shaping having zero roll-off [3]. The higher multiplicity QAM transmission of a 1024 level signal at 60 Gbit/s was demonstrated by using a frequency-domain equalization (FDE) technique and a digital back-propagation (DBP) method, resulting in an SE of 13.8 bit/s/Hz [4]. 50.53 Gbit/s 1024 QAM orthogonal frequency division multiplexing (OFDM) with an SE of 11.7 bit/s/Hz was also achieved by using a pilot-based phase noise mitigation technique [5]. With the aim of achieving even higher multiplicity, a 2048 QAM signal transmission with a capacity of 66 Gbit/s and an SE of 15.3 bit/s/Hz has been realized by using an analog digital converter (ADC) with an increased effective number of bits (ENOB) of 7 bits and a lower phase noise condition obtained with an improved OPLL circuit [6]. In addition, a 58.67 Gbit/s, 2048 QAM transmission over a 3-km-long 12-core single-mode-fiber was achieved with an aggregated SE of 205.4 bit/s/Hz [7]. In addition, a field trial over legacy metro links using a single carrier 400 Gbit/s 128 QAM signal has been reported [8].

On the other hand, the probabilistic shaping that modifies the probability mass function (PMF) of the amplitudes in a higher order QAM is recently receiving considerable attention as a method that increases the SE under a fixed power constraint and that can approach the Shannon limit [9–11]. It is shown in Ref [9]. that shaping gains of up to 1.4 dB OSNR can be derived by probabilistically shaped (PS) 64 QAM compared with the uniformly-shaped (US) 16 QAM. A low-complexity PS scheme based on prefix codes is proposed in Ref [10], and a widely flexible transponder is demonstrated by using PS-256 QAM. A constellation shaped 256 QAM/1024 QAM transmissions, in which the probability mass function of the amplitudes was designed by a dyadic approximation, has been realized and a transmission distance enhancement of 300 km was attained [11]. Furthermore, a field trial of PS-64 QAM transmission over a trans-oceanic distance [12] has also been recently reported.

In this paper, we experimentally demonstrate for the first time a 3 Gsymbol/s polarization division multiplexed PS-4096 QAM transmission in an all-Raman amplified 160-km transmission. Then, we compare the transmission performance dependence on the launch power with US-1024 QAM. PS-4096 QAM was set at the same entropy as a US-1024 QAM of 10 bit. The net data rate was 55.2 Gbit/s and a spectral efficiency of 15.3 bit/s/Hz was achieved. After the transmission with hard decision decoding, a shaping gain of 1.9 dB could be successfully obtained with PS-4096 QAM at the lower launch power. We provide a detailed description of the generation of the PS-4096 QAM signal and compare the performance of both modulation formats in terms of the general mutual information (GMI) for bit-wise hard decision decoding and bit-wise soft decision decoding.

## 2. Generation and characteristics of probabilistically shaped 4096 QAM

Figure 1 shows the block diagram we used to generate the PS-4096 QAM in our experiment. This configuration is based on the probabilistic amplitude shaping (PAS) system [13]. The input of the PAS is a discrete memoryless binary source. In the PAS, a sequence ${U}^{k}$of uniformly distributed binary input of length $k$ enters a distribution matcher (DM), which shapes uniformly distributed input bits to the amplitudes with a desired probability ${P}_{A}$ over the symbol alphabet $A=\{1,3,\mathrm{...},{2}^{m+1}-1\}$. *m +* 1 is the number of bits assigned to a symbol in each I and Q axis in an output QAM constellation. The *m* value was 5 when using PS-4096 QAM. The DM outputs a probabilistically shaped amplitude shift keying (ASK) signal sequence ${A}^{n}$ of length *n*. The remaining bits from the discrete memoryless source ${U}^{n}$ is transformed into a sign sequence ${S}^{n}$ in a binary phase shift keying (BPSK) mapper, where it converts the bit to the amplitude, i.e. 0 to 1 and 1 to −1. After that, an ASK sequence ${X}^{n}$ which has a desired entropy, is obtained by multiplying an amplitude sequence ${A}^{n}$ to a sign sequence ${S}^{n}$. ${X}_{i}$ takes the value of symbol $\chi =\{\pm 1,\pm 3,\mathrm{...},\pm {2}^{m}-1\}$ This means that a binary sequence ${U}^{n}$ is assigned to the most significant bit (MSB), which is the leading bit among the bits assigned to a symbol, in a sequence ${X}^{n}$. Then, an ASK sequence is scaled by a factor of $\Delta $, which is larger than 0. Lastly, the sequence is divided into two paths for the I and Q axis, respectively. In a usual PAS system, part of the MSB is prepared for a parity bit from the FEC encoder; however, all the MSBs were generated from a binary source in the work described in this paper. This is because we compare the transmission performance of both PS-4096 QAM and US-1024 QAM, where we did not apply an actual FEC in our experiment.

We adopted a constant composition distribution matcher (CCDM) [14] as a DM. In the first shaping process, the amplitude candidates, which are derived by quantizing the probability mass function of the desired distribution, are stored in the memory. The CCDM then picks them up one by one in accordance with the input binary sequence in each process. To generate a PS-4096 QAM constellation with an entropy of 10 bits, the PAS generates I and Q symbols each with an entropy of 5 bits. In the PAS system, one bit out of the desired entropy (bit/symbol) of each axis is assigned as the MSB as described. Thus, bit sequences with lengths of around 4/5 and 1/5 from discrete memoryless binary source were allocated in the paths to the DM and the BPSK mapper. As for the former path, the CCDM outputs distribution shaped 32 ASK signals with an entropy of 4 bits. The amplitude levels of the ASK signals are all positive numbers because the distribution of the output constellation is symmetrical around 0. Figure 2(a) represents the PMF used to construct the PS-4096 QAM in the CCDM. This distribution follows the Maxwell-Boltzmann distribution to achieve the ultimate shaping gain [15]. The transformation efficiency from the input bit sequence to the output amplitude sequence, which is an important characteristic of the CCDM, is decided by the length of the output amplitude sequence processed at one time. Figure 2(b) shows the ratio of the input bit length to the output symbol length as a function of the latter. Since the desired entropy of the 32 ASK signals is 4 in this case, the ideal ratio of the input bit length to the output amplitude length is 4. As the output symbol length increases, the ratio approaches 4, corresponding to the entropy of shaped 32 ASK signals. As shown in Fig. 2(b), the ratio was almost saturated at an output of 16,384, thus we chose an output length of 16,384. The number of input bits was 65,389. In the latter path in the PAS, a random bit sequence with a length of 16,384, which is the same as that of the CCDM output, was mapped into BPSK signals with amplitudes of +/− 1. Since the CCDM output is a positive number, namely 32 ASK with an entropy of 4 bits, 64 ASK signals with an entropy of 5 bits were obtained by multiplying the CCDM output with the BPSK signals. Then the 16,384 symbols with an entropy of 5 bits was split into 8,192 symbols for the I and Q axes. We then obtained PS-4096 QAM with an entropy of 10 bits by modulating these I and Q symbols simultaneously.

When we consider the hardware implementation of a probabilistically shaped constellation, we must note that the entropy of both modulation formats, PS-4096QAM and US-1024 QAM, are the same in the PAS system, i.e. 10 bits, however, the nominal number of information bits between the distribution matcher and distribution dematcher is increased to 12 bits because the output constellation is based on a 4096 QAM. Therefore, it is necessary to increase the FEC throughput in accordance with the shaping ratio. Although, this may cause net coding gain (NCG) degradation in the probabilistically shaped constellations under a certain power consumption constraint, we did not consider this factor in this paper. We assumed that the FEC can achieve the same performance in both modulation formats.

Figure 3 shows an example constellation diagram and probabilistic distribution of the amplitudes in PS-4096 QAM and US-1024 QAM with an entropy of 10 bits. PS-4096 QAM has a high probability at the center of the constellation, and it becomes lower as it moves towards the outside as shown in the figure. In the US-1024 QAM, the probability of each amplitude is not constant because we used pseudo random bit sequences as the input.

To estimate the performance improvement with PS-4096 QAM, we compared the GMI of both modulation formtas, which depends on the signal-to-noise ratio (SNR) of a bit-wise hard decision decoder (HDD-BW) and bit-wise soft decision decoder (SDD-BW). Symbol-wise hard and soft decision decoders are not considered in this paper because we focus on the bit-interleaved coded modulation (BICM) because of its low computational complexity compared with symbol-wise operation. Figure 4 shows the GMI as a function of the SNR in PS-4096 QAM and US-1024 QAM with an HDD-BW (GMI_{HDD-BW}) and SDD-BW (GMI_{SDD-BW}) calculated with a Monte Carlo simulation. The solid line in Fig. 4 is the channel capacity. The GMI_{SDD-BW} is defined by the following equation [16],

*N*is the number of symbols,

*H*is the entropy determined by the constellation size and shaping ratio,

*m*is the number of bits defined by the constellation size,

*c*is the binary code sequence, and

*λ*is the log-likelihood ratio (LLR) of each bit. The LLR of each bit is defined as follows by assuming an additive white Gaussian noise (AWGN) channel.

_{k}= 0 or 1} is the subset of the symbols whose k-th bit equals 0 or 1,

*y*is the received symbol,

_{l}*x*is one of the constellation points, and

*σ*is the noise variance. The GMI

^{2}_{HDD-BW}is derived by using a binary symmetric channel capacity function as follows [17],

*ε*represents error probability. Interestingly, the GMI

_{SDD-BW}of US-1024 QAM was almost the same as GMI

_{HDD-BW}of PS-4096 QAM in the high GMI region above 9 bits/QAM symbol, thus the same information rate is achievable in PS-4096 QAM even when using HDD while US-1024 QAM is using SDD. We used HD FEC with a 7% overhead (OH) [18] by considering a low power consumption and simplicity of implementation. In this case, we can obtain an SNR increase of approximately 1.8 dB in PS-4096 QAM compared with US-1024 QAM.

## 3. Experimental setup for PS-4096 QAM transmission

Figure 5 shows the experimental setup. The light source was an acetylene (C_{2}H_{2}) frequency stabilized fiber laser. The output wavelength was 1538.8 nm, and its linewidth was about 4 kHz. By using an arbitrary waveform generator (AWG), a PS-4096 QAM or US-1024 QAM signal of 3 Gsymbols/s was generated. The AWG operated at 9 Gsamples/s, in which the ENOB and 3 dB bandwidth of the D/A were 9 bits and 5 GHz, respectively. The length of the data frame was 8,192 symbols. The 16,384 symbol outputs from the CCDM were distributed to I and Q. We used a root-raised-cosine Nyquist filter with a roll-off factor of 0.2. Here, as shown in the inset of Fig. 4, a pilot tone, whose frequency is shifted by 1.8 GHz from the carrier frequency, is prepared adjacent to the signal spectrum. This tone was used for the optical phase tracking of the local oscillator (LO) in the OPLL operation. The QAM data and the pilot tone signal were coupled into a transmission fiber following a polarization division multiplexing emulator (PDME). The transmission fiber consisted of four 40-km spans of ultra-large effective area (ULA) fiber; its loss and A_{eff} were 0.18 dB/km and 153 μm^{2}, respectively. All span losses were compensated for with a Raman amplifier, resulting in a gain of 8.8 dB. At the receiver, the optical phase of the LO was adjusted to that of the transmitting laser using an OPLL by minimizing the phase error between the tone signal and a synthesizer operating at 1.8 GHz. The transmitted signal and the phase locked LO signal were then coupled to a 90° optical hybrid and detected with a balanced PD with a bandwidth of 43 GHz. The detected signal was A/D converted at 40 Gsamples/s, in which the ENOB and 3 dB bandwidth were 7 bits and 13 GHz, respectively. In the offline DSP, first, digital back propagation is performed, and then the hardware distortion is equalized in the frequency division, and pilot-assisted time domain equalization is employed. Pilot symbols were intermittently inserted in the data series at a rate of 1% on the time axis. It should be noted that this system employs OPLL for phase locking, which enables us to tracking the channel distortion less frequently and therefore reduce the pilot symbol ratio to 1%. The length of the header sequence required for frame synchronization was 50 symbols. When we locked the signal phase in the optical domain, the residual phase noise was compensated for by the maximum likelihood phase estimation (MLPE) [19]. To estimate the system performance, the pre-FEC BER was calculated from the equalized signal.

Figures 6(a) and 6(b) show the spectra obtained back-to-back and after a 160 km transmission, respectively. To increase the OSNR, we adopted four 40-km spans of ULA fiber with Raman amplifiers. We also replaced all the connectors, except for those between the 90° optical hybrid with balanced PDs, with angled physical contact (APC) connectors, to reduce the optical reflection. The OSNR was as high as 50.7 dB at back-to-back, and was reduced to 44.9 dB after a 160 km transmission with a launch power of −1 dBm.

## 4. Results and discussion

Figures 7(a) and 7(b) show the OSNR dependence of the pre-FEC BER for PS-4096 QAM and US-1024 QAM for back-to-back and 160 km transmissions, respectively. We evaluated the pre-FEC BER of both modulation formats with an HD-FEC of 7% (a FEC threshold of 2x10^{−3}) [18]. Although the use of the GMI instead of the pre-FEC BER has been proposed as a performance evaluation method that is independent of modulation format when using SD-FEC [16], the pre-FEC BER is also applicable when using HD-FEC. In Fig. 7(a), the SNR difference between PS-4096 QAM and US-1024 QAM was 1.7 dB at a 7% FEC threshold of 2x10^{−3}. This is similar to the shaping gain of 1.8 dB obtained in the numerical calculation shown in Fig. 4. In the high OSNR region of above 36 dB, we can see a floor in both modulation formats. This is because the maximum electrical SNR is limited by the ENOB of the A/D. In Fig. 7(b), the measured pre-FEC BERs of 1.98x10^{−4} and 1.65x10^{−4} in PS-4096 QAM and US-1024 QAM, respectively, were below the 2x10^{−3} HD-FEC limit. They were achieved for a launch power of −5 dBm, and the spectral efficiency reached as high as 15.3 bit/s/Hz taking account of the header for synchronization, pilot symbol OH, and a 7% FEC OH. We varied the launch power from −17 to −1 dBm in 2 dB increments. At the FEC limit, a 1.9 dB power margin was obtained by probabilistic shaping. Since the performance degradation in a transmission with such a low launch power is mainly caused by an insufficient SNR, the shaping gain, which brings an SNR increase, can directly extend the power margin. This margin agrees well with the SNR increase obtained in our numerical simulation in Fig. 4 with a 0.1 dB difference. This large gain is caused by the adoption of the HDD-BW as a decoder. The GMI difference between two modulation formats with the HDD-BW is larger than that with the SDD-BW as shown in Fig. 4. In the high launch power region from −7 dBm to −1 dBm, the BER of PS-4096 QAM is slightly worse than that of US-1024 QAM. This is because the nonlinear distortion increased when the PS modulation format had a higher PAPR than the US modulation format [20]. The PAPR of PS-4096 QAM was almost double that of US-1024 QAM. The constellations of PS-4096 QAM back-to-back and after a 160 km transmission with a launch power of −5 dBm are shown in Figs. 8(a) and 8(b), respectively.

If we take account of the FEC code rate, the transmission rate of the two schemes becomes slightly different. To realize the same transmission rate as US-1024 QAM, the entropy of the PS-4096 QAM should be increased from 10 bits to 10.13 bits. A numerical simulation shows that this 1.3% increase yields a 0.5 dB degradation in the SNR tolerance, but the SNR tolerance improvement of more than 1 dB still exists compared to US-1024 QAM.

## 5. Conclusion

We have successfully transmitted a single-channel PS-4096 QAM signal for the first time with a spectral efficiency of 15.3 bit/s/Hz in an all-Raman amplified 160 km transmission. Compared with US-1024 QAM, we successfully achieved a 7% HD-FEC limit (2x10^{−3}) at a 1.9 dB lower launch power by using probabilistic shaping. We have shown that even with a 1024 QAM class transmission where a large SNR is required, we can obtain an additional power margin by using PS-4096 QAM.

## Funding

This work is supported by the “Research and Development Project toward 5G Mobile Communication Systems” of the Ministry of Internal Affairs and Communications, Japan

## References and links

**1. **M. Nakazawa, K. Kikuchi, and T. Miyazaki, *High Spectral Density Optical Transmission Technologies* (Springer, 2010).

**2. **S. Okamoto, K. Toyoda, T. Omiya, K. Kasai, M. Yoshida, and M. Nakazawa, “512 QAM (54 Gbit/s) coherent optical transmission over 150 km with an optical bandwidth of 4.1 GHz,” in *Proceedings of European Conference on Optical Communication* (IEEE, 2010), paper PD2.3. [CrossRef]

**3. **R. Schmogrow, D. Hillerkuss, S. Wolf, B. Bäuerle, M. Winter, P. Kleinow, B. Nebendahl, T. Dippon, P. C. Schindler, C. Koos, W. Freude, and J. Leuthold, “512QAM Nyquist sinc-pulse transmission at 54 Gbit/s in an optical bandwidth of 3 GHz,” Opt. Express **20**(6), 6439–6447 (2012). [CrossRef] [PubMed]

**4. **Y. Koizumi, K. Toyoda, M. Yoshida, and M. Nakazawa, “1024 QAM (60 Gbit/s) single-carrier coherent optical transmission over 150 km,” Opt. Express **20**(11), 12508–12514 (2012). [CrossRef] [PubMed]

**5. **M.-F. Huang, D. Qian, and E. Ip, “50.53-Gb/s PDM-1024QAM-OFDM transmission using pilot-based phase noise mitigation,” in *Proceedings of Opto-Electronics and Communications Conference* (IEEE, 2011), paper PDP1.

**6. **S. Beppu, K. Kasai, M. Yoshida, and M. Nakazawa, “2048 QAM (66 Gbit/s) single-carrier coherent optical transmission over 150 km with a potential SE of 15.3 bit/s/Hz,” Opt. Express **23**(4), 4960–4969 (2015). [CrossRef] [PubMed]

**7. **D. Qian, E. Ip, M.-F. Huang, M.-J. Li, and T. Wan, “698.5-Gb/s PDM-2048QAM transmission over 3 km multicore fiber,” in *Proceedings of European Conference on Optical Communication* (IEEE, 2013), paper Th.1.C.5.

**8. **G. Khanna, T. Rahman, E. de Man, E. Riccardi, A. Pagano, A. C. Piat, S. Calabro, B. Spinnler, D. Rafique, U. Feiste, H. de Waardt, S. K. Huug, H. Bernd, D. Norbert, B. Tomislav, M. Bohn, and A. Napoli, “Single-carrier 400G 64QAM and 128QAM DWDM field trial transmission over metro legacy links,” IEEE Photonics Technol. Lett. **29**(2), 189–192 (2017). [CrossRef]

**9. **F. Buchali, F. Steiner, G. Böcherer, L. Schmalen, P. Shulte, and W. Idler, “Rate adaptation and reach increase by probabilistically shaped 64-QAM: An experimental demonstration,” J. Lightwave Technol. **34**(7), 1599–1609 (2016). [CrossRef]

**10. **S. Chandrasekhar, B. Li, J. Cho, X. Chen, E. C. Burrows, G. Raybon, and P. J. Winzer, “High-spectral-efficiency transmission of PDM 256-QAM with parallel probabilistic shaping at record rate-reach trade-offs,” in *Proceedings of European Conference on Optical Communication* (IEEE, 2016), paper Th.3.C.1.

**11. **M. P. Yankov, F. Da Ros, E. P. da Silva, S. Forchhammer, K. J. Larsen, L. K. Oxenløwe, M. Galili, and D. Zibar, “Constellation shaping for WDM systems using 256QAM/1024QAM with probabilistic optimization,” J. Lightwave Technol. **34**(22), 5146–5156 (2016). [CrossRef]

**12. **J. Cho, X. Chen, S. Chandrasekhar, G. Raybon, R. Dar, L. Schmalen, E. Burrows, A. Adamiecki, S. Corteselli, Y. Pan, D. Correa, B. McKay, S. Zsigmond, P. Winzer, and S. Grubb, “Trans-Atlantic field trial using probabilistically shaped 64-QAM at high spectral efficiencies and single-carrier real-time 250-Gb/s 16-QAM,” in *Proceedings of Optical Fiber Communication Conference* (Optical Society of America, 2017), paper Th5B.3. [CrossRef]

**13. **G. Böcherer, P. Shulte, and F. Steiner, “Bandwidth efficient and rate-matched low-density parity-check coded modulation,” IEEE Trans. Inf. Theory **63**(12), 4651–4665 (2015).

**14. **P. Shulte and G. Böcherer, “Constant composition distribution matching,” IEEE Trans. Inf. Theory **62**(1), 420–434 (2016).

**15. **F. R. Kschischang and S. Pasupathy, “Optimal nonuniform signaling for Gaussian channels,” IEEE Trans. Inf. Theory **39**(3), 913–929 (1993). [CrossRef]

**16. **A. Alvarado, E. Agrell, D. Lavery, R. Maher, and P. Bayvel, “Replacing the soft-decision FEC limit paradigm in the design of optical communication systems,” J. Lightwave Technol. **33**(20), 4338–4352 (2015). [CrossRef]

**17. **A. Sheikh, A. G. Amat, and G. Liva, “Achievable information rates for coded modulation with hard decision decoding for coherent fiber-optic systems,” J. Lightwave Technol. **35**(23), 5069–5078 (2017). [CrossRef]

**18. **J. G. Proakis, *Digital Communications* (McGraw Hill, 2000).

**19. **S. Zhang, P. Y. Kam, C. Yu, and J. Chen, “Laser linewidth tolerance of decision-aided maximum likelihood phase estimation in coherent optical M-ary PSK and QAM systems,” IEEE Photonics Technol. Lett. **21**(15), 1075–1077 (2009). [CrossRef]

**20. **T. Fehenberger, A. Alvarado, G. Böcherer, and N. Hanik, “On probabilistic shaping of quadrature amplitude modulation for the nonlinear fiber channel,” J. Lightwave Technol. **34**(21), 5063–5073 (2016). [CrossRef]