## Abstract

A novel, probabilistically shaped star-CAP-16/32 modulation based on constellation design with honeycomb-like decision regions is proposed in this paper. The proper geometric structural design of the star constellation, along with the probabilistic shaping, is able to achieve better improvement with regards to constellation figure of merit (CFM) and bit error rate (BER) performance. A 25-km standard single-mode fiber (SSMF) data transmission employing the proposed PS star-CAP modulation scheme is successfully demonstrated. Experiment results show that the proposed PS star-CAP-16 in _{${C}_{4,4,4,4}$} excels the traditional PS star-CAP-16 in _{${C}_{8,8}$} by 1.5 dB in receiver sensitivity at the BER of _{$1\times {10}^{-3}$}. At the same time, the novel PS star-CAP-32 with entropy of 4.4 bits/symbol defeats the uniform star-CAP-32 by 1.6 dB improvement under the same bit rate.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

As the bridge between customer premises network (CPN) and metropolitan area network/wide area network (MAN/WAN), optical access network must provide better bandwidth performance to accommodate the ever-growing traffic requirements of the emerging augmented reality/virtual reality (AR/VR), internet of things (IoT), cloud computing and other high-rate services. In the actual application scenario, the requirement for limited cost in the process of implementation and deployment gives intensity modulation/ direct detection (IM/DD) system an opportunity to stand out as a promising and economical solution due to its low cost, compact footprint, and low power consumption when compared with coherent system [1,2]. A range of modulation schemes based on IM/DD have been investigated including pulse amplitude modulation (PAM) [3], discrete multi-tone (DMT) modulation [4,5] and carrier-less amplitude/phase (CAP) modulation [6,7]. Among these techniques, CAP is considered as an attractive candidate since it can achieve high spectral efficiency owing to its multilevel signal constellations and spectral shaping [6]. Moreover, finite impulse response (FIR) filter can be used in CAP to generate quadrature signals without use of any mixers, which significantly and effectively reduces the system complexity and implementation cost [7].

Besides, probabilistic shaping (PS) technology, aimed at increasing the transmitted probabilities of the inner constellation points while reducing the probabilities of outside points, has attracted much more focuses and been a research hot spot in recent years, which can be attributed to its higher spectral efficiency and capacity [8]. This technology can significantly reduce the average constellation power and excel in the robustness against noise, thus improving the system performance. On one hand, in the cases of long-haul transmission, a PS-16QAM combined with low-density parity-check (LDPC) code has been applied in long-haul optical system with wavelength-division-multiplexed (WDM) to increase the system reach by up to 7% [9]. Similarly, in a single-carrier 400G data transmission, the implementation of PS-64QAM has achieved up to 300% reach enhancement over regular 64QAM [10]. And a C-band EDFA-only trans-Pacific transmission using 84 channels of 49 GBd truncated probabilistic constellation shaping 64QAM (TPCS-64QAM) and 7 spatially-coupled LDPC (SC-LDPC) codes to achieve a net throughput of 24.6 Tb/s and a spectral efficiency of 5.9 b/s/Hz after 10,285 km straight line has been reported [11]. On the other hand, in the cases of short-reach communication, PS is also widely applied. A net data rate of 28.95 Gbit/s/λ PS-1024-QAM discrete Fourier transform-spread (DFT-S) OFDM over 40km SSMF has been demonstrated in an IM/DD system, which can achieve 1.85 Gbit/s capacity increment [12]. Also, a PS scheme based on symbol-level labeling and rhombus-shaped modulation has been proposed in passive optical network (PON) system, which can outperform the traditional signaling approach by 2 dB in the received optical power [13]. Furthermore, the hybrid probabilistic and geometric shaping has also been widely studied. The generalized pairwise optimization algorithm is adopted to design geometric shaping multi-dimension constellation together with PS based on two objective functions: minimize the bit error rate (BER) or maximize the modulation capacity at the given signal noise ratio (SNR) and bits mapping [14]. 70.46 Tb/s transmission over 7,600 km and 71.65 Tb/s transmission over 6,970 km with C + L band EDFAs using a multidimensional coded modulation format with hybrid probabilistic and geometric constellation shaping are demonstrated [15]. And a novel geometric shaped 32QAM has been designed using the base points in the first quadrant such that it can work with probabilistic shaping achieved via quadrant folding and 2-dimensional distribution matcher [16]. However, most of the current probabilistic shaping studies are focused on square-shaped constellation while little has been conducted on star-shaped constellation. Specifically, star-shaped constellation is composed of multiple rings with various amplitudes, and the constellation points are uniformly distributed on each ring. This unique attribute of the star-shaped constellation can give a considerable room for flexibility and optimization in the process of constellation design. A great performance improvement can be achieved when applying probabilistic shaping in star-shaped constellation rather than square-shaped one, which has been verified in [17]. Therefore, the appropriate structural design of star-shaped constellation combined with probabilistic shaping technique can be a favorable approach to improve the system performance.

In this paper, we propose a novel probabilistically shaped star-CAP modulation based on constellation design with honeycomb-like decision regions. In this process, aimed at maximizing the constellation figure of merit (CFM), we increase the number of rings with different amplitudes in the star constellation and reduce the number of constellation points on each ring, in order to converge most constellation points inwards. Meanwhile, probabilistic shaping is employed to optimize the probability distribution of the constellation points in the proposed novel star constellation. The geometric structural design in parallel with probabilistic shaping is able to reduce the constellation average power and enhance the CFM, thus improving the system BER performance. On the basis of the proposed scheme, the novel PS star-CAP-16/32 modulation is elaborated to show the superiority. Theoretical analysis is given to verify the favorable performance of the proposed PS star-CAP-16/32, as well as the successful experimental demonstration in an IM/DD system.

## 2. Principle of probabilistically shaped star-CAP modulation based on constellation design with honeycomb-like decision regions

For a given constellation _{$C$}, the number of its points can be indicated by _{$\left|C\right|$}, namely the size of the constellation. When the size _{$\left|C\right|$} is determined, _{${d}_{\mathrm{min}}^{2}(C)$}, the minimum squared distance between its points, and _{$P(C)$}, the average power of the constellation, are the two key parameters impacting the SNR efficiency of the constellation. Specifically, the minimization of _{$P(C)$} in the case of a given _{${d}_{\mathrm{min}}^{2}(C)$}, or the maximization of _{${d}_{\mathrm{min}}^{2}(C)$} in the case of a given _{$P(C)$}, both will significantly mitigate the noise interference, thus bringing out a favorable improvement in SNR efficiency and BER performance. Hence, _{${d}_{\mathrm{min}}^{2}(C)$} to _{$P(C)$} ratio is introduced as CFM shown in Eq. (1), which can be a quantitative measurement index of the constellation performance [18].

_{$\alpha $}, both

_{${d}_{\mathrm{min}}^{2}(C)$}and

_{$P(C)$}will be multiplied by

_{${\alpha}^{2}$}, maintaining the value of CFM unchanged. Therefore, in the process of constellation design,

_{${d}_{\mathrm{min}}^{2}(C)$}is usually normalized to 1, while the reduction of average power

_{$P(C)$}is being fully considered in the orchestration of geometric structural design and probability distribution optimization, in order to improve the CFM and SNR efficiency, thus enhancing the robustness against noise and BER performance of the communication system.

The constellation mapping rule of the traditional star-CAP-16 modulation is illustrated in Fig. 1(a), where 16 points are uniformly distributed in the two concentric rings with different amplitudes, and every 8 points are distributed in a ring with the same phase difference of _{$\pi /4$}. This constellation is labeled as _{${C}_{8,8}$}. Also, Gray mapping is adopted to reduce the BER to its maximum extent in the same SER (symbol error rate). Figure 1(b) illustrates the principle of constellation design on geometric structure. As can be observed in Fig. 1(b), the Euclidean distance between the neighboring points in the inner ring is smaller, which is denoted as _{${d}_{\mathrm{min}}$} and marked with brown segment. In addition, _{${d}_{\mathrm{min}}$} is fixed as 1 and hence the set of constellation points _{$\Xi $} can be represented as follows:

_{$r(l)$}denotes the radius of the two rings and

_{$\varphi (i)$}is the phase of the constellation points. According to Eq. (1), the CFM of the traditional star-CAP-16 can be calculated as below:

Aimed at maximizing the CFM, a novel probabilistically shaped star-CAP-16 modulation based on constellation design with honeycomb-like decision regions is proposed shown in Fig. 2. In our proposed scheme, both constellation design on geometric structure and probability distribution optimization of the constellation points are given comprehensive consideration to significantly reduce the average power _{$P(C)$} and improve the CFM.

Firstly, the constellation design in the presence of geometric structure is carried out. Specifically, Fig. 2(a) depicts the constellation mapping rule of the novel star-CAP-16 modulation, in which the constellation is notated as _{${C}_{4,4,4,4}$}. Different from the traditional star-CAP-16, the constellation points are evenly distributed in the four concentric rings with various amplitudes, where each ring is occupied by four signal points with the same phase difference of _{$\pi /2$}. In this way, the reduction of constellation points on each ring will converge more points inwards to generally reduce the average power when the minimum Euclidean distance stays fixed. Additionally, as the probabilities of constellation points in each ring are discretely-chosen in the form of Maxwell-Boltzmann distribution, so when the number of rings increases, much more discrete values can be picked up to design the probabilities of constellation points, which provides an outstanding opportunity for the approximation of Maxwell-Boltzmann distribution to fully improve the optimization effects of probabilistic shaping. Also quasi-Gray mapping is adopted to reduce the BER in the same SER. Figure 2(b) elaborates the principle of the geometric structural design in the novel constellation, where the minimum Euclidean distance _{${d}_{\mathrm{min}}$} is marked by brown segment and fixed as 1. The four points in the innermost ring constitutes a square with the side length of _{${d}_{\mathrm{min}}$}. Based on the four sides of this square, four equilateral triangles can be constructed outwards, where four points can be located on the other vertexes to achieve the smallest radius of the second ring. Then the eight points in the third and outermost ring are established by stretching the eight points in the innermost and second rings by _{${d}_{\mathrm{min}}$} outwards in the direction of the radius respectively, maintaining the minimum Euclidean distance unchanged. In this way, the amplitude _{$r\left(l\right)$} and phase _{$\varphi \left(i\right)$} of the data modulation for the proposed novel star-CAP-16 can be expressed as follows:

Following the above procedure, PS is employed to optimize the probability distribution of the points in the new constellation. In this process, the geometric positions of the constellation points remain unchanged. And by increasing the probabilities of the points in the inner rings while at the same time decreasing those in the outer rings, the average power is reduced to improve the CFM. To be more specific, for the additive Gaussian noise (AWGN) channel, it has been proven that the Maxwell-Boltzmann distribution is the optimal choice for amplitude ring probabilities to minimize the average power of a given constellation [19]. And this Maxwell-Boltzmann distribution can be expressed as follows [20]:

where_{$\nu $}is optimized for the channel and different

_{$\nu $}indicates different probability distributions accompanying corresponding entropy.

_{${A}_{\nu}$}is utilized to ensure that all the probabilities are summed up to 1. And

_{$x$}denotes the points in the

_{$\Xi $}. In this paper, the constant composition distribution matching (CCDM) scheme is adopted as the distribution matcher (DM) to accommodate the optimization of probability distribution in the novel star-CAP-16 [21]. Figure 2(d) illustrates the histogram of the probability distribution when the entropy is 3.6 bits/symbol. It can be shown that after probabilistic shaping, the constellation points located in the same ring have the same probabilities, while the outward expansion of the ring brings down the probabilities so as to be subject to Maxwell-Boltzmann distribution.

Moreover, the comparison of CFM between the traditional star-CAP-16 and the proposed novel star-CAP-16 is carried out to theoretically analyze the enhancing effect of our scheme in presence of geometric structure and probabilistic shaping, which is illustrated in Table 1. It can be seen that when the entropy of both schemes are 4.0 bits/symbol in the absence of probabilistic shaping, the novel star-CAP-16 outperforms the traditional star-CAP-16 by 0.0831 CFM improvement, which can be attributed to the geometric shaping gain introduced by the convergence of constellation points in a multi-ring structure. And it is worth mentioning that this performance enhancement of the novel star-CAP-16 does not require higher digital signal processing (DSP) complexity. On top of the above analysis, the introduction of probabilistic shaping changes the probability distributions of both schemes in a way that the inner rings are of larger probabilities than the outer ones instead of uniform distribution, thus improving the CFM and reducing the entropy in certain ranges. However, due to the much larger amounts of rings in the proposed novel star-CAP-16, in which the radius of the outmost ring is approximately equal to that of the traditional star-CAP-16 while the innermost ring is much smaller, the probability pattern of the constellation points is more closed to the Maxwell-Boltzmann distribution and much more points are concentrated to the areas with lower energy in the constellation. When the entropy is 3.3 bits/symbol, the CFM gain introduced by PS for the novel star-CAP-16 is 0.5233, which is more than twice larger than the 0.2418 CFM gain of the traditional star-CAP-16. This huge advantage can be clearly displayed in the histogram in Fig. 3. In addition, the theoretical MI (mutual information) vs. SNR curves of both the traditional and proposed star-CAP-16 are displayed in Fig. 4, where the probability distribution with the entropy of 3.6 bits/symbol is selected. It can be seen that with or without the implementation of probabilistic shaping, our proposed novel scheme can achieve higher SNR gain compared with the traditional one. Moreover, with such higher shaping gain achieved, the necessity for the finer instrument performance (e.g., DAC and ADC with higher resolution) is not required since both the novel and traditional schemes have the same quantization order, thus enabling the application of the proposed scheme for high-speed optical communication in a practical and low-cost manner.

Similarly, embracing the goal of maximizing CFM, a novel probabilistically shaped star-CAP-32 modulation with honeycomb-like regions is realized by combining geometric structural design with probabilistic shaping shown in Fig. 5. The constellation mapping rule is illustrated in Fig. 5(a). Thirty-two constellation points are evenly distributed on the four concentric rings with different amplitudes in a way that the phase difference is _{$\pi /4$} for the same ring while the points are interleaved with phase difference of _{$\pi /8$} on the neighboring rings. Figure 5(b) describes the principle of geometric structural design for the new constellation. Wherein the minimum Euclidean distance _{${d}_{\mathrm{min}}$} is fixed as 1 and marked by brown segment in the schematic. Firstly, eight points are evenly distributed on the innermost ring with _{${d}_{\mathrm{min}}$} among them. These points make up a regular octagon, and from its all sides eight equilateral triangles can be formed outwards to determine the eight points on the second ring. Similarly, by further forming a rhombus outwards, the eight points can be obtained to ensure the third ring reach the smallest radius. Finally, in order to maintain the minimum Euclidean distance constant, the eight constellation points on the outmost ring can be located via extending the radius of the second ring by _{${d}_{\mathrm{min}}$}. In the above process of geometric structural design, the amplitude _{$r\left(l\right)$} and phase _{$\varphi \left(i\right)$} can be denoted as follows:

## 3. Experiment and results

An experiment employing IM/DD system, illustrated in Fig. 7, is carried out to verify the superiority of the proposed PS star-CAP-16/32 modulation. At the transmitter, the offline DSP is adopted to realize CAP modulation and an arbitrary waveform generator (AWG, TekAWG70002A) is used to generate the electrical CAP signal. Specifically, original input bits with independent Bernoulli (1/2) distribution are firstly transformed into a sequence of output symbols with a desired probability distribution by a distribution matcher (DM), followed by the procedure of constellation mapping mentioned above. Then the mapped symbols are up-sampled with a sampling factor of 4. It is worth mentioning that the sampling factor is determined by the AWG sampling rate to data baud rate ratio. For CAP generation, the real and imaginary parts of the up-sampled sequence are split and processed by two orthogonal digital shaping filters _{${h}_{I}(t)$} and _{${h}_{Q}(t)$}, respectively. The impulse responses of the two filters constitute a Hilbert transform pair, which can be expressed as [22]:

_{$g(t)$}is the baseband square-root-raised-cosine shaping filter with a roll-off coefficient of 0.2, and

_{${f}_{c}$}is the carrier frequency. Afterwards, the outputs of the filters are combined together in the form of subtraction to get an electrical CAP signal, which can be expressed as [1]:where

_{${x}_{n}$}and

_{${y}_{n}$}denote the real and imaginary components of the up-sampled signal,

_{$n$}is the symbol index, and

_{$T$}is the symbol period. The corresponding electrical waveform generated by AWG with maximum sampling rate of 25 GSa/s, amplified by an electrical amplifier (EA), is used to drive the Mach-Zehnder modulator (MZM). In our experiment, a continuous wave (CW) laser operated at 1550 nm with an optical power of 10 dBm serves as the light source, which is injected into MZM to fulfil the intensity modulation. In the end, the generated optical CAP signal is ready for transmission in a 25 km standard single-mode fiber (SSMF).

At the receiver, a variable optical attenuator (VOA) is placed after the fiber link to adjust the received optical power for the optical signal detection and conversion by a photodiode (PD) with bandwidth of 40 GHz. And a mixed signal oscilloscope (MSO, TekMSO73304DX) with sampling rate of 100 GSa/s is utilized to sample the electrical signal for further offline DSP corresponding to the one at the transmitter. In the offline DSP, the newly-generated signal _{$r(t)$}, resampled by MSO, are first passed through the two matched filters which are the time reverse of _{${h}_{I}(t)$} and _{${h}_{Q}(t)$} to separate the real and imaginary components [1]:

^{−9}or even zero. In this sense, the sequence length should be long enough for the BER performance comparison, and the results may not be obvious as the BER is too small to be clearly compared. Therefore, FEC coding/decoding is not adopted and the BER is calculated before inverse distribution matching in the experiment.

Figure 8 illustrates the measured BER as a function of received optical power for uniform star-CAP-16 signals in the constellation _{${C}_{8,8}$} and _{${C}_{4,4,4,4}$} before and after 25 km transmission, respectively. The baud rate is set as 5 Gbaud and according bit rate is 20 Gbit/s in the experiment. Due to the chromatic dispersion, noise interference and so on in fiber channel, the power penalty of about 0.3 dB is required for both modulation schemes after 25 km transmission. As can be seen from the two BER curves after transmission, when the received optical power is greater than −19.6 dBm, our proposed novel star-CAP-16 is superior to the traditional one in BER performance, and this trend is more favorable as the two curves deviate further from each other when the received optical power increases. Whereas BER performance for both schemes are deteriorating basically to the same numerical level when the received optical power is less than −19.6 dBm. At the BER of _{$1\times {10}^{-3}$}, the receiver sensitivity of the proposed novel star-CAP-16 is −16.4 dBm, which improves by 0.8 dB compared with −15.6 dBm of the traditional one. This advantage can be attributed to the well-designed geometric structure of our proposed constellation, where the inward convergence of most constellation points reduces the average power when the minimum Euclidean distance stays fixed, thus acquiring better BER performance. Moreover, the two received constellation diagrams at b2b case are also shown as insets in Fig. 8 when the received optical power is −17 dBm.

Figure 9 depicts the measured BER as a function of received optical power for both traditional and novel modulation schemes after 25 km transmission with/without PS, respectively. In order to achieve the same bit rate of 20 Gbit/s, the baud rate of uniform star-CAP-16 signal is fixed at 5 Gbaud while the baud rate of PS star-CAP-16 signal with the entropy of 3.6 bits/symbol is set as 5.56 Gbaud. It can be observed that PS star-CAP-16 in _{${C}_{4,4,4,4}$} and PS star-CAP-16 in _{${C}_{8,8}$} have got receiver sensitivities of −18.3 dBm and −16.8 dBm at the BER of _{$1\times {10}^{-3}$} respectively. The novel PS star-CAP-16 shows a 1.5 dB improvement in the case of same entropy and bit rate. From a different perspective, probabilistic shaping can improve the receiver sensitivity of the novel star-CAP-16 in _{${C}_{4,4,4,4}$} by 1.9 dB, while only 1.2 dB improvement can be obtained for traditional star-CAP-16 in _{${C}_{8,8}$}. The increased number of rings with different amplitudes in the proposed novel constellation is the key factor for this phenomenon, which enhances the performance of probabilistic shaping in presence of average power reduction. In addition, the received constellation diagrams of the four modulated signals are presented as insets (a-d) in Fig. 9 when the received optical power is −17 dBm. It can be seen that for both PS star-CAP-16 modulation schemes, the inner constellation points are of larger transmitted probabilities, and constellation diagrams are more clearly viewed than those of uniform star-CAP-16.

The BER curves of the proposed novel star-CAP-32 with different probability distributions after 25 km transmission are also measured, and the measured results are shown in Fig. 10. In the experiment, the baud rate of uniform star-CAP-32 is set as 5 Gbaud, while the baud rate of PS-star-CAP-32 with different entropies is set as 5.21, 5.43 and 5.68 Gbaud respectively, thus achieving the same bit rate of 25 Gbit/s. It can be shown that the better BER performance can be achieved with probabilistic shaping, and this BER performance can be better enhanced in the modulation schemes with smaller information entropy. At the BER of _{$1\times {10}^{-3}$}, the PS star-CAP-32 with entropy of 4.8, 4.6 and 4.4 bits/symbol outperform the traditional uniform star-CAP-32 by 0.9, 1.2 and 1.6 dB in receiver sensitivity, respectively, which corresponds with our theoretical analysis. However, it should be pointed out that another reason for the enormous gain obtained by PS under the same bit rate is at the cost of system spectral efficiency in some extent. The received constellation diagrams of modulation schemes with different probability distributions are also illustrated in Fig. 10. The clearness of the constellation diagrams indicates the better BER performance introduced by probabilistic shaping.

## 4. Conclusion

We have proposed a novel probabilistically shaped star-CAP-16/32 modulation based on constellation design with honeycomb-like decision regions, where the combination of geometric structural design and probabilistic shaping is capable of improving the CFM for the enhanced system performance. Theoretical analysis, as well as experimental demonstration in a 25 km IM/DD transmission system, is carried out to verify the superiority of our proposed PS star-CAP-16/32 modulation schemes. Experiment results show that at the BER of _{$1\times {10}^{-3}$}, the novel uniform star-CAP-16 in _{${C}_{4,4,4,4}$} outperforms the traditional counterpart in _{${C}_{8,8}$} by 0.8 dB in receiver sensitivity, while the proposed PS star-CAP-16 in _{${C}_{4,4,4,4}$} excels the traditional PS star-CAP-16 in _{${C}_{8,8}$} by 1.5 dB. At the same time, in the case of the same bit rate, the novel PS star-CAP-32 with entropy of 4.4 bits/symbol defeats the uniform star-CAP-32 by 1.6 dB improvement in receiver sensitivity. This outstanding advantage indicates that the proposed novel star CAP modulation scheme is capable of playing an important role in the access network to achieve better system performance.

## Funding

National Natural Science Foundation of China (NSFC) (61835005, 61822507, 61522501, 61475024, 61675004, 61705107, 61727817, 61775098, 61720106015, 61875248); Program 863 (2015AA015501, 2015AA015502); Beijing Young Talent (2016000026833ZK15); Fund of State Key Laboratory of IPOC (BUPT).

## References

**1. **Y. Gao, Q. Zhuge, W. Wang, X. Xu, J. M. Buset, M. Qiu, M. Morsy-Osman, M. Chagnon, F. Li, L. Wang, C. Lu, A. P. T. Lau, and D. V. Plant, “40 Gb/s CAP32 short reach transmission over 80 km single mode fiber,” Opt. Express **23**(9), 11412–11423 (2015). [CrossRef] [PubMed]

**2. **K. Zhong, X. Zhou, T. Gui, L. Tao, Y. Gao, W. Chen, J. Man, L. Zeng, A. P. T. Lau, and C. Lu, “Experimental study of PAM-4, CAP-16, and DMT for 100 Gb/s short reach optical transmission systems,” Opt. Express **23**(2), 1176–1189 (2015). [CrossRef] [PubMed]

**3. **S. Ohlendorf, S. Pachnicke, and W. Rosenkranz, “Multidimensional PAM with Pseudo-Gray coding for flexible data center interconnects,” IEEE Photonics Technol. Lett. **30**(12), 1143–1146 (2018). [CrossRef]

**4. **F. Li, X. Li, J. Zhang, and J. Yu, “Transmission of 100-Gb/s VSB DFT-spread DMT signal in short-reach optical communication systems,” IEEE Photonics J. **7**(5), 1–7 (2015).

**5. **F. Li, J. Yu, Z. Cao, J. Zhang, M. Chen, and X. Li, “Experimental demonstration of four-channel WDM 560 Gbit/s 128QAM-DMT using IM/DD for 2-km optical interconnect,” J. Lightwave Technol. **35**(4), 941–948 (2017). [CrossRef]

**6. **G. Stepniak, “Comparison of efficiency of N-dimensional CAP modulations,” J. Lightwave Technol. **32**(14), 2516–2523 (2014). [CrossRef]

**7. **L. Sun, J. Du, and Z. He, “Multiband three-dimensional carrierless amplitude phase modulation for short reach optical communications,” J. Lightwave Technol. **34**(13), 3103–3109 (2016). [CrossRef]

**8. **J. Shi, J. Zhang, X. Li, N. Chi, Y. Zhang, Q. Zhang, and J. Yu, “Improved performance of high-order QAM OFDM based on probabilistically shaping in the datacom,” *in Proc. OFC**2018*, paper W4G.6. [CrossRef]

**9. **C. Pan and F. R. Kschischang, “Probabilistic 16-QAM shaping in WDM systems,” J. Lightwave Technol. **34**(18), 4285–4292 (2016). [CrossRef]

**10. **Y. Zhu, A. Li, W.-R. Peng, C. Kan, Z. Li, S. Chowdhury, Y. Cui, and Y. Bai, “Spectrally-efficient single-carrier 400G transmission enabled by probabilistic shaping,” *in Proc. OFC**2017*, paper M3C.1. [CrossRef]

**11. **I. F. de Jauregui Ruiz, A. Ghazisaeidi, O. A. Sab, P. Plantady, A. Calsat, S. Dubost, L. Schmalen, V. Letellier, and J. Renaudier, “25.4-Tb/s transmission over transpacific distances using truncated probabilistically shaped PDM-64QAM,” J. Lightwave Technol. **36**(6), 1354–1361 (2018). [CrossRef]

**12. **J. Shi, J. Zhang, N. Chi, Y. Cai, X. Li, Y. Zhang, Q. Zhang, and J. Yu, “Probabilistically shaped 1024-QAM OFDM transmission in an IM-DD system,” *in Proc. OFC**2018*, paper W2A.44. [CrossRef]

**13. **X. Xu, B. Liu, X. Wu, L. Zhang, Y. Mao, J. Ren, Y. Zhang, L. Jiang, and X. Xin, “A robust probabilistic shaping PON based on symbol-level labeling and rhombus-shaped modulation,” Opt. Express **26**(20), 26576–26589 (2018). [CrossRef] [PubMed]

**14. **S. Zhang and F. Yaman, “Constellation design with geometric and probabilistic shaping,” Opt. Commun. **409**, 7–12 (2018). [CrossRef]

**15. **J.-X. Cai, H. G. Batshon, M. V. Mazurczyk, O. V. Sinkin, D. Wang, M. Paskov, W. W. Patterson, C. R. Davidson, P. C. Corbett, G. M. Wolter, T. E. Hammon, M. A. Bolshtyansky, D. G. Foursa, and A. N. Pilipetskii, “70.46 Tb/s over 7,600 km and 71.65 Tb/s over 6,970 km transmission in C+L band using coded modulation with hybrid constellation shaping and nonlinearity compensation,” J. Lightwave Technol. **36**(1), 114–121 (2018). [CrossRef]

**16. **S. Zhang, Z. Qu, F. Yaman, E. Mateo, T. Inoue, K. Nakamura, Y. Inada, and I. B. Djordjevic, “Flex-rate transmission using hybrid probabilistic and geometric shaped 32QAM,” *in Proc. OFC**2018*, paper M1G.3. [CrossRef]

**17. **B. Liu, Y. Zhang, K. Wang, M. Kong, L. Zhang, Q. Zhang, Q. Tian, and X. Xin, “Performance comparison of PS star-16QAM and PS square-shaped 16QAM (square-16QAM),” IEEE Photonics J. **9**(6), 1–8 (2017). [CrossRef]

**18. **G. D. Forney and L.-F. Wei, “Multidimensional constellations-part I: Introduction, figures of merit, and generalized cross constellations,” IEEE J. Sel. Areas Comm. **7**(6), 877–892 (1989). [CrossRef]

**19. **F. R. Kschischang and S. Pasupathy, “Optimal nonuniform signaling for Gaussian channels,” IEEE Trans. Inf. Theory **39**(3), 913–929 (1993). [CrossRef]

**20. **G. Böcherer, F. Steiner, and P. Schulte, “Bandwidth efficient and rate-matched low-density parity-check coded modulation,” IEEE Trans. Commun. **63**(12), 4651–4665 (2015). [CrossRef]

**21. **P. Schulte and G. Böcherer, “Constant composition distribution matching,” IEEE Trans. Inf. Theory **62**(1), 430–434 (2016). [CrossRef]

**22. **K. Zhong, X. Zhou, J. Huo, C. Yu, C. Lu, and A. P. T. Lau, “Digital signal processing for short-reach optical communications: a review of current technologies and future trends,” J. Lightwave Technol. **36**(2), 377–400 (2018). [CrossRef]