1. Introduction

When dealing with optical communication systems, there is still the lack of the general communication theory describing nonlinear fiber channels (where the dispersive effects intertwine with the fiber nonlinearity), in contrast to the classical additive white Gaussian noise (AWGN) communication channel. Thus, many fundamental and practically important questions related to the optimal coding, modulation, pulse-shaping, and channel equalization for the nonlinear optical transmission systems, remain either partially solved or even yet to be answered [1,2]. In particular, the optimal signal statistics (the Gaussian distribution for AWGN [3]) is not known for the transmission affected by the simultaneous action of fiber dispersion and nonlinearity. Addressing this problem numerically, in general, is not practically feasible due to the high computational cost of modelling high-speed transmission via dispersive nonlinear channels. Thus, there remains the research challenge related to developing practical transmitters and receivers with signal format and modulation inherently adjusted to nonlinear transmission. We note that currently-used transceivers are suboptimal, which brings about a cap on the achievable data rates and transmission distances. Recently, machine learning (ML) methods and, in particular, artificial neural networks (NNs) have been applied to the design of optical communication systems, see, e.g., recent Refs. [4–10] and numerous literature sources therein. Albeit it is rather difficult (if possible at all) to obtain the general conclusions on the optimal signal shaping and modulation in the realistic nonlinear fiber systems, it is possible to obtain some sub-optimal results by using specific ML techniques tailored to deal with complex nonlinear problems.

It is important to point out that the properties of optical fiber-channels strongly depend on signal parameters: the power is linked to the nonlinear effects, and the carrier pulse width defines the dispersive signal broadening and respective memory effects. Then, the transmission distance defines how essential the noise-induced corruptions are, the strength of nonlinear signal distortions, and the dispersive effects. Therefore, different nonlinear fibre channels can have rather distinct optimal signal modulations and coding.

End-to-end (E2E) learning approach, proposed in [11], is a machine learning technique offering a way to automatically tailor signal modulation and coding for an arbitrary communication link. The basic idea of E2E learning lies in representing the link from messages-in to messages-out as a single NN, differentiable with respect to the link parameters, to simultaneously optimize these parameters by using, e.g., the efficient gradient-descent-based optimization. Many applications of the E2E learning in different communication links then followed [12,13].

The first applications of E2E learning to optical communication systems were done for intensity modulation / direct detection (IM/DD) systems. In Refs. [14–17], the E2E learning of geometric constellation shaping (GS), i.e., optimal symbol locations, for IM/DD optical communication systems was proposed. Further, in [18] the E2E learning of waveforms was considered for the specific case of a link based on nonlinear frequency division multiplexing.

The following works dealt with the E2E learning of the optimal constellation points locations, i.e. on the GS, and the pre-distortion techniques in coherent systems. In [19–24], the E2E learning of single-symbol GS was considered for a coherent communication system. While [19–23] considered the distortions generated only by the optical channel, in [24] a more realistic link model that included the local oscillator laser noise, was studied. In [13,25–30], the E2E learning of GS, signal waveform, and nonlinear pre-distortion resistant to transmitter distortions was considered. However, the true complexity of nonlinear fiber-optic channel distortions was neglected in these works, namely, the distortions were either completely neglected, or modelled via a simplified Gaussian noise model. Conversely, in [31] the authors considered the joint E2E learning of GS and linear pre-distorter mitigating the fiber channel distortion. Nonetheless, the pre-distorter learnt in that work was rather trivial: it combined the Nyquist pulse shaper and chromatic dispersion compensator, and, hence, it actually did not contribute to the nonlinearity mitigation.

In our current paper, we propose the machine learning algorithm for E2E learning of the constellation shaping that takes into account the nonlinearities and memory present in optical channel distortions. With this algorithm, we jointly optimized the symbol locations in the constellation diagram, the symbol probabilities, and the nonlinear pre-distortion. The learnt transmitted signal distribution chooses the transmitted symbol based not only on the message sent in the corresponding time slot, as in the conventional constellation shaping, but also on the messages sent in the neighbouring time slots. Therefore, we refer to the resulting signal distribution as the multi-symbol constellation shaping (MSCS).

In the proposed E2E learning algorithm, we implement two key new features. The first one is the utilization of the auxiliary channel model based on perturbation theory [32], which allows us to reduce the computational efforts/resources needed to model the complex mixture of nonlinear and linear distortions taking place in fiber-optic links. Second, we implemented the training procedure for the simultaneous learning of symbol probabilities and locations according to [33], in which the accuracy of the procedure was demonstrated for the AWGN channel. These innovations let the algorithm learn the signal distribution outperforming a conventional Maxwell-Boltzmann (MB) one, where the latter is optimal for the AWGN channel [34]. In more details, we applied the proposed algorithm to a single-channel dual-prolarized 64 GBd transmission over 170 km standard single mode fiber (SMF) link, where the expected gains by nonlinear shaping are particularly high [35]. The learnt MSCS led to the bit-wise mutual information (BMI) gain of 0.47 bits/2D-symbol/polarization over the conventional MB shaping. Furthermore, we show that the proposed E2E learning is applicable for the cases when, because of hardware- or complexity-related limitations, we cannot use multi-symbol constellation shaping. For the same test case, we learnt a single-symbol joint probabilistic and geometric shaping showing 0.074 bits/2D-symbol/pol. BMI gain over the reference MB shaping. Moreover, for the case when geometric shaping is not an option, we learnt the single-symbol probabilistic shaping which outperforms the MB shaping by 0.043 bits/2D-symbol/pol. in terms of BMI. Note that the proposed method is rather generic and is applicable to improving the quality of transmission in an arbitrary coherent fiber-optic communication link. In [36] we successfully applied the similar E2E learning technique to a 64 GBd transmission over long-haul 30x80 km (2400 km) SMF link.

This work extends our previous approaches from Refs. [36,37]. The remainder of the paper is organized as follows: Section 2 describes in the proposed E2E learning algorithm; Section 3 describes the considered testcase and the results achieved by the proposed E2E learning; and Section 4 concludes the paper.

2. Models and the algorithm description

2.1 End-to-end learning

In the end-to-end learning approach, the whole system is implemented as a single NN from bits-in to bits-out, including transmitter, channel, and receiver. This enables the joint training of transmitter and receiver. The idea of the approach was first suggested in [11]. The scheme of end-to-end platform used in this paper is given in Fig. 1.

 

Fig. 1. Principal scheme of the end-to-end learning algorithm implemented in this paper. Blue denotes trainable blocks, dashed lines denote feedback from loss function. MF stands for matched filtering, CDC stands for chromatic dispersion compensation.

Download Full Size | PDF

2.2 Transmitter design, constellation shaping, and pre-distortion

We consider the two-stage transmitter, consisting of a sampler followed by a nonlinear pre-distorter. Note that the transmitter’s separability into autonomous stages, with each stage having its well-defined purpose, improves its interpretability and cost-efficiency [13].

In the sampler, the input data is mapped to some complex-valued constellation points ${s_m \in \mathbf {S} \in \mathbb {C},} |\mathbf {S}| = M$ and then transmitted over a channel. A fixed bit label $\mathbf {b}_m = [b_{m,1}, \ldots, b_{m,J}]$, ${b_{m,j} \in \{0,1\}},\, J=\lceil \log _2(M)\rceil$ is set for each symbol in the alphabet $\mathbf {b}_m \Longleftrightarrow s_m,\,\forall s_m \in \mathbf {S}$. The input data is generated in a way that symbols $s_m$ are sampled according to the discrete probabilistic distribution $p_S(s_m)\,\forall s_m \in \mathbf {S}$. In this work, we assume that for an arbitrary distribution $p_S$ there exists a mapper which maps the original input data stream in such a manner that the resulting symbols are independent and identically distributed according to $p_S$. Such an idealistic mapping can be achieved by a distribution matcher, e.g., the constant composition distribution matching (CCDM), with a long enough sequence.

Because of the imperfections of the communication channel, the information rate of the system depends on the locations of constellation points $\mathbf {S}$ and their probability distribution $p_S$. Therefore, it is possible to maximize the system information rate by optimizing symbol locations $\mathbf {S}$ and occurrence probabilities $p_S$. Optimizing $\mathbf {S}$ is called a geometric constellation shaping (GS), and optimizing $p_S$ is called a probabilistic constellation shaping (PS) [38]. Notably, for linear channels with AWGN, the family of Maxwell-Boltzmann (MB) distributions results in information rates very close to the optimal PS [34]

In this work, we consider a separate optimization of PS, GS, along with the joint optimization of both $\mathbf {S}$ and $p_S$; the joint optimization method is referred to as the joint shaping (JS). The sampler generated the dual-polarized signal $X = \{X_{h}, X_{v}\}$ with both polarization components $X_h, X_v$ sampled from the same alphabet $x_{i,h/v} \in \mathbf {S}$ according to the same distribution $p_S$.

The sampler was followed by a trainable nonlinear pre-distorter. The goal of the latter is to pre-compensate the nonlinear channel distortions through pre-processing the symbol sequence $X_{h/v}$ generated by the sampler, before the sequence is sent into the optical channel. The pre-compensation is made in such a way that the symbol sequence at the channel output, $Y$, approximates the sampled input symbol sequence. We consider a perturbation-based pre-distortion (PPD) [39], which adds to each transmitted symbol $x_{i,h/v}$ an additive correction $\Delta _\text {PPD} x_{i,h/v}$, depending on the characters transmitted in the neighboring slots in both polarizations $x_{i,h}, x_{i,v}$. The pre-distortion $\Delta _\text {PPD}$ is defined as a linear combination of cubic polynomials $T_{m,n}$ calculated from the symbols co-propagated with the pre-processed symbol at the neighboring time slots. In more detail, for any symbol $x_{i,h/v}$ transmitted at the $i$-th time slot in the H- or V-polarization, the pre-distortion takes the form:

The performance-to-complexity ratio of PPD is determined by the range of polynomials $T_{m,n}$ taken into consideration in a particular algorithm. Since the PPD is a linear regression over $T_{m,n}$ terms, the importance of each term in the trained PPD can be assessed by the absolute value of the coefficient $|C_{m,n}|$ corresponding to it. Hence, one can reach a cost-effective PPD by training an excessively complex one and then pruning it, i.e. removing terms $T_{m,n}$ with the absolute value smaller than the chosen cut-off value $T_{m,n}: |C_{m,n}| < C_\text {cutoff}$. This makes the cut-off value $C_\text {cutoff}$ an important hyper-parameter of PPD learning, defining the performance-to-complexity ratio of the resulting algorithm. We consider its optimization as part of E2E learning in Section 3.4.

2.3 RP-based channel model

The second autoencoder block, the auxiliary channel model, maps the signal generated by the transmitter $X$ to a sequence of symbols $Y$ collected by the receiver. The model includes both deterministic and stochastic distortions, expressed via the probability of receiving $Y$ given $X$ was transmitted, $p_{Y|X}$. The approximate channel model must be computationally simple and easy to differentiate, to allow for fast transmitter learning; but, simultaneously, it has to describe the distortion introduced by the channel accurately enough, so that the learned transmitter and receiver could emulate well a real-world communication link.

The nonlinear dispersive channel is typically modelled by the Manakov equations, see Eq. (3) below, which are simulated by a serial cascade of alternating convolutional and pointwise nonlinear operators; this solution scheme is referred to as the split-step Fourier method (SSFM) [40]. The SSFM can be represented by the convolutional NN consisting of many layers [31,41], and the complexity of such a convolutional NN makes the calculation of gradients of the model outputs over its inputs (in the back-propagation learning [42]) very slow and rather challenging. First, the learning process implies that all the intermediate states are stored, and it makes the process memory hungry. Second, the back-propagation through many layers often results in the numerical errors, leading to the infamous uncontrolled growth or vanishing of gradients [9,43]. One way to circumvent these problems is to consider a simpler auxiliary channel model. This approach, while typically offering the lesser degree of channel approximation precision than SSFM and, hence, the smaller shaping gain of E2E learned constellation shaping, can greatly reduce the numerical complexity of E2E learning. Among others, the following models were proposed for this purpose: the E2E learning using dispersion-free nonlinear channel model [44]; a nonlinear interference noise (NLIN) model [20,37,45], which considers nonlinear distortion as a AWGN process with a constellation-dependent variance [46,47]; or neglecting the optical channel nonlinearity at all by modelling its distortion as a fixed AWGN [13]. Nonetheless, all these models neglect the channel memory, i.e., inter-symbol interactions. On top of it, NLIN and AWGN-based channel models erase the information about the determinism of nonlinear distortions, modelling it as a stochastic noise. These two factors prevent the E2E learning of any memory-aware constellation shaping or nonlinearity pre-distorter, both of which rely on the information about the inter-symbol behavior of nonlinear distortion.

At the same time, NLIN-based auxiliary channel model can be useful for the limited task of E2E learning the single-symbol constellation shaping. In general, we expect the E2E learning via memory-aware auxiliary channel models to obtain higher shaping gain than NLIN, because of these models being free from the assumption of Gaussianity of the injected channel distortion. Nonetheless, the simplicity of NLIN model may lead to faster and cheaper training procedure. Therefore, we consider comparing NLIN with memory-aware channel models to be a fruitful, yet still open research question.

Recently, the data-driven approaches, learning the channel model properties from analysing the propagated data, were suggested [48,49], with [23] proposing the E2E GS learning in a coherent communication link through a data-driven auxiliary channel model. On one side, compared to the aforementioned approaches, the data-driven models do not require precise information about the optical channel properties. Nonetheless, the data-driven models are often less cost-effective since they require additional computational resources to learn the link parameters. In this paper, we assume the common case of the channel parameters being known in advance, which, therefore, renders the data-driven approach excessively complex.

Following our previous work [36], we propose here the application of first-order regular perturbation model (RP model) as an auxiliary channel model. Particularly, we suggest using the RP model for calculating both loss value and gradients, referred to as the forward and backward passes, correspondingly. In the following, we describe the RP model and the benefits of its application in more detail.

Consider the Manakov equations describing the evolution of the waveform of a dual-polarized optical signal $\mathbf {E}(z,t)=\mathbf {u}(z,t)\sqrt {f(z)}$ during its propagation over a fiber-optic link with lumped optical amplifiers (OAs) [50]:

The first-order regular perturbation (RP) [32,50,51] is an elaborate method to approximate $\mathbf {u}(z,t)$ in a weakly nonlinear regime. The principal scheme of RP model is given in Fig. 3. The channel output $\mathbf {u}(z,t)$ is approximated using the perturbations according to expressions:

The RP model can be better understood through its comparison with the split-step Fourier method (SSFM) applied for solving the Manakov equations [32], see Figs. 2, 3. SSFM is formulated as a sequence of alternating steps: the linear steps introducing the dispersive broadening $D_{\delta }[\cdot ]$ and accounting for linear losses, and the full nonlinear one, $\exp ({K}_{\delta, m})$, introducing Kerr nonlinear phase shift applied at the $m$-th step centered around point $z = (m - 0.5)\delta$:

 

Fig. 2. Principal scheme of the split-step Fourier method, Eq. (5). The scheme is given to illustrate the derivation of RP model, Eq. (4).

Download Full Size | PDF

 

Fig. 3. Principal scheme of the first-order regular perturbation (RP) model [32] introduced in Eq. (4).

Download Full Size | PDF

Compared to the “conventional” channel modelling by applying the full SSFM, the RP-model application for channel modelling has two main advantages in the context of end-to-end learning. First, the key advantage of RP is its parallel structure, i.e. its separability into the branches, where each particular branch is calculated via nearly the same routine. This RP property allows speeding up the computation of signal evolution and the computation of gradients via the "back-propagation" stage. Hence, the overall E2E learning is parallelizable in a straightforward way, which allows considerably speed-up its calculation by using modern multi-thread computational hardware, e.g. multicore processors and graphics processing units (GPUs). Second, the parallel structure of the RP model can increase the numerical stability of end-to-end learning. As noted in [52], the SSFM structure mimics the one of a typical convolutional NN made up of interchanging convolutions and point-wise nonlinearities. These complex multi-layered structures are prone to numerical errors in gradient estimation, referred to as infamous exploding/vanishing problem [9,43]. The simpler structure of RP model allows us to circumvent the gradient estimation problems by having a single nonlinear step per branch.

Notably, in addition to all the aforementioned benefits, the RP model offers a rather good approximation for the precise SSFM model in the practically important weakly nonliear regime [53] and around the optimal launch power level, where the link performance is the highest. At the same time, as expected, RP model poorly approximates optical channel at higher power levels. We measure how well does RP model approximate a precise SSFM one later in Sec. 3.2.

2.4 Receiver and performance metrics

At the receiver, the goal is to estimate the transmitted symbol $x_n$ from each received symbol $y_n \in \mathbf {Y}$. The optimal way is to decide according to the posterior probability $P_{X|Y}(x_n|y_n)$. This is equivalent to computing the posterior probability $P(\mathbf {b}_n|y_n)$ based on the bit labels ${\mathbf {b}_n = [b_{n,1},\ldots, b_{n,K}], b_{n,j} \in \{0,1\}}$. The alternative sub-optimal approach is to estimate the posterior probabilities $P(b_{n,j}|y_n)$ and decide on each $b_{n,j}$ individually. This approach corresponds to many practical receivers using bit-interleaved coded modulation with soft-decision binary forward error correction (FEC) codes. In such systems, the information rate determines (and is limited) by the bit-wise mutual information (BMI) Eq. (10) [38]. We consider here such a practical receiver.

We applied a commonly used mismatched Gaussian receiver (MGR) to approximate the bit posterior probabilities vector $q(b_{n,j}|y_n)$ [54,55]. In this approach, the conditional probability linking the channel input and output, $q_{Y|X}$, is assumed to obey a Gaussian distribution:

Applying the Bayes rule, we can write,

2.5 Optimization and training procedure

During the training, we optimize the symbol locations, probabilities, and pre-distorter parameters via a batch gradient descent procedure, i.e., by a repeated generation of the training symbols’ batches of fixed size and updating the trainable parameters using the numerical gradients of our objective function (averaged over the batches). The most challenging part here is the computation of gradient over individual symbol probabilities. We followed the training procedure proposed in [33]. This approach relies on a key consideration that the order of differentiation and expectation may not be exchangeable and this can affect the accuracy of numerical gradient calculation if the back-propagation algorithm is applied from some off-the-shelf toolboxes, like Tensorflow or PyTorch, without some necessary changes. To convey this point better, we reproduce here the algorithm presented in [33] in more detail.

Let $\bar {s}$ be a random constellation point generated by the transmitter according to symbol probability $P_S = \{p_S(s_1), p_S(s_2), \ldots, p_S(s_M)\}$. Since each symbol has a fixed bit labeling, we denote the corresponding random bit labeling by $\bar {\mathbf {b}}=[\bar {b}_1,\ldots,\bar {b}_K]$. Accordingly, the transmitter sends $\bar {x}=\bar {s}/\sqrt {\sum _{m=1}^{M} p_S(s_m) |s_m|^2}$. The power normalization is required to constrain the average power of the transmitter. Whatever the actual transmitter power is, it affects the channel distribution $p_{Y|X}(\bar {y}|\bar {x})$. The receiver receives symbol $\bar {y}$ according to $p_{Y|X}(\bar {y}|\bar {x})$ and computes the likelihood of each bit labels $\bar {b}_j$ according to the receiver’s trained posterior distribution $q(\bar {b}_j|\bar {y})$. The objective is to train $q(\bar {b}_j|\bar {y})$ for all $j$ at receiver as well as $P_S$ and the set of constellation points $\mathbf {S}$ to maximize the end-to-end BMI given by [38],

In our case of using MGR, no receiver training is needed. Nonetheless, if one instead employs another trainable model as a receiver, e.g. an artificial neural network, the receiver learning becomes simply equivalent to training of $K$ independent binary classifiers $q(\bar {b}_j|y)\,\forall j\in \{1,\ldots,K\}$. This optimization is a classical logistic regression, so we skip it here and refer to [56], where the authors detailed the training of such receivers.

The learning task of transmitter is more challenging. We should optimize $P_S$ and $S$ such that the aggregate BMI $\mathcal {I}$ is maximized. Since $P_S$ is constraint by $P_S(s_m)\geq 0$ and $\sum P_S(s_m)=1$, it is more recommended to train its unconstrained logarithm $\ell _m$ such that

Differentiation over pre-distortion parameters $C_{m,n}$ is followed similarly, i.e.

Training over $\ell _m$ is slightly different as several terms in objective function $\mathcal {I}$ depend on $\ell _m$. Let us first express the differentiation explicitly and then explain how one computes it using back-propagation algorithm. Differentiation of entropy $H(\bar {s})$ results

We explain now how to train the auto-encoder via batch gradient ascent algorithm to numerically maximize $\mathcal {I}$. Recall that the considered transmitter consists of two stages: sampler and pre-distorter. To generate a batch of size $N$, the sampler randomly draws with replacement $N$ indices from the current symbol distribution $P_S = \{p_S(s_1), p_S(s_2), \ldots, p_S(s_M)\},\,s_m \in \mathbf {S}$. Each drawn index, let say $s_n$, is mapped to a corresponding transmitted normalized symbol $x_n = s_n / \sqrt {\sum _{\omega =1}^{M} p_S(s_\omega ) |s_\omega |^2}$ with a given, fixed bit label $\mathbf {b}_n$, e.g. Gray bit labeling. The sampled normalized symbols $x_n$ and its corresponding bit-labels $\mathbf {b_n}$ are stacked, respectively, in the input symbol vector $\mathbf {X} = [x_1, x_2, \ldots, x_N]$ and the bit vector $\mathbf {B} = [\mathbf {b}_{1}, \ldots \mathbf {b}_{N}]$. The symbol power normalization is required to keep the batch power constant during the training.

The generated input batch is processed by a cascade of pre-distorter, followed by the channel model. The cascade maps input symbols $\mathbf {X}$ to the vector of channel output symbols $\mathbf {Y} = [y_1, y_2, \ldots, y_N]$ according to the cascade’s conditional probability density function $P_{Y|X}$. The vectors of input symbols $\mathbf {X}$, input bit-labels $\mathbf {B}$, and received symbols $\mathbf {Y}$ constitute the training dataset. The differentiability of both pre-distorter and channel model allows calculating the gradients of the output symbols over the input symbols and the pre-distorter parameters. The next step is to compute the gradient numerically and apply batch gradient ascent algorithm [42, Sec. 8.1.3]. Notably, gradient ascent, not descent as usually done in ML, should be applied there since we are interested in increasing the optimized metrics BMI. For instance, Eq. (14) is numerically estimated by,

Numerical computation of gradient for $\ell _m$ is slightly different. The derivatives of $\frac {\partial H(\bar {s})}{\partial \ell _m}$ are expressed in terms of optimized parameters in Eq. (16) and, hence, do not need to be computed via training data. The derivatives of $\frac {\partial \mathbb {X}(j)}{\partial \ell _m}$ must be however computed numerically. Let $\mathbb {Y}_m$ denote the subset of outputs $y_n$ in the batch for which the same constellation point $s_m$ produced their corresponding input symbol, i.e., $x_n = s_m / \sqrt {\sum _{\omega =1}^{M} p_S(s_\omega ) |s_\omega |^2}$. Let $N_m$ denote the number of output samples in $\mathbb {Y}_m$. Then, Eq. (17) can be partitioned into three terms

As a result, the numerical estimate of Eq. (18) is expressed by,

Finally, all the computed gradients were then used by the Adam optimizer [57] to train the respective communication system parameters. This completes the training process of our E2E system.

Remark: if one computes the derivatives from the numerical estimate of $\mathcal {I}$, i.e.

3. Results

3.1 Testcase

Having proposed the end-to-end learning algorithm for the general coherent fiber-optic communication link, we illustrate its benefits on the particular case of a single-span link. More specifically, we numerically consider the dual-polarized (DP) 64 GBd transmission of 256-symbol constellations over the 1x170 km single mode fiber (SMF) link. The SMF parameters were taken as: chromatic dispersion coefficient $D = 16.8$ ps/(nm*km), effective nonlinear coefficient $\gamma = 1.14$ $(\text {W*km})^{-1}$, loss coefficient $\alpha = 0.21$ dB/km. The span is followed by a lumped optical amplifier (OA) with the noise figure 4.5 dB. No transceiver phase noise was considered.

During the E2E system training, we used the RP model, Eq. (5), with $N_{br} = 100$ branches, as an auxiliary channel model. At the same time, the performance of the learnt constellations was estimated using the “precise” SSFM, i.e. via Eq. (4) channel model.

In our current work, we focus on a single-span link, because in such a case the nonlinearity-aware constellation shaping is expected to produce a considerable gain [35]. At the same time, we emphasize that our method is equally applicable for other fiber-optic communication systems, where, of course, the ultimate gain figures can be different. First, the training procedure, described in Section 2.5, was successfully applied in [33] to E2E learning the constellation shaping for the AWGN channel. The obtained results agreed with the analytical predictions for AWGN channel [58]. Particularly, the learned PS matched the performance of the "optimal" MB shaping Eq. (1), while the learned combination of PS and GS slightly outperformed MB, thanks to better ability to approach the truly optimal continuous Gaussian input symbol distribution. Second, in [36] the similar E2E learning technique was successfully applied to train the geometric constellation shaping and the nonlinear pre-distorter for the 64 GBd transmission over the long-haul 30x80 km (2400 km) SMF link.

Nonetheless, we expect the proposed E2E learning algorithm to reap smaller performance gains over MB shaping Eq. (1) in the links with bigger numbers of spans or co-propagating channels. We attribute it to the weakening of the overall gain obtainable by a nonlinearity-aware constellation shaping with the increase in link length [35] or number of co-propagated channels [59]. The channel distribution $p_{Y|X}$ in these testcases becomes more a Gaussian-like [47], for which MB shaping is optimal.

3.2 RP model channel approximation precision

We start from showing that RP model, despite being a simplified one, offers a decent approximation of the precise SSFM model. For the testcase, considered in this paper, we compared how do RP and SSFM models simulate the propagation of unshaped 256-QAM signal. In Fig. 4, first, we plot the signal-to-noise ratio (SNR) of the received signals after the chromatic dispersion compensation (CDC) was applied. One can see that the SNR is nearly the same for both the SSFM and RP models in the weakly nonlinear regime (up to 10 dBm), and, most notably, the correspondence is still good around the optimal launch power level of $P_\text {opt} \approx 10$ dBm. To demonstrate that this SNR-value-similarity comes directly from RP model correctly approximating the deterministic nonlinear distortions introduced by SSFM, we compared the outputs of the models in the noiseless scenario: we fed the same input signal to both models, while they injected no ASE noise, i.e. only the deterministic effects were considered. We quantified the difference between the outputs of RP $\mathbf {Y}_{\rm RP}$ and SSFM $\mathbf {Y}_{\rm SSFM}$ in terms of signal-to-distortion ratio (SDR), defined as $-20\log _{10}\left (\|\mathbf {Y}_{\rm SSFM}\| / \|\mathbf {Y}_{\rm RP}-\mathbf {Y}_{\rm SSFM}\|\right )$, also added to Fig. 4(b). We see there that up to the launch power of 10.5 dBm, SDR is at least 20 dB larger than the SSFM-modelled received SNR. This indicates that the approximation error of the RP model is much smaller than the total distortion in the link. To sum up, from Fig. 4 we readily see that the RP model renders a very good channel approximation for the case considered, and the deterministic mismatch between the models cannot noticeably affect the E2E system training.

 

Fig. 4. Comparison between channel models based on first-order regular perturbation (RP) and split-step Fourier method (SSFM) approximating the 64 GBd single channel dual-polarised transmission of unshaped 256QAM signal over 1x170 km SMF link.

Download Full Size | PDF

3.3 Learning the constellation shaping without pre-distorter

Once we have tested the accuracy of the RP model, we turn to the results of the E2E learning the constellation shaping.

In the first case, we consider learning the constellation shaping in a link without a pre-distorter, i.e. it was disabled and we learnt only PS and GS of separate symbols. This option is well suited for the use cases when there is no opportunity to implement a separate pre-distorter at the transmitter, typically, because of the limited complexity and/or power budget.

Before training, we initialized the encoder with a MB-shaped 256QAM constellation, defined in Eq. (1). To find the MB shaping parameter $\theta$ we, first, initialized the constellation as an unshaped 256QAM one and recorded the SNR produced by this constellation between the input and output of the auxiliary channel model. The MB shaping parameter $\theta$ was chosen as to have the optimal value for the AWGN channel, producing the same SNR level. The MB constellation with the $\theta$ parameter value found in this way was used as a starting seed for all the following learning of constellation shapings and its performance was used as a reference one.

We did the two types of constellation shaping learning. In the first experiment, we learnt just the PS. This is a preferable case for the links, where introducing the GS is not favored. The benefit of PS is that, while providing the performance gain, it keeps the square grid of QAM constellation intact (see Fig. 6(b)). Having the square grid of the transmitted constellation allows one to use the cost-effective blind DSP algorithms to recover the signal at the receiver [60] and lower the required precision of a digital-to-analog converter at the transmitter by limiting the range of signal powers along I- and Q- component present in the transmitted signal. In the second experiment, we jointly learnt the PS and GS to highlight the full single-symbol joint shaping (JS) gains reachable by the E2E learning strategy. For the PS, the training started from the reference MB shaped constellation. Then, the PS, learnt at the first stage, was used as an initial seed for the JS E2E learning.

The whole training procedure: constellation initialization as the MB-shaped one, learning the PS, and learning the JS, was done separately for a range of launch power levels. Figure 5 showcases the performance achieved by these shaped constellations in a validation SSFM simulation. The learnt PS (E2E-PS-256QAM) resulted in the 0.043 bits/2D-symbol/pol BMI gain on top of MB-shaping, and led to $\approx 0.25$ dBm increase in optimal power level. The learnt JS (E2E-JS-256) including both PS and GS, led to nearly doubling the performance gain: it resulted in 0.074 bits/2D-symbol/pol BMI gain on top of MB-shaping.

 

Fig. 5. The performance of the constellation shaping without pre-distortion: the reference Maxwell-Boltzmann (MB-256QAM) shaping, learnt probabilistic shaping (E2E-PS-256QAM), and the learnt joint probabilistic and geometric constellation shaping (E2E-JS-256).

Download Full Size | PDF

 

Fig. 6. Constellations applied at the optimal power level for the case of the link without nonlinear pre-distortion.

Download Full Size | PDF

The better performance of learned constellations can be partly explained by the fact that they produce a higher effective SNR in the link. Figure 7(a) shows that the effective SNR of the learned JS is higher than that of the reference MB shaping. Furthermore, we can quantify the difference between the constellations, leading to different SNR values. The extended Gaussian noise (EGN) model [47] suggests that the SNR of a signal propagated over a non-linear channel depends on the constellation and is inversely proportional to its 4th $\mu _4$ and 6th $\mu _6$ standard moments, where the k-th standard moment $\mu _k$ of the input symbol sequence $\mathbf {X}$ is defined as

 

Fig. 7. The comparison between the metrics of reference Maxwell-Boltzmann 256QAM (MB-256QAM) and the E2E learnt JS (E2E-JS-256) constellations.

Download Full Size | PDF

3.4 Learning the cost-effective pre-distorter via end-to-end learning

Once we have considered the single-symbol conventional constellation shaping, we move to the more advanced case of memory-aware shaping, i.e., the shaping when the transmitted symbol depends not only on the message transmitted in the corresponding time slot, but also on the messages transmitted in the neighbouring slots. To be effective, a constellation shaping should take into account the symbols co-propagating in the neighbouring slots, since they contribute to the deterministic nonlinear distortions introduced into the transmitted symbol of interest. We name this approach as the multi-symbol constellation shaping (MSCS).

Thankfully, the RP-based auxiliary channel model, Eq. (4), implemented into the E2E learning algorithm proposed in this work, allows the E2E learning of MSCS, since the RP introduces inter-symbol deterministic nonlinear distortions. Conversely, the EGN-based auxiliary channel model [47], suggested in previous works for end-to-end learning the coherent optical communications [19,45], does not support the MSCS learning: the EGN replaces the actual inter-symbol nonlinear distortion with the symbol-independent Gaussian noise and, therefore, prevents the E2E learning of dependencies between the neighbouring symbols.

We propose the MSCS implementation as a combination of single-symbol JS, described in the previous subsection, and the nonlinear PPD-based pre-distorter, Eq. (2), jointly trained in a single run of E2E learning. Before considering the MSCS training as a whole, we focus on the E2E learning for the cost-effective pre-distorter. We note that it is important to reduce the complexity of pre-distorter, inasmuch as its complexity defines the additional costs of MSCS implementation over the single-symbol JS, and, therefore, denominates the feasibility of the whole MSCS implementation.

As mentioned in Sec. 2.2, the PPD-based pre-distorter performance-to-complexity ratio is defined by the range of nonlinear perturbation terms $T_{m,n}$. An effective way to set it is by keeping in the algorithm only the terms $T_{m,n}$ for which the corresponding coefficient $\lvert C_{m,n} \rvert$ has the absolute value above the fixed cut-off threshold: $\lvert C_{m,n} \rvert > C_\text {cutoff}$. We refer to this approach as the pre-distorter pruning, in analogy with the similar technique from neural networks optimization [61].

We seek to find the optimal $C_\text {cutoff}$ value by a grid search. First, we loaded the unshaped 256QAM constellation to the sampler, and the constellation sampler from E2E learning. Then we trained the PPD with the range of coefficients $|m|,|n| \le 10$, and pruned it with the range of various cut-off values $C_\text {cutoff}$. We measured the performance of the generated family of E2E learnt pre-distorters on the precise SSFM channel model. The dependence of the link performance with the various pruned PPD, on the share of pruned coefficients $C_{m,n}$, is given in Fig. 8. The figure shows that when we keep around 25% of the most significant PPD terms $T_{m,n}$, adding the new ones leads to the negligible improvement in resulting algorithm’s performance. This pruning scheme will be used in the following MSCS learning.

 

Fig. 8. The dependence of the performance of pruned indirectly learned perturbation-based pre-distorter (PPD) with $|m|,|n| \leq 10$ on the margin of the most significant $C_{m,n}$ coefficients non-zeroed during the pruning procedure. The PPD training was done on the RP model, while the performance was measured in precise SSFM simulation. Inset: The distribution of PPD coefficients $C_{m,n}$ learned and pruned with the cut-off leaving 25% of the coefficients.

Download Full Size | PDF

Notably, the distribution of the learnt PPD coefficients $|C_{m,n}|$ agrees with the approximation suggested in [62], implying that $|C_{m,n}| \sim 1/|mn|$. The inset of Fig. 8 shows the distribution of the absolute value of coefficients $|C_{m,n}|$ in the aforementioned indirectly trained PPD with 25% coefficients left.

3.5 Learning the memory-aware constellation shaping

We have defined the cost-effective perturbation-based pre-distorter (PPD) in the previous subsection, and now we consider the E2E learning of the MSCS enabled by the pre-distorter. The MSCS estimates the performance gains reachable by the proposed E2E learning algorithm, when no complexity-limiting constraints are put on it. The set of non-pruned triplets $T_{m,n}$, defined during the PPD initialization, was kept fixed during the following E2E learning. After the initialization, we simultaneously optimized the PPD with joint probabilistic and geometric single-symbol constellation shaping, thereby arriving at the MSCS shaping.

The pre-distorter initialization followed by the E2E MSCS learning was done separately for a range of launch powers. The performance of the resulting learnt constellation measured on the precise SSFM channel model is given in Fig. 9. For comparison, we also plotted the performance of the single-symbol constellation shapings learnt in Sec. 3.3: MB-256QAM and E2E-JS-256. Compared to the reference MB constellation shaping, the MSCS led to the considerable improvement in the system’s BMI, adding 0.47 bits/2D-symbol/pol., and the optimal power level moved up by $\approx 1.25$ dBm.

 

Fig. 9. The performance of the end-to-end learnt multi-symbol constellation shaping (E2E-MSCS-256). For reference, we added here the performance of Maxwell-Boltzmann (MB-256QAM), learnt JS (E2E-JS-256), and the nonlinear perturbation-based pre-distorter trained on top of the fixed E2E-JS-256 constellation (E2E-JS-256 + PPD).

Download Full Size | PDF

To clarify the reason for the superior MSCS performance, we separately considered its contributing factors: PPD and fine-tuning the JS. To do this, we compared the performance of MSCS, involving the simultaneous training of JS and PPD, with the PPD trained on top of fixed pre-learnt single-symbol E2E-JS-256 (E2E-JS-256 + PPD). In the latter case, the constellation shaping isn’t by affected by the presence of PPD in the link, which allows us to isolate the performance gains of PPD. The results of this separate E2E learning are given in Fig. 9. Notably, PPD is found to be the main contributor to the MSCS performance: training the PPD on top of fixed JS already brought 0.39 bits/2D-symbol/pol BMI gain, while E2E learned MSCS brought only 0.012 bits/2D-symbol/pol BMI gain on top of this. Despite this result provides a valuable insight into the reasons for the performance of the MSCS, we consider it necessary to conduct a detailed study analysing different link testcases and deep learning hyperparameter values before making any sound conclusions. This study is left beyond the scope of this work.

4. Conclusion

The end-to-end learning of the optical coherent detection communication system offers a possibility to specialize the characteristics of the transmitted signal to the properties of the nonlinear channel. However, we underline that the two main problems are typically associated with the E2E learning implementation.

The first problem relates to the overall complexity of modelling the channel distortions, where the latter constitute an intricate mix of instantaneous nonlinear fiber responses intertwining with a dispersive pulse broadening. The modelling of modern high-baudrate links is especially difficult because of a huge dispersive memory, implying a considerable complexity and time consumption of the split step simulations. In our work, we address this issue by proposing a parallelizable simplified channel model based on the first-order regular perturbation [32].

The second challenge is the difficulty of introducing the concept of trainable discrete probability distribution into the machine learning algorithm. In our paper, this problem is addressed by adopting a novel training procedure proposed first in [33] for the AWGN channel.

The resulting composite solution, proposed and demonstrated in this work, made possible learning the joint probabilistic and geometric shaping of symbol sequences in coherent fiber-optic communication links. Despite the considered approach is still suboptimal, the computed multi-symbol joint probabilistic and geometric shaping has shown a considerable bit-wise mutual information (BMI) improvement of 0.47 bits/2D-symbol/pol over the conventional Maxwell-Boltzmann shaping for a single-channel 64 GBd transmission over the 170 km SMF link.

Moreover, we observed that the proposed end-to-end learning is applicable in the situations when, because of hardware or complexity limitations, we cannot use the multi-symbol shaping. For the same test case, we found that a single-symbol joint probabilistic and geometric shaping gives 0.074 bits/2D-symbol/pol BMI gain over the reference MB shaping. And in the case when geometric shaping is not possible, the single-symbol probabilistic shaping can be used: it also outperforms the MB shaping by 0.043 bits/2D-symbol/pol in terms of BMI.

We believe that the end-to-end learning approach, proposed in our work, can lay a path to finding the optimal signal distribution for a variety of nonlinear fiber-optic channels.